Open-Source Tool apitap Moves 10 Million DB Rows in Under 10 Seconds
A developer last week released apitap, an open-source Python library designed to transfer entire database tables between systems using a single function call and no configuration files. In tests on a 16-core machine, apitap moved 10 million rows from PostgreSQL to ClickHouse in 9.9 seconds, compared to 111 seconds for ingestr and over 1,800 seconds for dlt. The tool is built to run on minimal hardware, with memory capped by design so a 256 MB container can handle tables of any size by streaming data rather than loading it all at once. To avoid partial or corrupted loads, transfers stage into a shadow table before atomically swapping in, ensuring readers always see complete data. The developer's stated goal is to make the data-movement engine cheap enough that cloud database costs, not pipeline infrastructure, become the primary expense.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in