Why Bulk Database Inserts Stay Slow Even When Your Code Looks Fine

A common performance trap in database-heavy applications involves foreign key dependencies that force inserts to pause and retrieve auto-generated IDs before proceeding. When loading relational data — such as bands and their associated songs — each child record cannot be built until the parent's database-assigned ID is returned, creating an unavoidable sequential dependency. This means the entire load is split into distinct phases that cannot overlap, preventing parallel processing across tables regardless of batch size tuning. SQLAlchemy's insertmanyvalues feature, using INSERT...RETURNING, can speed up this process roughly 15 times by batching ID retrieval, but the fundamental sequencing constraint remains. The article argues the real bottleneck is architectural — an application waiting on the database to assign identities — rather than query performance or indexing.