Developer builds go-zerocsv to cut Go CSV parsing memory from 540MB to 5KB
A Go developer discovered that the standard library's encoding/csv package generates over 10 million heap allocations when parsing a 5-million-row CSV file, consuming roughly 540 MB of RAM. The root cause is the Read() method's design, which allocates a new string slice and individual heap strings for every row and field parsed. To address this, the developer built go-zerocsv, an open-source parser that uses a reusable 4 KB buffer and in-place typed scanning, reducing allocations to near zero on the hot path. Benchmarks on an AMD Ryzen 5 8400F running Go 1.26 show go-zerocsv parsing the same 5-million-row file in 302 ms with only 5 KB of memory, compared to 492 ms and 540 MB for the standard library. The library also improves CSV writing by accepting typed values directly, eliminating the temporary string conversions previously required by the stdlib writer.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in