Developer's compression tool scored 0/24 on CSV readability with GPT-4o-mini
A developer testing ctxfold, a lossless compression format designed for LLM prompts, discovered that its CSV encoder made data completely unreadable to AI models. In benchmark tests, GPT-4o-mini answered zero out of 24 lookup questions correctly from folded CSV, compared to a perfect score on raw CSV. The root cause was that the encoder stripped shared value prefixes into a header, forcing the model to reconstruct values through indirection rules it could not reliably follow. Switching to the more powerful GPT-4o improved scores only marginally and inconsistently, confirming the format itself was the problem. Rather than patching the encoder, the developer reclassified CSV folding as a pipeline-only feature in v0.1.4, meaning compressed CSV must be decompressed before being passed to a model.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in