Developer benchmarks his own PDF-to-Markdown tool against rivals, publishes losses

A developer who built a PDF-to-Markdown converter called CleanMD ran a structured benchmark comparing it against Microsoft's MarkItDown and Artifex's pymupdf4llm across five diverse public documents, including an RFC, a SCOTUS opinion, and an arXiv paper. The test measured section heading recall, code block detection, and table quality using ground truth derived from the documents themselves rather than any tool's output. CleanMD initially missed 11 RFC headings due to a font-parsing bug where hyphens triggered a second font subset, a flaw the developer fixed the same day but kept in the published results to avoid presenting a misleading after-only picture. pymupdf4llm led on heading recall and fence count, though many of its fences wrapped single inline code words rather than full blocks, while MarkItDown produced cleaner tables on the NIST document but detected zero headings in it. The developer published the benchmark openly, including the areas where his own tool underperformed, framing it as a transparency exercise rather than a marketing exercise.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in