SShortSingh.
Back to feed

Microsoft's MarkItDown Tool Cuts LLM Token Costs by Converting PDFs to Markdown

0
·1 views

Uploading PDFs directly to AI language models consumes a large number of tokens because models must process not just text but also fonts, layout data, and embedded formatting — a 20-page document can cost up to 70,000 tokens before a single query is made. Microsoft's open-source tool MarkItDown, which has surpassed 119,000 GitHub stars, addresses this by converting PDFs and other file types into clean Markdown text. Markdown strips away formatting noise while preserving meaningful structure such as headings, lists, and tables, reducing token usage by as much as 70 percent in favorable cases. Beyond token savings, mainstream LLMs are trained heavily on Markdown and tend to process it more accurately than raw PDF output. MarkItDown also ships an MCP server package that allows compatible AI clients and editors to trigger file conversion automatically, removing the need for manual export and re-upload steps.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI Is Turning Software Developers Into 'Pipeliners', Argues Tech Writer

A developer's core role has historically been that of an 'intelligent executor' — someone who translates incomplete human requirements into working code using judgment, not just technical skill. Unlike previous waves of automation that only removed mechanical tasks, AI is now capable of handling the judgment and gap-filling that defined the developer's value. This shift means the fundamental identity of the profession is changing, not merely its toolset. The emerging role, dubbed 'Pipeliner,' involves designing and managing the pathways through which AI executes work — deciding inputs, context, verification, and flow. Rather than building software directly, the Pipeliner's job is to architect how intelligence moves through a system to produce results.

0
ProgrammingDEV Community ·

How Contract Testing Catches Breaking API Changes Before They Reach Clients

Contract testing goes beyond schema validation by running executable fixtures that verify actual API responses match what clients expect. Subtle changes like wrapping search results in an object, renaming fields, or converting null values to zero can silently break mobile apps that cannot update immediately. Developers are advised to store representative response fixtures and assert on keys, types, nullability, and response envelopes for endpoints covering search, food lookup, and 404 cases. Edge-case fixtures — including sparse records, zero-calorie items, Unicode text, and duplicate barcodes — help surface real-world failures that clean sample data would hide. A post-deployment smoke test suite and a frozen baseline JSON file in the repository together provide verifiable evidence that API compatibility has been maintained across releases.

0
ProgrammingDEV Community ·

What ODbL License Really Means for Apps Using Open Food Facts Data

The Open Database License (ODbL) permits commercial use of databases like Open Food Facts, but attaches specific conditions around attribution, public display, and redistribution of derived databases. Developers building nutrition apps must distinguish between querying data for outputs, such as product pages or meal summaries, and actually distributing a substantial copy of the underlying database, as each carries different obligations. Attribution must be clearly visible to end users wherever data appears, not buried in documentation, with a recommended format citing Open Food Facts and linking to the licence. Images, logos, and other database contents may carry separate licensing terms beyond ODbL, requiring independent review before use. Engineers are advised to store source metadata, maintain a licence registry, and seek qualified legal counsel before launching any commercial product that relies on openly licensed nutrition data.

0
ProgrammingDEV Community ·

How to Build Reliable Barcode Lookup for Nutrition Apps Using GTIN Standards

Developers building nutrition apps must treat barcode scanning and product lookup as two distinct steps, since a valid scan does not guarantee a database match. Barcode values like UPC, EAN, and GTIN should be stored as text to preserve leading zeroes, and a modulo-10 check digit validation can reject invalid codes before any network request is made. When multiple database records share the same barcode due to imports or packaging updates, a consistent ranking rule — such as preferring records with images and higher confidence scores — should determine which result is shown. If no match is found, apps should offer fallback options like name-based search or manual nutrition entry rather than generating data from the barcode alone. Tracking metrics such as database hit rates, duplicate rates, and user corrections after a match can help teams improve lookup reliability over time.

Microsoft's MarkItDown Tool Cuts LLM Token Costs by Converting PDFs to Markdown · ShortSingh