SerpApi's Markdown Output Lets Local LLMs Fit More Web Search Data

A local model can only read the search results that fit within its context window, and early SerpApi responses often exceeded that limit, so the JSON Restrictor was used to trim them before they reached the model. SerpApi's new Markdown output represents the same data in fewer tokens than JSON, allowing the restrictor to keep more information and often letting the full response fit without trimming. The author demonstrates applying the JSON Restrictor to Markdown output in an open-source app that connects a model running in LM Studio to web search. The benchmark compares Markdown and JSON outputs across several search tools, using four local models on a 32K context window. Token sizes were estimated with OpenAI's tiktoken library, and the models were tested end to end on thirteen questions.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in