LLM reranker silently did nothing: token limit cut off replies, fallback hid the failure
A developer's text-to-SQL table selector included an optional LLM reranker that appeared to have no effect on results. Logging showed the model was being called but returned no answer, because newer Claude models spend tokens on internal reasoning and exhausted a 120-token limit before producing output. The reranker's error handling caught the failure and silently returned the original order, so existing tests passed while the feature was broken. The fix raised the token limit to 1024, added a test that fails below 512 tokens, and logged whether the model answered on each question. On public Spider and BIRD benchmarks, the reranked version placed the correct tables in the top five for substantially more questions than the baseline, though results reflect one model and measure table retrieval rather than final SQL accuracy.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in