Developer finds zero gain from fine-tuning embeddings, ships RAG engine as MCP server
A developer concluded a six-part series on a retrieval-augmented self-recall engine called RE-call by reporting that fine-tuning an embedding model on their own document corpus produced zero improvement in MRR and nDCG@10 scores. The null result was intentional and informative: the base model already achieved perfect retrieval on the small corpus, leaving no room for gains. Testing on a harder, jargon-heavy corpus showed the same fine-tuning approach yielded a +0.24 MRR improvement, demonstrating that fine-tuning only helps when the base model fails to cover domain vocabulary. The developer argues that publishing negative results saves teams from unnecessary GPU costs and reveals which retrieval regime they are operating in. The completed engine ships as an MCP server called recall_mcp, allowing AI agents such as Claude to query structured memory that includes trust verdicts, calibrated confidence, and explicit abstention signals alongside every result.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in