Developer Ditches Needle2 for llama.cpp and Granite After Local Tool-Calling Tests

A developer building a semantic shell — which maps natural language commands to local filesystem tools — initially tested Needle2, a small model designed for tool calling and structured extraction. While Needle2 worked in basic scenarios, problems emerged as the tool set grew beyond a handful of options, since its retrieval-based shortlisting risked excluding the correct tool before final selection. The developer also found that small models are highly sensitive to tool descriptions, requiring precise, disambiguating language rather than brief documentation-style text. Concerns about whether confidence scores across candidate groups were globally comparable added further uncertainty. Ultimately, the need for native C++ integration without external processes led the developer to switch to llama.cpp paired with the Granite model.