Host-held state stayed exact across 10 of 11 open LLMs from 1.5B to 32B parameters
In 3,200-step test runs, the host-held state matched the true state on every step for 10 of 11 open models, ranging from 1.5 to 32 billion parameters, with no corrupted steps. The setup used a grammar that named the coordinates, and the host checked every change the model proposed. Model size made a large difference when requests were phrased in ordinary words, where a 1.5B model read 2.31 percent of steps correctly versus 97.91 percent for the 32B model. The single failing model, DeepSeek-R1-Distill-Qwen-7B, was unable to produce readable replies to deletions, so the host dropped them and its state fell behind. The results suggest model size mainly affected speed and cost when the host enforces the check, but it matters greatly when the model must also interpret loose language.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in