LLM Agreement Study Finds Models Echo Their Own Priors, Not Message Content

A small art project encoding sentences as only word-length sequences prompted researchers to test whether large language models could recover intended meaning from that minimal signal. The experiment compared model readings of a true message's length sequence against readings of a different length sequence and against randomly assembled texts, measuring positional agreement across four runs. Results showed that treatment-arm agreement never consistently separated from the prior-control arm across all four runs, suggesting models were converging on shared priors rather than decoding actual message content. An early version of the study contained a methodological flaw where the random-baseline vocabulary was drawn from the control arm's own limited readings, artificially inflating the floor and nearly obscuring the real finding. The corrected conclusion is that when evaluating model self-consistency, the meaningful baseline is the model's own prior output, not random chance.