Do AI assistants give consistent answers? Testing six on 20 place questions
Asking an AI assistant the same question twice can produce different answers, a problem explored in PlaceBench, a test of six assistants on everyday questions about real places. Researchers asked 20 of the 150 questions three times each to measure how consistently each assistant stayed correct. Dots performed best, answering correctly all three times on 17 questions and never being wrong, while Instinct changed its grade on 15 of the 20 questions. The study also found that a correct first answer does not guarantee correct later ones, and it estimated the chance each assistant would be right on all three tries. The authors advise checking answers or asking repeatedly before relying on an assistant, while noting that the sample of 20 questions is small. The questions, answer keys, and grades have been published openly under a CC BY 4.0 license.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.



Discussion (0)
Log in to join the discussion and vote.
Log in