SShortSingh.
Back to feed

Do AI assistants give consistent answers? Testing six on 20 place questions

0
·3 views

Asking an AI assistant the same question twice can produce different answers, a problem explored in PlaceBench, a test of six assistants on everyday questions about real places. Researchers asked 20 of the 150 questions three times each to measure how consistently each assistant stayed correct. Dots performed best, answering correctly all three times on 17 questions and never being wrong, while Instinct changed its grade on 15 of the 20 questions. The study also found that a correct first answer does not guarantee correct later ones, and it estimated the chance each assistant would be right on all three tries. The authors advise checking answers or asking repeatedly before relying on an assistant, while noting that the sample of 20 questions is small. The questions, answer keys, and grades have been published openly under a CC BY 4.0 license.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingHacker News ·

Behind the Making of Jean-Michel Jarre's Oxygene Album

A feature on jeanmicheljarre.com examines how the electronic album Oxygene was created. The piece is linked from a Hacker News discussion that gathered 7 points and one comment. The source material provided is limited to the article link and discussion metadata, so the specific production details it covers cannot be confirmed here. The discussion appears to be modest in scale, reflecting limited community engagement at the time of posting.

0
ProgrammingDEV Community ·

Fine-Tuning and Adaptation Bridge Robot Foundation Models to Real-World Deployment

Robot foundation models learn reusable knowledge across many tasks and environments, but pretraining alone cannot ensure reliable performance once a robot is deployed. Because robot errors can cause collisions, damaged equipment, or unsafe contact with people, these models must be calibrated for geometric accuracy, dynamic feasibility, and safety, not just semantic correctness. Adaptation is needed to address gaps between limited training data and real deployment conditions, as well as differences in robot embodiment, sensor configurations, and task success criteria. The text frames adaptation as the step that turns broad, general competence into physically executable behavior suited to a specific machine and workspace.

0
ProgrammingDEV Community ·

Edge Deployment of Robot Foundation Models: Why Onboard Inference Matters

Robot foundation models convert perception, language, and learned skills into physical actions, so they must run close to sensors and actuators. Cloud-based inference adds network latency to the perception-action loop, which can impair safe and responsive behavior for mobile robots and humanoids. Network connectivity is also unreliable in many settings such as factories, warehouses, and disaster zones, so local execution enables continued autonomy when communication is lost. Local processing further reduces bandwidth demands by converting large multimodal sensor streams into compact representations before any external transmission. Together, these factors make edge deployment a core architectural requirement for robot foundation models.

0
ProgrammingDEV Community ·

Introduction to Robot Foundation Models and Vision-Language-Action Systems

Traditional robotics relies on task-specific engineering, so each robot typically needs its own perception, planning, and control pipeline, which is costly and hard to reuse. The physical world is highly varied and constantly changing, making fixed rules and narrowly trained models insufficient for many operating conditions. Foundation models offer a way to learn broadly reusable representations first and then adapt them to specific tasks or robot bodies. Robots generate heterogeneous data, including camera images, depth or LiDAR measurements, joint readings, language instructions, and action trajectories, which such models can align into a shared representation. The approach aims to improve generalization and transfer of knowledge across tasks and environments, making learning a more central part of robot design.