FrameFlip48: four models, one frame switch, two kinds of failure
This is a submission for the Kaggle Benchmarking Challenge. FrameFlip48 measures whether a model tracks the translation frame in a short robot-motion program. Each of 24 matched pairs has identical numbers and commands. One prompt says ACTIVE_FRAME = LOCAL; its counterpart says ACTIVE_FRAME = WORLD. Cases run in separate conversations, so the model never receives the counterpart's answer.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in