Benchmark ranking should exclude records where image alignment failed
The article argues that a numeric benchmark score can look comparable even when the measurement behind it differs. In the StagingBench records, drift_score sometimes holds an aligned architecture score and sometimes a raw fallback when alignment fails, and sorting on that field discards the distinction. The author recommends checking alignment status before sorting, averaging, or ranking, and treating each score as a measurement with a procedure and status. Two failed records are cited, MeltFlex room 01 with 20 inliers against a minimum of 25 (raw score 48.5) and REimagineHome room 04 with 10 inliers (raw score 40.1), both reported as unaligned. The author notes that passing alignment is only a prerequisite and does not establish that framing passed, and advises keeping failed records visible with their diagnostic reasons outside the ranking.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in