FreeCAD Fix benchmark ranks LLMs by geometry oracle, not AI judges
The LFORLA FreeCAD Fix benchmark tests language models on repairing a broken parametric FreeCAD script using identical prompts each time. Instead of relying on an LLM as judge, it runs each repaired script through real freecadcmd measurements to check the geometry it produces. Results are broken into seven per-axis rubrics, including Safety, Runs Clean, and Dimension Accuracy, with every run's full answer published for review. Qwen 3.8 27B currently leads the leaderboard at 99.33 percent, while Laguna S 2.1 sits lowest among those listed at 56.67 percent. The authors also submitted their own model, GLM 5.2, which scored 78.0 overall.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in