SShortSingh.
Back to feed

FreeCAD Fix benchmark ranks LLMs by geometry oracle, not AI judges

0
·5 views

The LFORLA FreeCAD Fix benchmark tests language models on repairing a broken parametric FreeCAD script using identical prompts each time. Instead of relying on an LLM as judge, it runs each repaired script through real freecadcmd measurements to check the geometry it produces. Results are broken into seven per-axis rubrics, including Safety, Runs Clean, and Dimension Accuracy, with every run's full answer published for review. Qwen 3.8 27B currently leads the leaderboard at 99.33 percent, while Laguna S 2.1 sits lowest among those listed at 56.67 percent. The authors also submitted their own model, GLM 5.2, which scored 78.0 overall.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Shopify customer merge fails: what the customerMerge docs say about blockers and gaps

A developer guide explains how to diagnose failed Shopify customer merges by checking userErrors, Customer.mergeable, and the customers/merge webhook. It notes that Shopify's documentation lists hard blockers such as subscriptions, store credit, and company contacts, but does not map every error code to a blocker. The customerMerge mutation rules determining which customer survives are summarized, along with its asynchronous job behavior and required access scope. The Help Center's nine profile-level merge blockers and the CustomerMergeErrorFieldType enum's 11 values are compared, with the author flagging mismatches and noting that merges are irreversible.

0
ProgrammingDEV Community ·

Company adds WebMCP tools to its own website as a test of the emerging agent standard

A web development firm that builds WebMCP integrations for clients has added the proposed standard to its own service page. WebMCP lets a page register named tools that AI agents can call directly, instead of having to guess how to interact with buttons and forms. The firm's setup includes a read-only tool for retrieving its service packages, a read-only FAQ search tool, and a tool that pre-fills its audit request form without submitting it. The article notes that the API is still early, with Chrome testing it as an origin trial and Firefox and Safari not announcing support, and that it has changed in 2026. The firm advises starting with read-only lookup tools and never exposing admin functions or customer data through them.

0
ProgrammingDEV Community ·

Vietnamese Discord bot Voxy reads voice-channel chat aloud, and real servers exposed new problems

Khanh Duy, an indie developer in Ho Chi Minh City, built Voxy, a Discord bot that reads a voice channel's text chat aloud for Vietnamese servers. The bot was designed to help members who cannot speak on voice, since Discord's built-in text-to-speech reads Vietnamese shorthand like "ko" as individual letters. Voxy converts common chat shorthand, numbers with units, laughter and links into natural spoken Vietnamese before synthesis. Each server can maintain its own pronunciation dictionary, and messages are capped at 40 words. The article notes that the bot initially suffered from message delays under heavy use, which the developer traced to a per-message REST call rather than the speech API itself.

0
ProgrammingDEV Community ·

Autor monta assistente de IA local para proteger dados pessoais, sem depender de nuvem

Um profissional de infraestrutura de TI descreve como criou um assistente pessoal de IA que roda inteiramente em casa, motivado pela preocupação com a privacidade de rotinas, finanças e anotações. Ele utiliza o Ollama, ferramenta open-source que executa modelos de linguagem localmente e expõe uma API acessível apenas na própria máquina. O texto alerta que a API não possui autenticação própria, de modo que expô-la à rede sem firewall ou proxy com token pode permitir acesso indevido. Para transformar o modelo em um assistente funcional, o autor integra o sistema a ferramentas de automação como o n8n, que permitem disparar ações conforme gatilhos e agendamentos. A solução é apresentada como replicável com componentes gratuitos, reservando serviços de nuvem apenas para tarefas que exigem modelos maiores.