SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Benchmark tests 10 LLMs on Blender 5.0 Python scripts across 119 prompts

0
·1 views

A developer submitted a benchmarking challenge to Kaggle that evaluates how accurately ten large language models generate working Python scripts for specific versions of Blender, the open-source 3D software. The test is necessary because Blender's bpy API changes significantly across major releases, meaning a script written for version 3.x often fails silently or crashes in 4.x or 5.0. Each of 30 scripting tasks was prompted across four Blender versions, with answers graded by actually running them in headless builds of the exact target version. GPT-5.5 led in execution success at 95%, while also costing the most at $4.44 of the $11.23 total spend, whereas smaller models like Gemini-3.8-Flash matched frontier performance at a fraction of the cost. A key finding was that many models could produce scripts that ran correctly but failed to flag the API changes involved, revealing a gap between functional output and genuine version awareness.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Webhook Signature Verification Alone Does Not Guarantee Request Authorization

Verifying an HMAC webhook signature only confirms that a payload originated from a vendor, not that the request is authorized to modify specific resources in your system. Developers must separately map incoming events to internal subjects and apply proper authorization checks — covering tenant, user, and resource — before executing any side effects. A correctly signed payload can still contain a spoofed tenant ID or an action the customer never permitted. Network trust and cryptographic authenticity do not substitute for a defined authorization policy. Experts recommend evaluating subject, action, and resource before mutating any system state.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Splits 2,000-Line Home Assistant Config into 8 Organised Files Using Python

A Home Assistant user refactored a bloated 2,144-line automations.yaml file — spanning 63 automations across heating, alarms, cameras, and more — into eight separate, topic-specific files. The main concern was silently breaking critical automations, such as heating controls managing 11 thermostats, during the refactor. To address this, the developer wrote a 40-line Python script to split the file and a separate verification script to confirm no automations were lost in the process. Classification was handled through substring matching on German-language automation aliases, routing each entry into a named bucket like 'heating' or 'alarm', with unmatched entries falling into a miscellaneous file. The project highlights a practical approach to managing growing smart-home configuration files without relying on metadata tags or manual reorganisation.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Build Identity-Aware AI Agents Using Microsoft Agent Framework and Auth0

A developer built a four-part tutorial series demonstrating how to add proper identity and authorization controls to AI agents using Microsoft Agent Framework and Auth0. The project centers on a fictional expense-approval agent where a manager interacts with an AI that retrieves reports, sends emails, and approves or rejects expenses on their behalf. Each agent capability — from reading expense data to taking actions — required its own distinct identity and authorization decision, rather than a single blanket permission. Auth0's Fine-Grained Authorization was used to filter accessible data before vector search runs, ensuring the underlying language model never processes information a user is not permitted to see. The series, published on the Auth0 blog, aims to address the identity and accountability gaps that most AI agent tutorials overlook entirely.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Veteran developers quietly adopting AI tools despite trust concerns, industry data shows

Experienced software engineers who once dismissed AI coding assistants are now routinely using them to ship features, driven not by direct management mandates but by subtle competitive pressure from peers and hiring signals. Developers report shipping AI-generated code they have not fully reviewed, drawing comparisons to trusting third-party libraries without reading their source. The shift has been rapid, with attitudes moving from mockery to reluctant adoption within roughly 18 months. Industry observers note that the skills now valued most are not raw coding ability but the capacity to clearly define problems, direct AI tools, and critically evaluate their output. Those most at risk are developers whose value was tied to memorised syntax and boilerplate speed, capabilities that AI tools now replicate reliably.

0 comments Read more at DEV Community
Benchmark tests 10 LLMs on Blender 5.0 Python scripts across 119 prompts · ShortSingh