LFORLA Benchmark Tests AI Agents on Building LEGO Sets From a Written Brief

·8 views

LFORLA has introduced a LEGO Set Build benchmark that asks AI agents to turn a written brief into an LDraw model file. Each agent must plan the build, source the required parts, and assemble the model under identical conditions for every provider. A deterministic oracle, rather than an LLM judge, scores outputs on axes such as file validity, spec compliance, and structural integrity. On the current leaderboard, Longcat 2.5 Preview leads with 98.83, while GLM 5.2, submitted by LFORLA itself, scored 78.00. The developers argue that fixed rubrics and deterministic checks make cross-provider comparisons fairer than judgments that can be swayed by persuasive rationales.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Argues Tool Engineering Will Replace Prompt Engineering as AI Skill of Choice

A developer writer contends that prompt engineering, once a prized skill, is becoming obsolete as language models grow more capable. He points to three drivers: models now grasp plain-language requests without elaborate scaffolding, very large context windows reduce the need for compression tricks, and fine-tuning and retrieval-augmented generation solve problems that prompt hacks once addressed. He proposes a successor discipline, Tool Engineering, centered on building, selecting, and orchestrating the tools AI agents use to act in the real world. The piece argues that a model without tools can reason but cannot take actions such as booking a hotel or running a database query. Developers who keep investing in prompt techniques, he warns, risk becoming increasingly irrelevant.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Recording Free Inference Calls in a Ledger Before Making Them

The article argues that free inference lanes fail less on quality than on accounting, since unrecorded spend cannot be subtracted from a budget. It proposes placing a ledger in front of the lane and recording each reservation before the upstream call is made. Because workers can die mid-call and retries can double spend, the ledger must survive crashes without manual cleanup. The author describes a lease-based design where each job holds an open budget row that is committed or released, with a sweeper expiring stale leases. The implementation uses only the Python standard library and a single SQLite file, suitable for a laptop or small always-on machine.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Unveils Gemini 4 Argon With 1M-Token Output, Rolling Out Gradually

Google DeepMind announced Gemini 4 Argon on September 30, 2026, positioning it for complex software engineering, knowledge work, and defensive cybersecurity. Its standout specification is an output limit of up to 1 million tokens, raised from 64K, which could support long multi-step tasks. Access is initially limited to trusted cyber defenders and testers, with broader developer and enterprise availability still to come. Google reports a 77.9% score on DeepSWE v1.1 along with an introductory API price of $2 per million input tokens and $10 per million output tokens. The article cautions that these are publisher-reported figures, not a controlled head-to-head comparison with OpenAI's GPT-6.1 Sol.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Log Model Call Metadata Instead of Payloads to Reduce Data Leak Risk

The article argues that AI features often leak sensitive data through logs rather than request bodies, since logging receives less security scrutiny. It proposes logging only a "shape" record for each model call, including a payload digest, byte size, field names, latency, and attempt count, instead of message content. Encryption is deemed insufficient because logs are decrypted when searched, exported, or shared in support bundles. The approach is enforced with a CI test that scans log fixtures for disallowed fields and sensitive patterns such as emails, bearer tokens, and private keys. The author also recommends including a deliberately leaky fixture to confirm the scanner actually detects leaks.

0 comments Read more at DEV Community