OpenAI Playbook Calls for Full Harness Disclosure in AI Model Evaluations

·1 views

OpenAI has released an official playbook urging that third-party evaluations of frontier AI models disclose the full testing environment, not just benchmark scores. The company argues that factors such as prompts, tool access, state management, compute budgets, and scoring mechanisms can materially alter observed model performance. This concern is illustrated through GPT-5.5 cyber-range tasks, where harness features like state compaction visibly shifted results. OpenAI identifies additional risks to evaluation validity, including reward hacking, data contamination, and sandbagging, and recommends these be reported alongside results. The guidance is framed as a transparency standard rather than a universal configuration, aiming to help readers judge whether evaluation conditions actually match the claims being made.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Presigned URLs vs Proxy Uploads: When to Use Each for File Storage

A software engineer has outlined the key trade-offs between presigned URLs and proxy-based file uploads for web applications. Presigned URLs, where the browser uploads directly to object storage via a short-lived credential, are recommended for most SaaS use cases because they avoid loading the app server with large payloads, reduce costs, and lower latency. Proxying uploads through a backend server is justified only in specific scenarios, such as compliance-mandated virus scanning, server-side file transformation before storage, or strict per-tenant quota enforcement. Security best practices for presigned URLs include having the backend generate the object key, setting short expiry windows, and keeping stored objects private with signed download links. Skipping these precautions, the author warns, effectively creates an unsecured public file drop with no access controls.

0 comments Read more at DEV Community

ProgrammingHacker News ·

The AI Aesthetic

Article URL: https://blog.jim-nielsen.com/2026/ai-aesthetic/ Comments URL: https://news.ycombinator.com/item?id=49117099 Points: 3 # Comments: 0

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Wetask Opens Task Runtime to External Workers with Go, TypeScript, and Python Clients

Wetask is releasing an update that extends its task runtime beyond embedded handlers to support external workers, including those running machine-learning jobs in Python, high-throughput backend work in Go, and application services in TypeScript. The platform introduces first-party clients for all three languages, allowing workers to claim task batches, renew long-running work, and settle results independently of the main process. To prevent stale workers from corrupting task state, Wetask assigns each claimed attempt a one-time fencing token, storing only its SHA-256 hash and rejecting duplicate completions. Workers can acknowledge or reject entire batches atomically, which also reduces durable storage overhead by recording settlements at the batch level rather than per task. Internal benchmarks show durable batch processing at 32 tasks per batch achieves around 2,402 tasks per second, roughly 27.8 times the single-task baseline, with Wetask noting that RabbitMQ remains ahead in replicated queue durability and operational maturity.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Tiered Architecture Beats LLM-as-Judge for AI Security, Developer Argues

A developer building an AI security tool argues that using a large language model to evaluate every incoming request is slow, costly, and itself vulnerable to jailbreaking. Instead, they propose a four-tier detection system that starts with cheap, fast methods like regex and classical machine learning before escalating to heavier models only when necessary. The first two tiers are deterministic and handle the vast majority of traffic at near-zero cost, while an optional transformer layer and a user-supplied LLM handle genuinely ambiguous cases. This approach means LLM-level inference costs are incurred only on a small fraction of requests rather than across all traffic. Because the always-on tiers run locally, the entire system can operate air-gapped inside a private network without contacting any external hosted model.

0 comments Read more at DEV Community