SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Model Aliases Are Not Versions: Why Silent Drift Breaks Evals

0
·1 views

Developers often report regressions with no config changes, assuming randomness, but model names can behave like moving pointers rather than fixed versions. Providers rotate aliases to point at newer snapshots, and the same alias may route to different backends over time, so logging the requested model, echoed model, timestamp, and base URL is recommended. Setting temperature to 0 does not guarantee identical outputs, since batching, floating-point ordering, and routing introduce variance. Fluent-looking output is also an unreliable signal of change, so the article suggests measuring self-consistency, output length, n-gram overlap, and error rates against a baseline. The author proposes a small, unexecuted drift-detection setup using a fixed prompt set and standard tooling.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI Reshapes Job Titles and Work Methods, Shifting Value Toward Engineering Judgment

An opinion piece argues that artificial intelligence is changing both job titles and the way work gets done across industries. It contends that AI tools are now widely accessible, so the key differentiator is how users apply them, including their methodology and ability to steer the tools toward useful results. In software development, the author says writing code is becoming a cheap, routine step, with value moving toward engineers who understand the broader discipline and can guide AI toward real-world solutions. The piece notes that AI also creates new roles, citing examples such as prompt engineering, RAG engineering, agentic AI engineering, Chief AI Officer, and AI business translator. It concludes that people who resist adaptation risk falling behind, and that thoughtful, moderate adoption of AI can strengthen skills and professional capabilities.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why coding agents turn CRLF files into whole-file diffs, and what Codex 0.162 changed

Coding agents that rewrite a Windows file with LF line endings can make git report every line as changed, even when only one word was edited. This happens because git stores the committed CRLF bytes, and the rewrite adds an invisible difference to each line. Developers have reported the problem with Claude Code, and Codex CLI's 0.162.0 release now preserves existing CRLF endings in its patch updates. The article includes a reproducible git demo showing that one changed word produces three deletions and three insertions. It explains how core.autocrlf and text attributes in .gitattributes decide where conversion occurs, and why files already committed with CRLF can escape normalization.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

GraphGuard Uses a Graph Database to Link Security Alerts Into Attack Stories

Security teams often face alert fatigue because tools display each alert as an isolated record that hides its connections to other events. The author argues that security data is naturally a graph, so the context needed to spot attacks lies in relationships between users, devices, IP addresses, and incidents. GraphGuard uses FalkorDB, a graph-native database, as the backbone for an AI investigation agent that traverses these connections quickly. The system stores investigations and evidence as persistent nodes, allowing the agent to recall past cases across sessions. The model is built from eight node types and ten relationship types, seeded with three realistic attack scenarios.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Remote-support session logs must leave the appliance to survive a rebuild, experts say

Remote-support sessions are recorded mainly to document changes made during maintenance, but appliance-local logs sit on an encrypted partition and are not centralised unless forwarding is configured. CISA advises forwarding appliance syslog and session events to a SIEM immediately, along with web application firewall, reverse proxy and perimeter firewall logs covering the exposure window. The article stresses that usable records must leave the appliance, be time-synchronised with other systems, and be retained beyond the appliance's lifecycle. It also highlights that missing sessions can signal altered logging, not just inactivity. Mandiant warns that intervals between intrusion and discovery in the BeyondTrust case were measured in weeks, and additional organisations may have been compromised before disclosure.

0 comments Read more at DEV Community