Developer Uses AI-Assisted Evals to Catch Faulty Support Triage Logic Before Shipping

·1 views

A developer building an AI-powered support-triage app discovered that the system was incorrectly classifying routine API key rotation requests as security incidents requiring human escalation. Although the app returned valid JSON with no crashes, the behavior violated the defined policy, which treated key rotation as a self-service how-to task. To systematically catch such errors, the developer integrated DeepEval via a Cursor agent skill, creating a golden dataset of test cases with expected outputs drawn from existing tickets and policy documents. DeepEval's built-in metrics checked both whether the app's output matched expected decisions and whether those decisions aligned with escalation rules, surfacing a clear failure on the key-rotation case. After narrowing the triage prompt to escalate only genuine security compromises or outages, all test cases passed, confirming the fix did not break previously correct behavior.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Hard-Learned Celery Production Tips: Monitoring, Retries, and Silent Failure Fixes

A software team sharing lessons from four years of Celery deployments describes how a silent worker failure once left 40,000 tasks unprocessed and delayed client invoices for two weeks. The core issue is that Celery, by default, acknowledges tasks before they run, meaning a worker crash silently drops the job with no alert raised. Setting task_acks_late and task_reject_on_worker_lost ensures failed tasks are requeued rather than lost, while worker memory and task-count limits prevent gradual degradation. The team also recommends deploying Flower with event tracking enabled to monitor queue depth and active worker counts, triggering alerts when thresholds are breached. For task reliability, they use exponential backoff retries with differentiated handling for expected errors versus genuine data-missing cases.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

REST vs GraphQL: How to Choose the Right API Style for Your Project

REST and GraphQL are two widely used API approaches, each with distinct trade-offs depending on the use case. REST organises data around fixed endpoints and HTTP methods, making it simple to implement, cache, and version, but it can lead to over-fetching or multiple round-trips for related data. GraphQL lets clients request exactly the fields they need in a single query, which benefits mobile apps on variable networks, though it introduces added complexity in setup, caching, and query management. Many teams adopt a hybrid approach, using REST for public-facing APIs and GraphQL internally to reduce frontend round-trips. The best choice depends on team familiarity, client needs, and whether the API is intended for public or controlled consumption.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How 'Codeburn' and Language Server Protocol Aim to Fix AI Coding Hallucinations

AI coding agents like Devin, Cursor, and Copilot Workspace frequently generate syntactically incorrect or logically flawed code — a problem known as hallucination — because they predict tokens based on training data rather than a repository's actual structure. Each hallucination wastes computational resources and pollutes the agent's context window, causing cascading errors in subsequent reasoning steps. The emerging 'Codeburn' architectural pattern attempts to address this by discarding invalid code paths in real time, before they are committed, using feedback from the development environment. Central to this approach is the Language Server Protocol (LSP), which supplies live data on abstract syntax trees, types, and dependencies to constrain what the model can generate. Together, Codeburn and LSP aim to shift the burden of code correctness away from probabilistic language models and onto deterministic static analysis tools.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How OSPF Hello Packets Filter Neighbor Relationships Before Adjacency Forms

An OSPF Hello packet does more than act as a keepalive sent every 10 seconds — it functions as a strict filter that determines whether two routers can begin a neighbor relationship at all. Before any adjacency or database exchange can occur, four fields in the received Hello packet must exactly match the local interface configuration: area ID, netmask, hello interval, and dead interval. If any of these fields differ, the relationship silently never starts, with no error sent to the remote router and often no visible log entry. This behavior explains a common troubleshooting scenario where two routers on the same segment both send Hellos yet never form an adjacency. A free educational series called 'Protocol in Code' explores this logic through a compact Python implementation available on GitHub.

0 comments Read more at DEV Community