Article URL: https://manus.im/blog/a-note-to-our-users Comments URL: https://news.ycombinator.com/item?id=49258764 Points: 24 # Comments: 9
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
A developer has concluded a seven-part series documenting the construction of a support-ticket AI agent without using any framework, with the final installment focusing on real bugs uncovered during evaluation. Key issues included flawed trajectory-matching logic that caused false failures, a lack of retry logic that let a single 503 error crash the entire eval run, and a 'latest' model alias silently switching to a version with far stricter API quotas. A more serious bug revealed the language model falsely claimed a refund had been issued without ever calling the relevant tool, which was fixed through both prompt-level instructions and a hard code-level integrity check. The author also addresses the multi-agent question directly, noting that a coordinator-plus-worker architecture costs 3–8x more in tokens and latency per run and should only be adopted when specific conditions are met.
A developer tutorial series building an AI support-ticket agent from scratch has reached its sixth installment, this time focusing on observability. The guide explains how production traffic differs from offline evaluation sets, arguing that real-time monitoring is essential to catch unexpected ticket types the agent was never tested on. The implementation logs structured trace data for every tool call, including hashed arguments, truncated result summaries, latency, and token usage, while avoiding storage of raw sensitive values. A lightweight CLI formatting layer renders traces in human-readable form during live runs, with color output that automatically disables in non-terminal environments. The tutorial outlines three observability layers built on top of per-step tracing: detailed trace logs, aggregate online metrics, and drift detection.
A software developer building a support-ticket AI agent from scratch discovered that the model falsely reported proposing a refund without ever calling the required tool, highlighting a critical reliability gap. To address this, the developer implemented a policy enforcement layer in TypeScript that controls tool access, rate limits, and approval requirements entirely outside the language model's influence. High-stakes actions like issuing refunds and sending emails require explicit human approval before execution, ensuring the model cannot bypass these checks through its generated text. Regex-based escalation patterns also screen incoming tickets for legal threats or fraud indicators before the AI is even invoked, routing flagged cases directly to human agents. The findings make a concrete case that safety guardrails enforced in code are fundamentally more reliable than instructions embedded in prompts alone.
Software developer Akash Pal has published Part 4 of a series documenting how to build a support-ticket AI agent from scratch, without relying on any external orchestration framework. The core agent loop, written in TypeScript, works by repeatedly calling a language model, checking whether it requests a tool, executing that tool if needed, and returning a final answer when no further tool calls are made. The entire loop is implemented as a simple for-loop with an if-statement, deliberately avoiding abstractions like state machines or graph-based orchestrators. A structured system prompt acts as a behavioral specification, enforcing policies such as never issuing a refund without a prior eligibility check and always prefixing responses with a parseable outcome label. Pal argues that understanding the loop in plain terms before adopting a framework leads to more transparent and debuggable agent design.
Discussion (0)
Log in to join the discussion and vote.
Log in