How monday.com Built Agent Evals Against Live Staging Instead of Mocks

·1 views

monday.com AI Engineering Director Dor Cohen shared insights in a September 15 webinar on why his team chose to run agent evaluations against a live staging cluster rather than mocked dependencies. Cohen demonstrated that mocks fall short because they drift from production, use unrealistically clean synthetic data, and cannot accurately represent stateful changes made by an agent. A key example showed an agent that appeared to complete a task correctly but had silently processed only 500 of 600 requested items — a flaw only caught by evaluating the full action trajectory, not just the final output. To connect eval runners directly to a real Kubernetes environment, monday.com adopted mirrord, a tool that mirrors live traffic, secrets, and file systems to a local or CI process. The approach gives their evals production-level fidelity, enabling them to catch edge cases and verify actual end states that mock-based testing would miss.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Linux fork() Uses Copy-on-Write and Why It Can Exhaust Memory

When a Linux process calls fork(), the kernel does not immediately duplicate physical memory; instead, it copies the page table hierarchy and marks all entries as read-only. Any subsequent write by either the parent or child process triggers a hardware page fault, prompting the kernel to allocate a new physical page and copy the data. This Copy-on-Write mechanism keeps fork() fast under low-write conditions but becomes costly when workloads involve heavy, sustained writes to large memory regions. Systems running in-memory databases like Redis can experience write amplification, TLB thrashing, and uncontrolled memory growth during background snapshot operations. In extreme cases, the cascade of micro-allocations exhausts available RAM and activates the kernel's Out-of-Memory killer, terminating processes unexpectedly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NPU, DPU, QPU Explained: Which Specialist Chips Are Production-Ready Now

Hardware vendors are aggressively promoting three chip types — NPUs, DPUs, and QPUs — but their real-world utility varies significantly. Neural Processing Units (NPUs) are already proving their worth in production for low-power, on-device AI inference tasks like keyword detection and camera segmentation on mobile and edge devices. Data Processing Units (DPUs), such as NVIDIA BlueField and AMD Pensando, are similarly mature, helping large-scale infrastructure operators offload high-throughput networking, storage, and security tasks from general-purpose CPUs. Quantum Processing Units (QPUs), by contrast, remain largely confined to research settings and are not yet viable for mainstream production workloads. Engineers are advised to evaluate each chip against specific workload requirements rather than adopting them based on vendor marketing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

EduVetta: New SaaS Tool Automates Repetitive Admin Tasks for Teachers

A developer has launched EduVetta, a SaaS platform designed to reduce the administrative workload teachers face outside of actual classroom instruction. The tool offers four core features: automatic lecture note summaries, AI-drafted question papers with answer keys, a simplified assignment submission system, and interactive quiz generation. Teachers can generate content from multiple input types including YouTube links, PDFs, or spoken requests, and students can access assignments without creating an account. Objective grading such as multiple-choice questions is automated, while subjective answers are intentionally left for human review to avoid errors that could harm students. The product is currently available with a free tier at eduvetta.com, though the developer notes that diagram- or equation-heavy subjects still require additional manual editing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Structured Data Contracts Can Make Backlink Audits Easier to Debug

A technical design note proposes a small, structured data contract to improve how backlink audit tools record and interpret results. Rather than returning a simple pass-or-fail flag, the proposed format captures fetch state, document scope, link presence, and index or referral status separately. The distinction matters because a page can be reachable while a specific link is missing, or a link can exist without any evidence of indexing. The article also outlines best practices such as using URL parsers instead of substring matching, resolving relative links before comparison, and preserving prior successful observations during outages. The author notes the piece was prepared with AI assistance and is a design proposal, not a production benchmark or description of Google's internal systems.

0 comments Read more at DEV Community