How to Test Unpredictable LLM Pipelines in CI Using a Three-Tier Contract Method

·1 views

Testing LLM-backed features in CI is challenging because language models like GPT-4 or Claude can return different valid outputs for the same input, breaking standard determinism assumptions. Many engineering teams respond by writing few tests or asserting on exact strings, both of which become unsustainable in production. A proposed contract-based approach splits testing into three tiers with different determinism guarantees, each mapped to a separate CI job. The first tier uses static JSON fixtures to validate output shape and field types without ever calling a live model, running in milliseconds on every commit. The second tier employs cassette-based replay, recording a real model response once and replaying it in subsequent CI runs to test parsing and business logic without hitting the network.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Enterprise RAG Systems Need Lineage Governance to Stay Secure

Retrieval-Augmented Generation (RAG) has become a core architecture for enterprise AI agents, pairing large language models with vector databases to handle complex business queries. However, traditional access control frameworks like Role-Based Access Control do not translate into vector embedding spaces, leaving sensitive data exposed to unauthorized retrieval. Key risks include privilege escalation through context injection, indirect prompt injection via malicious documents, and hallucinations caused by stale or outdated embeddings. To address these vulnerabilities, platform engineering teams are urged to implement Data, Context and RAG Lineage Governance, which enforces query-time authorization and cryptographic data provenance. This approach structures RAG pipelines into distinct security boundaries, embedding metadata and access control attributes directly alongside vector representations to maintain a verifiable audit trail.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Safe Slack Approval Workflow for Deleting Cloud Infrastructure

Developers building cloud cost-optimization tools face a critical challenge: allowing a Slack button click to trigger real infrastructure deletion without causing incidents. The FinOps Sentinel project demonstrates how to wire up a two-channel Slack interactivity system using incoming webhooks for outbound alerts and a verified HTTP callback endpoint for inbound approvals. Signature verification using Slack's signing secret is essential, as without it any unauthenticated POST from the internet could trigger destructive actions like deleting EBS volumes. Block Kit JSON messages are structured to surface key details such as resource ID, region, and estimated monthly cost so approvers can make informed decisions quickly, even on mobile. LLM-generated summaries are treated as untrusted display copy and kept isolated from action values to prevent injection-style misuse.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Rust Memory Layouts Explained: repr(Rust), Packed and Aligned Types

A technical deep-dive on DEV Community explores Rust's memory representation options for structs and other complex types. Unlike repr(C), Rust's default repr(Rust) allows the compiler to reorder struct fields — typically placing larger fields first — to minimize padding and reduce memory usage. The article demonstrates how a sample struct can shrink from 32 bytes under repr(C) to just 16 bytes with repr(Rust). It also covers repr(packed), which eliminates padding at the cost of slower or potentially crash-prone misaligned memory access, and repr(align(n)), which forces larger alignment to prevent false sharing across CPU cache lines in concurrent programs. Additional memory layouts for tuples, arrays, and unions are briefly outlined as well.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How an AI Consent Ledger Prevents Voice Agents From Ignoring Opt-Out Requests

AI voice and messaging agents can fail to honor a user's revocation request when consent is only updated in one channel while other workflows continue running unchecked. Unlike traditional apps that capture permissions at fixed moments, AI agents operate across voice, SMS, email, and CRM steps simultaneously, making consent a dynamic runtime state rather than a one-time setting. An AI consent ledger addresses this by maintaining an append-only record of all permission events and providing a fast policy-check layer that any agent must query before taking action. The ledger tracks who gave or withdrew consent, which channel and purpose it covers, and which workflow acted on it, ensuring a single revocation propagates across all related outreach. Builders are advised to implement both keyword detection and natural-language classifiers to catch opt-out signals in conversation, routing ambiguous cases to human review rather than relying solely on transcript summaries.

0 comments Read more at DEV Community