DevTime v0.4.0 verifies code claims after stub file misled developers about Stripe webhooks

·1 views

A developer discovered that Cal.com's open-source repository contains a Stripe webhook handler that returns only a 404 error, meaning the file's name implies functionality that does not actually exist. This finding prompted a pivot in the developer's own tool, DevTime, which as of version 0.4.0 acts as a code verification layer rather than a simple repository explainer. Users can submit plain-language claims about a codebase — such as whether webhooks verify signatures — and DevTime returns a status of SUPPORTED, WEAK, CONTRADICTED, or UNKNOWN, backed by specific file-level evidence. The tool distinguishes between absent evidence, which it marks as WEAK, and conflicting evidence, which it marks as CONTRADICTED, to avoid generating noise from every code gap. It also tracks staleness by fingerprinting evidence files, flagging verified claims when their supporting code changes.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Engineering Team Cuts RAG Pipeline Latency 40% Using Bayesian Search and Hybrid Retrieval

A development team overhauled their retrieval-augmented generation (RAG) pipeline after standard production deployments proved inadequate for legal contracts, API documentation, and customer support tickets. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% across content types. They combined vector search with BM25 keyword search using Reciprocal Rank Fusion, then applied a cross-encoder reranker to narrow 50 candidates down to 5, adding only 50ms of latency while improving recall by 15%. A query transformation layer was also introduced to handle poorly phrased user inputs before retrieval. The combined changes reduced overall query latency by 40%, bringing the pipeline from over one second per query to a significantly faster, more reliable system.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Engineering team cuts RAG pipeline latency 40% using Bayesian search and hybrid retrieval

A development team overhauled their retrieval-augmented generation (RAG) pipeline after standard production deployments revealed significant performance gaps across legal, API, and support document types. The team replaced fixed 512-token chunking with document-specific strategies — including recursive, semantic, and agentic chunking — achieving recall@10 scores between 91% and 97% depending on content type. They combined vector search with BM25 and cross-encoder reranking using Reciprocal Rank Fusion, which improved relevance correlation from roughly 0.75 to 0.92. A query transformation layer was also introduced to handle poorly formed user queries before retrieval. Together, these changes reduced end-to-end query latency by 40% while pushing overall recall@10 to 95%.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How One Team Built Automated LLM Evaluation That Catches 92% of Hallucinations

A software team discovered the cost of informal AI testing after deploying a RAG-based customer support assistant that served hallucinated responses to over 500 users before any issues were caught. The assistant fabricated a billing policy and pulled API rate-limit figures from a competitor's documentation, exposing a complete absence of automated evaluation. In the post-mortem, the team identified that academic benchmarks like MMLU are insufficient for production use cases and that domain-specific, automated evaluation pipelines are essential. They subsequently built a CI/CD-integrated evaluation system using a judge ensemble that scores responses on faithfulness, instruction-following, schema validity, and safety. The resulting pipeline now catches 92% of hallucinations before deployment by running versioned test cases against multiple automated judges with defined pass/fail thresholds.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Class vs Object in OOP: Key Differences Every Developer Should Know

In object-oriented programming (OOP), a Class acts as a blueprint that defines the rules and structure for creating objects, without occupying any memory itself. An Object is a real instance created from that class, holding actual values and consuming memory at runtime. Every class is built from two core components: attributes, which store data, and methods, which define behavior. A single class can be written once and used to create multiple objects, each with its own set of values. Notably, an object is not required to assign values to every attribute or use every method defined in its class.

0 comments Read more at DEV Community