Developer Scraps Similarity Algorithm After Duplicate Articles Slip Through Content Pipeline

·1 views

A developer running an automated content pipeline discovered that the same story had been published three times after a similarity-checking algorithm repeatedly failed to flag duplicate topics. The deduplication gate compared candidate titles against previously published ones using SequenceMatcher and Jaccard bigram scoring, rejecting anything above a 0.58 threshold. However, two titles covering the same Chinese military film scored just 0.187 because the key shared term appeared in different positions, and the algorithm measured character overlap rather than meaning. Rather than tuning the threshold or switching to a smarter model, the developer replaced the automated check entirely with a manual human review step. A validation guard was also added to the build pipeline to fail any future attempt to reintroduce a similarity-based deduplication algorithm.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Tailscale didn't stop the Hugging Face intrusion

Article URL: https://tailscale.com/blog/hugging-face-intrusion Comments URL: https://news.ycombinator.com/item?id=49127306 Points: 5 # Comments: 0

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Engineering Guide Explains Why QR Codes Fail in the Field and How to Fix Them

A technical guide published on DEV Community outlines the engineering decisions that determine whether a QR code scans reliably in real-world conditions, not just on screen. The piece frames QR codes as input/output systems, where payload size, version, error correction level, and module size must be matched carefully to the physical medium. It explains that the four error correction levels — L, M, Q, and H — offer trade-offs between capacity and damage tolerance, with higher correction increasing module density and reducing effective scanning distance at a fixed print size. The guide warns that a payload near a version boundary can be pushed into a denser, harder-to-scan version by even a single extra URL parameter. Practical recommendations include computing the required version before generating a code and sizing up the physical print area when higher error correction is necessary, such as when overlaying a logo.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Agent in CI Pipeline to Automate Multi-Repo Release Notes

A developer at Entire automated the company's weekly release notes, called Dispatches, by integrating an AI agent named Goose into a GitHub Actions CI pipeline. The task proved far more complex than a similar project built at Block, where a single repository with clean release tags made automation straightforward. Entire's Dispatches span multiple projects with different deployment cadences, feature flags, and mixed public-private visibility, making it difficult to determine which changes were safe to announce. The developer iterated through several approaches, eventually shifting from release-tag-based triggers to a time-window method using the previous Dispatch's publication date to avoid duplicating entries. The pipeline was designed not just to surface what changed, but to assess whether each change had fully shipped, was stable, and was appropriate for public communication.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev cuts Next.js CI/CD pipeline from 15 minutes to 4 by fixing artifact bloat

A developer discovered their Azure DevOps pipeline was packaging a 1.2 GB artifact — including full node_modules and source files — despite Next.js already generating a lean 24 MB standalone build directory. The fix involved updating the pipeline to copy only the .next/standalone folder into the deployment zip, shrinking the transferred artifact from 614 MB compressed down to 24 MB. This single change reduced the total pipeline runtime from roughly 15 minutes to about 4 minutes. The developer also added a dedicated /api/health route after finding health checks were hitting auth middleware and returning redirects instead of status responses. A separate cookie-naming bug that caused an infinite sign-in loop locally was resolved by basing the secure cookie prefix on the actual NEXTAUTH_URL protocol rather than NODE_ENV.

0 comments Read more at DEV Community