Quarantine Gates in Delta Live Tables: Isolating Bad Data Without Halting Pipelines

·5 views

A data engineer argues that failing entire pipelines over malformed records is counterproductive, especially in HIPAA- and PCI-regulated environments where most of a batch is still valid. The article recommends replacing strict fail-on-violation expectations in Delta Live Tables with expect_drop or expect plus a downstream filter. Violating records are routed to a dedicated quarantine table so that auditors can see exactly which records failed and why. It also suggests alerting on-call engineers when violation rates cross a threshold and querying DLT's event_log to track which expectations fail most often. Finally, it advises running new validation rules in logging-only mode before enforcing them.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why agent harnesses matter for energy and token efficiency

A post on Neuralwatt's site argues that the software harness wrapping an AI model affects how much energy and how many tokens a task consumes. The piece frames harness design as a lever for efficiency alongside the model itself. It was shared on Hacker News, where it drew five points and no comments at the time of capture. The provided source text contains only the link metadata, so the specific arguments and figures in the article could not be verified here.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Cast a Local MKV With Subtitles to Chromecast Directly From Chrome

A guide explains how to stream a local MKV video, with either embedded or separate SRT/VTT subtitles, to a Chromecast or Google TV using only the Chrome browser. The user opens a web app in Chrome, selects the file, configures subtitles, and presses Cast, with the file read locally and sent over the home network rather than uploaded. The article notes that playback stops if the tab closes or the device sleeps, and that large files are streamed in chunks, with limits for files needing conversion. Android users can follow the same steps, with a screen-mirroring fallback available. Troubleshooting advice focuses mainly on network issues, such as guest Wi-Fi isolation blocking device discovery.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Tool Statically Diffs Electron App Builds to Reveal Security-Relevant Changes

A developer built hexvero, a tool that compares two Electron app.asar bundles without running either build. It links preload-script bridge members to IPC handlers and reports window setting changes such as nodeIntegration and sandbox. Where channel names cannot be resolved statically, it flags them as unresolved rather than guessing or dropping them. The tool is read-only and produces a local web view and Markdown reports, but its findings describe code contents, not exploitability. On a blind test set it found only part of the IPC channels and version changes, and the author publishes those misses.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PayGuard Local: Offline AI Tool Sorts Freelancer Invoices Into Structured Records

A developer has released PayGuard Local, a desktop utility that uses AI to organize freelancers' invoices and receipts into structured JSON records. The tool runs entirely on the user's machine using Meta's open-weight Llama 3.2 model via Ollama, so client data never leaves the laptop. It is aimed at freelancers, student developers, and small agencies who currently sort bills and log payments by hand. The creator says local inference avoids recurring per-scan API costs and allows users to swap in other models or adapt the schema to local tax formats. The project was submitted to the Hacktoberfest Open-Source AI Challenge and is available on GitHub.

0 comments Read more at DEV Community