Shadow Rebuilds AI Video Render Queue on PostgreSQL to Fix Memory and Frame-Drop Issues

·1 views

Engineering team at Shadow replaced a Redis/RabbitMQ-based job queue with a PostgreSQL-native system after discovering that queue starvation — not GPU availability — was the primary bottleneck in their AI video rendering pipeline. The new architecture uses PostgreSQL's SKIP LOCKED feature and a zero-idle-RAM contract to ensure workers only hold tensors for their current frame batch, preventing memory spikes. The pipeline processes video generated by MiniMax's Hailuo H3 model, which maintains a continuous skeleton state across frames to produce temporally coherent motion at 24fps. To meet cinema standards, the team also implemented synthetic shutter blur derived directly from H3's joint velocity estimates, avoiding a costly optical flow pass. The redesigned system satisfies three core requirements: job persistence through worker crashes, bounded memory usage, and fair dispatch between long and short rendering tasks.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How Shadow rebuilt its media pipeline using bitemporal PostgreSQL and CIEDE2000 color calibration

Shadow's engineering team overhauled its media synthesis pipeline after persistent color drift problems emerged across 90-minute renders at 24 frames per second. The pipeline stitches four streams per frame — including diffusion, depth, specular, and emissive layers — within a strict 41.66ms per-frame budget. To ensure perceptual accuracy, the team adopted the CIEDE2000 color difference formula, which better matches human vision than standard RGB delta-E, particularly in the blue-gamut range where most emissive overlays appear. Frames are treated as immutable events, enabling a bitemporal data model in PostgreSQL that records both application and system timelines, replacing an earlier Redis Streams setup that struggled with replay and ordering guarantees. This architecture allows engineers to reconstruct exactly what the compositor processed at any given moment, making post-render debugging of perceptual artifacts significantly more reliable.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Generalize Transaction Scope in NestJS Across TypeORM and MongoDB

A common NestJS pattern wraps entire HTTP request handlers in a single database transaction, which works well when only database operations are involved. Problems arise when a use case must call an external service — such as an AI model or payment gateway — before writing to the database, leaving the transaction open for minutes and holding pooled connections and row locks. The article proposes extracting the transaction executor's contract into a small domain-level interface, decoupling transaction control from any specific ORM. This abstraction is validated with two separate implementations — one for PostgreSQL via TypeORM and one for MongoDB — running identical use cases and tests against both. The exercise also highlights which engine-level differences no abstraction can fully hide, offering a practical boundary for how far such a generalization can reliably reach.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Multi-stage Docker builds cut Node image size from 1 GB to a fraction

Most inherited Node.js Docker images exceed one gigabyte, even though the actual application may occupy only around 40 MB, with the remainder comprising compilers, dev dependencies, npm cache, and source files. This bloat slows deployments, increases security scan times, and widens the attack surface for vulnerabilities. Multi-stage Docker builds address this by using multiple FROM instructions, allowing developers to compile the app in an early stage and copy only the necessary production artifacts into a leaner final image. The final image can be based on a minimal variant like node:20-slim or a distroless image, and should install only production dependencies via npm ci --omit=dev rather than copying node_modules from the build stage. Proper layer ordering — copying lockfiles before source code — also preserves Docker's build cache, keeping repeated builds fast.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CSS text-box-trim removes phantom text spacing gaps, but developers still ignore it

A native CSS property called text-box-trim, now supported across all modern browsers, automatically removes the invisible spacing embedded in font metrics above and below text characters. For years, developers worked around this issue using negative margins, line-height calculations, and hardcoded values sourced from design files. The property works alongside text-box-edge to align text precisely to the cap height and alphabetic baseline, eliminating the need for manual hacks. Despite its practical utility, the feature remains widely overlooked, partly because existing design systems and codebases were built around the old workarounds. The Syntax Podcast recently highlighted text-box-trim alongside other underused CSS features, bringing it to the attention of a broader developer audience.

0 comments Read more at DEV Community