Developer builds parallel video pipeline processing 50 clips at once using BullMQ and FFmpeg
A developer built BatchEdits, a tool allowing content creators to upload and process up to 50 videos simultaneously, with features like silence removal, caption generation, and platform-specific cropping. The core architecture uses Next.js for the API layer, BullMQ for job queue management, FFmpeg for video processing, and Whisper for audio transcription running concurrently across a pool of workers. To avoid server bottlenecks, video uploads bypass the Next.js server entirely using presigned URLs that send files directly from the browser to Cloudflare R2 storage. Parallel uploads are throttled to five concurrent connections using the p-limit library to prevent overwhelming slower networks. Once uploaded, all video jobs are queued simultaneously and processed independently by separate workers, ensuring a single failure does not halt the entire batch.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in