Developers build fused video search using visual and transcript embeddings on a shared timeline

·1 views

A new developer tutorial demonstrates how to build semantic search inside videos by combining visual shot embeddings and transcript text chunks, both anchored to the same video timeline. The system uses shot detection via PySceneDetect, SigLIP-2 embeddings for visual frames, and Whisper for speech transcription, with results stored in a pgvector-backed database. At query time, both indexes are fused so that a natural-language search query returns a precise video timestamp rather than just a video ID. The approach addresses a core limitation where frame-only search misses spoken content and transcript-only search cannot identify visual elements like diagrams or slides. The tutorial provides working Python code for each pipeline stage, including shot boundary detection, midpoint frame extraction, and normalized vector embedding for both image and text inputs.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to detect and fix variable frame rate video uploads before they break your pipeline

Screen recording tools like OBS, QuickTime, Loom, and phone capture apps produce variable frame rate (VFR) video, which causes audio drift when processed by transcoders that assume a constant frame rate (CFR). The drift is subtle at first but becomes clearly noticeable around the four-minute mark, making it a silent but serious problem for platforms that accept user-uploaded tutorials, demos, or gameplay footage. A two-tier detection approach using ffprobe can identify VFR files: a cheap header comparison between r_frame_rate and avg_frame_rate serves as a first-pass filter, followed by a per-frame timestamp analysis to confirm irregular gaps. Once a VFR file is confirmed, FFmpeg 8's -fps_mode cfr flag can normalize it at ingest before it causes downstream issues. The article provides Python 3.11 scripts and ffprobe commands to automate both detection tiers efficiently.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Scroll-Aware HLS Video Preload Window Under Safari's Element Limit

A developer tutorial on DEV Community explains how to fix stuttering in vertical video feeds by building a directional preload window using hls.js. The approach attaches HLS sources only to off-screen items ahead of the user's scroll direction, while aggressively releasing elements outside the window to avoid crashing on Safari, which caps the number of active video elements. Unlike the browser's native preload attribute, which mobile browsers largely ignore, hls.js gives developers direct control over when segment fetching begins. The solution uses a reconciler pattern of roughly 120 lines, with separate buffer configurations for active and preloading items to avoid burning users' data plans. A small asymmetric window — four items ahead and one behind — is recommended to match real-world scroll behavior.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds Aura Router to add SPA navigation to HTML and Web Components projects

A developer created Aura Router after struggling to maintain both React and Web Components in the same project when client-side navigation was needed. The router is designed to add single-page application navigation to existing HTML and Web Components projects without introducing a second component model. It requires every public URL to return complete, indexable HTML, allows routes to be declared as Custom Elements, and ensures ordinary links still function without JavaScript. A live demo illustrates that Aura updates page content without triggering a full browser reload, while also working correctly with JavaScript disabled. The initial release, version 0.1.0, is available via npm under the package name @auraui/router.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Sliding Window Technique Cuts Algorithm Runtime from O(n²) to O(n)

The sliding window is a programming technique that uses two pointers, left and right, to maintain a moving subarray or substring without rescanning elements from scratch. As the right pointer expands the window, a state such as a frequency map or running sum is updated incrementally; when a constraint is violated, the left pointer advances to shrink the window. Each element enters and exits the window at most once, keeping total operations proportional to n and achieving O(n) time complexity. Classic problems like finding the longest substring with at most K distinct characters or the minimum subarray with a given sum drop from O(n²) brute-force solutions to linear time using this approach. A common pitfall is failing to update the tracked state when the left pointer moves, which causes the window's data to become stale and produce incorrect results.

0 comments Read more at DEV Community