Developer Builds Hand Gesture Controls for AI Coding Agent Using Webcam
A developer built a system that lets hand gestures control an Anthropic-powered AI coding agent, replacing keyboard prompts entirely. Three gestures were mapped to distinct actions: a pinch triggers code writing, a spinning index finger starts brainstorming, and two fingers mimicking running executes the generated code. The project used MediaPipe for hand landmark tracking, though the newer Tasks API caused GPU crashes on Mac, forcing a switch to the legacy API pinned at version 0.10.21. Gesture detection relied on distance thresholds, fingertip oscillation counts, and signed angle accumulation to distinguish intentional movements from accidental hand drift. The developer noted the unsandboxed setup was a deliberate choice to make the live demo more compelling, and plans to add a 3D avatar overlay for future recordings.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in