Developer's LLM agent hit 182s p95 latency, forcing a pivot from real-time to background use

A developer building Porchlight, a membership win-back tool, recorded 67 timed interactions with a live LLM agent and found a median response time of 29.2 seconds alongside a 95th-percentile latency of 182 seconds. The stark gap between the two metrics effectively ruled out any real-time, user-facing deployment of the agent. Porchlight is designed to conduct exit interviews with cancelled members, match their stated reasons for leaving against creator announcements, and draft personalised win-back messages. The developer also benchmarked the LLM against a simple keyword-matching baseline, finding the agent resolved 11 departures that the baseline completely missed across 54 test cases. The latency data, gathered before the product design was finalised, became the deciding factor in how the tool could realistically be deployed.