Engineering team cuts incident log query time from 6 minutes to 4 seconds

A software team experienced a 50-minute incident in June where slow log queries — taking six to eight minutes each — left engineers able to ask only seven questions during the entire outage. The root cause was a poorly structured log storage setup where all 40 services fed into a single daily index, forcing searches to scan up to 1.4 terabytes of data with no filtering. The team restructured their logging by routing data into per-service indices, tiering storage by age across fast and cheaper nodes, and defaulting search views to a one-hour window with service filters pre-applied. They also replaced manually typed queries with five pre-built buttons covering the most common incident questions, such as error counts by endpoint and slowest endpoints. The changes brought the median query time in recent incidents to under ten seconds, and the team now tracks query latency as a formal metric in post-incident reviews.