Five hard-learned lessons for scraping Indian government open data reliably

A developer maintaining Village Finder, an open-source project tracking over 78,000 Indian villages, has shared practical lessons from daily automated data pulls from Indian government APIs. One key finding is that the data.gov.in gateway's Web Application Firewall silently returns HTTP 502 errors when it detects default Python request signatures, rather than a standard 403, misleading developers into chasing infrastructure problems. The developer recommends setting a descriptive custom User-Agent string as a simple fix and adopting the Unix EX_TEMPFAIL exit code (75) to handle upstream outages gracefully without triggering false pipeline failure alerts. For undocumented government portals like myScheme, which lacks a public API, inspecting client-side network requests can reveal internal query endpoints, though these typically require matching Origin and Referer headers to avoid authentication errors. These lessons aim to help developers building civic data pipelines avoid days of unnecessary debugging.