How a Real Estate Data Tool Learned to Detect Silent Failures in City Permit Scrapers

AddressIntel, a real estate intelligence platform, has compiled 187,838 building permits from 20 jurisdictions across the San Francisco Peninsula, exposing how inconsistent and error-prone municipal permit portals can be. California cities use at least six different vendor portal systems, many requiring logins, JavaScript rendering, or real browser sessions, making automated data collection highly fragile. The platform identified two costly silent failures: a login rejection that caused a city's permits to vanish from the product unnoticed for weeks, and a search result cap on eTRAKiT portals that silently truncated results without any error, dropping the newest permits entirely. Unlike loud failures that trigger immediate alerts, these quiet errors returned plausible-looking data, allowing bad information — including a falsely zeroed demolition count — to reach published content. The write-up outlines how measuring each portal's result cap directly became a necessary validation step before any scraped data could be trusted.