Building a unified product recall database from seven government feeds is harder than it sounds

Multiple governments publish product recall data — including the EU's Safety Gate, France's RappelConso, openFDA, NHTSA, and CPSC — but integrating them into a single searchable database presents significant technical challenges. Each source uses a different data format and a different definition of what constitutes a recall event, making naive deduplication unreliable and requiring reconciliation at the identifier level. Product identifiers such as GTINs and lot codes are often buried in free-text fields, requiring regex parsing that rejects roughly one in five extracted values due to check-digit failures. Pagination limits in APIs like openFDA — which hard-caps the 'skip' parameter at 25,000 regardless of API key tier — force engineers to use date-windowed queries to access full datasets. A silent Worker crash during ingestion, which left six of seven data sources stalled while dashboards falsely reported healthy status, highlighted the need to track job state explicitly and isolate resource budgets across independent pipeline tasks.