Developer shares four hard-won lessons building a reliable SSL cert monitoring tool
A developer building an Apify actor to monitor Certificate Transparency logs for unexpected SSL certificates discovered that crt.sh, a free community-run CT log search service, returns HTML error pages instead of JSON during high load, making 404 and 5xx responses unreliable as 'no results' signals. Retry logic initially set to four attempts proved insufficient when the service returned six consecutive 502 errors during a bad patch, pushing the tool's 30-day failure rate to 46%, prompting an increase to eight attempts with capped backoff. A missing request timeout caused some fetch calls to hang indefinitely, bypassing retry logic entirely, which was fixed by adding an AbortController with a per-attempt timeout. Most critically, a field used for date filtering silently disappeared from crt.sh's JSON response, causing the actor to return zero results every time without throwing any error — the kind of silent failure especially dangerous in a security monitoring tool.
