Developer builds stdlib-only git secret scanner, battles packfile format edge cases
A developer built a secret-scanning tool during a hackathon that reads git repository object databases using only Python's standard library, with no third-party git bindings allowed. The tool was designed to detect API keys and credentials across a repo's full history, including long-deleted files. A major hurdle emerged when git's garbage collection compressed loose objects into packfiles — the format used by every repository cloned from GitHub — forcing a full packfile parser to be written from scratch. Two subtle bugs surfaced: an offset varint encoding that silently differed from the size varint, producing plausible but wrong data without crashing, and a recursive delta resolver that exponentially re-processed shared base objects on larger repositories. Both issues were caught through rigorous SHA checksum verification and fixed with memoisation, yielding a working scanner that handles real-world repositories.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.



Discussion (0)
Log in to join the discussion and vote.
Log in