SHA-256 Local Deduplication Cuts Redundant API Calls in File Intake Pipelines

Developers building file intake pipelines often encounter duplicate submissions from multiple sources, re-uploads, or retried jobs, leading to redundant proof records when APIs are called without prior deduplication. A two-layer approach addresses this: compute a SHA-256 digest locally before any API call, and skip submission if the digest has already been processed. A persistent local store of seen digests ensures the worker retains this knowledge across restarts, while chunked file reading keeps memory usage flat for large files. When the ProofLedger v1 API does receive a new submission, it returns a duplicate_of field pointing to the earliest canonical proof ID if the hash was previously anchored. Handling rate-limit responses by raising exceptions rather than silently skipping ensures backoff and retry logic can be managed deliberately at the call site.