Deduplicating job postings before LLM scoring to stop repeat judgments
An author's nightly job-matching agent kept sending identical postings to an LLM, paying for verdicts it had already produced. Because model verdicts are nondeterministic, the same posting could even receive different scores on different runs. A simple URL-based seen-set fails because reposts carry tracking parameters and edited postings keep stale verdicts under the same address. The solution instead fingerprints normalized posting content, with URL keys stripped of query strings, and stores the results in a lightweight SQLite table. A dedicated graph node performs this check so the model is never called for jobs already judged.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in