CryptoBench tests whether LLMs catch cryptographic API misuse in short Python snippets

CryptoBench is a small synthetic dataset of 54 Python snippets covering nine categories of cryptographic misuse, each mapped to a CWE. Each category includes vulnerable examples as well as secure controls, so that models flagging everything cannot look accurate. The author evaluated seven open local models through Ollama, running each snippet five times for a total of 1,890 trials. Only three models separated secure from insecure code, and detection rates varied widely across categories, with weak PRNG being the hardest to catch. The author suggests models recognize flagged function names but miss ordinary APIs used incorrectly, though this remains untested. The dataset and harness are publicly available for others to rerun on their own models.