A CLIP score of 0.85 does not equal 85% listing accuracy

·3 views

A high confidence score from an image-text model can seem more authoritative than the evidence behind it. This article examines a marketplace verifier from Eşya Bul that uses CLIP to compare listing photos against short text descriptions and keeps listings whose best positive score exceeds 0.85. The code shows the score only answers which supplied description best matches a photo, not whether a seller is honest, an item works, or payment is safe. Because softmax normalizes scores against the candidate set, the same logit can yield a much lower share when a strong alternative is added, and scores from different product groups are not directly comparable. The author recommends testing narrow, human-applicable labels, such as whether an image shows a complete appliance rather than a replacement part, and analyzing errors separately for category selection and the English-language descriptions.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Argues Tool Engineering Will Replace Prompt Engineering as AI Skill of Choice

A developer writer contends that prompt engineering, once a prized skill, is becoming obsolete as language models grow more capable. He points to three drivers: models now grasp plain-language requests without elaborate scaffolding, very large context windows reduce the need for compression tricks, and fine-tuning and retrieval-augmented generation solve problems that prompt hacks once addressed. He proposes a successor discipline, Tool Engineering, centered on building, selecting, and orchestrating the tools AI agents use to act in the real world. The piece argues that a model without tools can reason but cannot take actions such as booking a hotel or running a database query. Developers who keep investing in prompt techniques, he warns, risk becoming increasingly irrelevant.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Recording Free Inference Calls in a Ledger Before Making Them

The article argues that free inference lanes fail less on quality than on accounting, since unrecorded spend cannot be subtracted from a budget. It proposes placing a ledger in front of the lane and recording each reservation before the upstream call is made. Because workers can die mid-call and retries can double spend, the ledger must survive crashes without manual cleanup. The author describes a lease-based design where each job holds an open budget row that is committed or released, with a sweeper expiring stale leases. The implementation uses only the Python standard library and a single SQLite file, suitable for a laptop or small always-on machine.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Unveils Gemini 4 Argon With 1M-Token Output, Rolling Out Gradually

Google DeepMind announced Gemini 4 Argon on September 30, 2026, positioning it for complex software engineering, knowledge work, and defensive cybersecurity. Its standout specification is an output limit of up to 1 million tokens, raised from 64K, which could support long multi-step tasks. Access is initially limited to trusted cyber defenders and testers, with broader developer and enterprise availability still to come. Google reports a 77.9% score on DeepSWE v1.1 along with an introductory API price of $2 per million input tokens and $10 per million output tokens. The article cautions that these are publisher-reported figures, not a controlled head-to-head comparison with OpenAI's GPT-6.1 Sol.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Log Model Call Metadata Instead of Payloads to Reduce Data Leak Risk

The article argues that AI features often leak sensitive data through logs rather than request bodies, since logging receives less security scrutiny. It proposes logging only a "shape" record for each model call, including a payload digest, byte size, field names, latency, and attempt count, instead of message content. Encryption is deemed insufficient because logs are decrypted when searched, exported, or shared in support bundles. The approach is enforced with a CI test that scans log fixtures for disallowed fields and sensitive patterns such as emails, bearer tokens, and private keys. The author also recommends including a deliberately leaky fixture to confirm the scanner actually detects leaks.

0 comments Read more at DEV Community