OCR and VQA Pipeline Outperforms CLIP and BLIP for Marketplace Product Mismatch Detection

A two-part technical series on DEV Community details how CLIP and BLIP models fail to catch fine-grained product mismatches in online marketplaces, such as wrong brand or incorrect size. The core problem is architectural: both models compare images and descriptions as whole units, producing a single similarity score that cannot isolate which attribute caused a mismatch. The proposed solution replaces holistic scoring with a feature dictionary approach, where attributes like brand, size, variant, and container type are each verified independently. OCR is used to extract printed text directly from product images, while a Visual Question Answering model handles attributes that cannot be read as plain text. Tests on three real grocery products — including milk, chicken, and butter packaging — demonstrate that this pipeline can reliably flag individual attribute mismatches that earlier models missed.