Developer Builds RAG Pipeline Using Vision Language Model to Handle Mixed Document Formats

A developer has redesigned a semantic retrieval pipeline to address the challenge of processing heterogeneous input formats such as PDFs, spreadsheets, and images. The solution uses a Vision Language Model (VLM) as an extraction layer, converting each document page into an image and sending it to the model via the Ollama API for summarization and keyword extraction. The resulting text summaries are embedded and stored in PostgreSQL using the pgvector extension, enabling semantic search by an AI agent. The pipeline was optimized by switching from the Qwen3.5 20B model to the smaller 4B variant, improving processing speed and reducing resource consumption. Additionally, the model's built-in reasoning mode was disabled after it caused empty outputs on some pages by exhausting the token limit, with conciseness and retrieval quality remaining acceptable after the change.