Developer builds internal doc Q&A bot using RAG, shares lessons on embedding pitfalls

A software developer spent a weekend building a question-and-answer bot to help their team quickly search through 200-plus pages of internal documentation spread across Confluence, Google Docs, and PDFs. The project used a Retrieval-Augmented Generation (RAG) approach, combining OpenAI embeddings, a Pinecone vector database, and GPT-4 to generate answers from retrieved document chunks. Early attempts with fixed-character chunking and naive retrieval produced poor results, with relevant content often split across chunks or buried beyond the top results. The developer ultimately settled on a hybrid pipeline that chunks documents by paragraph, combines dense embeddings with keyword search, retrieves ten candidate chunks, and reranks them using a lightweight cross-encoder before passing the best three to GPT-4. The experience highlighted that chunking strategy and reranking logic are critical, often overlooked factors in building reliable document search systems.