Developer Builds Privacy-First RAG Assistant Using Ollama, ChromaDB and LangChain

A Master's student in AI & Big Data built a fully local retrieval-augmented generation (RAG) assistant during a software services company internship to help technicians query large PDF documentation libraries more efficiently. The system was designed with a strict constraint: no sensitive financial data could be sent to external cloud providers like OpenAI or Anthropic, making a fully local deployment essential. The solution uses Ollama to run Llama 3 locally, ChromaDB as a vector database, FastAPI for the RAG pipeline, and Streamlit for the user interface, all orchestrated via Docker Compose. During ingestion, over 2,100 pages of accounting and HR manuals were split into nearly 10,000 text chunks, embedded as vectors, and stored locally for similarity-based retrieval at query time. The approach eliminates the need for model fine-tuning, as the LLM reads only the retrieved document passages as context for each query rather than relying on pre-trained knowledge.