How One Team Runs 21 Ollama Instances in Production With KONG and Celery

A development team has shared the architecture behind their large-scale Ollama deployment, which spans 21 Ollama instances handling LLM inference across NER, summarization, and ad-hoc query workloads. Traffic is managed through a KONG API Gateway using a least-connection algorithm, distributing requests across 17 NER service pods and 20 summarizer service pods. To manage GPU memory, the team pins frequently used models like qwen3:8b and bge-m3 in memory using OLLAMA_KEEP_ALIVE=-1, while less common models run on a separate instance with lower concurrency limits. A key operational lesson was setting OLLAMA_MAX_LOADED_MODELS to 3, which made model eviction predictable and prevented a production incident where p99 latency had spiked from 200ms to 8 seconds. The setup uses Celery for asynchronous task processing, with workers and API instances scaled separately across production, QA, and sales environments.