Developer benchmarks six AI models on DigitalOcean Inference; cheapest model tops the test

A developer built a ~390-line open-source Python tool to simultaneously benchmark six large language models — including Llama, DeepSeek, Mistral, Qwen, and OpenAI's open-weight models — hosted on DigitalOcean's inference platform. The tool fires a single prompt at all six models in parallel, streaming responses side by side while tracking time-to-first-token and cost per run. A key finding was that DigitalOcean's inference endpoint uses an OpenAI-compatible API, meaning all six models could be queried through a single client by changing only the model string. The developer also discovered that using Gunicorn's default synchronous worker caused requests to queue sequentially, inflating total response time to over 10 seconds; switching to threaded workers cut that to 6.4 seconds. Notably, the lowest-cost model in the test delivered the best overall performance, challenging the common assumption that more expensive models are worth the trade-off.