Cheap LLM API Relays May Secretly Swap Models; Here's How to Verify

Some low-cost API relay services that offer access to flagship AI models like GPT or Claude may quietly substitute smaller, cheaper, or heavily quantized models while billing customers for the premium tier. The problem is difficult to detect because performance often appears normal initially and only degrades weeks later, after users have stopped actively monitoring. Asking the endpoint to self-identify is unreliable, as responses can be easily spoofed through system prompts or fine-tuning. More reliable detection methods include analyzing tokenizer behavior via token-count patterns, running objective capability tests with known answers, and probing context-window limits using hidden passphrases. A developer affiliated with one such gateway has published this verification playbook along with an open-source tool designed to automate the testing process against any OpenAI- or Anthropic-compatible endpoint.