LiteLLM gateway runs on Google Cloud and Azure, serving both model families with no API keys

A proof-of-concept deploys two AI agents, one on Google Cloud and one on Azure, that each have their own identity and store no secrets. Both reach models through a LiteLLM model gateway running in each cloud, sharing one configuration and one Postgres database for agents, allowed models and usage. Each gateway can call both Gemini and Azure OpenAI, so an agent gets the same model catalogue from either side. Instead of issuing virtual API keys, the gateway authenticates agents with their Microsoft Entra tokens, checking an app role and a per-agent model allowlist. During a demo, one gateway was switched off and the agent's next request was served by the other cloud without noticeable disruption.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in