LiteLLM Gateway
LiteLLM is an open-source AI gateway and SDK that exposes over 100 different LLM provider APIs through a single OpenAI-compatible interface, adding cost tracking, load balancing, guardrails, and logging. It can be used as a Python library or run as a standalone proxy server in front of multiple model providers.
What gets deployed with it
- PostgreSQL HA Cluster — required LiteLLM cost tracking, virtual key storage, and request logging persistence dependency.
Baseline topologies
| Topology | Minimum nodes | Roles |
|---|---|---|
| single-node-proxy | 1 | litellm-proxy |
| ha-proxy-pair | 2 | litellm-proxy |
What drives the cost
- Peak routed requests per second
- Peak inference request throughput routed through the gateway. Drives CPU and connection pool sizing.
- Connected model providers
- Distinct model providers or backends the gateway routes to. Affects connection management and routing complexity.
Where it runs
bare_metal, dedicated, cloud_vm
Size LiteLLM Gateway and get a cost estimate