vLLM
vLLM is an open-source inference and serving engine for large language models that uses PagedAttention to manage GPU key-value cache memory efficiently. It enables high-throughput continuous batching and distributed inference behind an OpenAI-compatible API.
Baseline topologies
| Topology | Minimum nodes | Roles |
|---|---|---|
| single-gpu-node | 1 | vllm-server |
| multi-gpu-node | 1 | vllm-server-tensor-parallel |
What drives the cost
- Largest model size (B parameters)
- Largest model being served, in billions of parameters. Primary driver for GPU VRAM requirements and host memory sizing.
- Peak concurrent requests
- Peak simultaneous inference requests. Drives KV cache memory sizing, continuous batching throughput, and host CPU scheduling overhead.
Where it runs
bare_metal, dedicated, cloud_vm
Size vLLM and get a cost estimate