Boyang Yan

vLLM

11 min read

vLLM (virtual large language model) is an open-source library from UC Berkley.

A key innovation for inference is PagedAttention, which is an efficient virtual memory/page method for storing the KV cache, which in longer contexts can end up committing more memory than the models themselves.

PagedAttention also allows continuous batching, which is a method for handling multiple requests to LLMs that enables less idle time.

Like DeepSpeed, it also supports optimized CUDA kernels for lower latency inference.

It is built on top of Megatron and can interface with DeepSpeed.

Structure

vLLM is easiest to understand as two layers:

  1. A user-facing inference/serving layer.
  2. A high-performance execution engine underneath it.

The central runtime flow is: Python API or HTTP request ↓ Input/tokenization and configuration ↓ EngineCore ↓ Scheduler + KV-cache manager ↓ Executor ↓ Worker / ModelRunner ↓ Model layers and GPU kernels ↓ Sampling and output processing

Repository map

Directory Purpose ━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ vllm Main Python package ──────────────────────── ───────────────────────────────────────────────────── csrc Native C++/CUDA/ROCm implementations ──────────────────────── ───────────────────────────────────────────────────── rust Rust components and protocol support ──────────────────────── ───────────────────────────────────────────────────── tests Tests, generally organized like the main package ──────────────────────── ───────────────────────────────────────────────────── examples Runnable examples for common features ──────────────────────── ───────────────────────────────────────────────────── benchmarks Serving, throughput, latency, and kernel benchmarks ──────────────────────── ───────────────────────────────────────────────────── docs User, contributor, and architecture documentation ──────────────────────── ───────────────────────────────────────────────────── requirements Platform, build, test, and lint dependencies ──────────────────────── ───────────────────────────────────────────────────── cmake Native extension build configuration ──────────────────────── ───────────────────────────────────────────────────── docker Container images and entrypoints ──────────────────────── ───────────────────────────────────────────────────── .buildkite and .github CI and repository automation

The main Python package

Entry points

vllm/entrypoints contains public interfaces:

  • vllm/entrypoints/llm.py:66 defines LLM, the high-level offline Python API.
  • vllm/entrypoints/cli/main.py:17 dispatches commands such as vllm serve and vllm bench.
  • entrypoints/openai/ implements the OpenAI-compatible server.
  • entrypoints/serve/ contains the newer general serving infrastructure.
  • entrypoints/pooling/, generate/, and speech_to_text/ support other task types.

The CLI executable itself is registered in pyproject.toml:43.

Configuration

vllm/config defines configuration by concern:

  • model and tokenizer
  • scheduler
  • KV cache
  • parallel execution
  • compilation
  • quantization
  • speculative decoding
  • multimodal processing

The aggregate object passed through the runtime is typically VllmConfig. CLI and Python arguments are translated into it through vllm/engine/arg_utils.py.

The V1 engine

Most new engine work belongs under vllm/v1. Important parts are:

  • vllm/v1/engine/async_llm.py:70: asynchronous user/server-facing engine.
  • vllm/v1/engine/llm_engine.py:48: synchronous engine used by LLM.
  • vllm/v1/engine/core.py:98: owns the central inference loop.
  • vllm/v1/core/sched/scheduler.py:70: selects requests and token work for each step.
  • v1/core/: scheduling and KV-cache bookkeeping.
  • v1/executor/: selects how workers run—single process, multiprocessing, Ray, and platform-specific variants.
  • v1/worker/: device workers and model runners.
  • v1/attention/: attention abstractions and backend selection.
  • v1/sample/: token sampling.
  • v1/spec_decode/: speculative decoding.
  • v1/structured_output/: constrained generation.

A useful distinction: AsyncLLM / LLMEngine request lifecycle, input/output, streaming EngineCore engine loop and coordination Scheduler decides what token work runs next Executor sends work to one or more workers ModelRunner prepares tensors and executes the model

Models and layers

vllm/model_executor is the model implementation layer:

  • models/: architecture implementations and model registry.
  • layers/: attention, linear layers, MoE, normalization, quantization, and related building blocks.
  • model_loader/: loading weights from supported formats.
  • kernels/: Python-facing kernel integration.

A model implementation usually resembles its Hugging Face counterpart structurally, but replaces ordinary PyTorch layers with vLLM-aware parallel, quantized, and cache-aware layers. vllm/models contains newer or more specialized model families with platform-specific implementations.

Hardware and performance

Several areas work together:

  • vllm/platforms: detects CUDA, ROCm, CPU, TPU, XPU, and other platforms.
  • vllm/distributed: tensor, pipeline, data, and expert parallelism plus communication.
  • vllm/compilation: torch.compile, graph rewriting, and CUDA graphs.
  • vllm/kernels: Triton and other Python-side kernels.
  • csrc: compiled low-level operations.

The native code is important, but it is not the best place to begin unless you are specifically investigating a kernel.

  • transformers_utils/: Hugging Face compatibility.
  • tracing/ and v1/metrics/: observability.
  • plugins/: extension points.

How one request travels

For offline inference: llm = LLM(model=”…”) outputs = llm.generate(prompts, sampling_params) The approximate path is:

  1. LLM validates arguments and builds VllmConfig.
  2. LLMEngine or AsyncLLM preprocesses and tokenizes the request.
  3. An EngineCoreRequest is sent to EngineCore.
  4. The scheduler admits the request and allocates KV-cache blocks.
  5. The executor asks workers to execute the selected tokens.
  6. A model runner performs the forward pass.
  7. Sampling chooses output tokens.
  8. Engine outputs return to the output processor.
  9. Tokens are detokenized and returned or streamed.

The serving path adds HTTP request parsing and OpenAI response formatting around essentially the same engine.

Tests

Tests mirror architectural concerns:

  • tests/v1/: core V1 engine behavior.
  • tests/model_executor/: layers and model execution.
  • tests/models/: model-specific correctness.
  • tests/kernels/: kernel correctness, not performance.
  • tests/entrypoints/: CLI and server APIs.
  • tests/distributed/: multi-device behavior.
  • tests/multimodal/, tests/lora/, tests/spec_decode/: feature suites.
  • tests/evals/: evaluation coverage for output-affecting changes.

For a change, first look for an existing nearby test file. This project explicitly prefers extending existing suites over creating one-off files.

Deploying vLLM on Kubernetes

https://docs.vllm.ai/en/latest/deployment/k8s.html

Deploying vLLM on VM

Pre-requires

VM setting

In VM → Hardware:

Click Display, set to Default (or VirtIO-GPU).

Edit your PCI Device (01:00.0) and UNTICK “Primary GPU”. Keep All Functions + PCI-Express checked.

Install NVIDIA driver on ubuntu

Inside the VM:

sudo apt update
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers autoinstall
sudo reboot
# after reboot
nvidia-smi
lspci -nnk | grep -iA3 nvidia

Install vLLM

sudo apt update
sudo apt install -y python3-venv python3-pip build-essential
python3 -m venv ~/venvs/vllm
source ~/venvs/vllm/bin/activate
python -m pip install -U pip wheel setuptools

Install PyTorch (CUDA build)

Pick the CUDA 12.x wheel from PyTorch’s selector. Example (CUDA 12.4 wheel—if the site shows cu126/cu128, use that instead):

pip install --index-url https://download.pytorch.org/whl/cu124 torch torchvision torchaudio

Install vLLM

pip install vllm

Sanity check

python - <<'PY'
import torch, vllm
print("CUDA available:", torch.cuda.is_available())
print("CUDA reported by PyTorch:", torch.version.cuda)
print("Torch:", torch.__version__)
print("GPU:", torch.cuda.get_device_name(0))
print("vLLM:", vllm.__version__)
PY

Hugging Face Create a token

Sign in at huggingface.co → click your avatar → Settings → Access Tokens → New token.

Name it and choose Role = Read (enough to download models).

Click Create and copy the token (looks like hf_********).

Use the token on your vLLM

# in your vLLM Python env
pip install -U "huggingface_hub[cli]" sentencepiece
git config --global credential.helper store
hf auth login          # paste your hf_ token when prompted
hf auth whoami         # sanity check

Accept the model terms (once, in browser)

Sign in at Hugging Face with the account you’ll use on the VM. Open: https://huggingface.co/google/gemma-3-4b-it Click Agree and access (or Request access) and confirm. If you might also use the base model, do the same for google/gemma-3-4b

Deploy Gemma 3 4B on your vLLM VM (RTX 2070, 8 GB)

# one-shot shell
export HUGGINGFACE_HUB_TOKEN=hf_xxxxxxxxx...   # (optionally: export HF_TOKEN=$HUGGINGFACE_HUB_TOKEN)
# systemd (recommended)
sudo tee /etc/systemd/system/vllm.env >/dev/null <<'EOF'
HUGGINGFACE_HUB_TOKEN=hf_xxxxxxxxx...
HF_HOME=/opt/models/.cache/huggingface
EOF
sudo chmod 600 /etc/systemd/system/vllm.env
# then in /etc/systemd/system/vllm.service under [Service]:
# EnvironmentFile=/etc/systemd/system/vllm.env
sudo systemctl daemon-reload && sudo systemctl restart vllm
HF_HOME=/opt/models/.cache/huggingface \
vllm serve google/gemma-3-270m-it \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 2048 \           # you can raise later (even 4096 fits)
  --max-num-seqs 1 \               # keep concurrency low at first
  --gpu-memory-utilization 0.80 \  # headroom for kernels
  --swap-space 2 \                 # your VM has 10GB RAM; 2GB is safe
  --download-dir /opt/models \
  --trust-remote-code
sudo mkdir -p /opt/models && sudo chown $USER:$USER /opt/models
HF_HOME=/opt/models/.cache/huggingface \
vllm serve google/embeddinggemma-300m \
--host 0.0.0.0 --port 8000 \
--download-dir /opt/models

firewall setting

Open the port if you’ll call it from outside the VM

sudo ufw allow from 192.168.1.0/24 to any port 8000 proto tcp

Health check

alpine-docker:~# curl -i http://192.168.1.243:8000/health
HTTP/1.1 200 OK
date: Mon, 27 Oct 2025 03:47:22 GMT
server: uvicorn
content-length: 0

Run it as a service

# /etc/systemd/system/vllm-embed.service
[Unit]
Description=vLLM - EmbeddingGemma-300M
After=network-online.target
Wants=network-online.target
 
[Service]
User=yanboyang713
Environment=HF_HOME=/opt/models/.cache/huggingface
# If the model is gated, add: Environment=HUGGINGFACE_HUB_TOKEN=hf_xxx
ExecStart=/home/yanboyang713/venvs/vllm/bin/vllm serve google/embeddinggemma-300m \
  --task embedding --host 0.0.0.0 --port 8001 --download-dir /opt/models
Restart=always
RestartSec=3
 
[Install]
WantedBy=multi-user.target
 
sudo systemctl daemon-reload
sudo systemctl enable --now vllm-embed
sudo systemctl status vllm-embed --no-pager

Deploying vLLM on UVA High-Performance Computing Systems

login

ssh -Y rhe9cf@login.hpc.virginia.edu

Load Python / Miniforge

UVA recommends Miniforge for Python on HPC; available versions can be checked with module spider miniforge, and the default can be loaded with module load miniforge.

cd /sfs/gpfs/tardis/home/rhe9cf/Projects/vllm
module purge
module load miniforge/24.11.3-py3.12

If that exact version is unavailable, check:

module spider miniforge

Then load the available Python 3.12 or Python 3.11 Miniforge module.

Create the development environment

Create a named environment:

conda create -n vllm-py312 python=3.12 -y
conda activate vllm-py312

Or create it inside the repository:

conda create -p ./conda-env python=3.12 -y
conda activate ./conda-env

Install uv

python -m pip install uv

Check the machine

From the repository root:

nvidia-smi
uv --version

If nvidia-smi fails, you may be on a login/CPU node and need to request a GPU node first. If uv not load, please, load uv

module avail uv
module load uv
module load gcc/11.4.0
gcc --version
g++ --version
 
export CC="$(command -v gcc)"
export CXX="$(command -v g++)"

Create the development environment

The project requires Python commands to use uv and .venv/bin/python, not system python3 or bare pip.

uv venv --python 3.12
source .venv/bin/activate
 
uv pip install -r requirements/lint.txt
pre-commit install

For Python development and normal inference, use precompiled native components:

VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
 
# See detailed progress
VLLM_USE_PRECOMPILED=1 uv pip install -vv -e . --torch-backend=auto

This is much faster than compiling CUDA/C++ locally.

If you intend to modify csrc/ or other native code, build locally instead:

uv pip install -e . --torch-backend=auto
# See detailed progress
uv pip install -vv -e . --torch-backend=auto

That can take substantially longer and requires a working CUDA compiler toolchain.

Obtain Hugging Face access

Use the official instruction checkpoint: google/gemma-4-E2B-it Open its Hugging Face page, accept any applicable access terms, and create a read token. Then authenticate without putting the token directly in shell history:

.venv/bin/hf auth login

If that executable is unavailable:

uv pip install huggingface_hub
.venv/bin/hf auth login

Official Gemma 4 checkpoints also include E4B, 12B, 26B-A4B, and 31B variants. Start with E2B unless your GPU capacity clearly supports a larger model. Official Google Gemma guidance (https://ai.google.dev/gemma/docs/get_started)

Verify the installation

.venv/bin/python -c "import vllm; print(vllm.__version__)"
.venv/bin/python -m vllm.entrypoints.cli.main --help

The normal installed command should also work:

vllm --help

Run an OpenAI-compatible server

Start conservatively with a short context to reduce KV-cache memory:

vllm serve google/gemma-4-E2B-it \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --port 8000

If the GPU does not support BF16, try:

vllm serve google/gemma-4-E2B-it \
    --dtype float16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --port 8000

A long Gemma context requires much more KV-cache memory. Do not begin with its maximum context length.

Send a test request

In another terminal:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "google/gemma-4-E2B-it",
      "messages": [
        {
          "role": "user",
          "content": "Explain paged attention in three short sentences."
        }
      ],
      "max_tokens": 128
    }'

Or use the OpenAI Python client:

.venv/bin/python - <<'PY'
from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="unused",
)
 
response = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": "Explain paged attention in three short sentences.",
        }
    ],
    max_tokens=128,
    temperature=0.2,
)
 
print(response.choices[0].message.content)
PY

Run it directly from Python

For a simple text test without a server:

.venv/bin/python - <<'PY'
from vllm import LLM, SamplingParams
 
llm = LLM(
    model="google/gemma-4-E2B-it",
    dtype="bfloat16",
    max_model_len=4096,
)
print(outputs[0].outputs[0].text)
PY

supported quantized checkpoint, CPU offloading, or a GPU with more memory. Gemma 4 E2B’s name describes effective parameters; its total BF16 loading requirement is higher than a conventional 2B model.

Choose storage location

Use /scratch for Hugging Face model cache and your vLLM environment, not /home, because /scratch is intended for large computational work. Note that /scratch is temporary, not backed up, and files not accessed for more than 90 days may be deleted.

mkdir -p /scratch/$USER/vllm/{envs,hf-cache,logs,scripts}
cd /scratch/$USER/vllm

Set cache paths:

export HF_HOME=/scratch/$USER/vllm/hf-cache
export HUGGINGFACE_HUB_CACHE=$HF_HOME/hub
export TRANSFORMERS_CACHE=$HF_HOME/transformers

Load Python / Miniforge

UVA recommends Miniforge for Python on HPC; available versions can be checked with module spider miniforge, and the default can be loaded with module load miniforge.

module purge
module load miniforge/24.11.3-py3.12

If that exact version is unavailable, check:

module spider miniforge

Then load the available Python 3.12 or Python 3.11 Miniforge module.

Create a vLLM environment

vLLM’s stable docs require Linux, Python 3.10–3.13, and NVIDIA GPUs with compute capability 7.5 or higher.

conda create -p /scratch/$USER/vllm/envs/vllm python=3.12 -y
source activate /scratch/$USER/vllm/envs/vllm

Install uv and vLLM:

python -m pip install uv
uv pip install vllm --torch-backend=auto
uv pip install openai

vLLM’s docs recommend creating a fresh Python environment and show uv venv —python 3.12; they also recommend uv for installing vLLM wheels.

Check installation:

python -c "import vllm; print(vllm.__version__)"
python -c "import torch; print(torch.cuda.is_available())"

The second command may show False on a login node. That is okay; test CUDA inside a GPU job.

Choose a model and GPU size

For your first test, use a small or medium model. Examples:

export MODEL_ID="Qwen/Qwen2.5-7B-Instruct"

Avoid choosing a model that violates UVA policy. UVA’s RC usage policy page says users must comply with acceptable-use rules and includes restrictions on downloading or using prohibited applications such as DeepSeek AI on RC resources.

First test: interactive GPU session

Use this for debugging before writing a batch script.

salloc -A <your_allocation> \
  -p gpu \
  --gres=gpu:v100:1 \
  -c 8 \
  --mem=16G \
  -t 01:00:00

You can use uva slurm script generator

Graph View

Backlinks