The vLLM agent deploys and manages a vLLM inference server: GPU node group provisioning, model weight caching, OpenAI-compatible API, continuous batching, HPA on queue depth, and Prometheus metrics — on AWS EKS (primary).

What it deploys

• EKS GPU node group — g5.xlarge (1× A10G, 24 GB VRAM) or p4d.24xlarge (8× A100) with GPU taint and NVIDIA device plugin
• vLLM Deployment — vllm/vllm-openai container image, OpenAI-compatible /v1/chat/completions and /v1/completions endpoints on port 8000
• HPA — scales 2 → 8 replicas on vllm_num_requests_running > 16 (custom Prometheus metric)
• S3 weights cache — model weights stored in S3, mounted via PVC at pod start (~3 min cold start for an 8B model)
• NLB Service — internal load balancer for in-cluster or VPC-internal consumers
• Prometheus ServiceMonitor — scrapes /metrics (TTFT/TPOT histograms, KV cache utilization, batch size, queue depth)
• Grafana dashboard — four golden signals + GPU memory utilization + request acceptance rate

Quickstart

1. Start the agent: “Deploy a vLLM server for Llama 3.1 8B Instruct.”
2. Gate 1 — model and requirements: Agent asks for model name, peak QPS, TTFT target, and TPOT target. Reply with your constraints.
3. Gate 2 — topology: Agent proposes g5.xlarge (A10G) with continuous batching, --gpu-memory-utilization 0.9. Confirm, or ask for A100 + tensor parallel for 70B models.
4. Gate 3 — sizing: Agent shows memory math: 8B FP16 fits in 24 GB with KV cache headroom. Confirms 2 min replicas, HPA to 8.
5. Gate 4 — weights storage: Agent proposes PVC + S3 cache (best cold-start speed). Approve.
6. Gate 5 — observability: Agent wires Prometheus + Grafana + alert rules (TTFT P95 > 2s, queue > 100, GPU util > 95%). Approve.
7. Gate 6 — review and deploy: Terraform plan shows EKS GPU node group, Helm NVIDIA device plugin, S3 bucket, vLLM Deployment, HPA. ~15 minutes (GPU node bootstrap takes longer).

After deploy the agent outputs:

• Internal NLB endpoint (e.g. http://vllm-svc.vllm:8000)
• Example curl to test: POST /v1/chat/completions
• S3 weights bucket name

Configuration options

GPU sizing reference

Common patterns

Testing the deployed API

curl http://vllm-svc.vllm:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Serving multiple LoRA adapters

vLLM 0.6+ supports hot-swap of LoRA adapters on the same base model fleet. Tell the agent: “Add a LoRA adapter my-org/llama-3-finetuned to the existing vLLM deployment.” The agent updates the Deployment args with --lora-modules and manages adapter loading without a full redeploy.

Scaling for higher QPS

Tell the agent: “Our TTFT P95 is climbing — we’re seeing 50 concurrent requests.”

The agent will:

1. Review current HPA thresholds and GPU utilization metrics
2. If a single A10G is saturated, propose upgrading to g5.12xlarge (4× A10G) with TP=4
3. Alternatively, increase max_replicas if horizontal scaling is preferred and latency budgets allow it

Troubleshooting

See also

• Concepts: Agents
• Concepts: Gates
• Reference: Agents list