intermediateServing ~30 min

Serve an LLM with vLLM

Deploy vLLM and send an inference request.

Scenario

You need a high-throughput OpenAI-compatible endpoint for a model.

Prerequisites
  • ·A GPU node with enough memory for the model
  • ·kubectl access
Tasks
0 / 3 tasks
  1. bash
    kubectl create deploy vllm --image=vllm/vllm-openai:latest -- \
  --model mistralai/Mistral-7B-Instruct-v0.2
kubectl patch deploy vllm --type=json -p='[{"op":"add","path":"/spec/template/spec/containers/0/resources","value":{"limits":{"nvidia.com/gpu":1}}}]'
  2. bash
    kubectl expose deploy vllm --port=8000
kubectl port-forward svc/vllm 8000:8000
  3. bash
    curl localhost:8000/v1/completions -H 'Content-Type: application/json' \
  -d '{"model":"mistralai/Mistral-7B-Instruct-v0.2","prompt":"Hello","max_tokens":16}'

    Expected · A JSON completion response.

Success criteria
  • ✓The vLLM pod is Ready
  • ✓The completion endpoint returns a response
Hints
  • ·Watch GPU memory — pick a model that fits.
  • ·First request is slow while weights load.