DGX Spark + vLLM Playbook

A practical, end-to-end guide for serving LLMs with vLLM on a DGX Spark (GB10 Grace Blackwell), assembled from NVIDIA’s official playbook, the vLLM team’s deep-dive, model cards, and battle-tested community setups.

⚠️Warning

Flag choices on Spark are model- and image-specific, not hardware-wide defaults. The recipes below are starting points that worked for their authors against a specific container tag. Validate against the exact image you run, and pin a known-good tag/digest for anything you depend on. Copying a flag from one model’s recipe to another can silently regress throughput or output quality.

1. The landscape: current playbooks & walkthroughs

When vLLM is the right engine (and when it isn’t)

Four engines run well on Spark — vLLM, SGLang, llama.cpp, and TensorRT-LLM — and the right pick is set by your workload, not a star rating. This guide is vLLM-centric because vLLM is the strongest fit for serving; here’s the honest boundary:

Rule of thumb: concurrency → vLLM/SGLang; one local user → llama.cpp; squeeze every last token/s on one stream → TensorRT-LLM. Confirm on your own box — all four ship runnable Spark containers.

2. Hardware facts that drive every config

DGX Spark is a GB10 Grace Blackwell SoC with 128 GB unified memory shared by CPU and GPU, on a consumer-Blackwell sm_121 GPU, ~273 GB/s bandwidth, and an ARM64 (aarch64) host. Four consequences shape everything below:

• One memory pool for everything. Weights, KV cache, the OS, page cache, and the runtime all draw from the same 128 GB. –gpu-memory-utilization is a fraction of that shared pool — leave headroom or you OOM with “free” RAM still showing.
• Memory-bandwidth bound for decode. Spark shines at small-batch, single-user interactive serving. Keep concurrency low (–max-num-seqs ≈ 1–4).
• NVFP4 MoE is the sweet spot. Quantized MoE with ~3–13 B active params (even at 25–120 B total) is the best fit: small active set = fast decode, and 4-bit weights free unified memory for KV cache. Prefer NVFP4/FP8/MXFP4 over BF16.
• Use sm_121-validated builds. The upstream stable vLLM image does not support GB10. Use the CUDA-13 nightly track (vllm/vllm-openai:cu130-nightly / :nightly) or NVIDIA’s NGC container (nvcr.io/nvidia/vllm:26.02-py3+).

Measured GB10 performance (what to actually expect)

These are real numbers measured on one DGX Spark with this guide’s bench script (§10) — not datacenter figures or vendor claims. Decode is bandwidth-bound, so active parameter count dominates: both of these few-billion-active MoEs sit near their bandwidth ceiling, while a dense model the same size crawls (Gemma-4-31B ≈ 6 tok/s). The Qwen3.6-vs-Gemma gap below is mostly speculative decode — Qwen3.6’s built-in MTP draft, which Gemma-4-26B-A4B lacks.

Speculative decoding is the single biggest single-stream lever on Spark — often a bigger win than swapping models. A small drafter proposes tokens the big model verifies in one pass, so accepted tokens come nearly free. On GB10, AEON-7’s measurements take a 27B from ~12 tok/s to ~33 tok/s single-stream — up to 2.7× — in the chart below, which sweeps speculative-token count against prompt length. Two things to read off it. First, throughput isn’t monotonic in token count: n=10 dips just under n=5 on short prompts, because more proposed tokens means more wasted work whenever one is rejected — expected variance, not a measurement error. Second, the win is sharply content-dependent, and content type swings it more than anything the chart varies — code and reasoning accept far longer blocks than free prose (~60 vs ~30 tok/s; §14.4). Full setup in §14.

Prefill / time-to-first-token is a different bottleneck — it’s compute-bound, and Marlin FP4 is slower for it. Short prompts hit first token in ~50–70 ms, but long prompts scale badly: at deep context (tens of thousands of tokens) a large model can take minutes to first token — ~133 s in one test. Keep prompts tight, and use –enable-prefix-caching so repeated system prompts aren’t re-prefilled.

Concurrency is where Spark surprises you. Single-stream reviews understate it — measured on one Spark (vLLM), the same Gemma-4-31B that does 6 tok/s solo reaches ~92 tok/s aggregate at 16 concurrent requests, and a small Qwen2.5-3B jumps 26 → 477 tok/s at the same concurrency (≈1,460 at 64). If you serve more than yourself, benchmark at the concurrency you’ll actually use, not at batch 1.

Picking a model for the job

(Throughput figures are the measured GB10 decode rates above; capability picks follow each model’s published evals — accuracy is quantization-, not hardware-, dependent, so a model’s published quality holds on Spark.)

3. Storing all models in ~/vllm

vLLM fetches weights through the Hugging Face stack, which caches under the directory named by HF_HOME. Point that at a host directory and mount it, and every model lives in ~/vllm, shared across containers.

mkdir -p ~/vllm

Pre-stage on the host (recommended) — files end up owned by you, the download happens once, and the first vllm serve doesn’t stall on a multi-GB pull:

pip install -U "huggingface_hub[cli]"        # one-time
export HF_HOME=~/vllm
export HF_TOKEN=hf_xxx                          # for gated models; accept the license on the HF page first
hf download nvidia/Qwen3.6-35B-A3B-NVFP4

The mount, in every docker run:

  -e HF_HOME=/models \
  -v ~/vllm:/models \

Inside the container HF now caches at /models (= ~/vllm on host); pre-staged weights at ~/vllm/hub/… appear at /models/hub/… automatically.

💡Tip

Force fully-local loading (air-gapped, no network checks) with -e HF_HUB_OFFLINE=1 once weights are staged. If you let the container (root) download instead of pre-staging, files in ~/vllm are root-owned — add –user $(id -u):$(id -g) if that matters.

4. Is volume mounting necessary?

Yes — treat it as mandatory. A container filesystem is ephemeral: a freshly downloaded 27–120 B model is destroyed when the container is removed, so without a mount you re-pull many GB and re-pay the 10–15 min weight-load on every run. The vLLM team’s guidance is “download once, mount everywhere.”

The one other volume worth adding is the torch.compile cache (the slow part of cold start), which lives separately from your models at /root/.cache/vllm:

  -v ~/vllm-compile-cache:/root/.cache/vllm \

The compile cache only pays off when you recreate a container (image/flag change) — a long-running –restart unless-stopped container compiles once anyway. It’s keyed to GPU arch + vLLM version + model + flags, so clear it (rm -rf ~/vllm-compile-cache/*) if a stale entry causes a startup hiccup after an upgrade. –ipc=host / –shm-size is shared memory (RAM), not a -v volume.

5. One-time host setup

nvidia-smi                                   # GPU + driver visible
docker ps                                    # else: sudo usermod -aG docker $USER && newgrp docker
docker run --rm --gpus all nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi
mkdir -p ~/vllm
export HF_TOKEN=hf_xxx                        # gated models
# NGC pulls 401? -> docker login nvcr.io  (user "$oauthtoken", password = NGC API key)

⚠️Warning

Unified-memory OOM valve. Because CPU and GPU share DRAM, the Linux page cache can hold memory CUDA can’t reclaim — an “OOM” well under 128 GB. If a big model fails to load after heavy file activity, flush caches first: sudo sh -c ‘sync; echo 3 > /proc/sys/vm/drop_caches’

6. Baseline single-Spark recipe (template)

Minimal, model-agnostic shape — swap the model handle and per-model flags from §7.

export HF_TOKEN=hf_xxx   # only for gated models

docker run -d --name vllm --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -e HF_HOME=/models \
  -v ~/vllm:/models \
  nvcr.io/nvidia/vllm:26.04-py3 \
  vllm serve <HF_MODEL_HANDLE> \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 65536 \
    --gpu-memory-utilization 0.85 \
    --max-num-seqs 4

Smoke-test (first request triggers JIT warmup — see §9):

curl -sS http://localhost:8000/v1/models | jq -r '.data[0].id'
curl -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<HF_MODEL_HANDLE>","messages":[{"role":"user","content":"12*17"}],"max_tokens":500}'
# model should answer 204 (= 12*17)

📝Note

Two container tracks — pick per recipe. NVIDIA’s NGC image nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0) is the default for many nvidia/… checkpoints. Several recipes below use the upstream vllm/vllm-openai:nightly / :cu130-nightly track (needed for the newest architectures and for DFlash). A given model may need a newer container than you have — vllm –version inside the image tells you.

❗Important

The two image families take the launch command differently — this is the #1 first-run mistake. Upstream vllm/vllm-openai:* images already have vllm serve as their entrypoint, so you pass only the model handle + flags straight after the image tag. Writing vllm serve again produces vllm serve vllm serve …, which fails with vllm: error: unrecognized arguments: serve . NVIDIA’s nvcr.io/nvidia/vllm:* images use a pass-through entrypoint, so those do take the full vllm serve …. That’s why every vllm/vllm-openai recipe below starts with the bare model handle, while the NGC recipes start with vllm serve. (If a vllm/vllm-openai container is crash-looping on this error, remove it first — docker rm -f — since –restart unless-stopped keeps respawning it and holds the name.)

❗Important

Quantization flag rule. For NVIDIA ModelOpt NVFP4 checkpoints (the nvidia/… repos), pass –quantization modelopt. For compressed-tensors NVFP4 (Unsloth, llm-compressor community builds), vLLM auto-detects — do not pass –quantization. Getting this wrong is a common first-run failure.

Container image → vLLM version (what actually ships, mid-2026)

The :nightly/:latest tags move, and a recipe written against one can break on a newer one (deprecated flags, changed kernels). Known-good mappings people run on Spark today:

🔥Caution

vLLM 0.23 deprecated the FlashInfer-MoE env vars (VLLM_USE_FLASHINFER_MOE_FP4, VLLM_USE_FLASHINFER_MOE_FP8, …) in favour of the –moe-backend flag (marlin, flashinfer_cutlass, flashinfer_trtllm, flashinfer_cutedsl). On a 0.23+ image they log “Unknown vLLM environment variable detected” and do nothing. On sm121 (GB10), FP4 MoE must use –moe-backend marlin — the GPU has no native FP4 compute, so dense layers run FLASHINFER_CUTLASS while MoE layers fall back to Marlin W4A16. These older env vars still work on the NGC 0.19 image; prefer the CLI flag everywhere for portability.

📝Note

!!!!! or empty output from an FP4 model is an sm121 kernel problem, not your prompt — the working path is Marlin (–moe-backend marlin for NVFP4, VLLM_MXFP4_BACKEND=marlin for MXFP4).

GB10 (sm121, compute capability 12.1) has no native FP4, and vLLM’s auto-fallback chain trips on it: FlashInfer-TRTLLM MoE is “SM100+ only,” and the CUTLASS FP4 GEMM it falls back to fails on sm120/sm121 ([FP4 gemm Runner] Failed to run cutlass FP4 gemm on sm120). Marlin (W4A16) is the path that works — which is why these recipes force it. Garbage output usually means an image without the sm121 Marlin/PTX fixes; use NGC 26.04-py3, a current nightly, or a community Spark fork (eugr/spark-vllm-docker, christopher_owen/spark-vllm-mxfp4-docker).

7. Per-model configurations

Grouped by family: Qwen → Gemma → NVIDIA → other. Each model’s published quality benchmarks live on its card (accuracy is quantization-, not hardware-dependent); all commands assume ~/vllm is your model store.

Quick comparison

(c-t = compressed-tensors auto-detect; env vars = the two sm_121a exports shown in 7.1.1.)

7.1 Qwen family

7.1.1 Qwen3.6-35B-A3B-NVFP4 — flagship MoE agent (official Spark recipe)

MoE with hybrid attention, 35B total / 3B active, multimodal (text/image/video), 262K context, Apache 2.0, ~19 GB NVFP4. NVIDIA’s recipe was written for an older upstream nightly; the command below is the version-corrected form that boots on current vllm/vllm-openai:nightly (vLLM 0.23.x) — see the two notes after it for what changed and why.

export HF_TOKEN=hf_xxx
docker run -d --name qwen36-35b --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models -e HF_TOKEN="$HF_TOKEN" \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly \
  nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8000 \
    --tensor-parallel-size 1 --trust-remote-code --dtype auto \
    --quantization modelopt --kv-cache-dtype fp8 \
    --moe-backend marlin \
    --gpu-memory-utilization 0.85 --max-model-len 65536 \
    --max-num-seqs 4 --max-num-batched-tokens 8192 \
    --enable-chunked-prefill --async-scheduling --enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml

Two changes from NVIDIA’s published recipe (verified on vLLM 0.23.x): (1) The env vars VLLM_USE_FLASHINFER_MOE_FP4=0 and VLLM_FP8_MOE_BACKEND=… are removed — the first was deprecated in vLLM 0.23 in favour of –moe-backend (you’ll see “Unknown vLLM environment variable detected” if you keep it), and the second was never a real vLLM variable. –moe-backend marlin is the correct, version-stable replacement and is required on sm121 (GB10 has no native FP4 compute, so MoE must use the Marlin W4A16 path). (2) –attention-backend flashinfer is dropped — forcing one global attention backend crashes at init_device on this hybrid model (it mixes full attention + Gated-DeltaNet/linear + Mamba layers); let vLLM auto-select per layer. Also note: the card’s –max-model-len 262144 reserves far too much KV cache on a 128 GB Spark — use 65536.

NVIDIA’s published card was later updated (and now lists –attention-backend flashinfer, –tool-call-parser qwen3_xml, –gpu-memory-utilization 0.4, and –load-format fastsafetensors). That flashinfer flag is the one this recipe deliberately drops — it crashed at init_device on the nightly used here for this hybrid model. If you’re on a newer build, it’s worth re-testing the card’s exact command; if it boots, prefer qwen3_xml as the tool-call parser (the card’s current choice) and 0.4 utilization for one user. If it crashes at init_device, fall back to the version-corrected command above. Either way, –moe-backend marlin stays required on sm121.

The last two lines are required for agent / tool-calling clients (Hermes, Cline, OpenWebUI tools, any framework that sends tools). Omit them and the model still loads — /v1/models returns 200 — but every agent request dies with HTTP 400: “auto” tool choice requires –enable-auto-tool-choice and –tool-call-parser to be set, because agents send tool_choice: “auto”. What each of the three parser-related flags does:

• –enable-auto-tool-choice — turns on automatic tool selection: the server honours a request’s tools list with tool_choice: “auto” and decides, per response, whether to emit a tool call. Without it vLLM serves plain chat but refuses any tool-calling request (the 400 above).
• –tool-call-parser qwen3_xml — parses the model’s emitted tool call back into a structured OpenAI-style tool_calls object. It must match the model’s output format, not your client: Qwen3.6 emits the qwen3_xml format. Don’t reach for the hermes parser just because your agent is named Hermes — the parser tracks the model family.
• –reasoning-parser qwen3 — splits the model’s block out of the visible answer into a separate reasoning_content field, so chain-of-thought doesn’t leak into the reply. Not needed to clear the 400, but you want it on this reasoning model.

If init_device fails instead, bisect: start from a minimal run (–quantization modelopt –moe-backend marlin –gpu-memory-utilization 0.85 –max-model-len 65536 –max-num-seqs 4 –reasoning-parser qwen3) and re-add –kv-cache-dtype fp8, prefix caching, chunked prefill, async scheduling, and MTP one at a time to find the offender.

💡Tip

MTP vs DFlash on this model. The recipe above uses the card’s built-in MTP (num_speculative_tokens:3) — the most reliable spec-decode path on a stock image, though it can occasionally fall into output loops. The alternative is DFlash with the dedicated ~0.5 B drafter z-lab/Qwen3.6-35B-A3B-DFlash, which avoids the loops and reaches ~50 tok/s at 2.7–4.4 accepted tokens/step. Swap the –speculative-config line for:

    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":15}' \

Two caveats that bite on Spark: (1) on quantized (NVFP4/FP8) weights under stock vLLM, DFlash acceptance collapses — only the first few of 15 tokens get accepted, so it slows down rather than speeds up. The fix is a DFlash-capable sm121 build (the production-validated ghcr.io/aeon-7/… image paired with AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4), or running a BF16 target instead (works on stock nightly, at much higher memory). (2) Keep –attention-backend triton_attn and set -e VLLM_USE_DEEP_GEMM=0; flash_attn crashes on sm121. Re-pull the drafter if you cloned it before 2026-04-19 (a long-context crash was fixed then). Full DFlash setup is in §14.

7.1.2 Qwen3.6-27B-NVFP4 — flagship-level coding in a 27B dense model (Unsloth)

Dense 27B with a vision encoder (text/image/video), hybrid Gated-DeltaNet + Gated-Attention layers, MTP-trained, 262K context (extensible to ~1M via YaRN), Apache 2.0, ~18 GB NVFP4 (compressed-tensors → no –quantization flag).

docker run -d --name qwen36-27b --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models \
  vllm/vllm-openai:nightly \
  unsloth/Qwen3.6-27B-NVFP4 \
    --trust-remote-code --dtype bfloat16 \
    --gpu-memory-utilization 0.85 --max-model-len 65536 \
    --max-num-seqs 4 --reasoning-parser qwen3

The card’s default –max-model-len is a conservative 4096 (“increase only after checking memory”). 65536 is comfortable on Spark; keep ≥128K only if you have headroom, since the model leans on long context for thinking. For coding agents add –tool-call-parser qwen3_coder. MTP is supported by the architecture — you can try –speculative-config ‘{“method”:“mtp”,“num_speculative_tokens”:3}’ and validate acceptance on your build. Thinking is on by default; pass chat_template_kwargs={“enable_thinking”:false} (or {“preserve_thinking”:true} for agents) per request.

💡Tip

Beyond 262K (YaRN, up to ~1M). This checkpoint is native to 262,144 tokens; for longer contexts enable static YaRN via an –hf-overrides block and the long-len escape hatch (mind that static YaRN can slightly dent short-context quality — only enable it when you actually need the length):

  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  ... unsloth/Qwen3.6-27B-NVFP4 --trust-remote-code --dtype bfloat16 \
    --hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}' \
    --max-model-len 1010000

💡Tip

DFlash spec-decode for this model. A dedicated drafter z-lab/Qwen3.6-27B-DFlash pairs with Qwen3.6-27B — use it instead of MTP by appending:

    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-27B-DFlash","num_speculative_tokens":15}' \

Until it merges, this drafter needs a vLLM build with interleaved-SWA support (PR #40898), plus the usual Spark rules: –attention-backend triton_attn and -e VLLM_USE_DEEP_GEMM=0 (never flash_attn on sm121). The §7.1.1 quantized caveat applies — on NVFP4 under stock vLLM, confirm tokens past the first few are accepted, or use the prebuilt ghcr.io/aeon-7/aeon-vllm-ultimate image, whose AEON-7/Qwen3.6-27B-…-DFlash target runs at num_speculative_tokens:12 (~37.6 tok/s vs ~10.5 eager). Full DFlash setup is in §14.

7.1.3 Qwen3-Coder-Next — 80B/3B-active agentic coder, FP8 (Qwen)

80B total / 3B active MoE (512 experts, 10 active + 1 shared) on a hybrid Gated-DeltaNet + Gated-Attention layout, 256K context, non-reasoning for fast code, Apache 2.0. It performs at the level of models with 10–20× more active params and drops into Claude Code / Cline / Qwen Code. On Spark, FP8 is the path — unlike NVFP4, FP8 kernels run cleanly on sm121 — and the FP8 checkpoint (~85 GB) fits a 128 GB Spark with room for KV cache (Unsloth’s 4-bit dynamic GGUF runs in ~46 GB on llama.cpp if you want more headroom). A community benchmark measures ~43 tok/s single-stream (≈41 @ 16K, ≈39 @ 32K), with prefill from ~5,200 up to ~10,600 t/s.

docker run -d --name qwen3-coder-next --ipc=host --restart unless-stopped \
  --gpus all --shm-size 32gb -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models \
  vllm/vllm-openai:cu130-nightly \
  Qwen/Qwen3-Coder-Next-FP8 \
    --served-model-name qwen3-coder-next --host 0.0.0.0 --port 8000 \
    --attention-backend flashinfer \
    --enable-prefix-caching \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --gpu-memory-utilization 0.85 --max-model-len 170000 \
    --max-num-seqs 4 --max-num-batched-tokens 32768

⚠️Warning

Two Spark-specific gotchas (community-verified). (1) The flags printed on the model card disable prefix caching for this architecture — fatal for coding agents, which re-send a growing prompt every turn (0% cache hits = re-prefill on every request). Add –enable-prefix-caching explicitly; vLLM marks it experimental for this arch, but it works. (2) The default FLASH_ATTN backend caps you near ~60K tokens at 0.85 util, whereas –attention-backend flashinfer fits ~170K without fp8-quantizing the KV cache. It’s non-reasoning, so no –reasoning-parser. Unsloth’s unsloth/Qwen3-Coder-Next-FP8 dynamic quant claims ~25% more throughput; for the validated community image and a dual-Spark cluster variant, see eugr/spark-vllm-docker.

7.1.4 Qwen3.5-122B-A10B-NVFP4 — best all-rounder that still fits one Spark (community)

The strongest “max quality on a single Spark” pick. A 122B-total / 10B-active multimodal MoE; the BF16 original is ~234 GB, but the NVFP4 quant is 75.6 GB, leaving ~52 GB for KV cache and overhead. Compressed-tensors (RedHat) → no –quantization. A Spark owner who benchmarked it called it the most well-balanced single-Spark model, measuring ~16 tok/s decode with prefill from ~1,850 up to ~6,300 t/s as context deepens — slower than the A3B models, but markedly higher quality.

docker run -d --name qwen35-122b --ipc=host --restart unless-stopped \
  --gpus all --shm-size 32gb -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models \
  vllm/vllm-openai:cu130-nightly \
  RedHatAI/Qwen3.5-122B-A10B-NVFP4 \
    --served-model-name qwen3.5-122b --host 0.0.0.0 --port 8000 \
    --trust-remote-code --moe-backend marlin \
    --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --enable-prefix-caching \
    --gpu-memory-utilization 0.85 --max-model-len 65536 \
    --max-num-seqs 4 --max-num-batched-tokens 8192

Needs a recent vLLM (build from main or current nightly) and Transformers 5+. –moe-backend marlin is the sm121-safe default; the community author actually ran –moe-backend flashinfer_cutlass on this model and found it faster than Marlin here — worth A/B-ing on your build. If the chat output looks malformed, the Qwen3.5 chat template needs the fix shipped in eugr/spark-vllm-docker (mount it via –chat-template).

7.2 Gemma family

❗Important

Use vllm/vllm-openai:gemma4-cu130, never the bare :gemma4 tag. The non--cu130 image is v0.18.2-dev and crashes on GB10 with RuntimeError: [FP4 gemm Runner] Failed to run cutlass FP4 gemm on sm120/sm121. After ~90 s startup (≈84 s weight load + torch.compile), a healthy Gemma-4 NVFP4 run logs Using NvFp4LinearBackend.FLASHINFER_CUTLASS for NVFP4 GEMM (dense) and Using ‘MARLIN’ NvFp4 MoE backend (MoE). If a MoE model logs CUTLASS_FP4 for MoE instead of MARLIN, your –moe-backend marlin didn’t apply — stop and fix, because sm121 has no native FP4 MoE kernel.

7.2.1 Gemma-4-31B-IT-NVFP4 — dense multimodal (NVIDIA)

Dense 30.7B, multimodal (text/image/video), 256K context, 140+ languages, hybrid local/global attention with p-RoPE, NVIDIA Open Model License (+ Gemma terms), ~21 GB NVFP4. (Dense — no –moe-backend needed; its linear layers use FLASHINFER_CUTLASS automatically.)

export HF_TOKEN=hf_xxx
docker run -d --name gemma4-31b --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models -e HF_TOKEN="$HF_TOKEN" \
  vllm/vllm-openai:gemma4-cu130 \
  nvidia/Gemma-4-31B-IT-NVFP4 \
    --quantization modelopt --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 --max-model-len 65536 --max-num-seqs 4 \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

📝Note

The card uses –tensor-parallel-size 8 on a server — on Spark use TP=1. Gated: accept the Gemma license + set HF_TOKEN. The command above enables reasoning by default (–reasoning-parser gemma4 + –default-chat-template-kwargs ‘{“enable_thinking”: true}’) — pass enable_thinking:false per request to turn it off. For tool calling, also add –enable-auto-tool-choice –tool-call-parser gemma4.

7.2.2 Gemma-4-26B-A4B-NVFP4 — MoE multimodal (NVIDIA)

MoE 25.2B total / 3.8B active (8 of 128 experts +1 shared), multimodal, 256K context, ~16.5 GB NVFP4 — a small, fast, high-quality Spark pick.

# 1. Download the community NVFP4 checkpoint — it bundles gemma4_patched.py,
#    which fixes vLLM's expert scale-key mapping bug (#38912). The official
#    nvidia/Gemma-4-26B-A4B-NVFP4 does NOT ship this patch.
hf download bg-digitalservices/Gemma-4-26B-A4B-it-NVFP4 \
  --local-dir ~/vllm/gemma4-26b-a4b-nvfp4

# 2. Serve, mounting the patched loader OVER the image's gemma4.py
docker run -d --name gemma4-26b --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -v ~/vllm/gemma4-26b-a4b-nvfp4:/models/gemma4 \
  -v ~/vllm/gemma4-26b-a4b-nvfp4/gemma4_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py \
  vllm/vllm-openai:gemma4-cu130 \
  --model /models/gemma4 --served-model-name gemma-4-26b \
  --quantization modelopt --moe-backend marlin --trust-remote-code \
  --kv-cache-dtype fp8 --max-model-len 65536 \
  --gpu-memory-utilization 0.85 --max-num-seqs 4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --enable-auto-tool-choice --tool-call-parser gemma4

⚠️Warning

This MoE needs three things on Spark: the patched gemma4_patched.py loader (the second -v mount above), –moe-backend marlin, and TP=1. Tool calls use the gemma4 parser — Gemma 4’s family-wide tool protocol (the <|tool_call>…<tool_call|> special tokens), parsed with –tool-call-parser gemma4.

Why the loader: the NVFP4 checkpoint carries per-expert scale tensors (.input_scale, .weight_scale, .weight_scale_2) that vLLM 0.19’s gemma4.py can’t map onto FusedMoE params, so a stock image dies at load with KeyError: ‘…experts.0.down_proj.input_scale’ (vLLM #38912). The community bg-digitalservices checkpoint ships gemma4_patched.py, and the second -v mount overlays it onto the image’s copy — that mount is the whole fix. (NVIDIA’s own nvidia/Gemma-4-26B-A4B-NVFP4, published 2026-05-01, has the same scale keys and isn’t yet proven with this patch, so the community checkpoint is the known-good route.) On sm121 the native CUTLASS FP4 MoE path emits garbage (NaN scales, !!!!!), so Marlin is mandatory — a healthy log shows Using ‘MARLIN’ NvFp4 MoE backend for MoE and FLASHINFER_CUTLASS for dense; if MoE says CUTLASS_FP4, the flag didn’t take. The model is also TP=1 only (expert-parallel yes, tensor-parallel no).

Measured on a Spark with this exact recipe (this guide’s bench script, 3-run median, ~1,527-token prompt): decode 49.6 tok/s answer-only and 51.1 tok/s with thinking (–think) — effectively the same ~50 tok/s, the small gap being cross-session noise. That’s expected: with no speculative draft in this recipe, decode is pure memory bandwidth and doesn’t care whether the model is reasoning or answering — unlike Qwen3.6, where the MTP draft made thinking genuinely faster (higher acceptance on structured reasoning). At –output-tokens 100 the thinking run was ~97/100 reasoning chunks and never reached the answer, so 51.1 is a pure-reasoning rate; raise the budget to capture thinking+answer. Prefill ~5–6.5k tok/s (TTFT 235–298 ms for the identical prompt) — compute-bound, so it swings run-to-run unlike the rock-stable decode; either way it tracks the ~6,200 tok/s prefill on Qwen3.6-35B, so GB10 prefill is ~6k tok/s largely regardless of which small-active MoE you run. This decode is the same ~50 tok/s plotted for this model in §2’s chart (the chart’s Gemma-4-26B-A4B bars are these very numbers).

💡Tip

Keep thinking ON for anything needing a step of reasoning. Gemma 4 ships thinking off by default in vLLM, but the recipe above turns it on server-wide via –default-chat-template-kwargs (method 2 below), so this command already reasons. (The prompt “I want to wash the car and the washing station is just 100m away. Should I walk there or should I drive?” returns gibberish thinking-off, and is answered correctly thinking-on.)

Control it three ways. (1) Per request: add chat_template_kwargs={“enable_thinking”: true} (curl) or extra_body={“chat_template_kwargs”: {“enable_thinking”: True}} (OpenAI client) — or false to override the recipe’s default for a single call — with max_tokens 4096+. (2) Server-wide: –default-chat-template-kwargs ‘{“enable_thinking”: true}’ on the docker run, the line already baked into the recipe above; every client gets it and per-request kwargs still override. (3) In Open WebUI: Admin Panel → Settings → Models → your model → Advanced Params → + Add Custom Parameter, name chat_template_kwargs, value {“enable_thinking”: true}. Heads-up: on the gemma4-cu130 image the reasoning may land inside content instead of the separate reasoning_content field (vLLM #38855) — the model still reasons; that split is a parser bug fixed in newer builds.

7.2.3 gemma-4-12B-coder (MTP-NVFP4) — Python specialist with a bundled MTP draft (community)

A weight-only NVFP4 (W4A16) build of a Gemma-4-12B Python-coding fine-tune that thinks before it codes: an 8.25 GB main model plus a 0.85 GB bundled MTP draft for ~1.6× lossless single-stream. It’s strong on hard algorithms (DP, graphs, segment/Fenwick trees, bitmask DP) and faithful bug-fixing/refactoring — genuine O(n²)→O(n) rewrites that preserve semantics — reasoning in the open before it writes. It’s de-refused (not safety-aligned), Python/algorithmic, and English-centric. vLLM auto-detects the NVFP4 weights, so pass no –quantization. (W4A16 is deliberate: the base is non-QAT, where full W4A4 collapses on this architecture, so activations stay BF16.) Model card — downloads, evals, provenance: https://huggingface.co/sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4

This section is a build-up walkthrough: each step adds one capability, then we assemble the full command at the end. It’s a single GB10 throughout — the card’s multi-GPU NCCL_P2P_DISABLE / –disable-custom-all-reduce flags are only for PCIe tensor-parallel and don’t apply on one Spark.

Step 1 — Download into its own folder under ~/vllm. The MTP draft ships inside the download, so one pull gets both the model and its draft:

export HF_HOME=~/vllm
hf download sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 \
  --local-dir ~/vllm/gemma4-12b-coder

The result — note the draft is a subfolder, which matters in Step 3:

~/vllm/gemma4-12b-coder/
├── config.json, *.safetensors …   ← 8.25 GB main model (NVFP4)
└── assistant/                      ← 0.85 GB MTP draft (google/gemma-4-12B-it-assistant)

Step 2 — Serve it (plain chat first). Mount the folder to /model; vLLM detects the quant on its own:

docker run -d --name gemma4-coder --ipc=host --shm-size 16gb --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -v ~/vllm/gemma4-12b-coder:/model:ro \
  vllm/vllm-openai:nightly \
  --model /model --served-model-name gemma4-coder \
  --max-model-len 16384 --gpu-memory-utilization 0.92 --trust-remote-code

Smoke-test once it reports ready (the first request also warms the JIT — see §9):

curl -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"gemma4-coder","messages":[{"role":"user","content":"Write a Python function to validate an IPv4 address."}],"max_tokens":600}'

Step 3 — Add the bundled MTP draft for ~1.6× (lossless). Append two flags to the run:

  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","model":"/model/assistant","num_speculative_tokens":3}'

How the draft (the assistant/ folder) works: it’s Google’s google/gemma-4-12B-it-assistant, a small companion the 12B ships with for exactly this. Each step it proposes the next few tokens; the 12B target then verifies them in one forward pass — and since decode on GB10 is bandwidth-bound, that pass costs about the same as the single token the target was going to produce anyway. Tokens the target agrees with are kept, the first miss is corrected, and it repeats. Because the target validates every token, the output is identical to running with no draft — only faster. The reason acceptance stays high even in long sessions: this assistant shares the target’s KV cache and rides on the target’s own activations instead of building its own context. Two requirements come with it: –kv-cache-dtype fp8 is mandatory (NVFP4 KV collapses the draft’s acceptance), and num_speculative_tokens is 3 here as a conservative value — the official Gemma-4 recipe suggests 4–8 for this exact 12B draft, so A/B both on your Spark with the §10 bench script.

Step 4 — Turn on thinking (it’s OFF by default in vLLM). The model was trained to reason first in Gemma’s native thinking channel, but vLLM won’t activate it unless you ask — and on hard problems, quality drops without it. Two ways to enable it:

• Per request — add “chat_template_kwargs”: {“enable_thinking”: true} to the JSON body (or extra_body={…} from the OpenAI SDK).
• Server-wide default (nicer for a coding box you run yourself) — add these two flags to the run:

  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \

Two lines worth noting:

• –default-chat-template-kwargs ‘{“enable_thinking”: true}’ — the new one. Makes every request think unless it sends enable_thinking:false to override.
• –reasoning-parser gemma4 — pairs with it. With thinking now on by default, this splits the <|channel>thought… block into a separate reasoning field so the chain-of-thought doesn’t sit inline in the answer. Drop this line if you’d rather see the thinking inline with the code.

Adding a flag is just that — drop another –flag into the flags block (after the image tag, alongside the others; order among them doesn’t matter). The one thing that bites people is shell quoting: wrap the JSON in single quotes so the inner double-quotes survive the shell.

Step 5 — (optional) tool calling. The card documents thinking + code, not tools, so treat this as best-effort and confirm the tool call is actually parsed into the structured field — a known Gemma-4 failure is the call staying inside the message text instead. Gemma 4’s own parser is gemma4 — not pythonic — and it needs the Gemma-4 tool chat template, which carries the protocol:

wget -P ~/vllm https://raw.githubusercontent.com/vllm-project/vllm/main/examples/tool_chat_template_gemma4.jinja
# then add to the run:
  -v ~/vllm/tool_chat_template_gemma4.jinja:/tmpl/gemma4.jinja:ro \
  --enable-auto-tool-choice --tool-call-parser gemma4 \
  --chat-template /tmpl/gemma4.jinja \

Tools and thinking coexist — Gemma 4 reasons about which tool to call, then calls it — so the Step-4 thinking flags stay on. If you need reliable agentic tool use, the coder isn’t the model for it: the author’s dedicated agentic variant, yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2 (tuned on τ²-bench), is. It’s GGUF today, so run it under llama.cpp.

The full single-Spark recipe (MTP + thinking on; add the Step-5 lines if you want tools):

docker run -d --name gemma4-coder --ipc=host --shm-size 16gb --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -v ~/vllm/gemma4-12b-coder:/model:ro \
  vllm/vllm-openai:nightly \
  --model /model --served-model-name gemma4-coder \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","model":"/model/assistant","num_speculative_tokens":3}' \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 16384 --gpu-memory-utilization 0.92 --trust-remote-code

After editing flags, recreate the container (–restart unless-stopped keeps the old one alive and holds the name): docker rm -f gemma4-coder, then rerun. Sampling the author recommends: temp 1.0, top_p 0.95, top_k 64, or temp 0 for deterministic coding.

❗Important

Needs a recent nightly — only it registers Gemma4UnifiedForConditionalGeneration, and the stable :latest tag doesn’t carry Gemma-4 MTP, so –speculative-config silently does nothing there. With the MTP draft you must keep –kv-cache-dtype fp8; NVFP4 KV breaks the draft.

🔥Caution

The build is de-refused / not safety-aligned — add your own guardrails. It’s a superb algorithm/debug assistant but can write look-ahead bias into pandas/numpy time-series & back-test code (its reasoning sometimes states the right rule while the code does the opposite). Gate quant/accounting code; don’t ship it unreviewed.

7.3 NVIDIA models (proprietary)

Models NVIDIA itself designed and trained (not quantizations of someone else’s weights). Both are hybrid Mamba-2 + Transformer MoE models with native function calling: the Nano is the natural single-Spark starting point, and the Nano-Omni adds video/audio/image understanding.

7.3.1 NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 — best single-Spark Nemotron (Spark-tested)

A hybrid Mamba-2 + MoE (52 layers: 23 Mamba-2, 23 MoE with 128 experts +1 shared / 6 active, 6 GQA attention), 30B total / 3.5B active, text-only, 1M context (256K default), unified reasoning + non-reasoning, NVIDIA Nemotron Open Model License, ~18 GB on disk. NVFP4 weights with FP8 KV cache; attention and the Mamba layers feeding it stay BF16, and quantization-aware distillation (QAD) recovers accuracy. NVIDIA lists DGX Spark in this model’s tested hardware and ships a Spark/Jetson-specific container.

export HF_TOKEN=hf_xxx
# one-time: fetch the custom reasoning parser into ~/vllm (mounted at /models)
wget -P ~/vllm \
  https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/resolve/main/nano_v3_reasoning_parser.py

docker run -d --name nemotron-nano --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models -e HF_TOKEN="$HF_TOKEN" \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  nvcr.io/nvidia/vllm:25.12.post1-py3 \
  vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
    --served-model-name nemotron-3-nano --port 8000 \
    --tensor-parallel-size 1 --trust-remote-code \
    --max-model-len 262144 --max-num-seqs 8 \
    --gpu-memory-utilization 0.85 \
    --kv-cache-dtype fp8 \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --reasoning-parser-plugin /models/nano_v3_reasoning_parser.py \
    --reasoning-parser nano_v3

❗Important

Use NVIDIA’s nvcr.io/nvidia/vllm:25.12.post1-py3 container for this model and fetch the nano_v3_reasoning_parser.py plugin (into ~/vllm, referenced at /models/…). –kv-cache-dtype fp8 is part of the official recipe here — unlike Qwen3.5, this model is built for it.

–max-num-seqs 8 is NVIDIA’s Spark-tested value; drop to 4 for more KV headroom at long context. The two VLLM_*FLASHINFER_MOE* env vars are valid on this Dec-2025 container; on a 0.23+ image they become no-ops (logged “Unknown…”) — drop them and add –moe-backend marlin (the required fallback if the FlashInfer FP4 MoE path ever errors on sm121). Driver: this image is built for NVIDIA driver ≥ 590.44. On a Spark still on 580.x it runs through CUDA forward-compatibility — a clean boot prints CUDA Forward Compatibility mode ENABLED … Using CUDA 13.1 driver 590.44 with kernel driver 580.x and proceeds normally. If that line instead reads compatibility mode is UNAVAILABLE while the container restart-loops, the shim couldn’t initialize: update the host driver to ≥ 590.44, or run on a nightly image (vllm/vllm-openai:cu130-nightly) matching your driver. (A –restart unless-stopped container failing for another reason — e.g. a bad flag — can momentarily print UNAVAILABLE on the colliding restart; fix the real error first.)

Reasoning is on by default — pass chat_template_kwargs={“enable_thinking”:false} per request to turn it off. The model also supports a reasoning_budget (cap internal reasoning tokens to hit latency targets). Sampling: temperature=1.0, top_p=1.0 for reasoning; 0.6 / 0.95 for tool calling. Give each request a high max_tokens (≈10,000): the model always emits a reasoning trace first, so a low cap truncates it mid-thought — which returns empty or garbled content and silently breaks tool-call parsing (the single most common “tool calling doesn’t work” cause on this model). For up to 1M context add -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and raise –max-model-len. With only 3.5B active params and ~18 GB resident, decode lands in the fast ~50–56 tok/s A3B class (the ~56 in §2’s table — it edges the attention-based A4B models because its 23 Mamba-2 layers carry a recurrent state rather than re-reading a growing KV cache each step), and KV headroom is large — an excellent default Spark model. The –gpu-memory-utilization 0.85 added above is the memory cap: it bounds vLLM’s slice of the shared 128 GB pool. With only ~18 GB of weights you can safely lower it (e.g. 0.5) to leave more host RAM for the page cache and other services, with no decode penalty — lowering it shrinks KV-cache capacity, not speed. Prefer this over a Docker –memory cgroup cap, which is risky on unified memory (GPU allocations draw from the same DRAM, so a hard cap can trip the OOM-killer mid-load).

7.3.2 Nemotron-3-Nano-Omni-30B-A3B-NVFP4 — fast omni-modal agent (video + audio + image + text)

The Nano family’s omni sibling: a Mamba2-Transformer hybrid MoE (31B / 3B active) with a CRADIO-v4 vision encoder and a Parakeet speech encoder, so it ingests video (mp4 ≤ 2 min), audio (wav/mp3 ≤ 1 hr), images, and text and emits text, JSON, tool calls, reasoning, and word-level transcription timestamps — built for OCR, document intelligence, meeting/video summarization, and GUI/browser agents. 256K context, A3B decode speed (the same ~50-tok/s class as the text Nano), NVFP4 ~20 GB. NVIDIA ships a dedicated Spark recipe.

export HF_TOKEN=hf_xxx
docker run -d --name nemotron-omni --ipc=host --restart unless-stopped \
  --gpus all --shm-size 16g -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models -e HF_TOKEN="$HF_TOKEN" \
  --entrypoint /bin/bash \
  vllm/vllm-openai:v0.20.0 -c \
  "pip install vllm[audio] && vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
     --served-model-name nemotron-omni --host 0.0.0.0 --port 8000 --trust-remote-code \
     --gpu-memory-utilization 0.8 --max-model-len 131072 \
     --max-num-seqs 8 --max-num-batched-tokens 32768 --enable-prefix-caching \
     --limit-mm-per-prompt '{\"video\":1,\"image\":1,\"audio\":1}' \
     --media-io-kwargs '{\"video\":{\"fps\":2,\"num_frames\":256}}' \
     --allowed-local-media-path=/ \
     --reasoning-parser nemotron_v3 \
     --enable-auto-tool-choice --tool-call-parser qwen3_coder"

❗Important

This recipe breaks two of the guide’s usual rules on purpose. (1) It pins vLLM to exactly v0.20.0 — the version NVIDIA validated for this model — instead of the nightly track. (2) It overrides the entrypoint to /bin/bash so it can pip install vllm[audio] before vllm serve: the stock image has no audio stack, and without it any audio input (including use_audio_in_video:true) fails. That’s the one case in this guide where you write vllm serve explicitly on a vllm/vllm-openai image — because the entrypoint is no longer vllm serve. Multimodal input needs –allowed-local-media-path and –limit-mm-per-prompt; tune video via –media-io-kwargs (drop to fps:1/num_frames:128 for 1080p clips). If you OOM, lower –gpu-memory-utilization to 0.70 and/or –max-model-len to 32768; for more KV headroom you can add –kv-cache-dtype fp8 (the card’s non-BF16 path uses it — validate output quality first). If FP4 output is garbled on sm121, add –moe-backend marlin (see §6). Sampling: thinking mode temperature 0.6, top_p 0.95; instruct mode temperature 0.2. BF16/FP8 weights and a non-reasoning Instruct variant live on the same nvidia/Nemotron-3-Nano-Omni-30B-A3B-* repo family.

On the stable NGC 26.04 image, this runs vision-only. NVIDIA’s 26.04 release notes do list Nemotron 3 Nano Omni as supported — but only by overriding the architecture to the previous-generation vision-language model: –hf-overrides=’{“architectures”:[“NemotronH_Nano_VL_V2”]}’. That loads the VL backbone and drops audio. So use 26.04 + that override if you only need image / video / text on the NGC track; stay on the v0.20.0 recipe above for the full omni stack (audio, use_audio_in_video, word-level transcription timestamps) — the version the model card actually requires.

8. Flag reference & tuning

💡Tip

Stability ↔ throughput slider. A plain run leaves KV-cache dtype unset, spec-decode off, CUDA graphs on, backends auto. A tuned run layers in FP8 KV cache, async scheduling, MTP/DFlash, and pinned FP4 backends — each validated against the exact model + prompt shape + batch pattern + vLLM release. Want ~2–3× faster interactive decode? See §14 (MTP & DFlash).

Memory & KV cache: why RAM looks “maxed out”

Coming from llama.cpp/Ollama, the first shock on Spark is that vLLM pins almost the whole 128 GB at startup. This is by design, not a leak. vLLM uses PagedAttention and pre-allocates one large KV-cache pool sized to –gpu-memory-utilization, grabbing that memory up front (on Spark’s unified pool it shows as system RAM) no matter how many requests are actually running. llama.cpp instead allocates KV lazily, sized to the -c you pass — so it looks far lighter for the same weights. Read it straight from the startup log:

Model loading took 21.94 GiB memory          ← weights (~same as llama.cpp)
Available KV cache memory: 76.49 GiB         ← the pre-allocated KV pool
GPU KV cache size: 4,843,965 tokens          ← pool capacity, in tokens
Maximum concurrency for 65,536 tokens per request: 73.91x

That token number is a pool capacity, not a context window — the single most common misread. Picture the KV pool as a parking lot measured in token-slots, and –max-model-len as how many slots one request may occupy. Then concurrency = pool_tokens ÷ max-model-len (here 4.84M ÷ 65,536 ≈ 74 simultaneous full-length requests). The max-num-seqs 4 × 65,536 ≈ 262K figure is just the minimum pool you’d want so four max-length requests can be in flight at once. The three knobs:

• –max-model-len — context window of a single request (prompt + output).
• –max-num-seqs — how many requests run concurrently.
• –gpu-memory-utilization — how big the pool is (total token capacity).

Empirically from that log, fp8 KV on this model costs ~16.6 KB/token (76.49 GiB ÷ 4.84M) — the conversion between “tokens of capacity” and GB.

To reclaim RAM (single user): the pool is wildly over-provisioned for one person — lower –gpu-memory-utilization. You only need pool_tokens ≥ max-num-seqs × max-model-len plus slack.

📝Note

–max-num-seqs does not shrink the pool — –gpu-memory-utilization does. max-num-seqs only caps how many sequences share the pool. To free RAM, lower the utilization; to fit a long context, make sure the (smaller) pool still holds ≥ one full-length sequence.

Walkthrough: 65K → 256K context (one user)

The baseline recipe ships –max-model-len 65536 for a safe first boot, but this model’s native ceiling is 262,144 (256K) — 4× longer, reachable with no YaRN and no quality trade-off, because you’re still inside what the model was trained on. The only cost is memory, and for a single user that cost is small: one 256K sequence holds 262,144 tokens of KV ≈ ~4 GB (at 16.6 KB/token, fp8). So a handful of concurrent 256K chats fit in a fraction of the pool — the 76 GB the baseline grabbed was never needed.

Step 1 — raise the context to native max. This single change is enough to work:

    --max-model-len 262144

Step 2 — right-size the pool for one user, reclaiming the RAM the longer context didn’t actually need:

    --max-model-len 262144 \
    --max-num-seqs 2 \
    --gpu-memory-utilization 0.4 \
    --kv-cache-dtype fp8

That leaves a ~1.4M-token pool ≈ ~5 full-length 256K sequences of headroom — plenty for one user plus Open WebUI’s background title/tag calls — while freeing ~58 GB versus the 0.85 default. Raise –gpu-memory-utilization for more concurrent 256K sessions, or drop –max-num-seqs to 1 if it’s strictly you. If you hit a Mamba/CUDA-graph size error when raising the context, reduce –max-cudagraph-capture-size (default 512).

📝Note

Going beyond native 256K needs YaRN — prefer not to. To push toward ~1M you’d add -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 plus a YaRN –hf-overrides block (factor 2.0 ≈ 524K, factor 4.0 ≈ 1M). But YaRN stretches the model past its training and degrades short-prompt quality, and a ~1M prefill on GB10 (273 GB/s, Marlin FP4) takes minutes — fits ≠ fast. Stay at native 256K unless you truly need more; the exact override for this model is in the vLLM Qwen3.6 recipe:

  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
    --max-model-len 1010000 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

9. Pre-warm the JIT

The first request after boot triggers Inductor + FlashInfer codegen (~25–60 s). Fire a tiny warmup on the same path as real traffic, then short prompts return in <0.5 s:

curl -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<served-model-name>","messages":[{"role":"user","content":"ping"}],"max_tokens":3}'

This is separate from weight load (10–15 min for a 120B) — address that with fastsafetensors/InstantTensor if needed.

10. Verify & monitor

curl -sS http://localhost:8000/health
curl -sS http://localhost:8000/v1/models | jq
curl -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<served-model-name>","messages":[{"role":"user","content":"Explain quantum computing briefly."}],"max_tokens":200,"stream":true}'

Prometheus telemetry is at /metrics (no extra service). Watch KV-cache utilization (vllm:kv_cache_usage_perc) and TTFT / inter-token-latency histograms. Healthy single-user behavior: prefix-cached later turns don’t spike, decode rate stays steady, KV usage stays well below the context limit.

💡Tip

Confirm the fast paths are live. Check the startup log to verify NVFP4 GEMM kernels actually engaged (you want a line like Using NvFp4LinearBackend.FLASHINFER_CUTLASS for NVFP4 GEMM (exact backend varies — FLASHINFER_CUTLASS or MARLIN on Spark)) — if it silently fell back, you lose the FP4 speed/memory win. For speculative decoding, confirm acceptance via curl -s localhost:8000/metrics | grep -i spec_decode (see §14.6).

Benchmark throughput on your own Spark

Don’t trust anyone’s tok/s — including this guide’s — until you’ve measured on your box, image, and flags. vLLM ships vllm bench. NVIDIA’s benchmarking playbook distinguishes offline (raw model throughput, no HTTP/scheduler) from online (end-to-end through the running server), parameterized by ISL/OSL (input/output sequence length).

# Offline — raw decode/prefill throughput inside the container (no server)
docker exec -it qwen36-35b vllm bench throughput \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 --quantization modelopt \
  --input-len 1024 --output-len 256 --num-prompts 64

# Online — against your already-running server (realistic: scheduler + KV cache)
docker exec -it qwen36-35b vllm bench serve \
  --backend openai-chat --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --base-url http://localhost:8000 --endpoint /v1/chat/completions \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --num-prompts 100 --request-rate 4

📝Note

For a true single-stream decode number (what most “X tok/s” reviews quote), set –num-prompts 1 / –max-concurrency 1; sweep –request-rate (or –max-concurrency) upward to find your aggregate ceiling and the latency knee — recall a small model can go from tens of tok/s solo to many hundreds aggregate (§2). Match ISL/OSL to your real workload (a RAG turn ≈ long input/short output; a coding turn ≈ short input/long output) — throughput differs a lot between them. The same vllm bench works for trtllm-bench/SGLang equivalents if you want a cross-engine comparison.

Measuring prefill vs decode (single request)

vllm bench above is the full benchmarking harness; the script below is a focused companion that reads prefill (prompt → first token) versus decode (the streaming speed) directly, against any OpenAI-compatible vLLM endpoint. By default it measures single-stream — what most “X tok/s” reviews quote — but pass –concurrency N and it fires N requests at once and reports aggregate output throughput, the single→aggregate jump that makes Spark interesting (§2). It warms up the JIT, sends sized prompts, and reports TTFT, prefill tok/s, and decode tok/s from the server’s exact token counts.

Several details keep its numbers honest on Spark:

• it forces full-length generation (ignore_eos) so the decode rate is measured over a real number of tokens, not a model that stops after a short reply;
• it makes every prompt unique so –enable-prefix-caching can’t replay cached KV and make prefill look artificially fast;
• it counts reasoning tokens too — Qwen3.6 and other reasoning models stream their block as reasoning_content, not content, so a content-only counter would see nothing;
• it reports min/median/max across –runs (prefill in particular swings run-to-run, so a lone median hides the spread), and if the server exposes speculative-decoding counters on /metrics it prints the draft acceptance rate, so you can see why decode is fast, not just that it is (§14.6).

One honest limitation: throughput is token-accurate (from the server’s usage block), but per-token timing assumes ~1 token per streamed chunk. Speculative decoding can pack several accepted tokens into one chunk — the tok/s figures stay correct, but chunk-derived notes become approximate.

Save it as bench_prefill_decode.py:

#!/usr/bin/env python3
"""
bench_prefill_decode.py - measure vLLM PREFILL vs DECODE speed on a DGX Spark.

  PREFILL = processing the whole prompt up to the first token (compute-bound;
            reported as TTFT and prompt_tokens / TTFT).
  DECODE  = generating tokens after the first (bandwidth-bound; the streaming speed).

What this version gets right:
  * ignore_eos forces the model to emit the FULL --output-tokens, so the decode
    rate is measured over enough tokens to be stable (a 22-token reply is not).
  * every request carries a unique nonce, so --enable-prefix-caching on the server
    can't reuse cached KV and make prefill look artificially fast.
  * --runs reports min/median/max - a lone median hides the run-to-run swing that
    prefill in particular shows on a bandwidth-shared box.
  * --concurrency N fires N requests at once and reports AGGREGATE output throughput,
    the single->aggregate story that makes Spark interesting (tens of tok/s solo,
    hundreds at load).
  * if the server exposes speculative-decoding counters on /metrics, the draft
    acceptance rate is printed, so you can see WHY decode is fast, not just that it is.

Reasoning models emit <think> as `reasoning_content`; this counts both that and
`content`. Thinking control: --think forces it ON (enable_thinking=true; needed for
Gemma 4, which defaults OFF), --no-think forces it OFF, and omitting both leaves the
model's own default (Qwen3 on, Gemma 4 off).

Caveat: throughput is token-accurate (from the server's usage block), but per-token
TIMING assumes ~1 token per streamed chunk. Speculative decoding can pack several
accepted tokens into one chunk; that makes the chunk-based notes approximate, but
the tok/s figures themselves stay correct.

Usage:
  python3 bench_prefill_decode.py \
      --base-url http://localhost:8000/v1 \
      --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
      --prompt-tokens 4000 --output-tokens 256 --runs 3
  # single-user latency:       --concurrency 1   (default)
  # aggregate throughput:      --concurrency 16
  # machine-readable output:   --json
  # --api-key defaults to $VLLM_API_KEY; pass --api-key only to override it
"""
import argparse
import json
import math
import os
import statistics
import sys
import time
import uuid
from concurrent.futures import ThreadPoolExecutor

try:
    import requests
except ImportError:
    sys.exit("This script needs `requests`:  pip install requests")


def build_prompt(approx_tokens, nonce):
    unit = "The DGX Spark is a compact GB10 system with 128 GB of unified memory. "
    reps = max(1, approx_tokens // len(unit.split()))
    # The nonce defeats prefix caching so prefill is actually measured each run.
    return f"[run {nonce}] Read the text below, then reply.\n\n" + unit * reps


def run_once(session, url, headers, model, prompt, max_tokens,
             temperature=0.0, enable_thinking=None, ignore_eos=True):
    """Stream one completion; return timing + token counts for a single request."""
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": temperature,
        "stream": True,
        "stream_options": {"include_usage": True},
        "ignore_eos": ignore_eos,          # vLLM extension: generate the full budget
    }
    if enable_thinking is not None:
        payload["chat_template_kwargs"] = {"enable_thinking": enable_thinking}

    t0 = time.perf_counter()
    t_first = None
    usage = finish = None
    n_content = n_reasoning = 0
    try:
        resp = session.post(url, headers=headers, json=payload, stream=True, timeout=900)
    except requests.exceptions.RequestException as e:
        sys.exit(f"Could not reach the server at {url}: {e}\n"
                 "Is vLLM up, and does --base-url end in /v1 ?")
    with resp as r:
        if r.status_code != 200:
            sys.exit(f"HTTP {r.status_code}: {r.text[:500]}")
        for raw in r.iter_lines():
            if not raw:
                continue
            line = raw.decode("utf-8", "replace")
            if not line.startswith("data: "):
                continue
            data = line[len("data: "):]
            if data.strip() == "[DONE]":
                break
            try:
                obj = json.loads(data)
            except json.JSONDecodeError:
                continue                    # tolerate keep-alive / partial frames
            if obj.get("usage"):
                usage = obj["usage"]
            for ch in (obj.get("choices") or []):
                delta = ch.get("delta") or {}
                # Detect generated text in ANY field (content / reasoning_content /
                # reasoning / ...) so field-name differences across builds don't hide it.
                field = next((k for k, v in delta.items()
                              if k != "role" and isinstance(v, str) and v), None)
                if field:
                    if field == "content":
                        n_content += 1
                    else:
                        n_reasoning += 1
                    if t_first is None:
                        t_first = time.perf_counter()
                if ch.get("finish_reason"):
                    finish = ch["finish_reason"]
    t_end = time.perf_counter()

    p_tok = usage.get("prompt_tokens") if usage else None
    c_tok = usage.get("completion_tokens") if usage else None
    ttft = (t_first - t0) if t_first is not None else None
    decode_t = (t_end - t_first) if t_first is not None else None
    prefill_tps = (p_tok / ttft) if (p_tok and ttft and ttft > 0) else math.nan
    decode_tps = ((c_tok - 1) / decode_t) if (c_tok and c_tok > 1 and decode_t and decode_t > 0) else math.nan
    return {"t0": t0, "t_end": t_end, "t_first": t_first, "p_tok": p_tok, "c_tok": c_tok,
            "finish": finish, "ttft": ttft, "prefill_tps": prefill_tps, "decode_tps": decode_tps,
            "n_content": n_content, "n_reasoning": n_reasoning}


def scrape_spec_metrics(session, metrics_url):
    """Return (accepted_tokens, draft_tokens) spec-decode counters, or None if absent."""
    if not metrics_url:
        return None
    try:
        r = session.get(metrics_url, timeout=10)
        if r.status_code != 200:
            return None
    except requests.exceptions.RequestException:
        return None
    accepted = draft = None
    for line in r.text.splitlines():
        if line.startswith("#"):
            continue
        try:
            value = float(line.rsplit(" ", 1)[-1])
        except (ValueError, IndexError):
            continue
        if "spec_decode_num_accepted_tokens" in line:
            accepted = value
        elif "spec_decode_num_draft_tokens" in line:
            draft = value
    if accepted is None and draft is None:
        return None
    return accepted, draft


def measure_batch(shared_session, url, headers, args, enable_thinking):
    """Fire args.concurrency requests at once; return (per-request results, aggregate tok/s)."""
    def one():
        # one Session per worker: requests.Session isn't guaranteed thread-safe
        sess = shared_session if args.concurrency == 1 else requests.Session()
        prompt = build_prompt(args.prompt_tokens, uuid.uuid4().hex[:12])
        return run_once(sess, url, headers, args.model, prompt, args.output_tokens,
                        temperature=args.temperature, enable_thinking=enable_thinking)

    c = max(1, args.concurrency)
    start = time.perf_counter()
    if c == 1:
        results = [one()]
    else:
        with ThreadPoolExecutor(max_workers=c) as ex:
            futures = [ex.submit(one) for _ in range(c)]
            results = [f.result() for f in futures]
    wall = time.perf_counter() - start
    out_tokens = sum(r["c_tok"] or 0 for r in results)
    agg_tps = out_tokens / wall if wall > 0 else math.nan
    return results, agg_tps


def fmt(x, nd=0):
    return "nan" if (x is None or (isinstance(x, float) and math.isnan(x))) else f"{x:.{nd}f}"


def stats(values):
    vals = [v for v in values if v is not None and not (isinstance(v, float) and math.isnan(v))]
    if not vals:
        return {"min": math.nan, "median": math.nan, "max": math.nan}
    return {"min": min(vals), "median": statistics.median(vals), "max": max(vals)}


def main():
    ap = argparse.ArgumentParser(description="Measure vLLM prefill vs decode throughput on a DGX Spark.")
    ap.add_argument("--base-url", default="http://localhost:8000/v1")
    ap.add_argument("--model", required=True, help="must match an id from /v1/models")
    ap.add_argument("--api-key", default=os.environ.get("VLLM_API_KEY"),
                    help="defaults to $VLLM_API_KEY; only needed if vLLM was launched with one")
    ap.add_argument("--prompt-tokens", type=int, default=2000, help="approx prompt size")
    ap.add_argument("--output-tokens", type=int, default=256, help="decoded each run (forced via ignore_eos)")
    ap.add_argument("--runs", type=int, default=1, help="measured runs; reports min/median/max")
    ap.add_argument("--concurrency", type=int, default=1,
                    help="requests fired at once; >1 reports AGGREGATE output throughput")
    ap.add_argument("--temperature", type=float, default=0.0,
                    help="sampling temperature (default 0; raise to measure spec-decode at your serving temp)")
    ap.add_argument("--think", action="store_true",
                    help="force thinking ON (enable_thinking=true; needed for Gemma 4, which defaults OFF)")
    ap.add_argument("--no-think", action="store_true", help="force thinking OFF (enable_thinking=false)")
    ap.add_argument("--no-warmup", action="store_true")
    ap.add_argument("--json", dest="as_json", action="store_true", help="emit results as a JSON object")
    args = ap.parse_args()

    if args.think and args.no_think:
        sys.exit("--think and --no-think are mutually exclusive")
    base = args.base_url.rstrip("/")
    url = base + "/chat/completions"
    metrics_url = (base[:-3].rstrip("/") + "/metrics") if base.endswith("/v1") else None
    headers = {"Content-Type": "application/json"}
    if args.api_key:
        headers["Authorization"] = f"Bearer {args.api_key}"
    # None => send nothing, let the model's own default decide (Qwen3 on, Gemma 4 off)
    enable_thinking = True if args.think else (False if args.no_think else None)
    log = (lambda *a: None) if args.as_json else (lambda *a: print(*a, flush=True))

    session = requests.Session()
    if not args.no_warmup:
        log("Warming up (JIT / torch.compile / CUDA-graph capture) at the measured prompt size ...")
        # warm up at the REAL prompt size so the first measured run isn't a recompile
        run_once(session, url, headers, args.model,
                 build_prompt(args.prompt_tokens, uuid.uuid4().hex[:12]),
                 max_tokens=8, temperature=args.temperature, enable_thinking=enable_thinking)

    spec_before = scrape_spec_metrics(session, metrics_url)
    mode = f"concurrency {args.concurrency}" if args.concurrency > 1 else "single-stream"
    think_s = " (thinking ON)" if args.think else (" (thinking OFF)" if args.no_think else "")
    log(f"Measuring {args.runs} run(s), {mode}: ~{args.prompt_tokens} prompt tokens, "
        f"{args.output_tokens} output tokens{think_s} ...")

    runs = []
    for i in range(args.runs):
        results, agg_tps = measure_batch(session, url, headers, args, enable_thinking)
        if any(r["t_first"] is None for r in results) and args.concurrency == 1:
            r0 = results[0]
            print("\nNo per-token deltas were streamed.", file=sys.stderr)
            print(f"  usage: prompt_tokens={r0['p_tok']} completion_tokens={r0['c_tok']} "
                  f"finish={r0['finish']}", file=sys.stderr)
            if r0["c_tok"]:
                print("  Tokens were generated but none were streamed. With thinking ON this is usually the\n"
                      "  reasoning parser buffering the <think> block: ignore_eos hit the cap before </think>\n"
                      "  closed, so nothing flushed. Fix: --no-think, or raise --output-tokens (e.g. 2048).",
                      file=sys.stderr)
            sys.exit(1)
        runs.append({
            "results": results, "agg_tps": agg_tps,
            "prefill": statistics.median([r["prefill_tps"] for r in results
                                          if not math.isnan(r["prefill_tps"])] or [math.nan]),
            "decode": statistics.median([r["decode_tps"] for r in results
                                         if not math.isnan(r["decode_tps"])] or [math.nan]),
            "ttft": statistics.median([r["ttft"] for r in results if r["ttft"]] or [math.nan]),
        })
        if args.runs > 1 and not args.as_json:
            if args.concurrency > 1:
                log(f"  run {i + 1}: aggregate {fmt(agg_tps)} tok/s, "
                    f"per-req decode {fmt(runs[-1]['decode'], 1)} tok/s")
            else:
                r0 = results[0]
                log(f"  run {i + 1}: prefill {fmt(r0['prefill_tps'])} tok/s, "
                    f"decode {fmt(r0['decode_tps'], 1)} tok/s (out {r0['c_tok']}, finish {r0['finish']})")

    spec_after = scrape_spec_metrics(session, metrics_url)
    acceptance = None
    if spec_before and spec_after and spec_before[1] is not None and spec_after[1] is not None:
        d_acc = (spec_after[0] or 0) - (spec_before[0] or 0)
        d_draft = (spec_after[1] or 0) - (spec_before[1] or 0)
        if d_draft > 0:
            acceptance = {"rate": d_acc / d_draft, "accepted": d_acc, "draft": d_draft}

    last = runs[-1]["results"][0]
    prefill, decode = stats([r["prefill"] for r in runs]), stats([r["decode"] for r in runs])
    ttft = stats([r["ttft"] for r in runs])
    ttft_ms = {k: (v * 1000 if (v is not None and v == v) else math.nan) for k, v in ttft.items()}
    agg = stats([r["agg_tps"] for r in runs])

    if args.as_json:
        out = {"model": args.model, "prompt_tokens": last["p_tok"], "output_tokens": last["c_tok"],
               "runs": args.runs, "concurrency": args.concurrency, "temperature": args.temperature,
               "thinking": enable_thinking, "ttft_ms": ttft_ms,
               "prefill_tps": prefill, "decode_tps_per_request": decode}
        if args.concurrency > 1:
            out["aggregate_output_tps"] = agg
        if acceptance:
            out["spec_decode_acceptance"] = acceptance
        print(json.dumps(out, indent=2))
        return

    def tri(d, nd):
        return (fmt(d["median"], nd) if args.runs == 1
                else f"{fmt(d['median'], nd)}   [{fmt(d['min'], nd)} .. {fmt(d['max'], nd)}]")

    spread = "" if args.runs == 1 else ", min/median/max"
    print(f"\n================ results ({args.runs} run{'' if args.runs == 1 else 's'}{spread}) ================")
    print(f"prompt tokens (prefilled) : {last['p_tok']}")
    print(f"output tokens (decoded)   : {last['c_tok']}   (finish: {last['finish']})")
    if last["n_reasoning"]:
        print(f"  thinking / answer (chunk) : ~{last['n_reasoning']} thinking / ~{last['n_content']} answer")
    print(f"TTFT  (prefill latency)   : {tri(ttft_ms, 0)} ms")
    print(f"PREFILL throughput        : {tri(prefill, 0)} tok/s   (prompt_tokens / TTFT)")
    if args.concurrency > 1:
        print(f"DECODE  (per request)     : {tri(decode, 1)} tok/s")
        print(f"DECODE  (aggregate x{args.concurrency})    : {tri(agg, 0)} tok/s   "
              f"<- total output across all {args.concurrency} streams")
    else:
        tpot = (1000.0 / decode["median"]) if (decode["median"] and decode["median"] == decode["median"]) else math.nan
        print(f"DECODE  throughput        : {tri(decode, 1)} tok/s   ({fmt(tpot, 1)} ms/token)")
    if acceptance:
        print(f"SPEC-DECODE acceptance    : {acceptance['rate'] * 100:.0f}%   "
              f"({int(acceptance['accepted'])} accepted / {int(acceptance['draft'])} drafted)")
    print("=" * 57)

    if last["p_tok"] is None:
        print("NOTE: server returned no usage block (older build / include_usage unsupported);")
        print("      token-based throughput is unavailable, but the TTFT timings are still valid.")
    if last["c_tok"] and last["c_tok"] < 64:
        print("WARNING: output < 64 tokens - decode number is noisy; raise --output-tokens.")
    # Fire only when chunks are FEW in absolute terms - this targets a buffering parser
    # without false-alarming on speculative decoding, which legitimately packs many
    # accepted tokens into one chunk.
    seen = last["n_content"] + last["n_reasoning"]
    if last["c_tok"] and last["c_tok"] >= 64 and 0 < seen < 8:
        print("NOTE: very few stream chunks for many tokens - either a buffering reasoning parser")
        print("      (try --no-think) or multi-token chunks from speculative decoding (expected).")
    if last["n_reasoning"] and not args.no_think:
        print("Reasoning model: thinking tokens count toward DECODE. Use --no-think for answer-only.")
    if args.concurrency > 1:
        print(f"AGGREGATE is total output across {args.concurrency} concurrent streams; "
              "per-request is what one user feels.")
    print("Prompts are unique per run (prefix-cache-proof) and forced to full length (ignore_eos).")


if __name__ == "__main__":
    main()

Run it single-stream on the Spark (median of 3 runs, answer-only):

python3 bench_prefill_decode.py \
  --base-url http://localhost:8000/v1 \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --prompt-tokens 4000 --output-tokens 256 --runs 3 --no-think

…and the same model under load — 16 concurrent requests, reporting aggregate throughput alongside per-request decode:

python3 bench_prefill_decode.py \
  --base-url http://localhost:8000/v1 \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --prompt-tokens 4000 --output-tokens 256 --runs 3 --concurrency 16 --no-think

Flags: –prompt-tokens sizes the prompt (TTFT scales ~linearly with it); –output-tokens is forced via ignore_eos; –runs N reports min/median/max; –concurrency N fires N requests at once and adds an aggregate-output-throughput line (leave at 1 for single-user latency); –temperature defaults to 0 — raise it to measure speculative-decode throughput at your real serving temperature, since acceptance is sampling-dependent; –no-think forces answer-only and –think forces thinking on (needed for models like Gemma 4 that default off — omit both for the model’s own default); –json emits a machine-readable result for sweeps or CI; –api-key defaults to $VLLM_API_KEY (pass it only to override). For a reasoning model, give thinking-on runs a large –output-tokens (e.g. 2048) so it can close and emit an answer.

Measured this way on a single DGX Spark — Qwen3.6-35B-A3B-NVFP4 on the §7.1.1 recipe (MTP-3, NVFP4), 3-run medians:

Four patterns, all of which generalize past this one model: (1) prefill is flat at ~6,200 tok/s regardless of prompt size, so TTFT scales linearly — predict it as prompt_tokens ÷ 6,200 (≈0.5 s at 3K, ≈1.0 s at 6K, ≈10 s at 64K). (2) thinking decodes faster than answering (~120 vs ~102 tok/s) because MTP acceptance is higher on structured reasoning than on free-form prose — the same content-dependence speculative decoding shows everywhere (§14). (3) longer context decodes slightly slower (125 → 117 tok/s from 3K → 6K) because each token re-reads a larger KV cache. (4) MTP is landing ~3 tokens per step (~80% acceptance at num_speculative_tokens:3), which is what lifts decode above the bare bandwidth ceiling. For an agent, felt latency is dominated by TTFT (prompt size) and how much the model thinks — not the per-token rate.

11. Connecting a frontend & agents (Open WebUI / Hermes Agent)

Once vLLM is serving, it exposes an OpenAI-compatible API at http://:8000/v1. Two common ways to put a face on it:

• Open WebUI — a chat frontend that talks straight to the model.
• Hermes Agent (Nous Research) — an autonomous agent (terminal, file ops, web search, memory) that uses your model as its brain and exposes its own OpenAI API; Open WebUI then connects to the agent. The chain is vLLM → Hermes Agent → Open WebUI.

❗Important

The Docker-networking gotcha that defeats everyone: Open WebUI (and Hermes) run in their own containers, so http://localhost:8000/v1 points at their container, not your Spark host — the connection fails even though vLLM is up. Use one of the options below. On Linux without Docker Desktop, host.docker.internal does not resolve unless the container was started with –add-host=host.docker.internal:host-gateway.

Putting both on one network is the cleanest long-term fix (works because vLLM serves on –host 0.0.0.0):

docker network create ai 2>/dev/null
docker network connect ai qwen36-35b
docker network connect ai open-webui
# then address vLLM as  http://qwen36-35b:8000/v1

11.1 Open WebUI → your model

In Open WebUI: Admin Settings → Connections → OpenAI → Manage (wrench icon) → Add New Connection → Standard / Compatible tab:

• API URL: http://<SPARK_LAN_IP>:8000/v1 (not localhost — see above)
• API Key: none (vLLM has no key unless you set –api-key/VLLM_API_KEY; any placeholder works)

Pick the model (it appears under its served name, e.g. nvidia/Qwen3.6-35B-A3B-NVFP4) and chat.

💡Tip

A big model can take longer to load than Open WebUI’s 10 s model-list fetch timeout — if the connection saves but no models appear, recreate Open WebUI with -e AIOHTTP_CLIENT_TIMEOUT_MODEL_LIST=30. Sanity-check reachability first from inside the container: docker exec -it open-webui curl -s http://<SPARK_LAN_IP>:8000/v1/models — if it lists the model, the UI will connect with the same URL.

11.2 Hermes Agent → your model → Open WebUI

Hermes Agent (Nous Research) is a separate install, not just a connection — it adds terminal, file ops, web search, memory, and skills on top of your model:

1. Install Hermes Agent (quickstart); confirm with hermes –version.
2. Point its brain at vLLM — in Hermes’s config (~/.hermes/.env), set its model-provider base URL to your Spark endpoint (http://<SPARK_LAN_IP>:8000/v1), model nvidia/Qwen3.6-35B-A3B-NVFP4, key none. (Exact variable names are in the Hermes quickstart; it accepts any OpenAI-compatible endpoint.)
3. Enable Hermes’s own API server so Open WebUI can reach it — add to ~/.hermes/.env:

   API_SERVER_ENABLED=true
   API_SERVER_KEY=your-secret-key
   API_SERVER_HOST=0.0.0.0     # default 127.0.0.1 is localhost-only; 0.0.0.0 lets the open-webui container reach it
   

4. Run the gateway (keep it up — tmux/screen or a systemd service): hermes gateway → listens on :8642.
5. Add Hermes as a second connection in Open WebUI (same Connections → OpenAI panel): URL http://<SPARK_LAN_IP>:8642/v1, API Key = your API_SERVER_KEY. The hermes-agent model appears in the dropdown; selecting it routes through the agent, with inline tool indicators as it works.

📝Note

For the agent to use tools reliably, the underlying model needs solid function-calling — on the vLLM side add –enable-auto-tool-choice –tool-call-parser qwen3_xml (Qwen3.6 has strong tool use; this is the same pair baked into the §7.1.1 recipe). If the Hermes connection tests OK but no model loads, it’s almost always a missing /v1 suffix; verify with curl http://<SPARK_LAN_IP>:8000/v1/models.

12. Troubleshooting

13. Scaling to multiple Sparks (brief)

A single Spark handles models up to ~100 GB of weights. For larger models or higher TP, link Sparks over the 200 Gb/s QSFP (ConnectX-7) ports and run a Ray cluster with –tensor-parallel-size = number of GPUs:

• 2 Sparks: direct QSFP cable. 3+: through a switch.
• Bind NCCL to the QSFP interface (NCCL_SOCKET_IFNAME=enP2p1s0f1np1); an Ethernet fallback costs 10–20× throughput.
• Mount the same ~/vllm on every node and stage weights on each.
• Easiest: the mark-ramsey-ri/vllm-dgx-spark scripts, or NVIDIA’s spark_cluster_setup.sh + the official multi-Spark playbook.

14. Speculative decoding on Spark: MTP & DFlash (step-by-step)

A small, cheap drafter proposes the next few tokens; the big model verifies them in one pass — accepted tokens are free. It’s a latency win that shines at low concurrency, exactly Spark’s profile, and can roughly 2–3× interactive decode speed with no quality loss. Three approaches matter on Spark, easiest first:

• n-gram (prompt-lookup) — zero setup, any model, any image. Free speed when your output repeats the input (code edits, RAG, JSON, agent loops); nothing on novel prose. Start here — it costs nothing to add.
• MTP — one flag on a model that ships MTP heads (Qwen3.6, gemma4-coder). No download; broad, modest speedup.
• DFlash — a trained block-diffusion drafter (needs vLLM ≥ 0.21 + an sm121 build). The biggest win (up to ~6×) on code/reasoning, but per-target setup.

Rule of thumb: try n-gram first; reach for MTP if the model has it; invest in DFlash for maximum single-stream speed on one specific model.

14.1 Do they need special models, Docker, or configs?

📝Note

Both consume KV cache for speculative tokens, so they trade peak throughput for latency. Keep them on for interactive use; for bursty/concurrent serving add –speculative-disable-by-batch-size 32.

Which models in this guide can use which method:

(Empty = no published speculator for that exact checkpoint today; “validate” = architecture supports it but confirm acceptance on your build.)

n-gram works on every model above — it needs no model support, just append its flag (§14.2); it’s simply less effective the less your output echoes your input.

14.2 n-gram (prompt-lookup) — zero everything

The cheapest speculative decoding there is: no drafter, no special image, no model support. vLLM scans the prompt + text-so-far for a repeat of the last few tokens and proposes whatever followed them last time. When your output reuses spans of the input — editing a file (most lines unchanged), quoting retrieved context (RAG), reformatting, structured/JSON output, or agent loops that echo schemas — acceptance is high and the speed-up is free, even on the stock NGC 26.04 image with any checkpoint.

# append to ANY §7 recipe — nothing else changes
    --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":5}'

num_speculative_tokens = tokens proposed per step (3–5 typical). prompt_lookup_min/max = shortest/longest history n-gram to match (smaller min = more, riskier matches). It works with fp8 KV cache and needs no ≥ 0.21 build — it’s core vLLM. The catch is the flip side of its strength: on novel prose there’s nothing to look up, so acceptance ≈ 0 — turn it off there, or use MTP/DFlash.

💡Tip

A close cousin, “method”:“suffix” (suffix decoding), keeps a small suffix-tree cache instead of a fixed-length lookup and often accepts more on repetitive output — –speculative-config ‘{“method”:“suffix”,“num_speculative_tokens”:8}’. Also drafter-free and auto-resolved; try it if n-gram’s acceptance is only lukewarm on your workload.

14.3 MTP — zero-download (≈5 min)

Just a flag on an MTP-capable model (the §7.1.1 Qwen3.6 and §7.2.3 gemma4-coder recipes already use it):

📝Note

Two flavors of MTP. Most MTP models (Qwen3.6, DeepSeek-style) have the prediction heads baked into the checkpoint — nothing extra to download, just the flag. A few ship a small paired draft model instead: gemma4-coder bundles a 0.4 B draft in assistant/, so you point “model” at it (/model/assistant) — see §7.2.3. Both use “method”:“mtp”; only the second needs the “model” field.

docker run -d --name mtp --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/vllm:/models -e HF_TOKEN="$HF_TOKEN" \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly \
  nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8000 --trust-remote-code \
    --quantization modelopt --kv-cache-dtype fp8 --moe-backend marlin \
    --gpu-memory-utilization 0.85 --max-model-len 65536 --max-num-seqs 4 \
    --reasoning-parser qwen3 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'

num_speculative_tokens is usually 1 (some recipes use 3). For lowest latency at low concurrency, the vLLM recipe suggests disabling prefix caching. MTP reduces throughput under load — pair with –speculative-disable-by-batch-size 32.

14.4 DFlash on Spark — plug-and-play container (easiest)

The ghcr.io/aeon-7/vllm-dflash image is a prebuilt sm_121 vLLM with DFlash baked in; it serves a 27B dense Qwen3.5 with a block-diffusion drafter. Decode is sharply content-dependent: from the ~12 tok/s stock-eager baseline (the §2 chart), code/reasoning reaches ~60 tok/s and free prose ~30, while random/adversarial text shows no gain. (The §2 chart sweeps speculative-token count and prompt length on generic content and peaks near ~33 tok/s; the further climb to ~60 here is the content effect — structured code accepts much longer accepted blocks than prose.)

# 1) download the target into ~/vllm
pip install -U "huggingface_hub[cli]"; export HF_TOKEN=hf_xxx
hf download AEON-7/DFlash-Qwen3.5-27B-Uncensored-NVFP4 \
  --local-dir ~/vllm/DFlash-Qwen3.5-27B-Uncensored-NVFP4

# 2) make a PERSISTENT API key — save it; you need the SAME value for every request
export VLLM_API_KEY=$(openssl rand -hex 32); echo "VLLM_API_KEY=$VLLM_API_KEY" >> ~/.bashrc
docker run -d --name vllm-dflash --runtime nvidia --network host --ipc host \
  --restart unless-stopped --ulimit memlock=-1:-1 \
  -v ~/vllm/DFlash-Qwen3.5-27B-Uncensored-NVFP4:/models/target:ro \
  -v ~/vllm/drafter-cache:/models/drafter-cache \
  -e MODEL_PATH=/models/target \
  -e SERVED_MODEL_NAME=dflash-qwen3.5-27b \
  -e DFLASH_DRAFTER=z-lab/Qwen3.5-27B-DFlash \
  -e DFLASH_NUM_SPEC_TOKENS=15 \
  -e MAX_MODEL_LEN=65536 -e MAX_NUM_SEQS=16 \
  -e GPU_MEMORY_UTILIZATION=0.85 -e MAX_NUM_BATCHED_TOKENS=32768 \
  -e VLLM_API_KEY="$VLLM_API_KEY" -e HF_TOKEN="$HF_TOKEN" \
  ghcr.io/aeon-7/vllm-dflash:latest
docker logs -f vllm-dflash    # ~5–7 min cold boot

# 3) test (note the Bearer token)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model":"dflash-qwen3.5-27b","messages":[{"role":"user","content":"Write a haiku about GPUs."}],"max_tokens":200}'

Key env vars: DFLASH_DRAFTER (HF id of the drafter; empty = plain vLLM), DFLASH_NUM_SPEC_TOKENS (15 single-stream), KV_CACHE_DTYPE (stays BF16 with DFlash). AEON’s tuned Spark defaults are MAX_NUM_SEQS=16 / MAX_NUM_BATCHED_TOKENS=32768 for the 128 GB pool — lower MAX_NUM_SEQS if you want pure single-user latency. Persist VLLM_API_KEY (don’t inline a one-shot $(openssl …) you can’t reproduce).

💡Tip

AEON also ships a newer unified image, ghcr.io/aeon-7/aeon-vllm-ultimate:latest, with a Qwen3.6-27B target (AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash) at num_speculative_tokens: 12 — currently their best-performing GB10 package (~37.6 tok/s single-stream vs ~10.5 stock-eager, with the z-lab sliding-window Qwen3.6-27B drafter). On Spark, leave the drafter’s attention backend at default — don’t set attention_backend inside the spec config.

14.5 DFlash the general way (other models)

DFlash uses vLLM’s speculators format; pass the drafter in –speculative-config. Available drafters today:

docker run --rm vllm/vllm-openai:nightly vllm --version   # confirm >= 0.21.0

# Laguna XS.2 (coding; up to 7 tokens/step, ~70% acceptance on code)
docker run -d --name laguna --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" -e HF_HOME=/models -e VLLM_USE_DEEP_GEMM=0 -v ~/vllm:/models \
  vllm/vllm-openai:nightly \
  poolside/Laguna-XS.2 --trust-remote-code \
    --enable-auto-tool-choice --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
    --speculative-config '{"model":"poolside/Laguna-XS.2-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}'

📝Note

DFlash needs vLLM ≥ 0.21.0 (PR #41880); the stock NGC 26.04 (0.19.0) won’t do it — use nightly or a prebuilt image. VLLM_USE_DEEP_GEMM=0 is required for all DFlash (DeepGEMM is incompatible with the DFlash draft path), not just Laguna. On sm121, set the target –attention-backend triton_attn — z-lab’s own Spark builds use Triton, and flash_attn/FlashInfer crash on GB10; leave the drafter’s backend at default. Gemma-4 DFlash needs Z Lab’s build (ghcr.io/z-lab/vllm-openai:gemma4-dflash-cu130). The canonical drafter repo is github.com/z-lab/dflash; no drafter for your model? Train one — the speculators project (v0.5.0+) ships a DFlash training tutorial. The 4B/8B drafters (Qwen3.5-4B, Qwen3-8B, plus a Llama-3.1-8B drafter in the same collection) target small, fast models — where DFlash pushes single-stream into the hundreds of tok/s, useful for high-concurrency serving or draft-heavy agent loops; set num_speculative_tokens to the block size (8 for concurrency, 16 for longer accepts at concurrency 1).

14.6 Verify it’s actually accelerating (don’t fly blind)

Speculative decoding only helps if the drafter’s tokens are accepted. Confirm it, then tune:

• Check acceptance. vLLM logs a spec-decode summary (draft acceptance rate and mean accepted length per step) and exposes it on /metrics:

  curl -s http://localhost:8000/metrics | grep -i spec_decode
  

  Healthy MTP runs accept most proposed tokens (mean accepted length well above 1); DFlash on code can accept whole blocks. If acceptance is near zero, the method isn’t helping — the drafter/target are mismatched or the content is low-predictability (random/adversarial text).
• Tune num_speculative_tokens (K). Raise K for DFlash (it verifies a whole block in one pass — 7–15 is normal); keep K low for MTP (1–3) since each extra token costs a verify pass. Watch decode tok/s as you change it — past the acceptance sweet spot, throughput drops.
• A/B against a dense run. Time the same prompt with the spec flag removed. If tok/s isn’t clearly higher and output is identical, drop speculation for that workload.
• Concurrency check. Both methods tax throughput under load; if you serve bursts, set –speculative-disable-by-batch-size 32 so vLLM auto-disables speculation when batches grow.

14.7 The gotchas that bite people

🔥Caution

• DFlash floor: vLLM ≥ 0.21.0 (AEON-7/Z-Lab image or nightly). MTP is fine on 0.19.0+.
• VLLM_USE_DEEP_GEMM=0 for DFlash — DeepGEMM is incompatible with the DFlash draft path (silent failure otherwise).
• On sm121, target –attention-backend triton_attn — flash_attn/FlashInfer crash on GB10; leave the drafter’s backend at default.
• DFlash KV cache stays BF16 — never combine with –kv-cache-dtype fp8. (The MTP gemma4-coder draft is the opposite — it needs fp8 KV.) Relatedly, FP8-KV checkpoints emit scrambled output on non-Hopper GPUs (Spark included) — e.g. Laguna-XS.2-FP8/INT4; use the BF16 checkpoint or disable FP8 KV.
• One drafter per target — a DFlash drafter is trained for a specific model.
• Gemma 4 paired-draft MTP needs current vLLM — if startup logs show SpeculativeConfig(method=‘draft_model’, …) for a gemma4-coder assistant/ checkpoint, your vLLM build lacks Gemma-4 MTP support for that path; upgrade rather than forcing it through generic draft-model decoding.
• Higher K is cheap with DFlash (whole block in one pass), not with MTP — keep MTP num_speculative_tokens low (1–3); DFlash runs 7–15.
• Both hurt throughput under load — add –speculative-disable-by-batch-size 32.
• Acceptance is content-dependent — big on code/structured output, smaller on prose, none on random text.

Compiled June 2026. Performance charts are measured single-Spark numbers: the §2 decode chart and the §10 Qwen3.6 chart are this guide’s own bench-script runs on one DGX Spark; the §2 DFlash chart is AEON-7’s published Spark measurement. Container tags and model handles move fast — pin a known-good image digest for anything you depend on.