# vLLM backend — Huihui Qwen3.8-27B-abliterated (AWQ W4A16)

Self-contained vLLM version of the parent `coding-stack-huihui-qwen3.8-27b` project.
Instead of the Q8 GGUF it serves the pre-quantized INT4 checkpoint
**`shawnw3i/Huihui-Qwen3.8-27B-abliterated-AWQ-MTP`** (auto-round AWQ, group_size 64,
18.65 GiB on disk) as the model **`foundation`**, behind the same OpenAI-compatible
proxy that injects a SearXNG `web_search` tool.

This is the fast backend: vLLM's paged KV cache, chunked prefill and Marlin INT4 kernels
are what make the 250k-token context practical on one 48 GB card. It is also the least
flexible one — one sequence at a time, no vision, no FP8 on this GPU.

Deliberately **not** included: the llama.cpp and Ollama compose files, the
LibreChat/MongoDB/Meilisearch stack, GGUF/mmproj handling, and the parent's shared
container names/API key.

```
                   ┌─────────────── web_search ──────────────► SearXNG :8889 (external)
                   │
Client ──► :11433 proxy (Flask) ──► :18080 vLLM (OpenAI API)
                   ▲
                   └── :18080 is also published directly, bypassing the proxy
```

Containers: `coding-stack-vllm` (`vllm/vllm-openai:nightly`) and
`coding-stack-proxy` (`python:3.11-slim`). Port **11433** and the proxy container name
clash with the parent and Ollama stacks — stop those first if you switch back and forth.

## Files

| File | Purpose |
|------|---------|
| `docker-compose.yaml` | vLLM server + web_search proxy (the whole stack) |
| `proxy-passthrough.py` | OpenAI facade on `:11433`; injects and executes `web_search` |
| `download-model.sh` | Pulls the AWQ checkpoint into the HF cache vLLM reads |
| `benchmark.sh` | TTFT + decode tok/s from a streaming request |
| `.gitignore` | Keeps caches and weights out of git |

There is no Modelfile and no import step: vLLM loads the HF checkpoint directly, and the
chat template comes from the checkpoint's `chat_template.jinja`.

## Requirements

- Docker + Docker Compose v2, NVIDIA driver + Container Toolkit
- **One whole 48 GB GPU** (RTX A6000 here): 18.65 GiB weights + ~15.5 GiB KV at 253952
  context + CUDA graphs/activations, at `--gpu-memory-utilization 0.93`
- **Compute capability ≥ 8.9 only for FP8** — the A6000 is 8.6, so weights and KV stay
  FP16 and INT4 comes from AWQ + Marlin kernels. Don't add `--kv-cache-dtype fp8`
  (or an FP8 checkpoint) on this card: it won't run.
- ~62 GB RAM is comfortable; vLLM reported 55 GiB available while loading 18.65 GiB
- Image: `vllm/vllm-openai:nightly` (~29 GB, **0.27.2rc1** at the time of writing,
  2 months old in this host's cache). It is a moving tag — re-pull deliberately.
- Optional: SearXNG at `host.docker.internal:8889` for `web_search`

The checkpoint is **already in this host's cache**:
`/storage-models/vllm/hub/models--shawnw3i--Huihui-Qwen3.8-27B-abliterated-AWQ-MTP`
(7 shards + `model_extra_tensors.safetensors`, size-verified in the parent project).
`download-model.sh` is a no-op when it is there.

## Quick start

```bash
cd vllm
./download-model.sh          # ~19 GB; no-op if the HF cache already has it
docker compose up -d         # first start: load + compile + CUDA graph capture (minutes)
docker compose ps            # wait for (healthy) — start_period is 900 s
./benchmark.sh               # TTFT + tok/s
```

Cold start is slow by design: vLLM compiles the model's kernels (`torch.compile`,
Triton GDN kernels) and captures CUDA graphs before it reports healthy. Expect several
minutes on first boot; subsequent boots reuse the compile cache in the image's
`/root/.cache` only if you persist it (this stack does not — add a volume if you care).

## Verify

```bash
# Direct to vLLM (OpenAI-compatible)
curl http://localhost:18080/v1/models
curl http://localhost:18080/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# Through the proxy (+ web_search injected)
curl http://localhost:11433/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

# Streaming (proxy fast-path: SSE passed straight through)
curl -N http://localhost:11433/v1/chat/completions -H "Authorization: Bearer x" \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hello"}],"stream":true,"max_tokens":64}'

curl http://localhost:18080/health
```

vLLM is started **without** `--api-key`, so any `Authorization` value works (or none).
Add `--api-key` and `BACKEND_API_KEY` to the proxy if the published port must be closed.

## Endpoints

| Port | Service | Notes |
|------|---------|-------|
| `11433` | proxy (Flask) | OpenAI `/v1/*`; injects `web_search`; everything else is transparent |
| `18080` | vLLM | OpenAI `/v1/*`, `/health`, `/metrics`; no auth |

Model name: **`foundation`** (`--served-model-name`), regardless of the HF repo name.

## Configuration

### vLLM flags (`docker-compose.yaml`)

| Flag | Value | Why |
|------|-------|-----|
| `--max-model-len` | `253952` | fits 18.65 GiB weights + ~15.5 GiB f16 KV on 48 GB; native max is 262144 |
| `--gpu-memory-utilization` | `0.93` | leaves ~3 GiB for the desktop/other processes |
| `--max-num-seqs` | `1` | one 250k sequence consumes the whole KV pool |
| `--max-num-batched-tokens` | `16384` | chunked prefill granularity for long prompts |
| `--enable-chunked-prefill` / `--enable-prefix-caching` | on | keeps a 250k-token agent prompt responsive and cached across turns |
| `--reasoning-parser` | `qwen3` | splits the reasoning trace out of `content` |
| `--tool-call-parser` | `qwen3_coder` | required for tool calls (the proxy's `web_search`) |
| `--enable-auto-tool-choice` | on | without it vLLM rejects requests that carry `tools` |
| `--language-model-only` | on | skips the vision tower (see Limitations) |
| `--dtype` | `float16` | the checkpoint's dtype; bf16 would double nothing but adds no benefit here |
| `--trust-remote-code` | on | auto-round checkpoint ships custom config |

The `VLLM_*` environment variables in the compose file are the parent project's tuning
for this exact card (Marlin atomic add, FlashInfer sampler, graph estimation, spawn
workers); they are not required for correctness.

### Changing context length

KV is f16 on the 16 full-attention layers only (the 48 GDN layers keep a small recurrent
state), so 64 KiB/token:

| Context | KV (f16) | Weights | Approx. total |
|---------|----------|---------|---------------|
| 32k | ~2.0 GiB | 18.65 GiB | ~22 GiB |
| 128k | ~8.2 GiB | 18.65 GiB | ~29 GiB |
| 253952 (default) | ~15.5 GiB | 18.65 GiB | ~37 GiB |
| 262144 (native max) | ~16 GiB | 18.65 GiB | ~38 GiB |

Lower `--max-model-len` (and drop `VLLM_ALLOW_LONG_MAX_MODEL_LEN` if you go below the
model's 262144) if you need to share the GPU. Going **up** is not possible without a
second card or a smaller/quantized KV cache — and FP8 KV needs CC ≥ 8.9.

## Thinking / reasoning

`--reasoning-parser qwen3` moves the trace out of `content`. The model keeps Qwen3.8's
default reasoning level, so it thinks by default and **reasoning tokens count against
`max_tokens`** — give it 2048–4096 for real questions. Verified on this build (vLLM
0.27.2rc1): the trace arrives in the message's **`reasoning`** field (streaming:
`delta.reasoning`), the answer in `content`, and the split is reported in
`usage.completion_tokens_details.reasoning_tokens`. (Older vLLM releases used
`reasoning_content`; the proxy re-streams either.)

Thinking controls pass through `chat_template_kwargs` (verified):

```bash
# no reasoning at all
curl localhost:18080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"foundation","messages":[{"role":"user","content":"Hi"}],
       "chat_template_kwargs":{"enable_thinking":false}}'

# shorter trace (named level from the Qwen3.8 template)
  ... "chat_template_kwargs":{"reasoning_effort":"low"}
```

With default settings a 300- and even a 1000-token reply came back as **pure reasoning**
(`finish_reason: "length"`, empty `content`) — this checkpoint thinks a lot. Set a high
`max_tokens` or lower the effort.

The proxy does not disable thinking and does not touch the `reasoning` field — it passes
completions through unchanged.

## Tools and `web_search`

`--enable-auto-tool-choice --tool-call-parser qwen3_coder` is what makes the proxy's
injected `web_search` tool usable. Verified: a `get_weather` request came back as
`tool_calls: [{"function":{"name":"get_weather","arguments":"{\"city\": \"Sofia\"}"}}]`
with `finish_reason: "tool_calls"`. Behaviour is the same as the other backends:

- **Non-streaming** through `:11433`: the proxy runs the search loop server-side
  (`MAX_SEARCH_ITER`, default 10) against SearXNG and returns only the final answer. The
  client's own tool calls are returned untouched, unless the model mixes them into a search
  turn (then they are stubbed — inherited edge case).
- **Streaming**: passed through natively; if the model calls `web_search`, the client gets
  the `tool_call` and must execute it.
- `SEARXNG_URL` / `SEARXNG_RESULTS` tune the backend (defaults
  `http://host.docker.internal:8889`, 5). `extra_hosts` is what resolves that hostname.
- When SearXNG returns nothing, this checkpoint keeps retrying with new queries and the
  loop ends with an explicit error instead of an answer — verified during testing (SearXNG
  had all engines rate-limited). The error now reports the counts:
  `Exceeded max search iterations (2 searches, 2 empty) — SearXNG returned no usable results`.
  Lower `MAX_SEARCH_ITER` (env on the proxy) if you prefer to fail fast.

## Pointing a client at it

```
base_url = http://localhost:11433/v1   # or http://localhost:18080/v1 to skip web_search
model    = foundation
api_key  = anything
```

Use the proxy for coding agents so `web_search` works, and set `max_tokens` high enough
for the reasoning trace. Direct vLLM (`:18080`) is also OpenAI-compatible, including
native `json_schema` structured output — the vLLM build does not need the grammar
stripping that llama.cpp did (dropped from this proxy).

## Limitations

- **No vision.** `--language-model-only` drops the visual tower and vLLM logs
  `running in text-only mode`. Remove the flag and lower `--max-model-len` if you need
  images; the processor configs are in the checkpoint.
- **One sequence at a time.** `--max-num-seqs 1`: a second concurrent request queues.
- **No FP8.** A6000 is compute capability 8.6; FP8 weights and FP8 KV need 8.9+.
- **No auth** out of the box (`--api-key` omitted).
- **Nightly image.** `vllm/vllm-openai:nightly` moves; `v0.18.0` (also cached on this host)
  predates `qwen3_5` support. Pin the digest if you need reproducibility.
- **INT4 is below the Q8 GGUF in fidelity.** AWQ keeps `mtp`, `lm_head`, the visual merger
  and some `linear_attn.in_proj_a/b` tensors in fp16, but the bulk of the weights are
  4-bit; the llama.cpp/Ollama stacks with `UD-DW-Q8_K_L` are the near-lossless option.

## Performance

Measured with `./benchmark.sh` on this RTX A6000 with the GPU otherwise idle (only ~0.3 GiB
of desktop usage), default settings, 253952 context:

| Metric | Value |
|--------|-------|
| decode (300 tokens) | **38.5 tok/s** |
| decode (1000 tokens) | 38.3 tok/s |
| TTFT (92-token prompt, warm) | **0.10 s** |
| prefill (9,062-token prompt) | ~1.8k tok/s (5.9 s wall, including 32 decoded tokens) |
| cold start → healthy | **~4 min** (06:52:28 → 06:56:41, kernels compiled + graphs captured) |
| VRAM at steady state | 42.9 GiB: 17.7 GiB weights/non-torch, 2.74 GiB peak activations, 0.04 GiB CUDA graphs, 23.6 GiB KV pool |

Comparison: llama.cpp with the Q8 GGUF measures **9.95 tok/s** on the same card, so vLLM
INT4 is **~3.9×** the decode rate (the parent project hoped for 2–3×). The trade is
fidelity: 4-bit AWQ vs near-lossless Q8.

The KV pool is **379,388 tokens** (65 KiB/token f16) — `--max-model-len 253952` only uses
~67% of it, so the context could be raised to the native 262144; the rest is headroom for
one long prompt, not a second sequence (`--max-num-seqs 1`).

Two implementation details that affect speed on this card (from the vLLM log):

- `Using MarlinLinearKernel for AutoAWQMarlinLinearMethod` — the INT4 path is Marlin.
- `Falling back to the Triton GDN decode path` — the fused CUDA GDN kernel requires a
  BF16 model with K=V=128; this FP16 checkpoint uses the Triton kernel instead.

## Troubleshooting

| Problem | Fix |
|---------|-----|
| `ValueError: No available memory for the cache blocks` / OOM at startup | lower `--max-model-len` (e.g. 131072) or `--gpu-memory-utilization`; free the GPU first (`nvidia-smi`) |
| Container stuck in `health: starting` | it is compiling/capturing graphs — watch `docker compose logs -f vllm`; allow up to 15 min on first boot |
| `model ... not found` / 404 | `./download-model.sh`; confirm `HF_HOME=/models` and the cache path under `/storage-models/vllm/hub` |
| AWQ kernel / quant errors | try `--quantization compressed-tensors`, or add `--enforce-eager` (slower but skips CUDA graphs) |
| Requests queue behind each other | expected with `--max-num-seqs 1`; raise it only if you lower `--max-model-len` |
| Tool calls rejected / not parsed | the parsers are mandatory: `--enable-auto-tool-choice --tool-call-parser qwen3_coder` |
| `reasoning` empty | check `--reasoning-parser qwen3` is present; it is what extracts the trace |
| Empty `content`, `finish_reason: "length"` | thinking ate the budget — raise `max_tokens` or pass `chat_template_kwargs: {"enable_thinking": false}` |
| `Exceeded max search iterations (N searches, M empty)` | SearXNG returned nothing; check its engines, or lower `MAX_SEARCH_ITER` on the proxy |
| Proxy returns 502 | `docker compose logs vllm` — usually a failed load; also confirm `BACKEND_URL=http://vllm:8000` |
| `web_search` → `Search failed` | SearXNG on `host.docker.internal:8889` is down (external to this stack) |
| Startup warning about multimodal limits | harmless: it is the text-only mode you asked for |

## Reset

```bash
docker compose down                      # keeps the HF cache and the image
# nuclear: delete the checkpoint cache (19 GB) and re-download with ./download-model.sh
sudo rm -rf /storage-models/vllm/hub/models--shawnw3i--Huihui-Qwen3.8-27B-abliterated-AWQ-MTP
```