Qwen3.8-27B (dense 27.8B hybrid Gated-DeltaNet VLM)¶

Qwen3.8-27B reports model_type: qwen3_5 despite the name — a dense 27.8B multimodal model whose 64 decoder layers are 48 Gated-DeltaNet (linear attention) + 16 full attention, one full layer in every four. Hidden 5120, intermediate 17408, vocabulary 248,320, plus a 27-block vision tower. mlx-lm 0.31.3 and mlx-vlm 0.6.3 already carry the architecture, so nothing needed porting; conversion takes about 18 seconds.

Two builds are published, and they target different machines.

¹ the agentic run was done on the 8-bit-extras tq3; the published tq3-mini differs only in the extras tier, at an identical 8.0052-vs-8.0551 perplexity.

Both fix a planted off-by-one end to end: given the task's pytest command, each build runs it, reads the file, makes the minimal edit, re-runs to green and explains with file:line. Vision works on both — shapes, colours, positions and grounded bounding boxes.

Take tq4 if it fits; take tq3-mini if it does not. At 4-bit, tq4 is faster as well as better than the 8-bit-extras tq3 for 2.3 GiB more — the same result the speed flip predicts, since the codebook path only earns its keep at 2-bit and 3-bit's 10-indices-per-uint32 packing costs more to unpack than 4-bit's clean 8.

The exception is the 16 GB tier, where nothing else reaches. tq3-mini-g64 is verified on a real 16 GB M4 Mac mini — not projected down from a bigger machine — at iogpu.wired_limit_mb=14336 and --prefill-step-size 256:

* The M4 Max sweep realized 246 / 844 / 2,040 / 5,043 prompt tokens — a constant 26 more than the mini's, worth 1.62 MiB of KV at 64 KiB/token. Read the comparison as close, not as an exact A/B.

Vision passes there too — OCR correct at 350×100 (12.44 GiB) and 700×200 (12.68 GiB). Peaks reproduced identically across two independent sweeps, and they run 0.09–0.27 GiB below the comparable prompts on the M4 Max — 55–170× more than that 26-token gap can account for — so the larger machine is the pessimistic estimate. Reproduce with benchmarks/bench_mini_qwen38.py; the full record is in bench_mini_qwen38_results.json.

Neither 4-bit build reaches that tier: affine 4-bit peaks at 16.16 GiB, more than a 16 GB machine has in total. Decode there is 3.7 tok/s — fine for document Q&A, summarization and vision calls, too slow for interactive chat.

The hybrid stack makes KV very cheap — only 16 of 64 layers cache anything, so 64 KiB/token, or 2.0 GiB at 32K context.

pip install "turboquant-mlx-full[vlm]"

python -m turboquant_mlx.convert_vlm \
  --hf-path Qwen/Qwen3.8-27B \
  --mlx-path ./Qwen3.8-27B-tq4-g64 \
  --bits 4 --group-size 64 --quantize-extras --extras-bits 8

python -m turboquant_mlx.generate_vlm \
  --model ./Qwen3.8-27B-tq4-g64 --prompt "..." --max-tokens 256

Measured on mlx-lm 0.31.3 with both mlx-vlm 0.6.3 and 0.6.13 — identical peaks and identical vision results on each, so the [vlm] floor of 0.6.12 reproduces these numbers.

--prefill-step-size is the memory knob for long prompts, and it is a large one. mlx-vlm only chunks prefill above the step size, so a 1342-token prompt runs unchunked at the 2048 default:

−4.5 GB for 2.1× slower prefill. 256 also keeps every TurboQuant layer on the fused Metal kernel, which materializes nothing. This is what makes the 24 GB target work.

lm_head stays off the polar path¶

With a 248,320-token vocabulary lm_head is 248320×5120 = 1.271B parameters, the largest matrix in the model. PolarQuantizedLinear has a fused Metal kernel only up to _QMM_MAX_TOKENS (256) tokens; past that it dequantizes the weight through full-size intermediates. Measured on exactly that shape at 3-bit/g64:

The same shape as an 8-bit affine layer costs +1.895 GiB at 2048 tokens — that is the logits array itself, with no weight-dequant term at all. So qwen3_5 joins muse_glimmer in VLM_SKIP_PATTERNS, and lm_head is left to the --quantize-extras affine path. End to end on a 1342-token prompt:

Polar buys 0.63 GB on disk and costs 10.5 GB at runtime — and that 33 GB peak is the difference between fitting a 36 GB Mac and not.

This is easy to miss because mlx-vlm's chunked prefill discards the chunk forward's return value, so MLX never evaluates the matmul at all. But chunking only engages above prefill_step_size (default 2048); a prompt of 257–2048 tokens takes the single-shot branch in generate/ar.py, which slices logits[:, -1, :] and therefore does evaluate lm_head across the whole sequence. That is an ordinary chat turn, not an edge case.

Against MLX's own affine 4-bit¶

mlx-community/Qwen3.8-27B-4bit is the same architecture at the same 4-bit / group-size-64 setting, with no MTP weights on either side — so this is a like-for-like comparison, run through the same harnesses and the same server.

The two TurboQuant columns differ only in what the non-polar remainder (embeddings, lm_head, vision tower) is stored at — --extras-bits 8 versus 4. They are separate builds with separate numbers, so they get separate columns.

Only the extras 8 build is published, and the vision/agentic runs were done on it; the extras 4 column is the size-matched control.

Take the affine build if you want 4-bit Qwen3.8-27B. It is 2.6× faster to decode at a lower peak, and the perplexity difference is 0.3% — real and stable in ordering across both corpus sizes, but far too small to trade that speed for.

The one place TurboQuant is structurally ahead is storage. At identical bit width and group size, on the same 17408×5120 projection:

The Hadamard rotation symmetrizes each group, so there is no zero-point to store — 5.5% leaner for free. That margin is a rounding error at 4-bit and the reason the codebook only earns its keep at 2-bit and below (see the speed flip).

Two cautions on reading the table. The vision split is not a capability gap: affine's single miss is one string (VOLTAGE 47 → VOLTAGE), and a five-string follow-up probe put it at 4/5, so treat 3/4-vs-4/4 as a tie. And the agentic wall-clock is not a speed metric — affine decodes 2.6× faster yet finished slower, because it spent more tool turns.

Reproduce both halves:

# 16K-token column (--chunks 32) and 32K-token column (--chunks 64);
# both records are checked in under benchmarks/
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
    ./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
    mlx-community/Qwen3.8-27B-4bit --chunks 32
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
    ./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
    mlx-community/Qwen3.8-27B-4bit --chunks 64
python -m turboquant_mlx.benchmarks.eval_vlm_vision \
    ./Qwen3.8-27B-tq4-g64 mlx-community/Qwen3.8-27B-4bit

Will it fit?¶

turboquant-plan --model <path> at 16K context, per machine:

Speed is the caveat. At ~11 tok/s decode and ~110–160 tok/s prefill an agentic turn takes about six minutes, against 33 s for Laguna-XS.2 at 3-bit. It passes, but this is a model to serve for quality rather than latency.