Benchmarks¶

Measured results, mostly on a 64 GB M4 Max, plus a 16 GB M4 Mac mini and a 48 GB M5 Pro where noted. For a download list by Mac memory, see Choose a model; each model's Hugging Face card has its full measurements.

Weight compression¶

* Estimated for 0.30.0, not measured. 0.30.0 makes ternary experts decode faster (the kernels no longer do an integer division per weight). These figures come from kernel timings at each model's expert shapes, calibrated against the model's measured speeds; the same estimate predicted 20.4 tok/s for Nemotron 3.5 VL, which measured 21.0. The 0.29.0 figures in parentheses were measured.

KV cache compression¶

KV cache compression projects to ~7 GB RAM saved at 131K context on GPT-OSS-120B and ~5 GB at 262K on Qwen3.5-122B. Roundtrip cosine similarity vs FP16: 0.983 at 3-bit, 0.995 at 4-bit.

On the Qwen3.5-122B KV rows: these were measured with symmetric 3-bit KV (demo_kv.py --tq-bits 3, short prompt), and the rates sit well below the model's fully-resident decode. With the Metal wired cap raised so the ~50 GB model stays fully resident, decode with the recommended mixed K8/V3 cache is ~24–25 t/s — and there fp16 edges out compression at short-to-moderate context, the gap widening as context grows. On this model KV compression is a memory win, not a speed win; the genuine decode speed-up is GPT-OSS-120B-specific. See the resident long-context sweep in #19.

Apple M5 Pro (48 GB)¶

First Metal4 / MTLGPUFamilyApple10 data point, contributed by @sbayer2 (#14, #16) — a new 48 GB tier between the 16 GB Mac mini and the 64 GB M4 Max. Reproduce with benchmarks/bench_m5_pro.py.

KV cache sweep on the 35B (resident, 3-run averages):

KV cache against context on the 35B (256 generated tokens; 128 at 63K):

FP16 decodes 1.14× faster at 65 tokens, 1.35× at 2.5K, 2.75× at 14.5K and 6.6× at 63K, so on this model KV compression saves memory, not time. Of the models measured, only GPT-OSS-120B decodes faster with a compressed cache (KV cache compression, 8.7 vs 6.4 tok/s), and not because of its size: Qwen3.5-122B, run resident on a 64 GB M4 Max, decodes faster with FP16 at every context (1.07× at 256 tokens, 1.20× at 4096; #19). The difference is GPT-OSS-120B's KV geometry.

122B expert streaming — parallel prefetch (--cache-budget-gb 30, 256 tokens):

A 1.23× decode speedup from --prefetch-workers 8, landing between the Mac mini (1.3×) and the M4 Max (1.67×). The M5 Pro MacBook Pro SSD is the limiter — parallel prefetch helps but doesn't saturate the way the M4 Max's higher-bandwidth SSD does.

122B expert streaming — cache-budget sweep (--prefetch-workers 8, 256 tokens):

The hit-rate curve flattens hard past 30 GB (only +0.7% for +8 GB), and throughput actually dips at 38 GB as peak Metal (42.2 GB) crowds the wired cap. 30 GB is the sweet spot on the 48 GB tier — 90%+ hit rate with ~14 GB of headroom for OS stability; pushing to 38 GB gives negligible gain while peaking uncomfortably close to the wired limit.

KV compression gives a consistent ~1.6× prompt-processing speedup for a ~12% decode cost (long-context decode behavior is in The speed flip). Expert-streaming hit-rate scales with the cache budget — 44.6% at 4 GB (16 GB mini) → 89.9% at 30 GB (48 GB), a ~7× throughput jump that fills the gap between the 16 GB and 64 GB tiers.

Stability near the memory ceiling: long-context prompt prefill close to the wired cap can starve the kernel watchdog (a watchdogd / AppleARMWatchdogTimer panic). A rapidly-growing KV cache makes Metal commit pages continuously (AGXG17XFamilyResidencySet _commitAddedAllocations), so the binding limit is allocation rate, not peak — a static 41 GB resident model (Nemotron) is stable, while a growing ~30 GB KV during a 63K-token prefill can panic. Practical limits on the 48 GB tier (#14): contexts up to ~14.5K are safe on the 35B with either KV config; 63K is feasible with K8/V3 (28.7 GB peak) but not fp16 (30.8 GB → panic). Keep headroom and close other apps for long-context runs near the cap.