Skip to content

Serve an OpenAI-compatible API

turboquant-serve is mlx_lm.server with a loader that understands TurboQuant models (quantization.mode = "turboquant" in config.json). Other models pass through unchanged, so it is a drop-in replacement for mlx_lm.server.

turboquant-serve --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64 --port 8080

It listens on 127.0.0.1:8080. Add --host 0.0.0.0 to accept other machines; there is no authentication or rate limiting, so only do that on a network you trust.

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 1024
  }'

The model field is the string you passed to --model, or default_model, mlx-lm's name for it. A request that names a different model makes the server load that one instead. Reasoning models return their thinking in message.reasoning and the answer in message.content; give them max_tokens >= 2048 so both fit.

For vision models, use turboquant-serve-vlm.

Options

All mlx_lm.server options work as usual: --host, --port, --temp, --top-p, --top-k, --max-tokens, --chat-template-args, --prefill-step-size, --prompt-concurrency, --prompt-cache-size, --prompt-cache-bytes and so on. The exception is KV quantization: TurboQuant's --kv-bits and --kv-group-size replace mlx-lm's, so --quantized-kv-start has no effect. TurboQuant adds the options below. turboquant-serve --help lists both sets (up to 0.29.0 it showed only mlx-lm's).

Option Default What it does
--kv-k-bits N --kv-v-bits M off Compress each request's KV cache. K8/V3 is the recommended mix. See KV cache compression.
--kv-bits N off The same width for K and V (legacy).
--kv-min-tokens N 0 Keep the first N cached tokens in FP16 (attention sinks). 128 is a good value.
--kv-group-size G 64 Rotation group size for KV quantization.
--kv-fused off Fused decode+attend kernel. Experimental; needs --kv-min-tokens 0.
--disk-cache [DIR] off Save prompt-cache checkpoints to disk, so a later request or a restart resumes instead of re-prefilling.
--disk-cache-budget-gb 10 Disk space for checkpoints (oldest evicted first).
--disk-cache-min-tokens 512 Don't checkpoint shorter prompts.
--disk-cache-save-every 1024 Tokens between checkpoints, including mid-prefill ones.
--disk-cache-no-prefill-checkpoints off Only checkpoint at the end of a request.
--prompt-cache-max-gb auto A byte cap on the in-memory prompt cache. auto caps it only when memory is tight.
--metal-cache-limit-gb auto Caps MLX's buffer-reuse cache after the model loads. auto acts only when memory is tight.
--rep-penalty, --rep-ctx from the model; 256 A default repetition penalty for every request.
--tool-syntax-greedy off Inside a <tool_call> block, pick the most likely token, so low-bit builds don't garble tool-call JSON. Tags via --tool-syntax-tags "<open>,</close>".
--prefill-keepalive off Sends real SSE chunks during a long prefill, so a proxy (for example claude-code-router) doesn't time a client out. --prefill-keepalive-interval (default 10 s).
--cache-budget-gb N\|auto off Serve a MoE bigger than RAM by streaming experts from disk. See below.
--max-active-experts K 4 Streaming only: read at most K experts per token. Use 0 for full routing in quality-sensitive or agent work.
--use-page-cache / --no-page-cache auto Streaming only: read through the OS page cache, or bypass it.
--wire-memory off Streaming only: wire weights and expert cache against memory pressure.
--no-hotlist — Streaming only: ignore a hot_experts.json in the model folder.
--ngram-offload off Qwen3.8-Flash-Next only: serve its n-gram table from disk-backed memory.
--prefill-stats [--prefill-stats-file F] off Diagnostic: log how much of each prompt is reused or re-prefilled.

KV compression serves one request at a time

TurboQuant KV caches can't be merged across requests, so any --kv-* option makes the server process requests one after another. That's right for one user; concurrent clients queue instead of batching.

Keep the cache across restarts: --disk-cache

With --disk-cache, KV-cache checkpoints are written per conversation by a background writer, and any later request, including the first after a restart or crash, resumes from the longest saved prefix. Measured: the first turn of a 16.3K-token conversation after a restart dropped from 21.1 s to 3.0 s on a 64 GB Mac (6.9×), with identical output. On a 16 GB mini, a 21K-token prefill that crashed resumed from token 18,432 (a 249 MB checkpoint, loaded in 2.3 s).

Hybrid models with a recurrent state (Qwen3.6, Qwen3.8, Nemotron) can't trim a checkpoint back to an earlier point, so they rely on mid-prefill checkpoints taken every --disk-cache-save-every tokens. Those are taken on the one-request-at-a-time path, so for hybrid models combine --disk-cache with a --kv-* option. On a 16 GB mini this let turn 2 of a 21K-token conversation reuse 20,480 of 21,250 tokens.

Near the memory limit

A server that fills most of memory can fail after a few turns with:

[METAL] Command buffer execution failed: Insufficient Memory
(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

The in-memory prompt cache grows with every conversation. In order of impact:

  1. Raise the GPU memory limit, once per boot: sudo sysctl -w iogpu.wired_limit_mb=57344 on a 64 GB Mac, 43008 on 48 GB. See Memory and macOS settings.
  2. Use a smaller prefill step on tight machines: --prefill-step-size 256 (or 128).
  3. Cap the prompt cache: --prompt-cache-max-gb 2, or --prompt-cache-size 1 to keep only one conversation. The auto guards above already do this when headroom is small. mlx-lm's own --prompt-cache-bytes is applied only when the server batches requests. With any --kv-* option or a --draft-model, or for a request that sets a seed or includes an image, requests run one at a time and only --prompt-cache-max-gb caps the cache.
  4. Close other GPU users (browsers, Electron apps, video editors).

Why the auto guards exist, measured on a 16 GB mini serving the 12.6 GB Qwen3.6-35B build: during a long prefill, MLX's buffer cache grew about 80 MB per 1K prompt tokens and crashed a 21K-token prefill at 14K while active memory stayed flat (--metal-cache-limit-gb caps it); and an agent conversation kept 8 cached states (1.14 GB) until the system paged and the GPU watchdog killed the server (--prompt-cache-max-gb caps it). With --disk-cache on, an evicted state is restored from disk, so a small cap costs little.

A model bigger than RAM

--cache-budget-gb loads the model through the expert-streaming path, so only the experts each token routes to are read from disk. This puts a ~50 GB 122B behind an OpenAI endpoint on a 16 GB Mac mini:

turboquant-serve \
    --model manjunathshiva/qwen3.5-122b-tq3 \
    --cache-budget-gb 4 \
    --kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128 \
    --prompt-concurrency 1 --port 8080

--cache-budget-gb auto sizes the expert cache from the machine. See Models bigger than RAM for the speed you can expect and the tuning options.

Recipes

  • A coding agent on a 16 GB Mac mini: see Local coding agents.

  • Nemotron-3-Super 3-bit on a 64 GB Mac:

    sudo sysctl -w iogpu.wired_limit_mb=57344
turboquant-serve --model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
    --port 8080 --prompt-cache-max-gb 2