Contents
Artificial Intelligence
On-premise AI architectures, local LLMs, RAG, autonomous agents. Intelligence you cannot afford to see switched off must be owned.
Explore →
Linux Services & Systems
Design, deployment and management of on-premise Linux infrastructure: from servers to AI hardware, with data that stays yours.
Explore →
Admina Enterprise
Open Source AI governance: audit trail, PII redaction, bidirectional ALLOW/BLOCK/REDACT policies on any model, local or remote.
Explore Admina →
Research & Development
Applied research and prototyping, from AI to infrastructure, in collaboration with universities and research centres.
Explore →Running a large language model on your own hardware, without sending it to the cloud, is almost always a memory problem: the model weights and the attention-state cache (the KV cache) have to fit in the RAM you have. turboquant-mlx is an open-source project (MIT licence, per its README) that tackles both fronts on Apple Silicon, porting the TurboQuant quantization method to MLX (Apple’s machine-learning framework) and extending it.
It is a young project by a single independent developer (manjunathshiva, a few dozen GitHub stars at the time of writing). It is worth carefully separating what comes from the original method and what the repo adds, and reading the numbers for what they are: the author’s own measurements, on specific hardware, not yet independently verified.
The problem: memory
Local inference is memory-bound. On Apple Silicon the RAM is unified, shared between CPU and GPU, which helps, but a 120-billion-parameter model in BF16 still takes about 240 GB in weights alone. That is what quantization is for: storing weights (and states) with fewer bits per parameter. OpenAI’s GPT-OSS models, for instance, are trained natively in MXFP4, a format at roughly 4.25 bits per parameter, which is what lets the 20B model fit in about 16 GB. That is the baseline TurboQuant-MLX measures itself against.
There is a second, often underestimated cost: the KV cache grows with context length and, at long contexts, can weigh as much as or more than the weights. Compressing the cache, therefore, matters as much as compressing the model.
The TurboQuant method, briefly
TurboQuant is described in “Online Vector Quantization with Near-optimal Distortion Rate”, an arXiv preprint (April 2025) by Amir Zandieh, Majid Daliri, Majid Hadian and Vahab Mirrokni, researchers associated with Google Research (one co-author is at NYU). It is a preprint, not yet a peer-reviewed publication; the repo calls it “Google’s TurboQuant”, but no affiliations appear on the arXiv abstract, so it is more accurate to attribute it to the authors.
The idea is elegant. To quantize a vector well, you first rotate it at random: the rotation makes the coordinates nearly independent of one another and concentrates them into a known distribution. From there you can quantize each coordinate on its own, with an optimal scalar quantizer, and still land close to the theoretical limit. The authors prove that the distortion then stays within a small constant factor (about 2.7 times) of the information-theoretic minimum.
Two properties matter. The method is data-oblivious (no calibration pass over data is needed) and online (it works on vectors arriving in a stream). That is why its natural application is the KV cache, which is generated token by token, rather than static weights. There is also an important detail for attention, which depends on inner products: MSE-optimal quantizers introduce a bias, so the paper adds a second stage, a 1-bit sketch on the residual (a quantized Johnson-Lindenstrauss transform) that makes the inner-product estimate unbiased.
The two applications shown in the paper are KV-cache compression for long contexts and similarity search (as a replacement for product quantization). On the KV cache the authors report quality neutrality at 3.5 bits per channel and marginal degradation at 2.5 bits per channel (a uniform, per-channel measure, not split into keys and values). One point to hold firm: weight quantization is not covered by the paper.
What turboquant-mlx adds
The repo does two things beyond the MLX port. First, it implements the paper’s recipe as a Hadamard rotation plus Lloyd-Max codebooks: the Hadamard transform is a fast, cheap way to realize the “random rotation”, and Lloyd-Max is the classic optimal scalar quantizer for the per-coordinate step. Second, and this is the author’s own contribution, it extends the idea to model weights, not just the KV cache. That extension belongs to the project, not the paper.
In detail, as the repo describes it:
- Weights: quantized at conversion time (from Hugging Face models) to 2, 3 or 4 bits. For Mixture-of-Experts (MoE) models the repo adds ternary 1.58-bit experts on a three-value codebook, and asymmetric schemes (lower-precision experts, more sensitive projections at higher bits).
- KV cache: quantized at runtime and dequantized back to float16 only when attention needs it, routed through MLX’s standard scaled dot-product attention. The recommended default is K8/V3 (8-bit keys, 3-bit values): a repo choice, distinct from the paper’s “bits per channel”.
- Expert streaming: for MoE models, keeping only the recently used experts in memory and reading the others from disk on demand, so a model that would not fit in RAM can still run, at a cost in speed.
Around this sits a command-line toolchain: turboquant-plan to estimate memory before even downloading, turboquant-convert to convert models, turboquant-generate for generation with compression flags, and turboquant-serve for an OpenAI-compatible API server. It installs with pip install turboquant-mlx-full and requires macOS on Apple Silicon (M1 onward), Python 3.10+, MLX and mlx-lm; for models of 20B and up the author recommends 64 GB of unified memory. The repo lists support for dense architectures (LLaMA, Qwen, Mistral) and MoE ones (GPT-OSS, DeepSeek-V2/V3, Qwen-MoE, Nemotron-3), plus hybrid attention/Mamba variants: that each one actually works is, again, the author’s claim. Note that the README describes the “rotation plus codebook” part but not the 1-bit residual sketch stage, so how much of that second half of the method the project covers is not spelled out.
The reported numbers
All the numbers below are the author’s own measurements, on specific hardware and without independent verification. Read them as orders of magnitude, not settled benchmarks, and do not add up rows from different machines.
- Perplexity (lower is better): on Qwen2.5-7B the author reports 8.92 with TurboQuant at 3-bit versus 13.37 with affine 3-bit quantization; on GPT-OSS-20B, 72.63 with TurboQuant at 4-bit versus 83.04 with MXFP4. These are single comparisons, one model each, in the author’s own benchmark.
- Memory and speed (on M4 Max): GPT-OSS-20B at 3-bit in 9.3 GB at 73 tok/s; GPT-OSS-120B at 3-bit in 48 GB at 44 tok/s, which the author says fits with its full 131K-token context in about 50 GB on a 64 GB MacBook; Qwen3.5-122B goes from about 240 GB in BF16 to about 50 GB at 3-bit, running at 26.5 tok/s.
- KV cache: on GPT-OSS-120B with 3-bit weights, the author measures the 3-bit cache as faster than the FP16 one (8.7 versus 6.4 tok/s) with about 3.8x cache-memory savings. On quality they report a cosine similarity of 0.983 at 3-bit and 0.995 at 4-bit against FP16, and on Qwen3.5-122B an output they describe, in their test, as indistinguishable from FP16.
- Expert streaming (16 GB Mac mini): on Qwen3.6-35B-A3B, with an 8 GB cache budget, the author reports 91% of experts already in memory and about 4.5 tok/s, running in under 4 GB of RAM thanks to streaming.
They convey what is possible on a Mac today, but they remain a single source: the quality of an aggressively quantized model has to be measured on your own use case.
Similar solutions
turboquant-mlx is not the only way to run large models locally. On Apple Silicon, mlx-lm already has its own quantization; outside the Apple world there are llama.cpp with the GGUF formats, and weight-quantization methods such as AWQ and GPTQ. For MoE models that do not fit in a GPU, offloading experts to RAM is the route taken by KTransformers. The TurboQuant method itself also has implementations other than this MLX one. What is distinctive about turboquant-mlx is bringing together, in a package built for Apple Silicon, very aggressive weight quantization, ternary experts and KV-cache compression.
Our take
The thread we care about is the usual one: when inference touches data or processes you cannot afford to send out, it pays to be able to run it in house, on your own hardware, with weights you can host. Quantization is the lever that makes this route practical, because it decides whether an open-weight model fits or not in the memory you have. A project like turboquant-mlx shows two useful things: that the local-inference stack on Apple Silicon (MLX, mlx-lm and the tools around them) is maturing fast, and that quantization research, KV cache included, reaches practical hands within weeks.
Caution is warranted, though. The numbers come from a single source, on specific hardware, and “indistinguishable from FP16” in one test is not “indistinguishable” for your workload: quality at 2 or 3 bits, with 1.58-bit experts, has to be verified on the real use case, not taken on trust. And it helps to keep separate two things the repo combines: the TurboQuant method, published as a preprint and designed for KV cache and vector search, and its extension to weights, which is the project’s contribution and not the paper’s. For anyone evaluating on-premise inference it is an interesting piece to try and measure, not a promise to put into production blindly.
References
- TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874)
- TurboQuant on Hugging Face Papers
- turboquant-mlx (manjunathshiva) on GitHub
- MLX, Apple’s framework (open source)
- OpenAI: introducing GPT-OSS (open-weight models)
- Hugging Face: MXFP4 quantization