Artificial Intelligence
On-premise AI architectures, local LLMs, RAG, autonomous agents. Intelligence you cannot afford to see switched off must be owned.
Explore →
Linux Services & Systems
Design, deployment and management of on-premise Linux infrastructure: from servers to AI hardware, with data that stays yours.
Explore →
Admina Enterprise
Open Source AI governance: audit trail, PII redaction, bidirectional ALLOW/BLOCK/REDACT policies on any model, local or remote.
Explore Admina →
Research & Development
Applied research and prototyping, from AI to infrastructure, in collaboration with universities and research centres.
Explore →What KTransformers is
KTransformers is a hybrid CPU/GPU inference engine released under the Apache-2.0 license by Tsinghua University’s MADSys Lab, Approaching.AI and 9#AISoft. It does something that sounds impossible: it runs enormous Mixture-of-Experts models, like DeepSeek-R1/V3 at 671 billion parameters, on a single 24 GB GPU (an RTX 4090) paired with plenty of system RAM, on the order of 512 GB. The work was published at SOSP 2025, the top systems conference.
Note. The performance figures here come from the project’s documentation and community benchmarks, not from independent testing by noze. They depend heavily on hardware, quantization and configuration, and refer to the versions indicated. We report them as indicative.
The idea: experts live in RAM, the hot path on the GPU
The trick comes from a property of MoE models: they are sparse. A model like DeepSeek-R1 has 671 billion total parameters, but for each token it activates only a small fraction, because the router picks a few experts out of the hundreds available. The vast majority of those parameters, for a given token, sit idle.
KTransformers exploits this asymmetry by distributing the model between GPU and CPU according to what is actually needed, and when. The hot-path components, the attention (MLA in DeepSeek’s case) and the KV cache, stay on the GPU, where memory bandwidth matters most. The experts of the MoE, which hold the bulk of the parameters but are touched rarely, live in system RAM, far more capacious and far cheaper than VRAM. It is the same shift in perspective we described for colibri and DwarfStar 4: the amount of VRAM stops being a wall and becomes one variable in a multi-tier system, where RAM and disk enter the hierarchy.
How it manages it: the CPU kernels matter
Moving experts into RAM has a cost: something has to compute them, and that something is the CPU. If the CPU-side compute is slow, the whole advantage disappears. This is where KTransformers’ technical contribution lies: very fast CPU kernels, written to exploit Intel AMX and AVX-512 instructions on INT4/INT8 quantizations. In the project’s benchmarks, the AMX-based MoE kernel reaches up to about 27.8× the speed of the llama.cpp implementation on a 2×32-core test (where llama.cpp stopped at around 10.31 tokens/s).
Around this sit the other techniques that make it practical:
- NUMA-aware memory management, for multi-socket CPU systems where RAM locality changes performance.
- Quantized GPU kernels (Marlin) and hybrid FP8 weights, with FP8 kernels extending context up to about 139K tokens on 24 GB of VRAM.
- A three-tier prefix cache (GPU, CPU, disk) to reuse work already done across the memory hierarchy.
The performance, taken as indicative, gives a sense of the jump: on the 671B, prefill reaches up to about 286 tokens/s in optimized configurations; generation is slower and depends heavily on hardware. In a server configuration with 8 L20 GPUs and a Xeon, the project reports about 227 tokens/s aggregate and 87 tokens/s output with 8 concurrent requests. It is not the speed of a dedicated cluster, but it is the difference between “this model does not run” and “this model runs”.
What runs, and on what hardware
KTransformers has moved fast behind the open-weight frontier. Version v0.6.3 (June 2026) claims support, often day-0, for a list that reads like a who’s who of open MoE: DeepSeek-V3/R1, Kimi K2/K2.5, GLM-5 and GLM-5.2, Qwen3-MoE and Qwen3-Next, MiniMax. On hardware it goes beyond NVIDIA, with support for AMD ROCm, Intel Arc and Ascend NPUs.
It is not only inference: the project has also added fine-tuning on modest hardware, integrating with LLaMA-Factory. An SFT run of DeepSeek-V3 goes, per the reported figures, at about 3.7 iterations per second on 4×RTX 4090 with roughly 80 GB of total memory, a result described as 6-12× faster than comparable alternatives. On integration, KTransformers hooks into SGLang for serving, works with the HuggingFace Transformers ecosystem, exposes OpenAI-compatible APIs and reads GGUF quantized formats.
Similar solutions
KTransformers is the most visible tip of a broader movement, and it is worth framing next to the other approaches, because the trade-offs differ.
- llama.cpp made selective offload mainstream: with the
--override-tensoroption you choose which tensors (typically the MoE experts) to keep on CPU and which on GPU. It is portable and universal; KTransformers instead bets on specialized CPU kernels (AMX) and beats llama.cpp precisely on CPU-side compute. - ik_llama.cpp, a fork of llama.cpp, pushes on quantizations and CPU kernels tuned for MoE, in a niche similar to KTransformers but in the llama.cpp tradition.
- PowerInfer and Fiddler attack the same problem from different angles: PowerInfer splits “hot” and “cold” neurons between GPU and CPU, Fiddler optimizes the expert compute path on CPU.
- And then there are the engines we have already written about: colibri, which streams the experts from NVMe disk rather than keeping them in RAM, and DwarfStar 4, which does the same for the KV cache.
The line that unites them is one: frontier MoE models are large but sparse, and sparsity is a lever. Some exploit it with RAM (KTransformers), some with disk (colibri), some by splitting hot and cold (PowerInfer). The practical result is the same: the frontier comes down from the data center to the server under the desk.
What we think
KTransformers is an important piece of a story we follow bit by bit. Every time a bigger open-weight model ships, from GLM 5.2 to Inkling to the 2.8-trillion-parameter Kimi K3 (with weights on the way), the practical question is always the same: does it actually run, and on what? Engines like KTransformers answer that yes, it runs, and on hardware a company can afford: a workstation with one GPU and plenty of RAM, not a cluster.
It is exactly the direction we work in. A frontier model running on-premise, where the data does not leave your perimeter, is no longer a theoretical exercise: it is a sovereignty choice made possible because the systems make it affordable. Then comes the other half of the job, governing what runs, with an audit trail and policies over any model, as Admina does. That is the sense of Open Intelligence, Secure Governance: own the operational floor, on Linux infrastructure that stays yours, and keep control of it. A 671-billion-parameter MoE on a single 4090, a year ago, sounded like a joke. Today it is a matter of kernels, RAM and how you distribute the work.