Kimi K3: Staying the Course on Linear Attention

Since Kimi K3 was released, most of the attention has gone to its 2.8 trillion parameters and the debate over whether it distilled from Fable.

Setting that aside, what stands out to me—having followed Kimi since K2—is that the team is still committed to its own path at the core architecture level. It was willing to make a bet, keep investing in it, and eventually scale that bet to a 2.8T-parameter model.

Attention sits at the heart of every LLM. One of K3's most important architectural changes is its large-scale use of Kimi Delta Attention, or KDA, an attention mechanism developed by Kimi.

When Kimi introduced KDA last year, it was still a bet on a technical direction. With K3, that bet is now running at 2.8T parameters.

What linear attention is trying to solve

KDA is the latest step in the evolution of linear attention.

The problem linear attention is trying to solve becomes especially painful with long context and long-running agent tasks. In full attention, the KV cache consumes more memory and bandwidth as the context grows. Longer context therefore means slower and more expensive inference.

Full attention keeps the keys and values for every previous token, then looks back through them when needed. Linear attention instead compresses incoming information into a fixed-size state. No matter how long the context becomes, that state does not keep growing.

Put simply, for most attention layers, the cost of generating the next token no longer increases with context length.

The trade-off is obvious: a fixed-size state cannot remember everything. It is a lossy compression of the context. The problem shifts from “how do I retrieve the right information?” to “what should I remember, and what should I forget?”

Teaching a model how to forget

In one sense, the evolution of linear attention over the past few years has been an attempt to answer a single question: how should a model forget?

KDA is Kimi's current answer. A simplified way to look at the progression is:

1. Basic linear attention keeps adding information to the state but cannot remove anything. Eventually, the state becomes noisy.
2. DeltaNet can use new information to correct an old association under the same key.
3. Gated DeltaNet adds forgetting, but everything within one attention head still decays at the same rate.
4. KDA makes forgetting more granular, down to individual feature dimensions, so different kinds of information can have different lifetimes. As an intuition, short-lived information such as syntax or references can fade faster, while longer-lived information such as entities or definitions can remain.

So the progression looks roughly like this:

remember → overwrite → forget → know that different things should be forgotten at different rates

It almost feels as if we are trying to crack human memory by improving attention.

Why KDA no longer needs RoPE

Another part I find fascinating is that KDA layers do not need RoPE positional embeddings.

With full attention, content similarity alone cannot distinguish the relative positions of tokens already in the context. Positional embeddings are added to tell the model what came first, what came later, and how far apart tokens are.

KDA's state, however, is updated sequentially. Information written earlier has gone through more decay by the time the model reaches the current token. This process of sequential updates and accumulated decay already carries implicit information about order and distance. The decay is also content-dependent and can differ across dimensions.

In other words, KDA internalizes position into the model's memory and forgetting mechanism.

K3 does not take this idea to an ideological extreme. Its actual architecture mixes three KDA layers with one MLA layer, keeping a small amount of full attention for precise retrieval instead of removing it entirely for the sake of technical purity.

Looking back, KDA's journey from a paper to a mechanism deployed at 2.8T parameters may matter more than the final number itself.

It is a concrete expression of the “taste” Kimi keeps talking about: choosing what to believe in, having the courage to make a bet, and staying with it step by step until you have carved out a path of your own in unexplored territory.