Kimi KDA, DeepSeek MLA and V4 CSA/HCA: Long-Context Cache and GPU Memory
[!NOTE] Executive Summary (TL;DR) Novel Attention Breakdown : Thorough analysis of Kimi K3's hybrid 69-layer KDA (Kimi Dynamic Attention) + 24-layer Gated MLA design, dispelling the myth of universal constant-state caching. Memory Scaling Profiles : Blending recurrent linear state updates with gate
Posted on Oct 21 • Originally published at blog.llmgo.top
MLA compresses each historical token's representation. KDA replaces a full historical cache in a recurrent layer with fixed-size state. DeepSeek-V4's CSA / HCA also compress the sequence dimension. These mechanisms need separate capacity, compute, and quality analysis.
KV_bytes = 2 × batch × sequence_length × layers × kv_heads × head_dim × bytes_per_element Enter fullscreen mode Exit fullscreen mode For batch=1, a KV hidden dimension of 2048, 80 layers, and FP16, each token needs 655,360 bytes: about 2.62GB at 4,000 tokens and 655.36GB at one million. These are decimal GB; divide bytes by 2^30 for GiB. The example is not a universal estimate for every large model.
MLA caches a low-dimensional latent representation and separate positional keys. Matrix absorption can avoid fully reconstructing historical multi-head K/V at every decode step. See MLA matrix absorption.
Cache still grows linearly with context. Its ratio depends on dimensions, layers, and precision. MLA is a trained architecture, not a lossless runtime switch for an arbitrary GQA checkpoint.
KDA includes gating, decay, and delta updates. S_t = S_(t-1) + ΔS_t is only an illustration. At fixed model dimensions, one recurrent layer has sequence-independent state size; distinguish per-token updates from total sequence processing.
The Kimi K3 model card specifies 69 KDA layers and 24 Gated MLA layers. The complete model therefore retains a context-dependent cache component. State snapshots, prefix reuse, and concurrency affect serving memory too; see prefix caching.
The DeepSeek-V4 model card names Compressed Sparse Attention and Heavily Compressed Attention. Sequence compression and sparse access require different accounting from V3 MLA's per-token latent cache. Use the checkpoint and engine's actual layout.
Without this information, numbers such as 8.6GB at 100K, 42GB at 1M, or 110 tokens/s are not reproducible benchmarks. Architectural claims require controlled workloads and quality checks. See the deployment guide.
No. MLA keeps compressed history; recurrent KDA layers have fixed state, but K3's Gated MLA layers retain caches.
It depends on model and backend. Weight quantization, KV precision, and recurrent-state precision are separate controls; test them separately. See the quantization guide.
No quality ranking follows from the recurrence alone. Measure matching tasks, sequence lengths, and reasoning budgets.
Originally published at LLM Systems Architecture on blog.llmgo.top.