In the abstract, it claimes “KimiK3 is built on Kimi Delta Attention and Attention Residuals, which improves information flow sequence length and model depth. Together with Stable LatentMoE, which effectively activates approximately 2.5x improvement in overall scaling efficiency over Kimi K2.”
Their contributions are summarized as follows:
- Pre-training at the open frontier. We train a 2.8T-parameter native multimodal MoE model with 104B activated
parameters and a 1M-token context window. KDA, AttnRes, Stable LatentMoE, refined data and training recipes
collectively improve overall scaling efficiency by approximately 2.5× over Kimi K2. - Reinforcement learning for multi-effort test-time scaling. We conduct RL across general, agentic, and coding
domains and multiple reasoning-effort levels, then consolidate the resulting capabilities into a unified model. - Infrastructure for multi-trillion-parameter, million-token intelligence. We introduce KDA systems co-designs;
MoonEP and memory-efficient infrastructure for 2.8T-parameter MoE pre-training; a co-located RL system with
resumable sandboxes for million-token agentic trajectories; and more infrastructure innovations. - An open frontier model. We release the full Kimi K3 model weights, making frontier intelligence available for
research, deployment, and further innovation.
In this blog, I would like to dive to KDA, as I see the fundamental connection between DFT (discrete Fourier Transformation) and KDA as both used factorization in improving compute efficiency.
Now in LLM training:
Citation from the paper directly:
