Moonshot AI has unveiled Kimi K3, a groundbreaking 2.8-trillion-parameter model boasting a million-token context window and advanced native vision capabilities. Touted as the world’s first open 3T-class model, Kimi K3 sets a new benchmark in the AI landscape.

Understanding Kimi K3

Kimi K3 is crafted as a sparse Mixture-of-Experts (MoE) model, integrating two significant architectural innovations: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). These updates redefine the flow of information across sequence length and depth within the model, tailoring it for complex tasks such as long-horizon coding and reasoning.

Moonshot AI proudly positions K3 as the first open model to breach the 2.8 trillion parameters threshold, maintaining leadership in open-model sizes over the past year. Despite this achievement, Moonshot acknowledges that K3’s performance still lags behind proprietary giants like Claude Fable 5 and GPT 5.6 Sol. However, within Moonshot’s evaluation framework, K3 consistently surpasses other tested models.

Innovative Architecture

The Kimi Delta Attention (KDA) mechanism is a hybrid linear attention model that significantly accelerates decoding processes in extensive token contexts, achieving speeds up to 6.3 times faster. Meanwhile, Attention Residuals (AttnRes) optimizes depth representation, enhancing training efficiency by approximately 25% with minimal cost increase.

K3’s design leverages sparsity through Stable LatentMoE, activating a select few experts to address routing and optimization challenges. Innovations like Quantile Balancing and Per-Head Muon ensure precise expert allocation and optimization of attention heads, while Sigmoid Tanh Unit (SiTU) and Gated MLA refine activation and attention selectivity.

Complementary to these architectural changes are refined training protocols and data strategies, which collectively offer a 2.5x improvement in scaling efficiency compared to its predecessor, Kimi K2.

Performance Insights

Detailed benchmarks reveal K3’s capabilities across various tests, with performance metrics framed by maximum reasoning efforts. K3 excels in domains like Program Bench and Automation Bench, outperforming peers like Fable 5 in specific areas. Nonetheless, it trails behind on benchmarks such as FrontierSWE and HLE-Full.

Practical Applications

Kimi K3’s versatile architecture supports diverse applications, from repo-scale engineering with minimal oversight to research reproduction involving extensive data analysis. Its multimodal capabilities integrate text, images, and video, facilitating advanced document parsing and deep research reporting.

Access and Utilization

K3 is accessible via multiple platforms including Kimi.com and the OpenAI SDK, with specific guidelines ensuring optimal usage. Pricing is straightforward, with flat rates for context processing, emphasizing efficient cache management to optimize costs.

Conclusion

• Kimi K3 stands as a pioneering 2.8T-parameter open MoE model, significantly advancing AI capabilities.
• Architectural innovations such as KDA and AttnRes enhance efficiency, offering substantial improvements over previous models.
• Despite competitive performance, K3 sets a new standard in open AI models, fostering further advancements in the field.