How Kimi Trains 896 Experts Without Collapse

Kimi K3 shocked the tech world: on coding, knowledge, and reasoning benchmarks, it goes toe-to-toe with the strongest closed-source models, at 2.8 trillion parameters, with only 104 billion active per token. This video explains the architecture behind that gap: Stable LatentMoE. We build it from first principles: Latent MoE, which makes 896 experts affordable, and the three mechanisms that keep training stable at 16-of-896 sparsity: RMSNorm on the routed branch, SiTU-GLU inside each expert, and Quantile Balancing on the router. Chapters: 0:00 Kimi K3: 2.8T open-weight model 01:10 FFN/MoE: Wh