LongCat-2.0 Debuts 1.6 Trillion Parameter MoE Architecture with Hybrid Parallelism

LongCat-2.0 is a newly detailed large language model featuring a 1.6 trillion parameter Mixture of Experts (MoE) architecture designed to improve scalability while keeping inference costs manageable. The model uses a 32-layer backbone with 16,000 experts organized into groups, enabling parallel processing across 128 GPUs at 98% utilization efficiency. Key technical features include dynamic sparse activation selecting 1–4 experts per token, 4-bit parameter quantization reducing memory use by 75%, and a hierarchical routing algorithm balancing content relevance with load distribution. Training runs on a 256-node cluster using RDMA-over-Converged-Ethernet interconnects, while precomputed routing tables cut batched inference overhead by 40%. Ongoing challenges include cold-start routing degradation, inter-node communication overhead, and GPU memory constraints that currently cap expert group sizes.