Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Overview

Switch Transformers is a neural network architecture that scales model parameters to trillions by introducing a sparsely activated mixture of experts (MoE) layer. It replaces dense feed-forward layers with multiple experts, routing each input token to only one expert per layer, which keeps computational cost constant as parameters grow. The paper demonstrates stable training and improved efficiency over dense models at equivalent compute budgets.

Best for

Best for
Researchers and engineers building or scaling sparse mixture-of-experts transformer models for language tasks

Use cases

• Training large language models with up to trillions of parameters on limited hardware
• Reducing inference latency in production NLP systems by activating only a fraction of parameters per token
• Benchmarking sparse MoE architectures against dense baselines for research purposes

Notes

Switch Transformers is a neural network architecture that scales model parameters to trillions by introducing a sparsely activated mixture of experts (MoE) layer. It replaces dense feed-forward layers with multiple experts, routing each input token to only one expert per layer, which keeps computational cost constant as parameters grow. The paper demonstrates stable training and improved efficiency over dense models at equivalent compute budgets.

Use cases

• Training large language models with up to trillions of parameters on limited hardware
• Reducing inference latency in production NLP systems by activating only a fraction of parameters per token
• Benchmarking sparse MoE architectures against dense baselines for research purposes

Pros

• Achieves massive parameter counts without proportional increase in FLOPs per token
• Simplifies training stability compared to prior MoE approaches with a single expert routing strategy
• Open-source paper with reproducible results on standard benchmarks

Cons

• Requires careful load balancing across experts to avoid routing collapse
• Memory footprint remains large due to storing all expert weights even when sparsely activated
• Not a drop-in replacement for existing dense models; needs custom implementation and tuning

Indexed from awesome-llm and enriched against its public facts.