Training Compute-Optimal Large Language Models

Overview

Chinchilla is a scaling law framework from a 2022 paper that determines the optimal allocation of compute between model parameters and training tokens. It demonstrates that many existing large language models are overparameterized relative to the data used, and provides a formula to minimize loss for a given compute budget.

Best for

Best for
Researchers and practitioners optimizing large language model training for compute efficiency

Use cases

• Determining the optimal parameter count for a given compute budget
• Deciding the number of training tokens to match model size
• Rethinking scaling strategies to improve compute efficiency

Notes

Chinchilla is a scaling law framework from a 2022 paper that determines the optimal allocation of compute between model parameters and training tokens. It demonstrates that many existing large language models are overparameterized relative to the data used, and provides a formula to minimize loss for a given compute budget.

Use cases

• Determining the optimal parameter count for a given compute budget
• Deciding the number of training tokens to match model size
• Rethinking scaling strategies to improve compute efficiency

Pros

• Empirically validated on multiple model sizes and datasets
• Reduces wasted compute by guiding resource allocation
• Widely cited and influential in the LLM community

Cons

• Derived from specific Transformer architectures and training setups, may not generalize universally
• Requires accurate estimates of total compute budget, which can be uncertain upfront
• Does not account for other factors like data quality or architectural innovations

Indexed from awesome-llm and enriched against its public facts.