Megatron-LM
by Community
Ongoing research training transformer models at scale
OSS
Megatron-LM
Added 1 June 2026
Overview
Megatron-LM is a Python framework for training large transformer models at scale, developed and maintained by NVIDIA. It provides distributed training optimizations and memory-efficient techniques to handle models that exceed single-GPU capacity.
Best for
Best for
ML engineers training large transformer models who need production-grade distributed training infrastructure
Use cases
- Training billion-parameter language models across multiple GPUs
- Reducing memory footprint and training time for large transformers
- Implementing pipeline parallelism and tensor parallelism strategies
Notes
Megatron-LM is a Python framework for training large transformer models at scale, developed and maintained by NVIDIA. It provides distributed training optimizations and memory-efficient techniques to handle models that exceed single-GPU capacity.
16,545 stars on GitHub. Last updated 2026-06-01.
Use cases
- Training billion-parameter language models across multiple GPUs
- Reducing memory footprint and training time for large transformers
- Implementing pipeline parallelism and tensor parallelism strategies
Pros
- Production-grade distributed training infrastructure from NVIDIA
- Significant memory and compute optimizations for large models
- Active research codebase with ongoing improvements
Cons
- Steep learning curve for distributed training concepts
- Requires multi-GPU or multi-node setup to be practical
- Community-driven with less formal support than commercial alternatives
Indexed from awesome-llm and enriched against its public facts.
Pros
- Production-grade distributed training infrastructure from NVIDIA
- Significant memory and compute optimizations for large models
- Active research codebase with ongoing improvements
Cons
- Steep learning curve for distributed training concepts
- Requires multi-GPU or multi-node setup to be practical
- Community-driven with less formal support than commercial alternatives
Pairs with
Other entries in the index that connect to this one. Click through to see the chain.
DeepSpeed
Community
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Colossal-AI
Community
Making large AI models cheaper, faster and more accessible
BLOOMZ&mT0
Community
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
GPT-NeoX
Community
An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries
Large Language Model Training in 2023
Community
Learn about large language model training with insights on large language model examples, model architecture, and model training guide.
Liger-Kernel
Community
Efficient Triton Kernels for LLM Training
nanotron
Community
Minimalistic large language model 3D-parallelism training
Axolotl
Community
Go ahead and axolotl questions
BMTrain
Community
Efficient Training (including pre-training and fine-tuning) for Big Models
Colossal-AI
Community
Making large AI models cheaper, faster and more accessible
DeepSpeed
Community
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
maxtext
Community
A simple, performant and scalable Jax LLM!
Mesh Tensorflow
Community
Mesh TensorFlow: Model Parallelism Made Easier
nanotron
Community
Minimalistic large language model 3D-parallelism training
torchtitan
Community
A PyTorch native platform for training generative AI models
Transformer Engine
Community
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide b
Get the free Developer’s Field Guide
A 27-page field guide to the AI coding workflow with Claude. Claude Code, MCP servers, the prompt patterns that work, and what to delegate. Free.
Enter your work email. We send it straight over, plus a few short notes worth knowing. Unsubscribe any time.