veRL

Overview

veRL is a Python framework for reinforcement learning post-training of large language models. It provides a flexible architecture for running RL workflows at scale, supporting distributed training across multiple GPUs and optimized inference pipelines. The framework handles reward modeling, policy optimization, and generation sampling in a modular design.

Best for

Best for
ML engineers building custom RL post-training pipelines for LLMs at scale

Use cases

• Fine-tuning LLMs with RL objectives like RLHF or DPO
• Running distributed RL experiments across GPU clusters
• Building custom reward models and policy optimization loops

Notes

veRL is a Python framework for reinforcement learning post-training of large language models. It provides a flexible architecture for running RL workflows at scale, supporting distributed training across multiple GPUs and optimized inference pipelines. The framework handles reward modeling, policy optimization, and generation sampling in a modular design.

21,691 stars on GitHub. Last updated 2026-06-01. Licensed Apache-2.0.

Use cases

• Fine-tuning LLMs with RL objectives like RLHF or DPO
• Running distributed RL experiments across GPU clusters
• Building custom reward models and policy optimization loops

Pros

• Modular architecture allows swapping components like reward models and optimizers
• Optimized for distributed training with efficient GPU utilization
• Active community project with 21k+ stars indicating adoption and maintenance

Cons

• Requires significant infrastructure and GPU resources to run effectively
• Steeper learning curve compared to higher-level fine-tuning APIs
• Documentation and examples may be limited relative to mainstream frameworks

Indexed from awesome-llm and enriched against its public facts.