Training language models to follow instructions with human feedback

Overview

InstructGPT is a method for fine-tuning language models using human feedback. It collects human-written demonstrations and comparisons to train a reward model, then uses reinforcement learning to optimize the language model to produce outputs preferred by humans. This approach significantly improves instruction-following and reduces harmful or untruthful responses compared to the base model.

Best for

Best for
Researchers and engineers aligning large language models to human preferences for safety and instruction-following

Use cases

• Fine-tuning an existing large language model to better follow user instructions
• Reducing toxic or biased outputs from a generative language model
• Aligning a model's behavior with human preferences for safe deployment

Notes

InstructGPT is a method for fine-tuning language models using human feedback. It collects human-written demonstrations and comparisons to train a reward model, then uses reinforcement learning to optimize the language model to produce outputs preferred by humans. This approach significantly improves instruction-following and reduces harmful or untruthful responses compared to the base model.

Use cases

• Fine-tuning an existing large language model to better follow user instructions
• Reducing toxic or biased outputs from a generative language model
• Aligning a model’s behavior with human preferences for safe deployment

Pros

• Demonstrates significant improvement in following instructions over base GPT-3
• Reduces the frequency of harmful and untruthful outputs
• Provides a reproducible framework for aligning language models

Cons

• Requires substantial human annotation effort for demonstrations and comparisons
• The RLHF process can be computationally expensive and unstable
• May still produce errors or biased responses despite alignment

Indexed from awesome-llm and enriched against its public facts.