RLHF

Definition

Reinforcement Learning from Human Feedback. The technique that makes language models actually useful as assistants. Humans rank model outputs by quality, a reward model learns these preferences, and the language model is fine-tuned to maximize the reward. How ChatGPT, Claude, and others learned to be helpful.

How It Works

RLHF (Reinforcement Learning from Human Feedback) is the training technique that turns a raw language model into a useful assistant. After pre-training on text prediction, the model generates responses that are technically fluent but not necessarily helpful or safe. RLHF uses human preferences to teach the model what "good" responses look like.

The process has three steps. First, human raters compare pairs of model responses and pick which one is better. Second, these preferences train a reward model — a separate model that predicts how highly a human would rate any given response. Third, the language model is fine-tuned using reinforcement learning (specifically PPO — Proximal Policy Optimization) to maximize the reward model's score. The result is a model that generates responses humans actually prefer.

RLHF is what made the difference between GPT-3 (impressive but unwieldy) and ChatGPT (useful and approachable). It's also imperfect. The model can learn to produce responses that look good to the reward model without actually being better — a problem called reward hacking. Alternatives and improvements include DPO (Direct Preference Optimization), which skips the reward model entirely, and Constitutional AI, which reduces the need for human labelers. But the core idea — learning from human preferences — remains central to making AI systems that people actually want to use.