In the previous lesson, you fine-tuned a pretrained LLM on instruction-response pairs. The resulting model can follow instructions — it knows the format of a helpful answer. But format is not the same as quality.
An SFT model can still produce outputs that are technically on-format but harmful, factually wrong, excessively verbose, or just unhelpful. The problem: next-token prediction doesn't encode human preferences. The model learned to imitate the training data, but it never learned which responses humans actually prefer over others.
🎯
The alignment problem
Same question, very different answer quality
Consider a user asking: "What is Python?" An SFT model might produce any of these responses:
Response A: "Python is a high-level programming language known for readability." — Helpful, concise, correct.
Response B: "Python is a snake found in tropical regions." — Technically correct, but unhelpful in a programming context.
All three are valid next-token predictions given the training data. But humans strongly prefer A over B, and B over C. RLHF teaches the model this preference ordering.
The gap between imitation and alignment:
SFT learns to imitate good examples. RLHF learns to rank outputs by quality. This ranking signal is what transforms a capable-but-unreliable model into a consistently helpful assistant. It's the difference between "can sometimes produce good answers" and "reliably produces the best answer."
The RLHF pipeline
RLHF happens in three stages, each building on the previous one. This is the process that OpenAI used to create ChatGPT from GPT-3.5, and that Anthropic uses for Claude.
Why not just train on "good" responses?
Because quality is relative, not absolute. Labelers often disagree on what makes a response "good." But they find it much easier to say "A is better than B." RLHF leverages this comparative signal — it's easier for humans to rank than to rate.
Preference data format
The foundation of RLHF is preference data — pairs of responses where human labelers have chosen which one is better. Each data point contains a prompt, a chosen (preferred) response, and a rejected response.
preference_dataset.pyRead-only — run locally with PyTorch
Expected output
Preference dataset size: 3 pairs
Example pair:
Prompt: What is Python?
Chosen: Python is a high-level programming language known for readability.
Rejected: Python is a snake found in tropical regions.
Scale of real preference data:
OpenAI used ~33,000 comparison pairs for InstructGPT. Anthropic uses hundreds of thousands. The quality and diversity of preference data is often more important than the quantity — labelers need clear guidelines, and edge cases need careful handling.
Reward model training
The reward model takes a (prompt, response) pair and outputs a scalar reward score — a single number indicating how good the response is. It's trained on the preference data: the chosen response should get a higher reward than the rejected response.
reward_model.pyRead-only — run locally with PyTorch
After training on thousands of preference pairs, a well-trained reward model produces scores like these:
Trained reward model scoring
Reward model scoring:
Q: What is Python?
A: Python is a high-level programming language known for readability.
Reward: 0.92
Q: What is Python?
A: Python is a snake found in tropical regions.
Reward: 0.31
Q: What is Python?
A: idk lol
Reward: 0.05
Reward model training loop
The loss function is simple: for each preference pair, the chosen response should score higher than the rejected response. We use the Bradley-Terry model — a log-sigmoid of the reward difference.
reward_training.pyRead-only — run locally with PyTorch
Expected output
=== Training Reward Model ===
Epoch 1/5: Accuracy: 62.3%
Epoch 3/5: Accuracy: 78.1%
Epoch 5/5: Accuracy: 85.4%
Accuracy interpretation:
A reward model accuracy of 85% means it correctly identifies the human-preferred response 85% of the time. This is typical — human labelers themselves only agree about 73-80% of the time on difficult comparisons, so a model above 80% is learning the consensus well.
PPO — optimizing the policy
With a trained reward model, we can now optimize the language model (the "policy") using Proximal Policy Optimization (PPO). The idea: generate responses, score them with the reward model, and update the policy to produce higher-scoring responses.
The critical constraint is KL divergence — we penalize the policy for drifting too far from the original SFT model. Without this, the model learns to exploit the reward model's weaknesses (reward hacking) instead of genuinely improving.
🔄
The PPO loop
Generate, score, update — with guardrails
Sample prompts from the training set
Generate responses using the current policy
Score responses with the frozen reward model
Compute KL penalty between current policy and SFT reference model
Update policy to maximize: reward - beta * KL_divergence
Repeat for thousands of steps
Policy (LLM)
->
Generate Response
->
Reward Model
->
Update Policy
PPO requires 4 models in memory:
(1) The policy being trained, (2) a frozen copy of the SFT model as the reference for KL, (3) the frozen reward model, and (4) a value model for PPO's advantage estimation. This is why PPO-based RLHF is computationally expensive — and why researchers looked for alternatives.
DPO — Direct Preference Optimization
In 2023, Rafailov et al. proposed DPO as a simpler alternative to PPO. The key insight: you can skip the reward model entirely and optimize directly on preference pairs. DPO reformulates the RLHF objective as a simple classification loss.
dpo_loss.pyRead-only — run locally with PyTorch
Expected output
DPO vs PPO:
PPO: Policy -> Generate -> Reward Model -> Update Policy (complex, 4 models)
DPO: Policy -> Preference pairs -> Direct update (simpler, 2 models)
Why DPO is gaining traction:
DPO produces comparable results to PPO on most benchmarks, but needs only 2 models (policy + frozen reference) instead of 4. It's simpler to implement, faster to train, and more stable. Meta's Llama 2 used PPO, but Llama 3 switched to DPO. Many labs now prefer DPO or its variants (IPO, KTO) over PPO.
DPO training loop
Here's a complete DPO training loop. Unlike PPO, there's no generation step — we train directly on pre-collected preference pairs.
dpo_training.pyRead-only — run locally with PyTorch
The DPO loss decreases (good). Chosen rewards increase — the policy is giving higher probability to preferred responses. Rejected rewards decrease — lower probability for bad responses. The gap between chosen and rejected rewards widens, meaning the model is learning to distinguish quality.
Run it yourself
Try RLHF components locally
Copy the reward model and DPO loss function from above and run them on your machine:
Start with the PreferenceDataset class — add your own prompt-chosen-rejected triples and see how the reward model scores them. Then try implementing the DPO loss and watch the chosen/rejected reward gap widen during training.
Why this matters for LLMs
RLHF is what separates GPT from ChatGPT
GPT-3 could generate text, but it often produced toxic, unhelpful, or factually incorrect outputs. RLHF transformed it into ChatGPT — a model that's helpful, harmless, and honest. The same technique powers Claude, Gemini, and every major assistant.
The shift from PPO to DPO reflects a broader trend in ML: replacing complex multi-stage pipelines with simpler end-to-end objectives. DPO achieves similar alignment quality with half the infrastructure — fewer GPUs, shorter training runs, and more predictable results.
ChatGPT: PPO-based RLHFLlama 3: DPOClaude: Constitutional AI + RLHFPreference data > Raw data
Challenge
Put it into practice
Challenge 1: Build a preference dataset
Create 10 preference pairs for a customer support chatbot. Each pair should have a prompt (customer question), a chosen response (helpful, professional), and a rejected response (rude, lazy, or factually wrong). Think about what makes a response truly "preferred" — tone, accuracy, completeness, or all three?
Consider edge cases: what if the rejected response is factually correct but rude? Or polite but wrong? Labelers often disagree on these. In practice, labeling guidelines specify the priority: safety > factual accuracy > helpfulness > conciseness. Your dataset should include examples that test these boundaries.
Challenge 2: Implement the full DPO pipeline
Take the DPO loss function from this lesson and integrate it with the MiniGPT model from Lesson 14. Create a frozen reference copy of the model, generate some synthetic preference pairs, and run the DPO training loop. Monitor the chosen/rejected reward gap — it should widen over training.
Use ref_model = copy.deepcopy(policy) and ref_model.eval() to create the frozen reference. For synthetic data, take any text and create "chosen" (original) vs "rejected" (randomly shuffled words) pairs. The DPO loss should drive the policy to prefer the coherent version.
Quiz
Question 1 of 5
Why is SFT alone insufficient for building a reliable AI assistant?
Question 2 of 5
What does the reward model learn to predict?
Question 3 of 5
What is the purpose of the KL divergence constraint in PPO-based RLHF?
Question 4 of 5
What is the main advantage of DPO over PPO?
Question 5 of 5
In preference data, what does each training example consist of?
0
/5
You understand how to align LLMs with human preferences.
Up next in Arc E
Next: Evaluation & Deployment ->
Your model is aligned. Now measure it — perplexity, BLEU, human evals, and safety benchmarks. Then deploy it with proper guardrails, rate limiting, and monitoring.