The 3-stage training pipeline

Every modern LLM goes through three training stages. Each stage teaches the model something different, uses different data, and has different compute requirements.

LLM Training Pipeline Stage 1: Pre-training Objective: Next-token prediction Data: Trillions of tokens Compute: 1000s of GPU-hours Teaches: language & knowledge Stage 2: SFT Objective: Follow instructions Data: 10K-100K examples Compute: 10s of GPU-hours Teaches: instruction following Stage 3: RLHF / DPO Objective: Align with humans Data: Preference pairs Compute: 10s of GPU-hours Teaches: helpfulness & safety Raw text (web, books, code) Pre-training uses 99%+ of total compute. SFT and RLHF are comparatively cheap. Compute cost distribution: Pre-training ~98% SFT RL Pre-training is where the model learns language. SFT and RLHF shape how it uses that knowledge.
Why three stages?
Pre-training gives the model broad knowledge (but it just completes text). SFT teaches it to follow instructions ("write a poem" instead of continuing random text). RLHF/DPO aligns it with human preferences — making it helpful, harmless, and honest. Each stage builds on the previous one.

Pre-training

Pre-training is the most expensive and most important stage. The model learns language, facts, reasoning, and code by predicting the next token across trillions of tokens of text.

🎯
Objective
Next-token prediction (Causal Language Modeling)

Given tokens t_1, t_2, ..., t_n, predict t_{n+1}. The loss is cross-entropy between the predicted probability distribution and the one-hot target.

Loss Function

Cross-entropy: -sum(y * log(p))
Averaged over all positions in the sequence.

Training Data

Web text (CommonCrawl, C4), books, Wikipedia, code (GitHub), scientific papers. Trillions of tokens total.

Compute

Thousands of GPU-hours. Llama 2 70B: ~1.7M GPU-hours on A100s. GPT-3: estimated ~3,600 petaflop-days.

What It Learns

Grammar, facts, reasoning patterns, code syntax, multilingual ability, basic math — all from next-token prediction alone.

Key hyperparameters

HyperparameterTypical RangeNotes
Learning rate1e-4 to 3e-4Peak LR after warmup; larger models use lower LR
Warmup steps1,000 - 5,000Linear warmup from 0 to peak LR
LR scheduleCosine decayDecay to ~10% of peak LR over training
Batch size0.5M - 4M tokensTotal tokens per gradient step; ramp up during training
Gradient clipping1.0Max gradient norm; prevents loss spikes
Weight decay0.1Applied to all weights except biases and LayerNorm
OptimizerAdamWbeta1=0.9, beta2=0.95 (GPT-3 / Llama)
Context length2K - 128K tokensGPT-3: 2K, Llama 2: 4K, GPT-4: 128K
Cost reality check:
Pre-training a 70B model from scratch costs $1-5M+ in cloud GPU time. This is why most teams start with an existing pre-trained model and fine-tune. Only a handful of organizations (OpenAI, Anthropic, Google, Meta) do pre-training at frontier scale.

Scaling laws

Scaling laws tell us how model performance improves as we increase model size (N), dataset size (D), and compute (C). Two landmark papers define the field.

Kaplan et al. (2020) — OpenAI Scaling Laws

Loss decreases as a power law with model size:

L(N) ∝ N-0.076

Key insight: performance scales predictably with parameters. Bigger models are reliably better, not randomly better. This means you can predict performance before spending millions on training.

Hoffmann et al. (2022) — Chinchilla Scaling Laws

The compute-optimal ratio between model size and data:

D ≈ 20N    (optimal tokens ≈ 20x parameters)

This paper showed that GPT-3 (175B params, 300B tokens) was undertrained. A 70B model trained on 1.4T tokens (Chinchilla) outperformed the 175B GPT-3. The lesson: don't just scale the model, scale the data too.

Quick reference: compute-optimal training

ParametersOptimal Tokens (20x N)Example Model
1B20B tokensTinyLlama
7B140B tokensLlama 2 7B (2T actual*)
13B260B tokensLlama 2 13B
70B1.4T tokensChinchilla, Llama 2 70B
175B3.5T tokensGPT-3 used only 300B*

* Many recent models train well beyond 20x N (Llama 2 used 2T tokens for 7B) because inference compute savings from a smaller, better-trained model outweigh extra training cost.

Fine-tuning methods comparison

After pre-training, fine-tuning adapts the model to specific tasks or behaviors. The key question: how many parameters do you update?

Method Params Updated Memory Speed Forgetting Risk
Full fine-tuning 100% Very high Slow High
LoRA 0.1 - 1% Low Fast Low
QLoRA 0.1 - 1% Very low Fast Low
Prefix tuning <0.1% Very low Very fast Very low
Full Fine-tuning

Updates all model parameters. Best performance but requires full-model memory (optimizer states = 3x model size). Risk of catastrophic forgetting — the model "forgets" pre-training knowledge.

LoRA (Low-Rank Adaptation)

Inserts small r x d adapter matrices into attention layers. Only trains the adapters (0.1-1% of total params). Merges back at inference for zero latency overhead. The go-to method for most fine-tuning today.

QLoRA

LoRA + 4-bit quantization of the base model. Fine-tune a 65B model on a single 48GB GPU. Uses NormalFloat4 (NF4) data type + double quantization. Minimal quality loss vs full-precision LoRA.

Prefix Tuning

Prepends learnable "virtual tokens" to the input at each layer. No modification to model weights. Very parameter-efficient but less expressive than LoRA. Good for lightweight task adaptation.

When to use each:
  • Full fine-tuning: You have abundant compute and data, and need maximum performance on a narrow domain
  • LoRA: The default choice — works for most use cases, good quality-to-cost ratio
  • QLoRA: You have limited GPU memory (consumer hardware, single GPU)
  • Prefix tuning: Quick experiments, multi-task serving (one base model, many prefixes)

RLHF vs DPO

After SFT, alignment training makes the model helpful, harmless, and honest. Two approaches dominate: RLHF with PPO (the original) and DPO (the simpler alternative).

RLHF (PPO) vs DPO — Head to Head

RLHF with PPO

  • 4 models required: policy, reference, reward model, value model
  • Two-phase: first train a reward model, then optimize policy with PPO
  • More compute: 4 models in memory simultaneously
  • Complex: reward hacking, KL penalties, PPO hyperparams
  • Mature: Used by OpenAI (ChatGPT), proven at scale
  • Flexible: Reward model can be reused across policy updates
vs

DPO (Direct Preference Optimization)

  • 2 models required: policy and reference only
  • Single-phase: directly optimize on preference pairs
  • Less compute: no reward model or value model
  • Simpler: standard supervised loss, fewer hyperparams
  • Newer: Used by Meta (Llama 2), growing adoption
  • Less flexible: needs new preference data per iteration

Preference data format

Both RLHF and DPO require preference pairs — a prompt with a preferred (chosen) and dispreferred (rejected) response:

{ "prompt": "Explain quantum computing", "chosen": "Quantum computing uses qubits...", "rejected": "I don't know what that is..." }

Typical datasets: 50K-200K preference pairs. Human-labeled is best, but AI-labeled (constitutional AI, RLAIF) works well and is cheaper.

The trend:
DPO is becoming the default for new projects because it is simpler and produces comparable results. But RLHF with PPO still has advantages when you need iterative online learning or when your reward model generalizes well across many policy updates.

Common training issues and fixes

LLM training is notoriously fragile. Here are the most common problems and how to fix them.

Problem Symptoms Fixes
Loss spike Sudden jump in loss, may or may not recover Reduce learning rate, gradient clipping (max_norm=1.0), skip bad batches, reduce batch size
Loss plateau Loss stops decreasing, model stops improving Increase data diversity, check for duplicate data, warm restart with lower LR, increase model capacity
Catastrophic forgetting Model loses pre-trained abilities after fine-tuning Use LoRA instead of full fine-tuning, lower learning rate (1e-5 to 5e-5), mix pre-training data into fine-tuning
Mode collapse Model repeats the same outputs, low diversity Increase KL penalty (RLHF), diversify training data, increase temperature, check for data imbalance
OOM (out of memory) CUDA OOM error during training Gradient checkpointing, mixed precision (bf16), reduce batch size, use DeepSpeed ZeRO, use QLoRA
NaN loss Loss becomes NaN, training crashes Lower learning rate, check for division by zero, use bf16 instead of fp16, clip gradients more aggressively
Slow convergence Loss decreases very slowly Increase learning rate, check data pipeline (shuffling, batching), increase batch size, verify tokenizer
Debug checklist for new training runs:
  1. Overfit on a single batch first — if loss doesn't go to ~0, you have a bug
  2. Check learning rate schedule — plot it before training
  3. Monitor gradient norms — they should be stable, not exploding
  4. Validate data pipeline — print 10 random samples, check tokenization
  5. Start small — train a tiny model first, scale up once it works

Evaluation metrics

How do you know if your LLM is any good? Different metrics measure different capabilities.

Metric What It Measures When to Use Range
Perplexity How well the model predicts next tokens (exp of cross-entropy loss) Pre-training evaluation, language model comparison Lower = better (1 = perfect)
BLEU N-gram overlap with reference text Machine translation, text generation with reference 0-100, higher = better
ROUGE Recall-oriented overlap (ROUGE-1, ROUGE-L) Summarization quality 0-1, higher = better
MMLU Multi-task accuracy across 57 subjects (STEM, humanities, social science) General knowledge and reasoning benchmark 0-100%, higher = better
HumanEval Functional correctness of generated code (pass@k) Code generation ability 0-100%, higher = better
HellaSwag Commonsense reasoning — pick the right sentence completion Reasoning and common sense evaluation 0-100%, higher = better
For pre-training

Track perplexity on a held-out validation set. Lower perplexity = better language modeling. Compare against baselines at the same model size.

For fine-tuning / alignment

Use task-specific benchmarks: MMLU for knowledge, HumanEval for code, MT-Bench for chat quality. Always include human evaluation for subjective quality.

Deployment options

Once your model is trained, you need to serve it efficiently. Different tools optimize for different use cases.

Tool Best For Key Feature Format
vLLM High-throughput production serving PagedAttention — 24x throughput vs naive serving HuggingFace, AWQ, GPTQ
Ollama Easy local deployment One-command setup, built-in model library GGUF (llama.cpp)
TGI (Text Generation Inference) Production serving (HuggingFace) Continuous batching, tensor parallelism, Paged Attention HuggingFace safetensors
llama.cpp CPU inference, edge devices Pure C++, runs on laptops/phones, highly optimized GGUF

Quantization trade-offs

📦
Memory vs Quality
Choosing the right precision
PrecisionBits/Param7B Model SizeQuality LossUse Case
FP323228 GBNone (baseline)Training only
FP16 / BF161614 GBNegligibleStandard inference
INT887 GBMinimal (<1% on benchmarks)Production serving
INT443.5 GBSmall (1-3% on benchmarks)Local / edge deployment

Rule of thumb: INT4 quantization halves memory twice (4x reduction) with only a few percent quality loss. For most applications, INT4 with GGUF or GPTQ is the sweet spot for local deployment.

Deployment decision tree:
  • Production API with high throughput? Use vLLM with FP16 or INT8
  • Quick local experimentation? Use Ollama with INT4 GGUF
  • HuggingFace ecosystem / Docker? Use TGI
  • Edge / mobile / CPU only? Use llama.cpp with INT4 GGUF

Putting it all together

This cheatsheet covers the full journey from raw text to deployed model

Every LLM you interact with — ChatGPT, Claude, Gemini, Llama — went through this exact pipeline: pre-training on trillions of tokens, supervised fine-tuning on curated instruction data, and alignment via RLHF or DPO. The difference between a good LLM and a great one often comes down to data quality at each stage, not architectural innovations.

In the lessons linked below, you build each of these stages from scratch — training your own mini-GPT, fine-tuning with LoRA, and implementing DPO alignment.

Pre-training Scaling Laws LoRA / QLoRA RLHF vs DPO vLLM / Ollama INT4 Quantization
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've studied the full training pipeline.
Now train the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →