The 3-stage training pipeline
Every modern LLM goes through three training stages. Each stage teaches the model something different, uses different data, and has different compute requirements.
Pre-training
Pre-training is the most expensive and most important stage. The model learns language, facts, reasoning, and code by predicting the next token across trillions of tokens of text.
Given tokens t_1, t_2, ..., t_n, predict t_{n+1}. The loss is cross-entropy between the predicted probability distribution and the one-hot target.
Cross-entropy: -sum(y * log(p))
Averaged over all positions in the sequence.
Web text (CommonCrawl, C4), books, Wikipedia, code (GitHub), scientific papers. Trillions of tokens total.
Thousands of GPU-hours. Llama 2 70B: ~1.7M GPU-hours on A100s. GPT-3: estimated ~3,600 petaflop-days.
Grammar, facts, reasoning patterns, code syntax, multilingual ability, basic math — all from next-token prediction alone.
Key hyperparameters
| Hyperparameter | Typical Range | Notes |
|---|---|---|
| Learning rate | 1e-4 to 3e-4 | Peak LR after warmup; larger models use lower LR |
| Warmup steps | 1,000 - 5,000 | Linear warmup from 0 to peak LR |
| LR schedule | Cosine decay | Decay to ~10% of peak LR over training |
| Batch size | 0.5M - 4M tokens | Total tokens per gradient step; ramp up during training |
| Gradient clipping | 1.0 | Max gradient norm; prevents loss spikes |
| Weight decay | 0.1 | Applied to all weights except biases and LayerNorm |
| Optimizer | AdamW | beta1=0.9, beta2=0.95 (GPT-3 / Llama) |
| Context length | 2K - 128K tokens | GPT-3: 2K, Llama 2: 4K, GPT-4: 128K |
Scaling laws
Scaling laws tell us how model performance improves as we increase model size (N), dataset size (D), and compute (C). Two landmark papers define the field.
Kaplan et al. (2020) — OpenAI Scaling Laws
Loss decreases as a power law with model size:
Key insight: performance scales predictably with parameters. Bigger models are reliably better, not randomly better. This means you can predict performance before spending millions on training.
Hoffmann et al. (2022) — Chinchilla Scaling Laws
The compute-optimal ratio between model size and data:
This paper showed that GPT-3 (175B params, 300B tokens) was undertrained. A 70B model trained on 1.4T tokens (Chinchilla) outperformed the 175B GPT-3. The lesson: don't just scale the model, scale the data too.
Quick reference: compute-optimal training
| Parameters | Optimal Tokens (20x N) | Example Model |
|---|---|---|
| 1B | 20B tokens | TinyLlama |
| 7B | 140B tokens | Llama 2 7B (2T actual*) |
| 13B | 260B tokens | Llama 2 13B |
| 70B | 1.4T tokens | Chinchilla, Llama 2 70B |
| 175B | 3.5T tokens | GPT-3 used only 300B* |
* Many recent models train well beyond 20x N (Llama 2 used 2T tokens for 7B) because inference compute savings from a smaller, better-trained model outweigh extra training cost.
Fine-tuning methods comparison
After pre-training, fine-tuning adapts the model to specific tasks or behaviors. The key question: how many parameters do you update?
| Method | Params Updated | Memory | Speed | Forgetting Risk |
|---|---|---|---|---|
| Full fine-tuning | 100% | Very high | Slow | High |
| LoRA | 0.1 - 1% | Low | Fast | Low |
| QLoRA | 0.1 - 1% | Very low | Fast | Low |
| Prefix tuning | <0.1% | Very low | Very fast | Very low |
Updates all model parameters. Best performance but requires full-model memory (optimizer states = 3x model size). Risk of catastrophic forgetting — the model "forgets" pre-training knowledge.
Inserts small r x d adapter matrices into attention layers. Only trains the adapters (0.1-1% of total params). Merges back at inference for zero latency overhead. The go-to method for most fine-tuning today.
LoRA + 4-bit quantization of the base model. Fine-tune a 65B model on a single 48GB GPU. Uses NormalFloat4 (NF4) data type + double quantization. Minimal quality loss vs full-precision LoRA.
Prepends learnable "virtual tokens" to the input at each layer. No modification to model weights. Very parameter-efficient but less expressive than LoRA. Good for lightweight task adaptation.
- Full fine-tuning: You have abundant compute and data, and need maximum performance on a narrow domain
- LoRA: The default choice — works for most use cases, good quality-to-cost ratio
- QLoRA: You have limited GPU memory (consumer hardware, single GPU)
- Prefix tuning: Quick experiments, multi-task serving (one base model, many prefixes)
RLHF vs DPO
After SFT, alignment training makes the model helpful, harmless, and honest. Two approaches dominate: RLHF with PPO (the original) and DPO (the simpler alternative).
RLHF with PPO
- 4 models required: policy, reference, reward model, value model
- Two-phase: first train a reward model, then optimize policy with PPO
- More compute: 4 models in memory simultaneously
- Complex: reward hacking, KL penalties, PPO hyperparams
- Mature: Used by OpenAI (ChatGPT), proven at scale
- Flexible: Reward model can be reused across policy updates
DPO (Direct Preference Optimization)
- 2 models required: policy and reference only
- Single-phase: directly optimize on preference pairs
- Less compute: no reward model or value model
- Simpler: standard supervised loss, fewer hyperparams
- Newer: Used by Meta (Llama 2), growing adoption
- Less flexible: needs new preference data per iteration
Preference data format
Both RLHF and DPO require preference pairs — a prompt with a preferred (chosen) and dispreferred (rejected) response:
Typical datasets: 50K-200K preference pairs. Human-labeled is best, but AI-labeled (constitutional AI, RLAIF) works well and is cheaper.
Common training issues and fixes
LLM training is notoriously fragile. Here are the most common problems and how to fix them.
| Problem | Symptoms | Fixes |
|---|---|---|
| Loss spike | Sudden jump in loss, may or may not recover | Reduce learning rate, gradient clipping (max_norm=1.0), skip bad batches, reduce batch size |
| Loss plateau | Loss stops decreasing, model stops improving | Increase data diversity, check for duplicate data, warm restart with lower LR, increase model capacity |
| Catastrophic forgetting | Model loses pre-trained abilities after fine-tuning | Use LoRA instead of full fine-tuning, lower learning rate (1e-5 to 5e-5), mix pre-training data into fine-tuning |
| Mode collapse | Model repeats the same outputs, low diversity | Increase KL penalty (RLHF), diversify training data, increase temperature, check for data imbalance |
| OOM (out of memory) | CUDA OOM error during training | Gradient checkpointing, mixed precision (bf16), reduce batch size, use DeepSpeed ZeRO, use QLoRA |
| NaN loss | Loss becomes NaN, training crashes | Lower learning rate, check for division by zero, use bf16 instead of fp16, clip gradients more aggressively |
| Slow convergence | Loss decreases very slowly | Increase learning rate, check data pipeline (shuffling, batching), increase batch size, verify tokenizer |
- Overfit on a single batch first — if loss doesn't go to ~0, you have a bug
- Check learning rate schedule — plot it before training
- Monitor gradient norms — they should be stable, not exploding
- Validate data pipeline — print 10 random samples, check tokenization
- Start small — train a tiny model first, scale up once it works
Evaluation metrics
How do you know if your LLM is any good? Different metrics measure different capabilities.
| Metric | What It Measures | When to Use | Range |
|---|---|---|---|
| Perplexity | How well the model predicts next tokens (exp of cross-entropy loss) | Pre-training evaluation, language model comparison | Lower = better (1 = perfect) |
| BLEU | N-gram overlap with reference text | Machine translation, text generation with reference | 0-100, higher = better |
| ROUGE | Recall-oriented overlap (ROUGE-1, ROUGE-L) | Summarization quality | 0-1, higher = better |
| MMLU | Multi-task accuracy across 57 subjects (STEM, humanities, social science) | General knowledge and reasoning benchmark | 0-100%, higher = better |
| HumanEval | Functional correctness of generated code (pass@k) | Code generation ability | 0-100%, higher = better |
| HellaSwag | Commonsense reasoning — pick the right sentence completion | Reasoning and common sense evaluation | 0-100%, higher = better |
Track perplexity on a held-out validation set. Lower perplexity = better language modeling. Compare against baselines at the same model size.
Use task-specific benchmarks: MMLU for knowledge, HumanEval for code, MT-Bench for chat quality. Always include human evaluation for subjective quality.
Deployment options
Once your model is trained, you need to serve it efficiently. Different tools optimize for different use cases.
| Tool | Best For | Key Feature | Format |
|---|---|---|---|
| vLLM | High-throughput production serving | PagedAttention — 24x throughput vs naive serving | HuggingFace, AWQ, GPTQ |
| Ollama | Easy local deployment | One-command setup, built-in model library | GGUF (llama.cpp) |
| TGI (Text Generation Inference) | Production serving (HuggingFace) | Continuous batching, tensor parallelism, Paged Attention | HuggingFace safetensors |
| llama.cpp | CPU inference, edge devices | Pure C++, runs on laptops/phones, highly optimized | GGUF |
Quantization trade-offs
| Precision | Bits/Param | 7B Model Size | Quality Loss | Use Case |
|---|---|---|---|---|
| FP32 | 32 | 28 GB | None (baseline) | Training only |
| FP16 / BF16 | 16 | 14 GB | Negligible | Standard inference |
| INT8 | 8 | 7 GB | Minimal (<1% on benchmarks) | Production serving |
| INT4 | 4 | 3.5 GB | Small (1-3% on benchmarks) | Local / edge deployment |
Rule of thumb: INT4 quantization halves memory twice (4x reduction) with only a few percent quality loss. For most applications, INT4 with GGUF or GPTQ is the sweet spot for local deployment.
- Production API with high throughput? Use vLLM with FP16 or INT8
- Quick local experimentation? Use Ollama with INT4 GGUF
- HuggingFace ecosystem / Docker? Use TGI
- Edge / mobile / CPU only? Use llama.cpp with INT4 GGUF
Putting it all together
This cheatsheet covers the full journey from raw text to deployed model
Every LLM you interact with — ChatGPT, Claude, Gemini, Llama — went through this exact pipeline: pre-training on trillions of tokens, supervised fine-tuning on curated instruction data, and alignment via RLHF or DPO. The difference between a good LLM and a great one often comes down to data quality at each stage, not architectural innovations.
In the lessons linked below, you build each of these stages from scratch — training your own mini-GPT, fine-tuning with LoRA, and implementing DPO alignment.
You've studied the full training pipeline.
Now train the career to match.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →