Pre-training vs fine-tuning

Building an LLM happens in two distinct phases, and they teach the model fundamentally different things.

Pre-training is unsupervised. You feed the model billions of tokens of raw text and train it to predict the next token. This teaches the model language itself — grammar, facts, reasoning patterns, code syntax, even some world knowledge. But a pre-trained model has no concept of "instruction" or "answer." Ask it to summarize a paragraph and it might just continue writing more paragraphs.

Supervised fine-tuning (SFT) is the fix. You take the pre-trained model and train it on a dataset of (instruction, response) pairs. Now the model learns a new skill: given an instruction, produce a helpful response. This is the step that turns GPT into ChatGPT.

🔑
The key insight
Pre-training learns language. SFT learns behavior.

Think of pre-training as reading every book in the library. The model knows everything — but it doesn't know how to be helpful. SFT is like job training: you show the model thousands of examples of good behavior (clear answers, step-by-step reasoning, polite refusals) and it learns to mimic that pattern.

The remarkable thing is that SFT doesn't need much data. While pre-training requires billions of tokens, SFT typically uses only 10,000-100,000 high-quality instruction-response pairs. The knowledge is already there from pre-training — SFT just teaches the model how to access and present it.

From Raw Model to Assistant — The 3-Stage Pipeline Stage 1: Pre-training Billions of tokens Next-token prediction Learns: language Stage 2: SFT 10K-100K examples Instruction-response pairs Learns: behavior Stage 3: RLHF Human preferences Reward model + PPO Learns: alignment GPT This lesson Next lesson Pre-training is expensive (weeks on many GPUs). SFT and RLHF are cheap (hours on one GPU).
Why SFT is so efficient:
Pre-training builds the model's knowledge from scratch — it needs to learn everything about language. SFT only needs to reshape the model's behavior. The model already knows how to write summaries, translate text, and answer questions — it just doesn't know that's what you're asking it to do. SFT bridges that gap with remarkably little data.

Instruction following — the SFT dataset format

An SFT dataset is a collection of examples, each with three fields: instruction (what the user wants), input (optional context), and output (the ideal response). This is the format popularized by Stanford's Alpaca dataset.

instruction_dataset.py Read-only — run locally with PyTorch
Expected output
Instruction format example: instruction: Summarize the following text in one sentence. input: The transformer architecture was introduced in the 2017 paper... output: The transformer, introduced in 2017, replaced recurrence with self-attention... Dataset size: 4 examples --- Formatted prompt --- ### Instruction: Summarize the following text in one sentence. ### Input: The transformer architecture was introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al. It replaced recurrence with self-attention, enabling parallel processing and achieving state-of-the-art results on machine translation. ### Response: The transformer, introduced in 2017, replaced recurrence with self-attention for parallel processing and better translation results.
Real-world SFT datasets:
Stanford Alpaca used 52K instruction-response pairs generated by GPT-3.5. Meta's Llama 2 was fine-tuned on ~27K high-quality examples. The quality of the data matters far more than the quantity — a few thousand well-crafted examples often outperform millions of noisy ones. This is why companies like Anthropic and OpenAI invest heavily in data curation.

LoRA — Low-Rank Adaptation

Fine-tuning all the parameters of a large model is expensive. A 7B-parameter model needs ~28GB of GPU memory just for the weights, plus optimizer states that triple the memory requirement. LoRA (Low-Rank Adaptation) solves this by freezing the base model and training only small adapter matrices.

🧮
The core idea
Approximate weight updates with low-rank matrices

Instead of updating a full weight matrix W (shape d x d), LoRA decomposes the update into two small matrices: A (shape d x r) and B (shape r x d), where r is the "rank" — typically 4, 8, or 16. The effective weight becomes W + A x B. Since r is tiny compared to d, this requires dramatically fewer trainable parameters.

LoRA Weight Decomposition W' = W + delta_W = W + A x B W 768 x 768 Frozen FROZEN + A 768x8 x B 8x768 = delta_W 768 x 768 (low rank) TRAINED W: 589,824 params (frozen) | A+B: 12,288 params (trained) | 97.9% savings
lora_layer.py Read-only — run locally with PyTorch
Expected output
=== LoRA Parameter Savings === Original W: 768x768 = 589,824 params LoRA A: 768x8 = 6,144 LoRA B: 8x768 = 6,144 LoRA total: 12,288 params Savings: 97.9% Ratio: 48x fewer parameters
Why LoRA works:
The key insight from the original paper (Hu et al., 2021) is that weight updates during fine-tuning have low "intrinsic rank." Even though W is a 768x768 matrix, the actual change needed to adapt the model to a new task can be captured by matrices of rank 4-16. By exploiting this, LoRA reduces trainable parameters by 97%+ while matching full fine-tuning performance.

When to fine-tune vs prompt

Not every task needs fine-tuning. With powerful base models, you can often get good results just by writing a better prompt. Here's when each approach makes sense.

🤔
Decision framework
Prompting vs fine-tuning — a practical guide
FactorUse promptingUse fine-tuning
Data availableFew examples (0-50)Hundreds to thousands of examples
Task complexityCan describe in natural languageHard to describe, easier to show
Output formatFlexible / standardVery specific format required
LatencyLong prompts are OKNeed fast responses (shorter prompts)
Cost per requestHigher (long prompts)Lower (knowledge baked in)
DomainGeneral knowledgeSpecialized domain (medical, legal)
Iteration speedMinutes to try new approachesHours/days for each experiment
Rule of thumb:
Start with prompting. If you can't get the quality you need after serious prompt engineering, and you have at least 500 high-quality examples, consider fine-tuning. Fine-tuning is a one-way door — it's harder to iterate on than prompts, and it requires infrastructure to train, evaluate, and deploy.

Applying LoRA to an existing model

To fine-tune with LoRA, you freeze all base model parameters and replace selected linear layers with LoRA-wrapped versions. Typically you apply LoRA to the attention layers (Q, K, V projections) — these are the layers that most benefit from task-specific adaptation.

apply_lora.py Read-only — run locally with PyTorch
Expected output
=== Fine-tuning Mini-GPT with LoRA === Trainable params: 12,288 / 2,104,064 (0.6%) Frozen params: 2,091,776
Where to apply LoRA:
In practice, LoRA is most commonly applied to the Q and V projection matrices in attention layers. Some implementations also include the K projection and the output projection. The rank r controls the capacity of the adapter — rank 8 is a common default, but for complex tasks you might use 16 or 32.

SFT training loop

The training loop for SFT is nearly identical to pre-training — the only difference is the data. Instead of raw text, you feed instruction-response pairs. The model still trains with next-token prediction, but now it predicts the tokens of the response given the instruction.

sft_training.py Read-only — run locally with PyTorch
Expected output
=== Fine-tuning Mini-GPT with LoRA === Trainable params: 12,288 / 2,104,064 (0.6%) Frozen params: 2,091,776 Epoch 1/3: Loss: 2.8431 Epoch 2/3: Loss: 1.4523 Epoch 3/3: Loss: 0.8912 Before SFT: "Summarize: The transformer..." -> "the the the architecture was..." After SFT: "Summarize: The transformer..." -> "The transformer replaced recurrence with attention for parallel processing."
Training tip — learning rate matters:
SFT typically uses a much smaller learning rate than pre-training (1e-4 to 5e-5 vs 3e-4 to 1e-3). You want to gently adjust the model's behavior without destroying the knowledge it learned during pre-training. This is sometimes called "catastrophic forgetting" — and LoRA naturally guards against it because the base weights are frozen.

Run it yourself

Run SFT with LoRA locally

Copy the code from the sections above into a single file and run it on your machine:

pip install torch
python sft_training.py

Try changing the LoRA rank from 8 to 4 or 16 and observe how the trainable parameter count changes. With rank 4, you'll have ~6,144 trainable params (0.3%). With rank 16, you'll have ~24,576 (1.2%). The sweet spot depends on your task complexity.

Why this matters for LLMs

SFT is how every modern assistant is built

Every LLM assistant you've used — ChatGPT, Claude, Gemini, Llama Chat — started as a raw pre-trained model that could only complete text. SFT transformed each of them into instruction-following assistants. The technique you just learned is the exact same process, just scaled up.

LoRA has become the de facto standard for efficient fine-tuning. Hugging Face's PEFT library, QLoRA (which combines LoRA with 4-bit quantization), and frameworks like Axolotl all build on the same principle: freeze the base, train small adapters. Engineers who understand this pipeline are building custom AI products at companies across Africa, India, and Silicon Valley.

SFT: behavior shaping LoRA: 97.9% parameter savings 0.6% trainable params Same architecture, new skills

Challenge

Put it into practice
Challenge 1: Build a domain-specific SFT dataset

Create 10 instruction-response pairs for a specific domain (e.g., customer support for an e-commerce store, medical Q&A, or code review). Format them using the instruction/input/output schema from this lesson. Think about what makes a "good" response — conciseness, accuracy, tone.

For e-commerce support: instructions like "How do I return a product?", "What is your shipping policy?", "My order hasn't arrived." Each response should be helpful, concise, and include any necessary steps. The input field is optional — use it when there's additional context (like an order number).
Challenge 2: Experiment with LoRA rank

Modify the LoRA implementation to apply adapters to both the QKV projection AND the output projection in each attention block. How many trainable parameters does this give you with rank=8? How about with rank=4? Calculate the savings percentage for each configuration.

The QKV projection is (128, 384) and the output projection is (128, 128). With LoRA rank=8 on both: QKV gets 128*8 + 8*384 = 4,096 + 3,072 = no, wait — you need to calculate it as (in + out) * rank for each layer. For QKV: (128 + 384) * 8 = 4,096. For out_proj: (128 + 128) * 8 = 2,048. Per block: 6,144. Times 4 blocks: 24,576 trainable params total.

Quiz

Question 1 of 5
What does supervised fine-tuning (SFT) teach a pre-trained language model?
Question 2 of 5
In LoRA, what happens to the original weight matrix W during fine-tuning?
Question 3 of 5
What are the three fields in a standard SFT instruction dataset?
Question 4 of 5
When should you consider fine-tuning instead of prompt engineering?
Question 5 of 5
Why does LoRA initialize matrix B to zeros?
0
/5
You understand the SFT pipeline that creates every AI assistant.
Up next in Arc E

Next: RLHF — Reinforcement Learning from Human Feedback ->

SFT teaches the model to follow instructions. RLHF teaches it to follow them well — using human preference data and reward models to align the model's outputs with what humans actually want.

Next: RLHF ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You just built the SFT pipeline.
Now build the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →