📚 What is LoRA?

Low-Rank Adaptation (LoRA) is a technique that fine-tunes large models by adding small trainable matrices to each layer, leaving pre-trained weights frozen. Instead of updating all 7 billion parameters in a model, LoRA updates only 1-2 million — reducing memory and compute dramatically.

W_new = W_original + A × B

Where A and B are low-rank matrices (much smaller than the original weight matrix W).

🏢 Rebecca's Fintech Story

💳
ML Engineer at NovaPay
Anomaly Detection in Real-Time Transactions

Rebecca works at NovaPay, a fintech company processing 5M transactions daily. She wants to fine-tune Llama 2 to detect fraudulent patterns in transaction descriptions. Full fine-tuning requires 80 GB VRAM; her RTX 4090 has only 24 GB. With LoRA, she trains on the same GPU in 2 hours.

24 GB
Available VRAM
80 GB
Full Fine-Tune Needs
3.2 GB
LoRA Uses

⚙️ How LoRA Works

The Core Principle

During fine-tuning, weight changes in neural networks have low intrinsic dimensionality. LoRA exploits this: instead of updating a 4096×4096 weight matrix, we train two smaller matrices: A (4096×8) and B (8×4096). This reduces parameters from 16.8M to 65K — a 260x reduction.

Rank Selection: The r Parameter

The rank (r) controls the size of A and B. Common values: r=8 (lightweight), r=16 (balanced), r=32 (expressive). Higher rank = more parameters but also more memory and slower training.

lora_config.pyLoRA Configuration
Output
trainable params: 1,179,648 || all params: 6,738,415,616 || trainable%: 0.0175

Alpha and Scaling

The lora_alpha parameter scales the LoRA weights. The effective update is (α/r) × LoRA_update. With lora_alpha=16 and r=8, the scale is 2.0 — tuning this affects learning dynamics.

🎯 Selecting Target Modules

You don't need to adapt every layer. LoRA commonly targets key projection matrices:

  • q_proj (query projections) — capture task-specific attention patterns
  • v_proj (value projections) — learn domain-specific information
  • k_proj (key projections) — optional, less impact
  • out_proj / dense — sometimes added for encoder-decoder models
target_modules_guide.pyChoosing Modules by Model

🔢 QLoRA: Quantization + LoRA

QLoRA combines 4-bit quantization with LoRA, enabling fine-tuning of 70B models on a single 24 GB GPU. The model is quantized to 4-bit, then LoRA adapters are frozen on top.

qlora_setup.py4-Bit Quantization with LoRA
Output
Memory: 3.24 GB # vs 140 GB for full 70B model

🆚 LoRA vs Full Fine-Tuning

When to Use LoRA

  • ✓ Limited GPU memory (< 40 GB VRAM)
  • ✓ Fine-tuning large models (7B+)
  • ✓ Multiple domain adapters (finance, legal, medical)
  • ✓ Fast iteration cycles
  • ✓ Production deployment with minimal overhead

When to Use Full Fine-Tuning

  • ✓ Abundant compute (8x GPU cluster)
  • ✓ Maximum model quality required
  • ✓ Smaller models (< 1B parameters)
  • ✓ Complete domain shift (e.g., medical to poetry)

🔌 Adapter Layers & PEFT Library

The PEFT library supports multiple parameter-efficient methods beyond LoRA:

  • Adapter — Bottleneck modules between layers (fewer params than LoRA)
  • Prefix Tuning — Learnable prefix tokens at input
  • Prompt Tuning — Virtual tokens (no task-specific architecture changes)
  • LoRA — Low-rank weight updates (best balance)
  • IA3 — Learned scaling of intermediate activations
peft_methods.pyComparing PEFT Methods

🚀 Training with LoRA

train_lora.pyComplete Training Loop

⚠️ Common Mistakes

Not Freezing Pre-Trained Weights
Always set target_modules to limit which layers get trained. If you set all layers trainable, you lose the memory advantage.
Rank Too Low
Using r=1 or r=2 can bottleneck learning on diverse tasks. Start with r=8 or r=16, then tune down if memory-constrained.
Wrong Module Names
Module names vary by architecture. Use model.named_modules() to inspect, or check HuggingFace docs for your model family.
Forgetting to Merge LoRA Before Inference
For inference, either keep the adapter loading overhead, or call model.merge_and_unload() to fuse LoRA weights back into the original model.

📋 LoRA & PEFT Cheatsheet

Parameter Meaning Typical Value
r Low-rank dimension 8, 16, 32
lora_alpha Scaling factor (usually 2x r) 16, 32, 64
target_modules Which layers to adapt ["q_proj", "v_proj"]
lora_dropout Dropout in LoRA layers 0.05, 0.1
load_in_4bit Enable 4-bit quantization True / False
bnb_4bit_quant_type Quantization dtype "nf4" (best), "fp4"

💼 Interview Q&A

LoRA fine-tunes large models by training small low-rank matrices while keeping pre-trained weights frozen, reducing memory by 90%.
Fine-tuning weight changes have low intrinsic dimensionality — you can represent the update ΔW = A × B where A and B are much smaller than W. This exploits the observation that pre-trained models contain redundancy.
LoRA adds low-rank adapters. QLoRA combines LoRA with 4-bit quantization of the base model, enabling 70B parameter models to run on a 24GB GPU.
Start with r=8 for lightweight tasks. Use r=16 for balanced quality/efficiency. Higher ranks (r=32, r=64) improve expressiveness but increase memory. Profile on your hardware.
Two options: (1) Load base model + LoRA adapter separately (slight overhead), (2) Call model.merge_and_unload() to fuse LoRA into weights (no overhead, single file). Option 2 is preferred for inference-heavy production.

🎯 Test Your Knowledge

Question 1 of 5
Which weight matrices in a Transformer typically get adapted by LoRA?
Question 2 of 5
If r=8 and lora_alpha=16, what is the scaling factor for LoRA updates?
Question 3 of 5
What's the main memory advantage of QLoRA over full fine-tuning of a 70B model?
Question 4 of 5
When should you use full fine-tuning instead of LoRA?
Question 5 of 5
What does model.merge_and_unload() do in LoRA?
0
/ 5
Great job!
Review the sections above and retry to reinforce your learning.

✨ Key Takeaways

  • LoRA fine-tunes large models with 90% less memory by training only low-rank adapter matrices.
  • The rank (r) and lora_alpha parameters control the trade-off between expressiveness and efficiency.
  • QLoRA combines 4-bit quantization with LoRA to fit 70B models on a single 24GB GPU.
  • Target key projection matrices (q_proj, v_proj) — not all layers need adaptation.
  • For production, use model.merge_and_unload() to fuse LoRA weights and eliminate adapter overhead.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've mastered parameter-efficient fine-tuning.
Ready to deploy production LLMs.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →