📚 What is LoRA?
Low-Rank Adaptation (LoRA) is a technique that fine-tunes large models by adding small trainable matrices to each layer, leaving pre-trained weights frozen. Instead of updating all 7 billion parameters in a model, LoRA updates only 1-2 million — reducing memory and compute dramatically.
Where A and B are low-rank matrices (much smaller than the original weight matrix W).
🏢 Rebecca's Fintech Story
Rebecca works at NovaPay, a fintech company processing 5M transactions daily. She wants to fine-tune Llama 2 to detect fraudulent patterns in transaction descriptions. Full fine-tuning requires 80 GB VRAM; her RTX 4090 has only 24 GB. With LoRA, she trains on the same GPU in 2 hours.
⚙️ How LoRA Works
The Core Principle
During fine-tuning, weight changes in neural networks have low intrinsic dimensionality. LoRA exploits this: instead of updating a 4096×4096 weight matrix, we train two smaller matrices: A (4096×8) and B (8×4096). This reduces parameters from 16.8M to 65K — a 260x reduction.
Rank Selection: The r Parameter
The rank (r) controls the size of A and B. Common values: r=8 (lightweight), r=16 (balanced), r=32 (expressive). Higher rank = more parameters but also more memory and slower training.
Alpha and Scaling
The lora_alpha parameter scales the LoRA weights. The effective update is (α/r) × LoRA_update. With lora_alpha=16 and r=8, the scale is 2.0 — tuning this affects learning dynamics.
🎯 Selecting Target Modules
You don't need to adapt every layer. LoRA commonly targets key projection matrices:
- q_proj (query projections) — capture task-specific attention patterns
- v_proj (value projections) — learn domain-specific information
- k_proj (key projections) — optional, less impact
- out_proj / dense — sometimes added for encoder-decoder models
🔢 QLoRA: Quantization + LoRA
QLoRA combines 4-bit quantization with LoRA, enabling fine-tuning of 70B models on a single 24 GB GPU. The model is quantized to 4-bit, then LoRA adapters are frozen on top.
🆚 LoRA vs Full Fine-Tuning
When to Use LoRA
- ✓ Limited GPU memory (< 40 GB VRAM)
- ✓ Fine-tuning large models (7B+)
- ✓ Multiple domain adapters (finance, legal, medical)
- ✓ Fast iteration cycles
- ✓ Production deployment with minimal overhead
When to Use Full Fine-Tuning
- ✓ Abundant compute (8x GPU cluster)
- ✓ Maximum model quality required
- ✓ Smaller models (< 1B parameters)
- ✓ Complete domain shift (e.g., medical to poetry)
🔌 Adapter Layers & PEFT Library
The PEFT library supports multiple parameter-efficient methods beyond LoRA:
- Adapter — Bottleneck modules between layers (fewer params than LoRA)
- Prefix Tuning — Learnable prefix tokens at input
- Prompt Tuning — Virtual tokens (no task-specific architecture changes)
- LoRA — Low-rank weight updates (best balance)
- IA3 — Learned scaling of intermediate activations
🚀 Training with LoRA
⚠️ Common Mistakes
target_modules to limit which layers get trained. If you set all layers trainable, you lose the memory advantage.model.named_modules() to inspect, or check HuggingFace docs for your model family.model.merge_and_unload() to fuse LoRA weights back into the original model.📋 LoRA & PEFT Cheatsheet
| Parameter | Meaning | Typical Value |
|---|---|---|
r |
Low-rank dimension | 8, 16, 32 |
lora_alpha |
Scaling factor (usually 2x r) | 16, 32, 64 |
target_modules |
Which layers to adapt | ["q_proj", "v_proj"] |
lora_dropout |
Dropout in LoRA layers | 0.05, 0.1 |
load_in_4bit |
Enable 4-bit quantization | True / False |
bnb_4bit_quant_type |
Quantization dtype | "nf4" (best), "fp4" |
💼 Interview Q&A
🎯 Test Your Knowledge
✨ Key Takeaways
- LoRA fine-tunes large models with 90% less memory by training only low-rank adapter matrices.
- The rank (r) and lora_alpha parameters control the trade-off between expressiveness and efficiency.
- QLoRA combines 4-bit quantization with LoRA to fit 70B models on a single 24GB GPU.
- Target key projection matrices (q_proj, v_proj) — not all layers need adaptation.
- For production, use model.merge_and_unload() to fuse LoRA weights and eliminate adapter overhead.
You've mastered parameter-efficient fine-tuning.
Ready to deploy production LLMs.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →