🇬🇭
Meet Kwame
NLP Researcher in Accra, Ghana

Kwame is building a conversational AI for customer support in Twi (Ghanaian language) and English. He has collected 500 instruction-response pairs (mixed language support). His team wants a lightweight model that runs on edge devices (inference on phones/tablets). He chose Llama 3.2 1B (1.3B parameters) with LoRA fine-tuning—trainable on a single GPU, deployable on phones, and capable of domain-specific adaptation. This project walks through the entire pipeline: data preparation, training with torchtune, evaluation, and inference.

500
Instruction examples (domain-specific)
1.3B
Model size (lightweight)
~4h
Training time (1x GPU)

📋 Project Overview & Architecture

This mini project is split into 4 phases:

Phase 1: Data Preparation

Create or collect 500 instruction-response pairs. Format as JSON. Validate format, check for duplicates, explore length distributions. Output: instruction-tuning-500.json

Phase 2: Training Setup & Configuration

Download Llama 3.2 1B. Write a LoRA config (rank=8, alpha=16). Define optimizer, batch size, learning rate. Prepare a torchtune recipe or training script.

Phase 3: Training & Monitoring

Fine-tune on your 500 examples for 3-5 epochs. Monitor loss curves, validation metrics. Save checkpoints. Track training time and GPU memory usage.

Phase 4: Evaluation & Inference

Evaluate on held-out test set (100 examples). Compute BLEU, ROUGE-L. Run inference on new prompts. Compare before/after model outputs. Showcase improvements.

🔧 Phase 1: Data Preparation

Start by creating your instruction dataset. Here is Kwame's template for customer support scenarios:

Create instruction-tuning-500.json
Dataset validation output
Created 500 examples in instruction-tuning-500.json Dataset stats: Instruction length: 3-12 words Output length: 15-85 words Total examples: 500 Unique instructions: 498 (minimal duplication)
Distribution of instruction and output lengths (500 examples)
Histogram showing word count distribution for instructions and outputs
Instructions average 6-8 words, outputs average 35-45 words. This is typical for Q&A/customer support. Your model will learn to produce short questions and longer, detailed answers.

⚙️ Phase 2: Training Setup & LoRA Config

Configure LoRA parameters and training hyperparameters. For Llama 3.2 1B with 500 examples:

lora_config.py - LoRA Configuration

Why These LoRA Settings?

rank=8: Low rank for 1B model. Rank 16 or higher risks overfitting on 500 examples. rank=8 adds ~655K trainable params (0.05% of model).
lora_alpha=16: Standard scaling (alpha/rank = 2). Controls LoRA contribution magnitude.
target_modules=[q_proj, v_proj]: Query and value projections capture attention patterns. Including MLP would add more params but marginal quality gain for small dataset.

🚀 Phase 3: Training Loop

Use torchtune or Hugging Face Trainer. Here is the Trainer approach (simpler for small projects):

train.py - Fine-Tuning Script
Training progress (3 epochs)
Epoch 1/3 Step 20 | Loss: 3.12 | LR: 2.5e-4 Step 40 | Loss: 2.87 | LR: 2.9e-4 Step 60 | Loss: 2.54 | Eval Loss: 2.31 | BLEU: 28.4 ... Epoch 2/3 | Loss: 2.18 | Eval Loss: 1.95 | BLEU: 35.2 ... Epoch 3/3 | Loss: 1.87 | Eval Loss: 1.73 | BLEU: 39.8 Final checkpoint saved to ./llama-1b-finetuned Total training time: 3h 47min (1x A100 GPU)
Training and validation loss curves across 3 epochs
Line graph showing loss decreasing over epochs
Loss decreases smoothly from 3.12 to 1.73 (training) and 2.31 to 1.73 (validation). No overfitting—validation tracks training closely. Stable learning indicates good hyperparameter choices.

📊 Phase 4: Evaluation & Inference

Evaluate the fine-tuned model on test set using BLEU, ROUGE-L, and manual inspection:

evaluate.py - Compute Metrics
Evaluation metrics
BLEU-4 Score: 0.3542 ROUGE-L (F-measure): 0.5231 Sample predictions (test set): Instruction: "How do I reset my password?" Reference: "To reset your password, go to the login page..." Generated: "Go to the login page, click 'Forgot Password', follow email steps." Instruction: "What payment methods do you accept?" Reference: "We accept Visa, Mastercard, PayPal, MTN Mobile Money..." Generated: "We accept major credit cards (Visa, Mastercard), PayPal, and mobile money."
Before vs After fine-tuning: comparison of base model vs fine-tuned outputs
Side-by-side comparison showing improvement in response quality
Base model (zero-shot) generates generic or irrelevant responses. Fine-tuned model produces domain-specific, helpful answers matching customer support context. BLEU/ROUGE improvements confirm better semantic alignment with reference answers.

🚀 Inference & Deployment

Deploy your fine-tuned model for inference. For edge devices (phones), use quantization:

inference.py - Inference Script
Inference output (live)
=== Fine-Tuned Llama 1B Inference === Q: How do I track my order? A: Check your order confirmation email for a tracking link. Status updates will be sent via SMS and email. Processing typically takes 1-2 business days. Q: What is your refund policy? A: We offer a 30-day return policy. Items must be unused and in original packaging. Refunds process in 5-10 business days after we receive your return. Q: Do you ship internationally? A: Yes, we ship to 15 West African countries. Standard shipping (7-10 days) and Express (3-5 days) available. Free shipping on orders over GHC500. Q: I forgot my password. What should I do? A: Visit the login page and click "Forgot Password". You'll receive an email with reset instructions. Check your spam folder if you don't see it.
🏆 Mini Challenge: Extend the Dataset

Task: Collect 100 more instruction-response pairs in your domain (make them realistic—order status, billing issues, product questions). Retrain the model with 600 total examples (500 train, 100 test). Measure if BLEU and ROUGE improve.

Hint: More diverse data typically improves generalization. Check if the model generalizes to questions it hasn't seen before (e.g., variations of "How do I track my order?").

💼 Interview Questions

Instruction tuning fine-tunes a pretrained LLM on instruction-response pairs to teach it to follow domain-specific commands and tasks. It's important because pretrained models are general-purpose; instruction tuning adapts them to specific use cases (customer support, legal analysis, etc.). With just 500 examples, you can build a domain-expert model that outperforms zero-shot base models by 50-80% on quality metrics.
LoRA trains only 0.05% of model parameters (vs 100% for full fine-tuning). Benefits: (1) Memory: 28 GB → 8 GB for 7B models. (2) Speed: ~2x faster training. (3) Portability: Save only 10 MB adapter weights, not full 7 GB model. (4) Quality: 95%+ of full fine-tuning. For 500-5000 examples, LoRA is preferred because it prevents overfitting while maintaining performance.
BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between generated and reference text (precision-focused). Ranges 0-1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures recall of n-grams and longest common subsequences. BLEU is better for translation; ROUGE for summarization. For question-answering, ROUGE-L (longest common subsequence) is more forgiving—exact phrasing varies but semantic meaning is key. Typical values: BLEU 0.25-0.4, ROUGE-L 0.45-0.65 for customer support.
Domain shift: Production queries differ from training (e.g., typos, slang, longer context). Data leakage: Test set is too similar to training (not held-out). Model brittleness: 500 examples may not cover edge cases (unusual questions, requests in Twi vs English). Solutions: (1) Collect production failures and retrain. (2) Create diverse test set. (3) Add data augmentation (paraphrase, backslation). (4) Use ensemble with keyword fallback. (5) Monitor in production with metrics.
(1) Quantization: int8 or int4 reduces model size from 2.6 GB to 1 GB (int8) or 650 MB (int4). Minimal accuracy loss (1-2%). (2) Pruning: Remove low-importance weights, further reducing size. (3) Knowledge distillation: Train a smaller model to mimic fine-tuned model, then quantize. (4) Inference optimization: Use llama.cpp or MLX (Apple Silicon), optimized for mobile. (5) On-device caching: Cache responses to common queries. (6) Hybrid approach: Simple classifier selects predefined templates for 80% of queries (customer support), use LLM for 20% (complex). Final deployment: ~200-500 MB on-device, <1 sec inference on modern phones.

✓ Quick Quiz

Question 1 / 5
You have 500 labeled instruction-response pairs. How should you split them for training and evaluation?
Question 2 / 5
What LoRA parameters are appropriate for a 1B model with 500 examples?
Question 3 / 5
Why should you mask loss on instruction tokens (label them -100)?
Question 4 / 5
Your fine-tuned model achieves BLEU 0.35 on the test set. Is this good?
Question 5 / 5
To deploy your fine-tuned Llama 1B to phones (2 GB limit), what is your strategy?
0

📋 Quick Cheatsheet

Task Command / Code Notes
Create dataset json.dump(examples, open("data.json", "w")) Format: {instruction, output} per example
Load base model AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B") Requires HF token for Llama weights
Apply LoRA LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"]) 0.05% trainable params for 1B model
Train (HF Trainer) Trainer(model, args, train_dataset, data_collator).train() Handles loss masking, logging, checkpoint saving
Compute BLEU corpus_bleu(references, hypotheses) Use weights=(0.25, 0.25, 0.25, 0.25) for BLEU-4
Compute ROUGE-L rouge_scorer.RougeScorer(['rougeL']).score(ref, hyp) Better for Q&A; less sensitive to phrasing
Generate inference model.generate(input_ids, max_new_tokens=100, temperature=0.7) Use do_sample=True for diversity
Quantize (int8) from_pretrained(..., load_in_8bit=True) Reduces 2.6 GB → 1 GB with 1-2% loss
Save LoRA adapter model.save_pretrained("adapter_weights") Saves 10-50 MB; merge with base model for inference
Loss masking labels[:, :instruction_len] = -100 Prevents loss on input; only output tokens contribute

⚠️ Common Mistakes

Not masking instruction tokens during loss computation
If you compute loss on instruction tokens, the model learns to predict instructions (copying input), not generate novel responses. Always set labels[:, :instruction_len] = -100 to mask instruction tokens.
Using too high learning rate for LoRA
LoRA is learning adapters on top of frozen weights. Learning rate 1e-3 (standard) is safe. If loss explodes, reduce to 5e-4 or 3e-4. High LR can corrupt adapter weights and cause poor fine-tuning.
Not splitting train/test before data inspection
If you inspect all 500 examples, then split, test set leakage occurs—model has seen examples it evaluates on. Always split first, then explore only training data.
Relying on BLEU alone for evaluation
BLEU is n-gram overlap and can be misleading (e.g., paraphrasing gets low BLEU despite high semantic quality). Use BLEU + ROUGE-L + manual evaluation.
Ignoring class imbalance in instruction types
If your 500 examples are 400 "How do I..." and 100 "Tell me about...", the model may overfit to "How" questions. Ensure diverse instruction types and evaluate per category.

✦ Key Takeaways

  • Instruction tuning adapts pretrained LLMs to domain-specific tasks with 500-5000 examples. Results in 50-80% quality improvement over zero-shot.
  • LoRA fine-tunes only 0.05% of parameters, reducing memory and training time by 60-70%. Quality is 95%+ of full fine-tuning for small datasets.
  • Always mask loss on instruction tokens—force the model to generate novel outputs, not copy inputs. Set labels to -100 for instruction positions.
  • Evaluate with multiple metrics: BLEU (n-gram overlap), ROUGE-L (recall-focused), and manual inspection. No single metric captures quality.
  • Deploy to mobile with int8 quantization (2.6 GB → 1 GB) for on-device inference. Balance accuracy and size for edge devices.
  • Kwame's fine-tuned Llama 1B achieves BLEU 0.35 + ROUGE-L 0.52 on customer support Q&A, generalization to unseen questions, and 4-hour training on 1x GPU.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've built a production-grade fine-tuned LLM.
Now master advanced architectures and deployment.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →