👨‍💻
Meet Arjun
Junior ML Engineer at HealthTech Startup in Hyderabad

Arjun is building a domain-specific LLM for medical literature summarization. His team has Llama 3.2 7B as a base model and 5,000 instruction-response pairs in the healthcare domain. Instead of building custom training loops, Arjun chose torchtune—PyTorch's native framework—to leverage recipes, distributed FSDP training, and battle-tested dataset pipelines. He needs to scale across 4 GPUs and evaluate the fine-tuned model on unseen medical text.

Llama 7B
Base model (pretrained)
5.2K
Instruction examples (domain)
4x A100
GPUs (distributed)

🚀 Why torchtune?

Writing LLM training loops from scratch is error-prone and reinvents solutions every time. torchtune is PyTorch's native framework for production LLM fine-tuning. Key advantages:

1. Recipe-Based Flexibility

Pre-built recipes (YAML configs) for LoRA, full fine-tuning, QLoRA, and DPO. No more boilerplate. Swap model, dataset, and hyperparams without touching code.

2. Distributed FSDP Out of the Box

Fully Sharded Data Parallel (FSDP) scales across GPUs/TPUs with automatic memory optimization. Arjun's 7B model trains on 4 A100s without manual pipeline parallelism.

3. Battle-Tested Dataset Pipelines

Handles common formats: instruction-output pairs, chat/conversation data, code. Built-in support for padding, packing, and efficient batching. No custom data loaders.

⚙️ torchtune CLI & Recipes

torchtune uses a CLI-first workflow. You define your training in a recipe (YAML) and run it with the tune command:

Installation and Basic Commands
torchtune ls output
Available recipes: - lora_finetune_single_device - lora_finetune_distributed - full_finetune_single_device - qlora_single_device - dpo_single_device ...

📋 Recipe Structure & Configs

A recipe is a YAML file that defines the entire training pipeline. Here is Arjun's LoRA fine-tuning recipe:

lora_finetune_distributed.yaml
The recipe centralizes all configuration. Arjun can experiment by editing YAML—swap to QLoRA by changing _component_, or scale to 8 GPUs by modifying FSDP settings.

🔗 Distributed Training with FSDP

Fully Sharded Data Parallel (FSDP) partitions model weights and optimizer states across GPUs. torchtune configures FSDP automatically based on your recipe.

Framework comparison: data parallelism, model parallelism, and FSDP efficiency
Bar chart showing throughput for different parallelization strategies

FSDP Sharding Strategies

FULL_SHARD: Fully shard model, optimizer, and gradients. Best memory efficiency, ~4x reduction on 4 GPUs. (Arjun's choice for 7B model on 4x A100s)
SHARD_GRAD_OP: Shard gradients and optimizer states; replicate model. Balanced memory/communication.
NO_SHARD: Data parallel only. Low overhead, but each GPU holds full model. Needed for very large models with limited GPU memory.

Running Arjun's Distributed LoRA Training
FSDP training progress (4 GPUs)
[GPU 0] Initializing FSDP model... [FSDP] Sharding: FULL_SHARD | CPU Offload: False [FSDP] Memory per GPU: ~28 GB (7B model + optim states) Epoch 1/3 | Step 1/500 | Loss: 3.12 | LR: 3e-4 Epoch 1/3 | Step 50/500 | Loss: 2.34 | LR: 3e-4 Epoch 1/3 | Step 100/500 | Loss: 1.98 | LR: 3e-4 ...Saving checkpoint at step 200 Validation loss: 1.87 | BLEU: 42.3

📊 Dataset Preparation & Formats

torchtune supports multiple dataset formats. For instruction tuning, the standard format is JSONL or JSON with instruction, output (and optional input):

healthcare_instructions.json (Arjun's data)

Dataset Processing in torchtune

torchtune's instruct_dataset automatically:
1. Loads JSON and validates format
2. Tokenizes instruction → output pairs
3. Packs multiple examples into max_seq_len (2048 tokens)
4. Applies causal masking (loss only on output, not instruction)
5. Handles padding and sampling efficiently

No manual data loader code needed!

Loading & validating data programmatically

🎯 Evaluation & Inference

After training, Arjun evaluates the fine-tuned model on a held-out validation set using BLEU, ROUGE, and custom metrics:

Evaluate fine-tuned model
Evaluation results
Validation Loss: 1.87 BLEU Score: 42.3 ROUGE-1 (F): 55.8 ROUGE-L (F): 48.2 Inference Example: Prompt: "Summarize acute MI management..." Response: "Acute myocardial infarction requires immediate reperfusion—either percutaneous coronary intervention (PCI) or thrombolysis. Administer dual antiplatelet therapy (aspirin + P2Y12 inhibitor), anticoagulation, and beta-blockers. Risk stratification guides further management."

🔄 LoRA vs Full Fine-Tuning

LoRA (Low-Rank Adaptation)

Adds small rank-8 matrices to attention layers. Trains only ~1% of parameters. Memory: 28 GB (vs 60+ GB for full). Speed: ~1.8x faster. Quality: 95%+ of full fine-tuning.

Full Fine-Tuning

Updates all 7B parameters. Maximum flexibility. Memory: 60+ GB. Speed: slower. Quality: marginal improvement over LoRA (0.5-1 BLEU). Use only if you have abundant compute.

Arjun chose LoRA for his 7B model because it trains in 4 hours on 4 A100s, vs 12+ hours for full fine-tuning. The medical domain adaptation is excellent despite training only 1% of parameters.

🏆 Mini Challenge: Customize a Recipe

Task: Modify Arjun's recipe to use QLoRA (quantized LoRA) instead of regular LoRA. Change the model component to torchtune.models.llama2.qlora_llama2, set quant_cfg: "int4", and reduce batch size to 1 (memory constraint with quantization). Run on a single GPU.

Hint: QLoRA quantizes the base model to int4, reducing memory to ~8 GB. Arjun could then fine-tune on a single A100 with the same quality as multi-GPU LoRA.

💼 Interview Questions

torchtune is PyTorch's native framework for LLM fine-tuning. It provides (1) recipe-based configuration (YAML), eliminating boilerplate, (2) pre-built FSDP distributed training, (3) battle-tested dataset pipelines, (4) support for LoRA, QLoRA, full fine-tuning, DPO, and more. Custom loops are error-prone and reinvent solutions every time. torchtune is battle-tested and production-ready.
A recipe is a YAML file that defines the entire training pipeline: model architecture, LoRA config, dataset, tokenizer, optimizer, learning rate scheduler, FSDP settings, batch size, and checkpointing. You run a recipe with tune run recipe_name --config recipe.yaml. No Python code changes needed—just edit YAML to experiment.
Fully Sharded Data Parallel (FSDP) partitions model weights, optimizer states, and gradients across GPUs. Each GPU holds a shard of the full model. Regular data parallelism replicates the full model on each GPU (memory-inefficient). FSDP reduces memory by 4x on 4 GPUs. Drawback: more communication between GPUs. For large models like 7B+, FSDP is essential.
LoRA (Low-Rank Adaptation) adds small rank-k matrices to attention layers: h = Wx + BAx, where W is frozen pretrained weight, BA is trainable LoRA. Advantages: (1) trains only ~1% of parameters (vs 100% for full fine-tuning), (2) memory reduction: 28 GB vs 60+ GB for 7B model, (3) faster training: ~2x speedup, (4) quality: 95-99% of full fine-tuning (often imperceptible difference). Drawback: less flexible than full fine-tuning, but tradeoff is worthwhile for most use cases.
(1) Use inference-only FSDP (not training FSDP)—distribute model across GPUs but keep batch inference on single GPU for lower latency. (2) Quantize the model post-training (int8, int4 with bitsandbytes). (3) Use flash attention (Triton kernels) for faster attention computation. (4) Batch requests (accumulate inference queries, run together). (5) Profile with PyTorch profiler to find bottlenecks—often communication between GPUs. (6) For production, convert to optimized format (torch.compile, ONNX). (7) Use batched beam search instead of greedy decoding if quality permits.

✓ Quick Quiz

Question 1 / 5
You need to fine-tune a 7B model on 4 A100 GPUs with a custom healthcare dataset. Should you use LoRA or full fine-tuning?
Question 2 / 5
What does FSDP's FULL_SHARD strategy do?
Question 3 / 5
How does torchtune's packing feature improve dataset efficiency?
Question 4 / 5
What is the memory advantage of LoRA over full fine-tuning for a 7B LLM?
Question 5 / 5
Your fine-tuned healthcare model performs well on your validation set but performs poorly on real-world medical notes. What is the likely issue?
0

📋 Quick Cheatsheet

Task Command / Code Notes
Install torchtune pip install torchtune Requires PyTorch 2.0+
Download model tune download meta-llama/Llama-2-7b-hf Downloads to ~/.cache/torchtune
List recipes tune ls See all available recipes
Run LoRA (single GPU) tune run lora_finetune_single_device --config recipe.yaml For dev/prototyping
Run LoRA (4 GPUs) tune run --nproc_per_node 4 lora_finetune_distributed --config recipe.yaml Uses FSDP automatically
Custom dataset source: json | data_files: /path.json Format: {instruction, output, input} per example
Pack sequences packed: true Concatenates examples, reduces padding waste
LoRA config lora_rank: 8, lora_alpha: 16, dropout: 0.05 Typical values; adjust for your task
FSDP strategy sharding_strategy: FULL_SHARD Best memory efficiency; use for 7B+ models
Inference model.generate(tokens, max_seq_len=2048, temperature=0.7) Use load_checkpoint() to load fine-tuned weights

⚠️ Common Mistakes

Using the same learning rate for LoRA and base model
LoRA is trainable while the base model is frozen. Use default LoRA LR ~3e-4, matching typical head fine-tuning rates. Do not use 1e-5 (too slow) or 1e-2 (too aggressive).
Ignoring data format validation
Check your JSON has required keys: instruction and output. Missing or misnamed fields cause silent failures. Use torchtune.datasets.validate_instruct_dataset() before training.
Not setting max_seq_len correctly
Default 2048 is fine for most LLMs, but check your model's max position embeddings. Set too low and you truncate examples; too high and you waste memory. Profile: print token lengths of your dataset.
Disabling packing for small datasets
Packing concatenates examples into full sequences, reducing padding and improving GPU utilization. Always enable for instruction tuning: packed: true.
Forgetting to adjust loss masking
torchtune automatically masks loss on instruction tokens (you only learn from output). Verify your labels array: -100 for instruction, actual loss value for output. Custom datasets must handle this.

✦ Key Takeaways

  • torchtune is PyTorch's native LLM fine-tuning framework. Use recipes (YAML) instead of custom training loops. No boilerplate, battle-tested, production-ready.
  • Recipes define everything: model, dataset, tokenizer, optimizer, FSDP, batch size, checkpointing. Edit YAML to experiment—no Python code changes.
  • FSDP partitions model across GPUs, reducing memory by 4x on 4 GPUs (FULL_SHARD strategy). Essential for fine-tuning 7B+ models.
  • LoRA trains only 1% of parameters, reducing memory from 60+ GB to 28 GB, and training time by ~2x. Quality is 95-99% of full fine-tuning—preferred for most use cases.
  • torchtune datasets handle JSON instruction-output pairs, automatic tokenization, packing, and loss masking. No custom data loaders.
  • Arjun's healthcare model (LoRA + FSDP on 4 A100s) trains in 4 hours and achieves strong BLEU/ROUGE on medical examples. Domain drift in production is the main challenge—retrain on real data if needed.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You've mastered torchtune.
Now build production LLMs with distributed training.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →