Arjun is building a domain-specific LLM for medical literature summarization. His team has Llama 3.2 7B as a base model and 5,000 instruction-response pairs in the healthcare domain. Instead of building custom training loops, Arjun chose torchtune—PyTorch's native framework—to leverage recipes, distributed FSDP training, and battle-tested dataset pipelines. He needs to scale across 4 GPUs and evaluate the fine-tuned model on unseen medical text.
🚀 Why torchtune?
Writing LLM training loops from scratch is error-prone and reinvents solutions every time. torchtune is PyTorch's native framework for production LLM fine-tuning. Key advantages:
1. Recipe-Based Flexibility
Pre-built recipes (YAML configs) for LoRA, full fine-tuning, QLoRA, and DPO. No more boilerplate. Swap model, dataset, and hyperparams without touching code.
2. Distributed FSDP Out of the Box
Fully Sharded Data Parallel (FSDP) scales across GPUs/TPUs with automatic memory optimization. Arjun's 7B model trains on 4 A100s without manual pipeline parallelism.
3. Battle-Tested Dataset Pipelines
Handles common formats: instruction-output pairs, chat/conversation data, code. Built-in support for padding, packing, and efficient batching. No custom data loaders.
⚙️ torchtune CLI & Recipes
torchtune uses a CLI-first workflow. You define your training in a recipe (YAML) and run it with the tune command:
📋 Recipe Structure & Configs
A recipe is a YAML file that defines the entire training pipeline. Here is Arjun's LoRA fine-tuning recipe:
_component_, or scale to 8 GPUs by modifying FSDP settings.
🔗 Distributed Training with FSDP
Fully Sharded Data Parallel (FSDP) partitions model weights and optimizer states across GPUs. torchtune configures FSDP automatically based on your recipe.
FSDP Sharding Strategies
FULL_SHARD: Fully shard model, optimizer, and gradients. Best memory efficiency, ~4x reduction on 4 GPUs. (Arjun's choice for 7B model on 4x A100s)
SHARD_GRAD_OP: Shard gradients and optimizer states; replicate model. Balanced memory/communication.
NO_SHARD: Data parallel only. Low overhead, but each GPU holds full model. Needed for very large models with limited GPU memory.
📊 Dataset Preparation & Formats
torchtune supports multiple dataset formats. For instruction tuning, the standard format is JSONL or JSON with instruction, output (and optional input):
Dataset Processing in torchtune
torchtune's instruct_dataset automatically:
1. Loads JSON and validates format
2. Tokenizes instruction → output pairs
3. Packs multiple examples into max_seq_len (2048 tokens)
4. Applies causal masking (loss only on output, not instruction)
5. Handles padding and sampling efficiently
No manual data loader code needed!
🎯 Evaluation & Inference
After training, Arjun evaluates the fine-tuned model on a held-out validation set using BLEU, ROUGE, and custom metrics:
🔄 LoRA vs Full Fine-Tuning
LoRA (Low-Rank Adaptation)
Adds small rank-8 matrices to attention layers. Trains only ~1% of parameters. Memory: 28 GB (vs 60+ GB for full). Speed: ~1.8x faster. Quality: 95%+ of full fine-tuning.
Full Fine-Tuning
Updates all 7B parameters. Maximum flexibility. Memory: 60+ GB. Speed: slower. Quality: marginal improvement over LoRA (0.5-1 BLEU). Use only if you have abundant compute.
Arjun chose LoRA for his 7B model because it trains in 4 hours on 4 A100s, vs 12+ hours for full fine-tuning. The medical domain adaptation is excellent despite training only 1% of parameters.
Task: Modify Arjun's recipe to use QLoRA (quantized LoRA) instead of regular LoRA. Change the model component to torchtune.models.llama2.qlora_llama2, set quant_cfg: "int4", and reduce batch size to 1 (memory constraint with quantization). Run on a single GPU.
Hint: QLoRA quantizes the base model to int4, reducing memory to ~8 GB. Arjun could then fine-tune on a single A100 with the same quality as multi-GPU LoRA.
💼 Interview Questions
tune run recipe_name --config recipe.yaml. No Python code changes needed—just edit YAML to experiment.
✓ Quick Quiz
📋 Quick Cheatsheet
| Task | Command / Code | Notes |
|---|---|---|
| Install torchtune | pip install torchtune |
Requires PyTorch 2.0+ |
| Download model | tune download meta-llama/Llama-2-7b-hf |
Downloads to ~/.cache/torchtune |
| List recipes | tune ls |
See all available recipes |
| Run LoRA (single GPU) | tune run lora_finetune_single_device --config recipe.yaml |
For dev/prototyping |
| Run LoRA (4 GPUs) | tune run --nproc_per_node 4 lora_finetune_distributed --config recipe.yaml |
Uses FSDP automatically |
| Custom dataset | source: json | data_files: /path.json |
Format: {instruction, output, input} per example |
| Pack sequences | packed: true |
Concatenates examples, reduces padding waste |
| LoRA config | lora_rank: 8, lora_alpha: 16, dropout: 0.05 |
Typical values; adjust for your task |
| FSDP strategy | sharding_strategy: FULL_SHARD |
Best memory efficiency; use for 7B+ models |
| Inference | model.generate(tokens, max_seq_len=2048, temperature=0.7) |
Use load_checkpoint() to load fine-tuned weights |
⚠️ Common Mistakes
torchtune.datasets.validate_instruct_dataset() before training.packed: true.✦ Key Takeaways
- torchtune is PyTorch's native LLM fine-tuning framework. Use recipes (YAML) instead of custom training loops. No boilerplate, battle-tested, production-ready.
- Recipes define everything: model, dataset, tokenizer, optimizer, FSDP, batch size, checkpointing. Edit YAML to experiment—no Python code changes.
- FSDP partitions model across GPUs, reducing memory by 4x on 4 GPUs (FULL_SHARD strategy). Essential for fine-tuning 7B+ models.
- LoRA trains only 1% of parameters, reducing memory from 60+ GB to 28 GB, and training time by ~2x. Quality is 95-99% of full fine-tuning—preferred for most use cases.
- torchtune datasets handle JSON instruction-output pairs, automatic tokenization, packing, and loss masking. No custom data loaders.
- Arjun's healthcare model (LoRA + FSDP on 4 A100s) trains in 4 hours and achieves strong BLEU/ROUGE on medical examples. Domain drift in production is the main challenge—retrain on real data if needed.
You've mastered torchtune.
Now build production LLMs with distributed training.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →