Master PyTorch DataLoaders, batch size tuning, and data augmentation. Samuel at AgriSense (Kisumu): 2,000 crop disease images, batch 32 = 87.24% accuracy, without augmentation = overfitting, with augmentation = robust generalization.
📦 Batching🔄 Data Augmentation🎯 Pipelines🔧 Advanced📖 ~20 min
Lesson 09 of Deep Learning Track (Arc B)
🌾
Real-world scenario
Samuel at AgriSense (Kisumu, Kenya): building a crop disease classifier from 2,000 phone photos. Small dataset + augmentation = production model.
Samuel is an agricultural data scientist at AgriSense in Kisumu, Kenya. His task: classify crop diseases (3 types) from low-resolution phone photos taken by farmers in the field. Challenge: only 2,000 labeled images total (6 months of farmer uploads). Using batch size 32, his accuracy reaches 87.24%. Without augmentation, the model overfits—training loss 0.2542, validation loss explodes to 0.6+ (poor generalization). Adding augmentation (horizontal flip, rotation, crop) increases training loss to 0.4068 (learning slower) but validation loss drops to 0.48 (robust to unseen variations). Batching and augmentation are the lifeline for real-world small datasets.
2,000
Images (small)
KSh 450K
Monthly budget
87.24%
Accuracy (batch 32)
PyTorch Dataset & DataLoader: the training pipeline foundation
Why batching? Training on one sample at a time is slow. Training on all 2,000 at once requires huge memory. Batching (e.g., 32 samples per step) balances speed and stability. Gradients average across the batch, reducing noise and improving convergence.
Dataset class: Wraps your data (images, labels). Implements __len__ (total samples) and __getitem__ (fetch one sample + apply transforms).
DataLoader: Iterator that combines Dataset with batching, shuffling, and multi-worker loading. Transparently feeds batches to your training loop.
Transforms: Preprocessing applied per-sample. Training transforms include augmentation (RandomFlip, Rotate); validation transforms only normalize (no augmentation).
Creating a custom Dataset class
Start with a simple Dataset wrapping a CSV (image paths + labels). Fashion-MNIST example (60K training, 10K test, 28x28 grayscale, 10 classes):
custom_dataset.pyCustom Dataset class for Fashion-MNIST
Output
Training samples: 60000
Test samples: 10000
Sample: (tensor(28x28), label=9)
DataLoader wraps your Dataset and handles batching, shuffling, and parallelization. Key parameters:
dataloader_setup.pyCreate DataLoaders for training and validation
Output
Training batches per epoch: 1563
Val batches per epoch: 79
Batch shape: (32, 1, 28, 28)
Labels shape: (32,)
Label values: [3, 5, 9, 1, 0]
Key takeaway: DataLoader automatically batches data, shuffles, and loads in parallel. With num_workers=4, 4 CPU processes prefetch the next batch while the GPU trains—no idle time.
Batch size effect: accuracy trade-off
Batch size is critical. Smaller batches = noisier gradients (more updates per epoch, may oscillate). Larger batches = smoother gradients (fewer updates, faster per-batch).
Fashion-MNIST accuracy vs batch size (10 epochs, Adam lr=0.001). Batch 32 achieves 87.24%, batch 512 drops to 84.81%.
Sweet spot: batch 32. Larger batches (256, 512) train fewer steps per epoch—less opportunity to converge to good minima. Batch 32 = 1,875 steps per epoch vs batch 512 = 117 steps. More steps = more updates = better accuracy.
Image transforms with torchvision.transforms
Transforms are applied per-sample, before batching. Compose chains multiple transforms. Training transforms include augmentation; validation uses only normalization.
Data augmentation: RandomHorizontalFlip, RandomRotation, RandomCrop
Augmentation increases dataset diversity without collecting more images. Randomly flip, rotate, crop during training. Validation uses original images only (no augmentation).
augmentation_transforms.pyTraining with augmentation, validation without
Output
Training with augmentation (flip, rotate, crop)
Validation without augmentation (original images)
Augmentation effect: train loss vs validation robustness
Without augmentation, training loss converges quickly but validation loss diverges (overfitting). With augmentation, training loss converges slower (noisier samples) but validation loss stays low (robust).
Fashion-MNIST: 10 epochs with/without augmentation. Without: train loss 0.2542 (tight fit). With: train loss 0.4068 (loose fit, better generalization).
Augmentation trades training loss for generalization. Train loss increases 60% (0.2542 → 0.4068), but validation performance is more robust. For small datasets (2,000 images), this is essential.
Sample Fashion-MNIST images: 20 random samples, 2 per class. Note low resolution and varied orientations.
How AI systems use this
Every production ML model trains on batches. ImageNet classifiers use batch 256 (distributed across 8 GPUs). Language models use batch 1,024+. Batch size is tuned empirically: too small is slow, too large diverges or overfits (fewer updates per epoch). Data augmentation is standard practice: crop, flip, color jitter, cutmix, mixup. Samuel's crop disease classifier at AgriSense uses batch 32 + heavy augmentation because 2,000 images is small. In contrast, medical imaging (CT scans) often uses light augmentation (flip only, no rotation) to preserve anatomical integrity. The principle is universal: batching + augmentation = robust, efficient training.
Data PipelineBatchingAugmentationOverfitting Prevention
🎯 Mini Challenge: batch size shootout
Task: Train a simple CNN on Fashion-MNIST with batch sizes 16, 32, 64, 128, 256 for 5 epochs. Plot accuracy vs batch size. Which batch size converges fastest (lowest validation loss after 5 epochs)? Why?
Hint 1: Smaller batches = more updates per epoch. 60K samples ÷ 16 = 3,750 batches vs ÷ 256 = 234 batches.
Hint 2: Track validation loss, not training loss. Larger batches may have lower train loss but worse validation (overfitting).
Expected: Batch 32-64 typically wins on Fashion-MNIST. Trade-off: batch 16 may be too noisy, batch 256 too few updates.
Interview questions: test your understanding
Batch 32 computes 1,875 weight updates per epoch (60K ÷ 32). Batch 512 computes only 117 updates. More updates = more optimization steps = better convergence. Larger batches have smoother gradients but fewer chances to escape bad minima.
Shuffling randomizes batch composition each epoch. If images are ordered by class (all shirts, then all pants), early batches have different distributions than later batches, confusing the optimizer. Shuffling ensures each batch is a representative mini-sample of the dataset.
Training augmentation increases dataset diversity: the model sees the same logical image (e.g., a flipped shirt) in many forms, learning invariant features. Validation without augmentation tests true generalization: can the model classify unmodified images? Augmenting validation would artificially inflate accuracy metrics.
Batch 8. With 2,000 images, batch 8 = 250 updates per epoch vs batch 128 = 16 updates. Small datasets need more frequent updates. Batch 8 is noisier but has more optimization opportunities. Large batches risk underfitting on small datasets. However, batch 8 needs careful learning rate tuning (may diverge with standard lr).
(1) Add aggressive augmentation: RandomRotation(45°), RandomAffine, ColorJitter. Increase dataset diversity. (2) Increase batch size to 64/128 to regularize (fewer updates, smoother gradients). (3) Add dropout or L2 regularization. Root cause: overfitting on 60K is usually solvable with augmentation + regularization, not architecture changes.
Quiz: 5 questions to test your knowledge
Question 1 of 5
Batch size 32 achieved what accuracy on Fashion-MNIST?
Question 2 of 5
Samuel at AgriSense has 2,000 crop images. Why is data augmentation critical for him?
Question 3 of 5
With 60K samples and batch 32, how many updates per epoch?
Question 4 of 5
Why is shuffling important in the training DataLoader?
Question 5 of 5
Training loss with augmentation (0.4068) is higher than without (0.2542). Is this a problem?
0
/5
Excellent! You understand DataLoaders and augmentation.
Cheatsheet: DataLoaders & augmentation at a glance
Randomizes batch order each epoch. Prevents order bias, improves convergence.
No shuffle val
shuffle=False
Deterministic validation. Reproducible results, no impact on accuracy.
Augmentation (train)
RandomHorizontalFlip, RandomRotation
Increase diversity. Train loss ↑, validation loss ↓. Essential for small datasets.
No augment (val)
Only normalize
Test true generalization. Original images only.
Normalize
Normalize((mean,), (std,))
Scale inputs to zero-mean, unit-variance. Improves training stability.
num_workers
num_workers=4
4 CPU processes prefetch batches. GPU never idles. Speed ↑ 2-3x.
Common mistakes: what breaks and how to fix it
Mistake 1: Using very large batch sizes (256+) on small datasets
Few updates per epoch = underfitting. 2,000 images with batch 256 = 8 updates/epoch. Use batch 16-32 instead (125-250 updates). Or use gradient accumulation: accumulate gradients over 8 batches of size 32, then update.
Mistake 2: Forgetting to set shuffle=False on validation DataLoader
Shuffled validation can cause non-deterministic results. If validation is deterministic, you can detect overfitting reliably. Always set shuffle=False for validation/test.
Mistake 3: Applying augmentation to both training and validation
Validation should test unmodified images. Augmenting validation inflates accuracy metrics. Create separate transforms: train_transform (with augmentation), val_transform (normalization only).
Mistake 4: Using num_workers=0 (single CPU process)
Slow! CPU loads one batch while GPU trains (bottleneck). Use num_workers=4 or CPU_COUNT//2. Parallel loading hides loading latency. Minor boost on SSDs, major on spinning disks.
Key takeaways
DataLoaders and augmentation are the unglamorous heroes of deep learning. Samuel's crop disease classifier wouldn't work with raw 2,000 images—batching + augmentation unlocks 87.24% accuracy.
Batching is fundamental: Batch size 32 achieves 87.24%, batch 512 only 84.81%. More updates per epoch = better convergence.
Augmentation beats more data: 2,000 images + augmentation + generalization > 2,000 raw images + overfitting. Flip, rotate, crop are free data.
Training ≠ validation: Separate transforms. Train with augmentation (learn invariant features). Validate on original images (test robustness).
Shuffling matters: Randomize batch order each epoch. Prevents order bias (e.g., all shoes first, then all shirts).
num_workers matters: Parallel loading hides I/O latency. GPU never idles. 2-3x speedup on SSDs, 10x+ on spinning disks.
Normalize inputs: Scale to zero-mean, unit-variance. Stabilizes training, makes learning rates more universal.