Tensor Creation & Basics
| Operation | Code | Description |
|---|---|---|
| zeros | torch.zeros(2, 3) | Create tensor of zeros with shape [2, 3] |
| ones | torch.ones(2, 3) | Create tensor of ones |
| rand | torch.rand(2, 3) | Uniform random [0, 1) |
| randn | torch.randn(2, 3) | Normal distribution N(0, 1) |
| arange | torch.arange(0, 10, step=2) | Create sequence [0, 2, 4, 6, 8] |
| linspace | torch.linspace(0, 1, 5) | Create 5 evenly spaced values [0 to 1] |
| tensor | torch.tensor([[1, 2], [3, 4]]) | Create from Python list |
| from_numpy | torch.from_numpy(numpy_array) | Convert numpy array to tensor |
python Basic tensor operations
Building Models with nn.Module
nn.Module Pattern
All PyTorch models inherit from nn.Module. Define layers in __init__ and forward pass logic in forward().
python Minimal nn.Module example
| Layer | Code | Input → Output |
|---|---|---|
| Linear | nn.Linear(10, 64) | [batch, 10] → [batch, 64] |
| Conv2d | nn.Conv2d(3, 16, kernel_size=3) | [batch, 3, H, W] → [batch, 16, H', W'] |
| LSTM | nn.LSTM(64, 128, num_layers=2) | [batch, seq_len, 64] → [batch, seq_len, 128] |
| Embedding | nn.Embedding(vocab_size, 64) | [batch, seq_len] → [batch, seq_len, 64] |
| BatchNorm1d | nn.BatchNorm1d(64) | Normalize features across batch |
| Dropout | nn.Dropout(0.3) | Drop 30% of activations (train only) |
Training Loop Pattern
python Complete training loop
DataLoader & Dataset
python Custom Dataset class
| Parameter | Example | Description |
|---|---|---|
| batch_size | 32 | Samples per batch |
| shuffle | True | Randomize order each epoch (for training) |
| num_workers | 4 | Parallel data loading processes |
| pin_memory | True | Pin to GPU memory (faster transfer) |
| drop_last | True | Drop last incomplete batch |
GPU & CUDA Operations
python GPU & device management
Loss Functions & Optimization
| Task | Loss Function | Usage |
|---|---|---|
| Multi-class classification | nn.CrossEntropyLoss() | Output: logits [batch, num_classes]. Input labels: [batch] (class indices) |
| Binary classification | nn.BCEWithLogitsLoss() | Output: logits [batch, 1]. Input labels: [batch, 1] (0 or 1) |
| Regression | nn.MSELoss() | Mean squared error between predictions and targets |
| Ranking/Margin | nn.MarginRankingLoss() | Ranking losses for contrastive learning |
| Optimizer | Code | When to Use |
|---|---|---|
| SGD | optim.SGD(params, lr=0.01, momentum=0.9) | Stable, good for fine-tuning |
| Adam | optim.Adam(params, lr=0.001) | Default choice; adaptive learning rates |
| AdamW | optim.AdamW(params, lr=0.001) | Adam with weight decay (better regularization) |
| RMSprop | optim.RMSprop(params, lr=0.001) | Good for RNNs |
Saving & Loading Models
python Save and load checkpoints
Common Patterns & Best Practices
Pattern 1: Evaluation Mode
Always set model.eval() before evaluation and model.train() before training. This disables dropout and batch norm updates.
python Evaluation pattern
Pattern 2: Gradient Accumulation
Process multiple batches before updating weights (useful when batch size is limited by GPU memory).
python Gradient accumulation
Pattern 3: Learning Rate Scheduling
Decay learning rate during training for better convergence.
python Learning rate scheduling
Pattern 4: Early Stopping
Stop training if validation loss doesn't improve.
python Early stopping
Debugging & Common Issues
| Issue | Solution |
|---|---|
| RuntimeError: Expected all tensors on same device | Move all tensors to same device: x.to(device), model.to(device) |
| RuntimeError: grad can only be implicitly created for scalar outputs | Loss must be scalar. Use loss.backward() not loss[0].backward() |
| CUDA out of memory | Reduce batch size, use gradient accumulation, clear cache: torch.cuda.empty_cache() |
| NaN or Inf in loss | Reduce learning rate, check for numerical issues in data, clip gradients |
| Model not learning (loss stuck) | Check learning rate (too high or too low), verify data normalization, inspect gradients |
python Inspect gradients & debug
Africa & India's Best AI Academy
Master PyTorch with production-grade expertise.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →