Tensor Creation & Basics

Operation Code Description
zeros torch.zeros(2, 3) Create tensor of zeros with shape [2, 3]
ones torch.ones(2, 3) Create tensor of ones
rand torch.rand(2, 3) Uniform random [0, 1)
randn torch.randn(2, 3) Normal distribution N(0, 1)
arange torch.arange(0, 10, step=2) Create sequence [0, 2, 4, 6, 8]
linspace torch.linspace(0, 1, 5) Create 5 evenly spaced values [0 to 1]
tensor torch.tensor([[1, 2], [3, 4]]) Create from Python list
from_numpy torch.from_numpy(numpy_array) Convert numpy array to tensor
python Basic tensor operations

Building Models with nn.Module

nn.Module Pattern

All PyTorch models inherit from nn.Module. Define layers in __init__ and forward pass logic in forward().

python Minimal nn.Module example
Layer Code Input → Output
Linear nn.Linear(10, 64) [batch, 10] → [batch, 64]
Conv2d nn.Conv2d(3, 16, kernel_size=3) [batch, 3, H, W] → [batch, 16, H', W']
LSTM nn.LSTM(64, 128, num_layers=2) [batch, seq_len, 64] → [batch, seq_len, 128]
Embedding nn.Embedding(vocab_size, 64) [batch, seq_len] → [batch, seq_len, 64]
BatchNorm1d nn.BatchNorm1d(64) Normalize features across batch
Dropout nn.Dropout(0.3) Drop 30% of activations (train only)

Training Loop Pattern

python Complete training loop

DataLoader & Dataset

python Custom Dataset class
Parameter Example Description
batch_size 32 Samples per batch
shuffle True Randomize order each epoch (for training)
num_workers 4 Parallel data loading processes
pin_memory True Pin to GPU memory (faster transfer)
drop_last True Drop last incomplete batch

GPU & CUDA Operations

python GPU & device management

Loss Functions & Optimization

Task Loss Function Usage
Multi-class classification nn.CrossEntropyLoss() Output: logits [batch, num_classes]. Input labels: [batch] (class indices)
Binary classification nn.BCEWithLogitsLoss() Output: logits [batch, 1]. Input labels: [batch, 1] (0 or 1)
Regression nn.MSELoss() Mean squared error between predictions and targets
Ranking/Margin nn.MarginRankingLoss() Ranking losses for contrastive learning
Optimizer Code When to Use
SGD optim.SGD(params, lr=0.01, momentum=0.9) Stable, good for fine-tuning
Adam optim.Adam(params, lr=0.001) Default choice; adaptive learning rates
AdamW optim.AdamW(params, lr=0.001) Adam with weight decay (better regularization)
RMSprop optim.RMSprop(params, lr=0.001) Good for RNNs

Saving & Loading Models

python Save and load checkpoints

Common Patterns & Best Practices

Pattern 1: Evaluation Mode

Always set model.eval() before evaluation and model.train() before training. This disables dropout and batch norm updates.

python Evaluation pattern

Pattern 2: Gradient Accumulation

Process multiple batches before updating weights (useful when batch size is limited by GPU memory).

python Gradient accumulation

Pattern 3: Learning Rate Scheduling

Decay learning rate during training for better convergence.

python Learning rate scheduling

Pattern 4: Early Stopping

Stop training if validation loss doesn't improve.

python Early stopping

Debugging & Common Issues

Issue Solution
RuntimeError: Expected all tensors on same device Move all tensors to same device: x.to(device), model.to(device)
RuntimeError: grad can only be implicitly created for scalar outputs Loss must be scalar. Use loss.backward() not loss[0].backward()
CUDA out of memory Reduce batch size, use gradient accumulation, clear cache: torch.cuda.empty_cache()
NaN or Inf in loss Reduce learning rate, check for numerical issues in data, clip gradients
Model not learning (loss stuck) Check learning rate (too high or too low), verify data normalization, inspect gradients
python Inspect gradients & debug
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Master PyTorch with production-grade expertise.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →