Easy Questions (Fundamentals)
These cover neural network basics, activation functions, and core concepts. Expected at junior/entry-level interviews.
Answer: Backpropagation is the algorithm used to train neural networks. It computes gradients of the loss with respect to weights by propagating error backward through the network.
How it works:
- Forward pass: Input flows through network, producing predictions
- Loss computation: Compare predictions to targets, compute loss
- Backward pass: Use chain rule to compute ∂Loss/∂Weight for each layer
- Weight update: Subtract gradient × learning_rate from weights
Example: For a 3-layer network: δ_output → δ_hidden2 → δ_hidden1 → ∂L/∂w for each weight.
Key insight: The chain rule lets us reuse gradients from later layers, making it efficient.
Answer: Activation functions introduce non-linearity to neural networks.
Why needed: Without them, stacking layers is useless (composition of linear functions is still linear). Networks can only learn linear relationships.
Common activations:
- ReLU: max(0, x) — simple, fast, standard in hidden layers
- Sigmoid: 1/(1+e^-x) — outputs [0,1], used for binary classification
- Tanh: (e^x - e^-x)/(e^x + e^-x) — outputs [-1,1], stronger than sigmoid
- Softmax: exp(x_i) / Σexp(x) — used for multi-class outputs
Rule: Use ReLU for hidden layers (fast, avoids vanishing gradients). Use softmax for classification outputs.
Supervised learning: Train on labeled data (X, y). Minimize prediction error on targets.
- Classification: predict discrete labels (cat, dog)
- Regression: predict continuous values (price, temperature)
- Loss: Cross-entropy, MSE
Unsupervised learning: Train on unlabeled data (X only). Find structure, patterns, or representations.
- Clustering: group similar examples (K-means)
- Dimensionality reduction: compress data (PCA, Autoencoder)
- Representation learning: learn useful features (Word2Vec, VAE)
- Loss: Reconstruction error, clustering objective
Key difference: Supervised has ground truth targets; unsupervised discovers patterns without them.
Overfitting: Model learns training data too well, including noise. Performs poorly on unseen data.
Signs: Training loss ↓ but validation loss ↑. Large gap between train and test accuracy.
Prevention strategies:
- More data: Larger datasets reduce overfitting
- Regularization: L1/L2 penalty on weights (prefer smaller weights)
- Dropout: Randomly drop activations during training (breaks co-adaptations)
- Early stopping: Stop training when validation loss plateaus
- Data augmentation: Artificially increase dataset (rotation, flip, crop)
- Simpler models: Fewer layers/parameters for small datasets
Trade-off: Prevent overfitting, but don't underfit (model too simple).
Gradient descent: Iteratively update weights in direction of negative gradient to minimize loss.
Update rule: w_new = w_old - learning_rate × ∇Loss
Variants:
- Batch GD: Use all training data per update (stable, slow)
- Stochastic GD (SGD): Use one sample per update (noisy, fast, escapes local minima)
- Mini-batch GD: Use small batch (32, 64, 128) per update (practical default)
- Momentum: Accumulate past gradients (faster convergence)
- Adam: Adaptive learning rate per parameter (most popular, works well by default)
Learning rate: Too high → diverge. Too low → slow or stuck. Often use learning rate scheduler (decay over time).
Medium Questions (CNNs & RNNs)
These cover convolutional and recurrent architectures, training challenges, and optimization techniques.
Convolutional layer: Applies a small learnable filter (kernel) across spatial locations to extract features.
Operation: Element-wise multiplication + sum (convolution) of kernel with image patch.
- Kernel size: 3×3, 5×5, 7×7 (small receptive field)
- Multiple kernels: Each learns different features (edges, textures, patterns)
- Early layers: Detect low-level features (edges)
- Later layers: Detect high-level features (shapes, objects)
Advantages over fully-connected:
- Parameter sharing: Same kernel applied everywhere (fewer parameters)
- Translation invariance: Detects features regardless of position
- Spatial structure: Preserves local relationships
Example: Conv2d(3, 32, kernel_size=3) has 3×3×3×32 = 864 parameters (vs 100K+ for fully-connected).
Batch normalization: Normalize layer inputs to have mean=0, std=1 across a batch.
Benefits:
- Faster training: Stabilizes gradients, allows higher learning rates
- Reduces internal covariate shift: Distribution of layer inputs doesn't change drastically
- Acts as regularizer: Noise from batch sampling reduces overfitting
- Less sensitive to initialization: Networks train from scratch better
Training vs inference:
- Training: Normalize using batch statistics (mean, var)
- Inference: Use running average of batch statistics (avoid batch-size dependency)
Note: If batch size is very small (<16), batch norm can be noisy. Use layer norm or group norm instead.
Vanishing gradient problem: In RNNs, gradients shrink exponentially as they propagate backward through many timesteps. Weights barely update.
Cause: Repeated multiplication by values <1 (sigmoid derivative ≤ 0.25). After 20-30 steps, gradient ≈ 0.
Consequence: Network forgets long-range dependencies. Can't learn patterns >5-10 timesteps apart.
LSTM solution: Gating mechanism (forget gate, input gate, output gate) controls information flow.
- Cell state: "Highway" that carries information with minimal transformation
- Forget gate: Decides what to discard from previous state
- Input gate: Decides what new information to add
- Output gate: Decides what part of cell state to expose
Result: Gradients can flow across 100+ timesteps without vanishing.
Dropout: Randomly set p% of activations to 0 during training.
How it works:
- Training: Randomly deactivate neurons (e.g., 30% probability)
- Inference: Use all neurons (scaled by 1-p to maintain expected value)
Intuition: Forces network to learn redundant representations. No single neuron becomes critical. At test time, ensemble of "sub-networks" votes.
Benefits:
- Reduces overfitting (prevents co-adaptation)
- Acts as ensemble method
- Improves generalization
Typical values: 0.2-0.5 (20-50% drop rate). Use higher rates for large networks or small datasets.
Important: Disable dropout at test time (model.eval() in PyTorch sets dropout to 0%).
Attention mechanism: Allows model to focus on relevant tokens/words in a sequence.
Key components:
- Query (Q): What am I looking for? Derived from current token
- Key (K): What am I offering? Derived from all tokens
- Value (V): What information should I retrieve? Derived from all tokens
Computation: Attention(Q, K, V) = softmax(Q·K^T/√d_k)·V
- Q·K^T: Compute similarity between query and all keys (dot product)
- softmax: Normalize to probability distribution (weights)
- ·V: Weighted sum of values (retrieve relevant information)
Multi-head attention: Multiple attention heads in parallel, each focusing on different aspects. Outputs concatenated.
Self-attention: Q, K, V all come from same sequence (token attends to all tokens including itself).
Benefit: Captures long-range dependencies directly (no sequential bottleneck like RNNs).
Hard Questions (Advanced Topics)
These cover advanced architectures, optimization nuances, and production challenges.
Problem: Self-attention is permutation-invariant (order doesn't matter). "Cat sat on mat" = "mat on sat cat". This is wrong for language.
Solution: Positional encoding. Add position-dependent values to token embeddings.
Standard approach (sinusoidal):
- PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
- PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
- Where pos = position, i = dimension index
Why sinusoidal?
- Periodic: Model can extrapolate to longer sequences
- Deterministic: No additional parameters to learn
- Unique per position: Each position has unique encoding
Alternative: Learnable position embeddings (like in BERT). Simpler but can't extrapolate beyond training length.
Problem: Poor initialization can cause vanishing/exploding gradients or slow convergence.
Strategies:
- Zero initialization: Don't use! All neurons learn identical features (symmetry breaking fails)
- Small random (Gaussian): N(0, 0.01) works okay but not optimal
- Xavier/Glorot: Var(w) = 1/(n_in + n_out). Maintains similar activation scale across layers. Good for tanh/sigmoid
- He initialization: Var(w) = 2/n_in. Designed for ReLU. Prevents vanishing gradients with ReLU
Example: He init: w ~ N(0, √(2/fan_in))
Why it matters:
- Good init → converges in 10 epochs
- Bad init → converges in 1000 epochs (or not at all)
Practice: Use PyTorch defaults (torch.nn initializes with He). For custom layers, use torch.nn.init module.
Transfer learning: Use weights from model trained on large dataset (ImageNet, Wikipedia) as initialization for new task with smaller dataset.
Motivation: Early layers learn generic features (edges, textures) that transfer across tasks. Later layers are task-specific.
Workflow:
- Step 1: Pre-train on large dataset (e.g., ImageNet with 1M images)
- Step 2: Load pre-trained weights
- Step 3: Replace final layer(s) for new task
- Step 4: Fine-tune: Train on small dataset, often with lower learning rate
Fine-tuning strategies:
- Full fine-tune: Update all layers (needs more data, higher LR)
- Freeze early layers: Only train final layers (needs less data, works with tiny datasets)
- Layer-wise learning rates: Lower LR for early layers, higher for later (best)
Example impact: Training ResNet from scratch on 100 samples fails. Transfer learning from ImageNet succeeds easily.
Batch size: Number of samples processed before updating weights.
Trade-offs:
| Small batch (16-32) | Noisier updates → better regularization. May escape local minima. Slower per-epoch but fewer epochs. Good for limited GPU memory. |
| Large batch (256-512) | Smoother gradients → faster convergence per step. Requires higher LR. May get stuck in sharp minima (worse generalization). Faster wall-clock training. |
Generalization: Small batch often generalizes better (noisier gradients act as regularizer). But recent research shows large-batch + careful LR scheduling can match small-batch generalization.
Practical guideline: Use largest batch size that fits GPU memory. If validation accuracy drops, reduce batch size or use learning rate warmup/scheduling.
Quantization: Convert model weights/activations from float32 to lower precision (int8, float16, etc.).
Benefits:
- Smaller model size: float32 → int8 = 4x compression. ResNet50: 100MB → 25MB
- Faster inference: int8 operations faster than float32 on many hardware
- Lower memory: Fits on mobile/edge devices
- Lower latency: Reduced bandwidth, cache misses
Types:
- Post-training quantization: Quantize pre-trained model (simple, minimal retraining)
- Quantization-aware training: Train with quantization in mind (better accuracy, more complex)
Trade-off: Accuracy loss (typically 1-3% for int8). Usually acceptable.
Example: Deploy BERT on phone: float32 → int8 = 110MB → 27MB, 4x faster inference.
Effect: Batch norm smooths the loss landscape, making it more convex and easier to optimize.
Mechanisms:
- Reduces internal covariate shift (gradients don't change drastically across batches)
- Makes gradients more predictable, allowing higher learning rates
- Reduces dependency on weight initialization
Consequence: Wider range of learning rates work. Faster training without tuning LR carefully.
Note: Large batch size makes batch statistics unreliable. Solution: use layer norm or group norm for small batches.
Residual connection (skip connection): Connect input directly to output: output = F(x) + x
Problem it solves: Very deep networks are hard to train (vanishing gradients, degradation problem).
Why it helps:
- Gradient flow: Gradients bypass layers through shortcut path
- Identity shortcut: If F(x) is bad, network can learn F(x)≈0 and just use input (easy fallback)
- Easier optimization: Learning residuals easier than learning full transformation
Impact: Enables training of 100+ layer networks (ResNet-152). Without skip connections, training becomes unstable.
Modern application: Every modern architecture uses residual connections: ResNet, BERT, GPT, etc.
Data augmentation: Artificially increase dataset by applying transformations (rotation, crop, flip, color jitter, etc.).
How it improves generalization:
- Larger effective dataset: More training examples → model sees more variations
- Invariance: Model learns to be robust to transformations (e.g., rotation-invariant for images)
- Regularization: Acts as implicit regularizer (harder to memorize)
Examples:
- Images: Random rotation, crop, flip, brightness adjustment
- Text: Back-translation, synonym replacement, random deletion
- Audio: Pitch shift, time stretch, background noise
Best practice: Use domain-appropriate augmentation. For medical images, avoid unrealistic transforms.
Modern approach: AutoAugment, RandAugment, MixUp, CutMix automatically find best augmentation policies.
Sigmoid: σ(x) = 1/(1+e^-x). Output range: [0, 1]
Softmax: σ(x_i) = e^(x_i) / Σe^(x_j). Outputs sum to 1 (probability distribution)
When to use:
- Sigmoid: Binary classification (2 classes), or multi-label classification (multiple labels per example)
- Softmax: Multi-class classification (mutually exclusive classes)
Example:
- Sigmoid: Email spam detection (spam or not). Image tagging (cat, dog, plant - all can be true)
- Softmax: Image classification (cat, dog, or bird - only one true)
Key difference: Sigmoid independent per output. Softmax enforces sum-to-1 constraint (competitive).
Loss: Sigmoid → BCEWithLogitsLoss. Softmax → CrossEntropyLoss.
Exploding gradient: Gradients grow exponentially, causing unstable training (NaN, Inf losses).
Cause: Repeated multiplication by values >1 during backpropagation through many layers.
Solutions:
- Gradient clipping: Clip gradients to max norm. torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
- Batch normalization: Normalizes activations, stabilizes gradients
- Lower learning rate: Reduce LR to prevent large updates
- Careful weight initialization: He init prevents gradient explosion with ReLU
Gradient clipping example: If ||gradient|| > max_norm, scale gradient down: gradient = gradient × (max_norm / ||gradient||)
When to use: RNNs/LSTMs over long sequences, very deep networks, or when loss suddenly becomes NaN.
Land your dream deep learning role.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →