Easy Questions (Fundamentals)

These cover neural network basics, activation functions, and core concepts. Expected at junior/entry-level interviews.

▼
Question 1
What is backpropagation and how does it work?
Easy

Answer: Backpropagation is the algorithm used to train neural networks. It computes gradients of the loss with respect to weights by propagating error backward through the network.

How it works:

  • Forward pass: Input flows through network, producing predictions
  • Loss computation: Compare predictions to targets, compute loss
  • Backward pass: Use chain rule to compute ∂Loss/∂Weight for each layer
  • Weight update: Subtract gradient × learning_rate from weights

Example: For a 3-layer network: δ_output → δ_hidden2 → δ_hidden1 → ∂L/∂w for each weight.

Key insight: The chain rule lets us reuse gradients from later layers, making it efficient.

▼
Question 2
What are activation functions? Why do we need them?
Easy

Answer: Activation functions introduce non-linearity to neural networks.

Why needed: Without them, stacking layers is useless (composition of linear functions is still linear). Networks can only learn linear relationships.

Common activations:

  • ReLU: max(0, x) — simple, fast, standard in hidden layers
  • Sigmoid: 1/(1+e^-x) — outputs [0,1], used for binary classification
  • Tanh: (e^x - e^-x)/(e^x + e^-x) — outputs [-1,1], stronger than sigmoid
  • Softmax: exp(x_i) / Σexp(x) — used for multi-class outputs

Rule: Use ReLU for hidden layers (fast, avoids vanishing gradients). Use softmax for classification outputs.

▼
Question 3
What is the difference between supervised and unsupervised learning?
Easy

Supervised learning: Train on labeled data (X, y). Minimize prediction error on targets.

  • Classification: predict discrete labels (cat, dog)
  • Regression: predict continuous values (price, temperature)
  • Loss: Cross-entropy, MSE

Unsupervised learning: Train on unlabeled data (X only). Find structure, patterns, or representations.

  • Clustering: group similar examples (K-means)
  • Dimensionality reduction: compress data (PCA, Autoencoder)
  • Representation learning: learn useful features (Word2Vec, VAE)
  • Loss: Reconstruction error, clustering objective

Key difference: Supervised has ground truth targets; unsupervised discovers patterns without them.

▼
Question 4
Explain overfitting and how to prevent it.
Easy

Overfitting: Model learns training data too well, including noise. Performs poorly on unseen data.

Signs: Training loss ↓ but validation loss ↑. Large gap between train and test accuracy.

Prevention strategies:

  • More data: Larger datasets reduce overfitting
  • Regularization: L1/L2 penalty on weights (prefer smaller weights)
  • Dropout: Randomly drop activations during training (breaks co-adaptations)
  • Early stopping: Stop training when validation loss plateaus
  • Data augmentation: Artificially increase dataset (rotation, flip, crop)
  • Simpler models: Fewer layers/parameters for small datasets

Trade-off: Prevent overfitting, but don't underfit (model too simple).

▼
Question 5
What is gradient descent and what are its variants?
Easy

Gradient descent: Iteratively update weights in direction of negative gradient to minimize loss.

Update rule: w_new = w_old - learning_rate × ∇Loss

Variants:

  • Batch GD: Use all training data per update (stable, slow)
  • Stochastic GD (SGD): Use one sample per update (noisy, fast, escapes local minima)
  • Mini-batch GD: Use small batch (32, 64, 128) per update (practical default)
  • Momentum: Accumulate past gradients (faster convergence)
  • Adam: Adaptive learning rate per parameter (most popular, works well by default)

Learning rate: Too high → diverge. Too low → slow or stuck. Often use learning rate scheduler (decay over time).

Medium Questions (CNNs & RNNs)

These cover convolutional and recurrent architectures, training challenges, and optimization techniques.

▼
Question 6
How do convolutional layers work? What are kernels and how do they extract features?
Medium

Convolutional layer: Applies a small learnable filter (kernel) across spatial locations to extract features.

Operation: Element-wise multiplication + sum (convolution) of kernel with image patch.

  • Kernel size: 3×3, 5×5, 7×7 (small receptive field)
  • Multiple kernels: Each learns different features (edges, textures, patterns)
  • Early layers: Detect low-level features (edges)
  • Later layers: Detect high-level features (shapes, objects)

Advantages over fully-connected:

  • Parameter sharing: Same kernel applied everywhere (fewer parameters)
  • Translation invariance: Detects features regardless of position
  • Spatial structure: Preserves local relationships

Example: Conv2d(3, 32, kernel_size=3) has 3×3×3×32 = 864 parameters (vs 100K+ for fully-connected).

▼
Question 7
What is batch normalization and why does it help?
Medium

Batch normalization: Normalize layer inputs to have mean=0, std=1 across a batch.

Benefits:

  • Faster training: Stabilizes gradients, allows higher learning rates
  • Reduces internal covariate shift: Distribution of layer inputs doesn't change drastically
  • Acts as regularizer: Noise from batch sampling reduces overfitting
  • Less sensitive to initialization: Networks train from scratch better

Training vs inference:

  • Training: Normalize using batch statistics (mean, var)
  • Inference: Use running average of batch statistics (avoid batch-size dependency)

Note: If batch size is very small (<16), batch norm can be noisy. Use layer norm or group norm instead.

▼
Question 8
What is the vanishing gradient problem and how do LSTMs solve it?
Medium

Vanishing gradient problem: In RNNs, gradients shrink exponentially as they propagate backward through many timesteps. Weights barely update.

Cause: Repeated multiplication by values <1 (sigmoid derivative ≤ 0.25). After 20-30 steps, gradient ≈ 0.

Consequence: Network forgets long-range dependencies. Can't learn patterns >5-10 timesteps apart.

LSTM solution: Gating mechanism (forget gate, input gate, output gate) controls information flow.

  • Cell state: "Highway" that carries information with minimal transformation
  • Forget gate: Decides what to discard from previous state
  • Input gate: Decides what new information to add
  • Output gate: Decides what part of cell state to expose

Result: Gradients can flow across 100+ timesteps without vanishing.

▼
Question 9
What is dropout and how does it work?
Medium

Dropout: Randomly set p% of activations to 0 during training.

How it works:

  • Training: Randomly deactivate neurons (e.g., 30% probability)
  • Inference: Use all neurons (scaled by 1-p to maintain expected value)

Intuition: Forces network to learn redundant representations. No single neuron becomes critical. At test time, ensemble of "sub-networks" votes.

Benefits:

  • Reduces overfitting (prevents co-adaptation)
  • Acts as ensemble method
  • Improves generalization

Typical values: 0.2-0.5 (20-50% drop rate). Use higher rates for large networks or small datasets.

Important: Disable dropout at test time (model.eval() in PyTorch sets dropout to 0%).

▼
Question 10
Explain the attention mechanism in Transformers.
Medium

Attention mechanism: Allows model to focus on relevant tokens/words in a sequence.

Key components:

  • Query (Q): What am I looking for? Derived from current token
  • Key (K): What am I offering? Derived from all tokens
  • Value (V): What information should I retrieve? Derived from all tokens

Computation: Attention(Q, K, V) = softmax(Q·K^T/√d_k)·V

  • Q·K^T: Compute similarity between query and all keys (dot product)
  • softmax: Normalize to probability distribution (weights)
  • ·V: Weighted sum of values (retrieve relevant information)

Multi-head attention: Multiple attention heads in parallel, each focusing on different aspects. Outputs concatenated.

Self-attention: Q, K, V all come from same sequence (token attends to all tokens including itself).

Benefit: Captures long-range dependencies directly (no sequential bottleneck like RNNs).

Hard Questions (Advanced Topics)

These cover advanced architectures, optimization nuances, and production challenges.

▼
Question 11
What is positional encoding and why is it needed in Transformers?
Hard

Problem: Self-attention is permutation-invariant (order doesn't matter). "Cat sat on mat" = "mat on sat cat". This is wrong for language.

Solution: Positional encoding. Add position-dependent values to token embeddings.

Standard approach (sinusoidal):

  • PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
  • PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
  • Where pos = position, i = dimension index

Why sinusoidal?

  • Periodic: Model can extrapolate to longer sequences
  • Deterministic: No additional parameters to learn
  • Unique per position: Each position has unique encoding

Alternative: Learnable position embeddings (like in BERT). Simpler but can't extrapolate beyond training length.

▼
Question 12
Explain weight initialization strategies and why they matter.
Hard

Problem: Poor initialization can cause vanishing/exploding gradients or slow convergence.

Strategies:

  • Zero initialization: Don't use! All neurons learn identical features (symmetry breaking fails)
  • Small random (Gaussian): N(0, 0.01) works okay but not optimal
  • Xavier/Glorot: Var(w) = 1/(n_in + n_out). Maintains similar activation scale across layers. Good for tanh/sigmoid
  • He initialization: Var(w) = 2/n_in. Designed for ReLU. Prevents vanishing gradients with ReLU

Example: He init: w ~ N(0, √(2/fan_in))

Why it matters:

  • Good init → converges in 10 epochs
  • Bad init → converges in 1000 epochs (or not at all)

Practice: Use PyTorch defaults (torch.nn initializes with He). For custom layers, use torch.nn.init module.

▼
Question 13
What is transfer learning and how does it work?
Hard

Transfer learning: Use weights from model trained on large dataset (ImageNet, Wikipedia) as initialization for new task with smaller dataset.

Motivation: Early layers learn generic features (edges, textures) that transfer across tasks. Later layers are task-specific.

Workflow:

  • Step 1: Pre-train on large dataset (e.g., ImageNet with 1M images)
  • Step 2: Load pre-trained weights
  • Step 3: Replace final layer(s) for new task
  • Step 4: Fine-tune: Train on small dataset, often with lower learning rate

Fine-tuning strategies:

  • Full fine-tune: Update all layers (needs more data, higher LR)
  • Freeze early layers: Only train final layers (needs less data, works with tiny datasets)
  • Layer-wise learning rates: Lower LR for early layers, higher for later (best)

Example impact: Training ResNet from scratch on 100 samples fails. Transfer learning from ImageNet succeeds easily.

▼
Question 14
What is batch size and how does it affect training and generalization?
Hard

Batch size: Number of samples processed before updating weights.

Trade-offs:

Small batch (16-32) Noisier updates → better regularization. May escape local minima. Slower per-epoch but fewer epochs. Good for limited GPU memory.
Large batch (256-512) Smoother gradients → faster convergence per step. Requires higher LR. May get stuck in sharp minima (worse generalization). Faster wall-clock training.

Generalization: Small batch often generalizes better (noisier gradients act as regularizer). But recent research shows large-batch + careful LR scheduling can match small-batch generalization.

Practical guideline: Use largest batch size that fits GPU memory. If validation accuracy drops, reduce batch size or use learning rate warmup/scheduling.

▼
Question 15
What is quantization and why is it important for model deployment?
Hard

Quantization: Convert model weights/activations from float32 to lower precision (int8, float16, etc.).

Benefits:

  • Smaller model size: float32 → int8 = 4x compression. ResNet50: 100MB → 25MB
  • Faster inference: int8 operations faster than float32 on many hardware
  • Lower memory: Fits on mobile/edge devices
  • Lower latency: Reduced bandwidth, cache misses

Types:

  • Post-training quantization: Quantize pre-trained model (simple, minimal retraining)
  • Quantization-aware training: Train with quantization in mind (better accuracy, more complex)

Trade-off: Accuracy loss (typically 1-3% for int8). Usually acceptable.

Example: Deploy BERT on phone: float32 → int8 = 110MB → 27MB, 4x faster inference.

▼
Question 16
What is batch normalization's effect on the loss landscape?
Hard

Effect: Batch norm smooths the loss landscape, making it more convex and easier to optimize.

Mechanisms:

  • Reduces internal covariate shift (gradients don't change drastically across batches)
  • Makes gradients more predictable, allowing higher learning rates
  • Reduces dependency on weight initialization

Consequence: Wider range of learning rates work. Faster training without tuning LR carefully.

Note: Large batch size makes batch statistics unreliable. Solution: use layer norm or group norm for small batches.

▼
Question 17
What are residual connections and why do they matter?
Hard

Residual connection (skip connection): Connect input directly to output: output = F(x) + x

Problem it solves: Very deep networks are hard to train (vanishing gradients, degradation problem).

Why it helps:

  • Gradient flow: Gradients bypass layers through shortcut path
  • Identity shortcut: If F(x) is bad, network can learn F(x)≈0 and just use input (easy fallback)
  • Easier optimization: Learning residuals easier than learning full transformation

Impact: Enables training of 100+ layer networks (ResNet-152). Without skip connections, training becomes unstable.

Modern application: Every modern architecture uses residual connections: ResNet, BERT, GPT, etc.

▼
Question 18
How does data augmentation improve generalization?
Hard

Data augmentation: Artificially increase dataset by applying transformations (rotation, crop, flip, color jitter, etc.).

How it improves generalization:

  • Larger effective dataset: More training examples → model sees more variations
  • Invariance: Model learns to be robust to transformations (e.g., rotation-invariant for images)
  • Regularization: Acts as implicit regularizer (harder to memorize)

Examples:

  • Images: Random rotation, crop, flip, brightness adjustment
  • Text: Back-translation, synonym replacement, random deletion
  • Audio: Pitch shift, time stretch, background noise

Best practice: Use domain-appropriate augmentation. For medical images, avoid unrealistic transforms.

Modern approach: AutoAugment, RandAugment, MixUp, CutMix automatically find best augmentation policies.

▼
Question 19
Explain the difference between softmax and sigmoid, and when to use each.
Hard

Sigmoid: σ(x) = 1/(1+e^-x). Output range: [0, 1]

Softmax: σ(x_i) = e^(x_i) / Σe^(x_j). Outputs sum to 1 (probability distribution)

When to use:

  • Sigmoid: Binary classification (2 classes), or multi-label classification (multiple labels per example)
  • Softmax: Multi-class classification (mutually exclusive classes)

Example:

  • Sigmoid: Email spam detection (spam or not). Image tagging (cat, dog, plant - all can be true)
  • Softmax: Image classification (cat, dog, or bird - only one true)

Key difference: Sigmoid independent per output. Softmax enforces sum-to-1 constraint (competitive).

Loss: Sigmoid → BCEWithLogitsLoss. Softmax → CrossEntropyLoss.

▼
Question 20
What is the exploding gradient problem and how to fix it?
Hard

Exploding gradient: Gradients grow exponentially, causing unstable training (NaN, Inf losses).

Cause: Repeated multiplication by values >1 during backpropagation through many layers.

Solutions:

  • Gradient clipping: Clip gradients to max norm. torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
  • Batch normalization: Normalizes activations, stabilizes gradients
  • Lower learning rate: Reduce LR to prevent large updates
  • Careful weight initialization: He init prevents gradient explosion with ReLU

Gradient clipping example: If ||gradient|| > max_norm, scale gradient down: gradient = gradient × (max_norm / ||gradient||)

When to use: RNNs/LSTMs over long sequences, very deep networks, or when loss suddenly becomes NaN.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Land your dream deep learning role.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →