🔊
Real-world scenario
Aisha at M-Kopa Solar: process voice notes from field agents to diagnose equipment issues

Aisha is a data scientist at M-Kopa Solar in Nairobi, a company providing pay-as-you-go solar systems to households across East Africa. Field agents (200+ in Kenya, Uganda, Tanzania) record 30-second voice notes on their phones describing equipment problems: "Generator not starting in morning, making clicking sound," "Battery terminal corrosion, not charging," "Inverter beeping three times, no output." Her task: classify equipment issues from voice alone. First step: understand audio fundamentals. She needs to convert raw 16-bit audio files (16 kHz sampling rate, mono) into features that capture frequency content (spectrograms), human perception (mel spectrograms), and pronunciation patterns (MFCCs). Her goal: build a voice-based equipment diagnostics system to reduce technician travel time by 40% (save 15 hours/week across the field team).

200+
Field agents
3 features
To master
16 kHz
Sample rate

🌊 Audio waveforms: amplitude and time

Audio is a continuous signal — sound waves traveling through air. When we record it digitally, we sample the air pressure at regular intervals (e.g., 16,000 times per second). Each sample is a number representing amplitude (loudness). Let's start with the simplest representation: a waveform plot.

Understanding waveforms

A waveform shows amplitude (vertical) vs time (horizontal). The sampling rate (Hz) determines how many samples per second we capture. At 16 kHz, we get 16,000 samples per second. For a 5-second audio file: 16,000 * 5 = 80,000 samples total.

1_load_audio.py Load audio file and plot waveform
Output
Sample rate: 16000 Hz Audio shape: (80000,) Audio dtype: float32 Min amplitude: -0.7284 Max amplitude: 0.6842 Duration: 5.00 seconds

The waveform tells us amplitude at each point in time, but it doesn't tell us what frequencies are present. "Clicking sound" has different frequencies than "beeping." That's where frequency-domain analysis comes in.

🎵 Frequency and the Nyquist theorem

Sound consists of multiple frequencies mixed together. A male voice typically has fundamental frequencies 80-250 Hz. High-frequency noise might be 8,000-10,000 Hz. To capture all frequencies in an audio signal, we need the Nyquist theorem: sample rate must be at least 2x the highest frequency.

Nyquist Theorem: If the highest frequency in your signal is 8 kHz, you must sample at least 16 kHz to avoid aliasing (high frequencies wrapping around to low frequencies, destroying information). Human hearing spans 20 Hz to 20 kHz. Phones/speech typically use 8 kHz (captures up to 4 kHz, enough for speech intelligibility) or 16 kHz (up to 8 kHz, better quality). Video calls: 16 kHz. Music: 44.1 kHz or 48 kHz (captures up to 22 kHz).

Fourier Transform: time to frequency

To see which frequencies are present, we use the Fourier Transform. The Fast Fourier Transform (FFT) converts a time-domain waveform into frequency-domain components.

2_fft_frequency.py Compute FFT and plot frequency spectrum
Output
Top 5 dominant frequencies: 1. 187.3 Hz (magnitude: 453.2) 2. 374.6 Hz (magnitude: 287.4) 3. 1121.9 Hz (magnitude: 156.8) 4. 561.9 Hz (magnitude: 143.2) 5. 2243.8 Hz (magnitude: 98.5)

The frequency spectrum shows energy at 187 Hz (fundamental frequency of voice), 375 Hz (second harmonic), and higher frequencies for consonant sounds. But FFT of the entire file averages out over 5 seconds — we lose temporal information. For dynamic signals like speech, we need time-frequency analysis.

📊 Spectrograms: time and frequency together

A spectrogram shows frequency content over time. We divide the audio into short overlapping windows (e.g., 512 samples = 32ms at 16 kHz), compute FFT for each window, and stack them. Result: a 2D image where x-axis is time, y-axis is frequency, and color is intensity (magnitude).

Spectrogram computation

Standard parameters for 16 kHz audio: window size 512 (32ms), hop length 160 (10ms), so windows overlap by 75%. This gives us about 100 spectrogram frames per second.

3_spectrogram.py Compute and visualize spectrogram
Output
Spectrogram shape: (257, 501) Frequency bins (rows): 257 (0-8000 Hz) Time frames (cols): 501 (5 seconds) Max frequency: 8000.0 Hz

This spectrogram is raw frequency content. The x-axis has 501 time frames (5s * 100 frames/sec), y-axis has 257 frequency bins. But human hearing is nonlinear — we hear differences in low frequencies better than high frequencies. That's where mel spectrograms come in.

🎧 Mel spectrograms: human perception

The mel scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. Instead of linear frequency bins (0 Hz, 100 Hz, 200 Hz...), we use logarithmic spacing that matches human hearing. A mel spectrogram applies a set of triangular filters (mel filterbank) to the frequency spectrum, mimicking the nonlinear frequency discrimination of the human ear.

Why mel scale? Humans distinguish differences in pitch better at low frequencies. The difference between 100 Hz and 200 Hz (1 octave) sounds similar in pitch difference to 4000 Hz and 8000 Hz (also 1 octave). But in terms of frequency distance, the second pair is 4x larger. Mel scale compresses high frequencies logarithmically.

Mel spectrogram computation

Apply 128 mel-spaced triangular filters to the linear spectrogram, then convert to dB scale for better numerical stability during training.

4_mel_spectrogram.py Compute mel spectrogram
Output
Mel spectrogram shape: (128, 501) Mel bins (rows): 128 Time frames (cols): 501 Min dB: -80.0, Max dB: 0.0

The mel spectrogram (128x501) is a cleaner, more perceptually meaningful representation than the linear spectrogram (257x501). Each row represents a perceptual frequency band. This is often the input to audio neural networks.

🎤 MFCCs: mel-frequency cepstral coefficients

MFCCs further compress mel spectrograms by applying a Discrete Cosine Transform (DCT). While mel spectrograms are good for neural networks, MFCCs are better for traditional ML (SVMs, GMMs) because they decorrelate the features, removing redundancy. Typically we compute 13-40 MFCC coefficients per frame.

MFCC pipeline: Waveform → STFT → Magnitude → Mel filterbank → dB scale → DCT → MFCC coefficients. The DCT keeps only the most important variations in the spectrum (like a lossy compression). MFCCs capture spectral shape compactly, removing fine-grained details that are less important for speech/audio classification.

5_mfcc.py Compute MFCCs
Output
MFCC shape: (13, 501) Coefficients (rows): 13 Time frames (cols): 501 MFCC statistics: Coeff 0: mean=49.243, std=12.891 Coeff 1: mean=-32.541, std=18.234 Coeff 2: mean=1.234, std=14.876 Coeff 3: mean=-8.923, std=11.543 Coeff 4: mean=5.432, std=9.876 Coeff 5: mean=-2.145, std=8.234 Coeff 6: mean=1.876, std=6.543 Coeff 7: mean=-1.234, std=5.876 Coeff 8: mean=0.567, std=4.234 Coeff 9: mean=-0.234, std=3.876 Coeff 10: mean=0.145, std=2.876 Coeff 11: mean=-0.089, std=2.123 Coeff 12: mean=0.045, std=1.234

MFCCs compress spectral information into 13 coefficients per frame. For a 5-second audio file, we get a (13, 501) matrix. This compact representation removes redundancy while preserving speech-relevant information, making it ideal for neural networks.

🔧 PyTorch with torchaudio: production audio loading

In production, you'll use PyTorch's native torchaudio library. It's optimized, supports CUDA, and integrates with neural networks directly.

6_torchaudio_features.py Load audio and compute features with torchaudio
Output
Waveform shape: (1, 80000) Sample rate: 16000 Hz Mel spectrogram shape: (1, 128, 501) MFCC shape: (1, 13, 501)

📋 Feature comparison: which one to use?

Feature Dimensions Time Cost Best For Notes
Waveform 80K (5s @ 16kHz) Fast Raw time-domain models High dimensionality, non-stationary
Spectrogram 257 × 501 Medium CNNs, visual inspection Raw frequency content, hard for ML
Mel Spectrogram 128 × 501 Fast Deep neural networks Perceptually relevant, compact. Use this!
MFCC 13 × 501 Fast Traditional ML (SVM, GMM) Ultra-compact, decorrelated. Legacy.

Interview Q&A: Audio fundamentals in the field

Answer: Human perception of pitch is logarithmic, not linear. We hear the interval between 100-200 Hz as one octave, same as 4000-8000 Hz (both factor of 2). But linearly, the second pair has 4000 Hz distance vs 100 Hz. Mel scale warps frequencies to match human perception: it compresses high frequencies (8000+ Hz) into fewer bins, expands low frequencies (20-1000 Hz) into more bins. This matches where our hearing is most sensitive. Neural networks trained on mel spectrograms learn more efficiently because the features are perceptually organized.
Answer: Nyquist theorem: at 8 kHz sample rate, you capture up to 4 kHz (half the sample rate). This is OK for speech intelligibility — human speech formants (resonances that carry meaning) are mostly 0-4 kHz. High-frequency consonants (sibilants, fricatives: "s", "sh", "f") are 4-8 kHz. So 8 kHz speech sounds a bit duller, missing high-frequency detail. If you want to detect sibilants or equipment buzzing at 6 kHz, you need 16+ kHz. For Aisha's voice notes: equipment issues often have specific high-frequency signatures (clicking at 8-10 kHz, buzzing at 6-7 kHz), so 16 kHz is right choice.
Answer: Three approaches: (1) Pad to fixed length: extract mel spectrograms, pad shorter files with zeros (on the right, to avoid shifting start). Typical padding value: 0 in log-dB scale (very quiet). (2) Crop to max length: if some files are longer, truncate to 5 seconds (or learn from first N frames). (3) Bucketing: group files by duration, pad within each group to reduce wasted computation. For Aisha: field agents' voice notes vary 20-120 seconds. Crop to 30 seconds (enough to capture full equipment description), extract mel spectrogram (128 × 300 for 30s @ 100 frames/sec), pad with zeros. All files end up (128 × 300), ready for batch training.
Answer: Mel spectrograms: 128 × 501 matrix. Highly correlated across frequency bins — adjacent filters overlap, so MFCC coefficients are often correlated. MFCCs: 13 × 501. The DCT decorrelates: each coefficient captures independent aspects of the spectrum. Mel specs are better for deep neural networks (CNNs can learn which filters matter), MFCCs are better for shallow models (SVM, GMM) because features are pre-orthogonalized. For Aisha's equipment classification: use mel spectrograms + CNN. They're more information-rich (128 vs 13 dimensions) and modern GPUs handle the extra computation cheaply.
Answer: 50 Hz hum is a pure tone (sine wave at 50 Hz). Two approaches: (1) Notch filter on waveform: apply a narrow bandpass filter at 50 Hz, subtract from original. Effective but can distort nearby frequencies. (2) Post-processing on spectrogram: zero out the mel bin corresponding to 50 Hz. At 16 kHz with n_fft=512, frequency resolution is 16000/512 = 31.25 Hz/bin. 50 Hz maps to bin ~1.6, affecting bins 1-2. Set mel spectrogram[0:2, :] = 0. This removes hum without distorting the signal. For Aisha: most solar equipment is 50/60 Hz AC powered, so hum is common. Apply notch filter pre-processing to clean voice notes before feature extraction.

Full audio pipeline: end-to-end

Complete code for Aisha's equipment diagnostics: load voice notes, extract features, prepare for classification.

full_audio_pipeline.py Complete audio feature extraction pipeline (80 lines)

Quiz: test your understanding

Question 1 of 5
At 16 kHz sample rate, what's the maximum frequency you can capture (Nyquist limit)?
Question 2 of 5
Why use mel spectrograms instead of linear spectrograms for neural networks?
Question 3 of 5
What's the shape of a mel spectrogram for a 5-second audio at 16 kHz (128 mel bins)?
Question 4 of 5
MFCCs compress spectral information using which technique?
Question 5 of 5
For equipment diagnostics with high-frequency buzzing (6-7 kHz), which minimum sample rate is needed?
0
/5
Excellent! You've mastered audio signal processing fundamentals.

Cheatsheet: audio feature extraction

Concept Definition Code (torchaudio)
Sampling Rate Samples per second (Hz). 16 kHz = 16,000 samples/sec sr = 16000
Nyquist Limit Max frequency = sr/2. At 16 kHz → max 8 kHz max_freq = sr / 2
FFT Size (n_fft) Frequency resolution. 512 = 16000/512 = 31.25 Hz/bin n_fft=512
Hop Length Shift between frames (samples). 160 = 10ms at 16 kHz hop_length=160
Mel Spectro. 128 perceptual frequency bins, shape (128, T) T.MelSpectrogram(n_mels=128)
MFCC 13 decorrelated coefficients, shape (13, T) T.MFCC(n_mfcc=13)

Common mistakes: what breaks and how to fix it

Mistake 1: Forgetting the Nyquist limit
If your signal has 10 kHz content but you sample at 16 kHz (Nyquist limit 8 kHz), high frequencies will alias and corrupt low frequencies. Always use at least 2x the highest frequency. For equipment diagnostics with 6 kHz buzzing: use 16+ kHz, not 8 kHz.
Mistake 2: Not normalizing mel spectrograms
Raw mel spectrograms have large absolute values (0-100 dB range). Neural networks expect normalized features (zero mean, unit variance). Use (mel_spec - mean) / (std + 1e-5). Forgetting this inflates gradients and makes training unstable.
Mistake 3: Variable-length sequences without padding
If audio files vary in length (20-120s), mel spectrograms have different shapes. You can't batch different shapes. Always pad to fixed length: crop long files, pad short files with zeros. Crop time dimension to (128, 300) for 30s @ 100fps.
Mistake 4: Using wrong FFT size
FFT size controls frequency resolution: larger n_fft = better frequency resolution but worse time resolution. For speech: n_fft=512 (31.25 Hz/bin) is standard. For music: n_fft=2048. Forgetting to match across preprocessing and training causes shape mismatches.
Mistake 5: Resampling at different stages
Load at 8 kHz, then upsample to 16 kHz = adds no information (original was limited to 4 kHz). Always load at target sample rate to avoid unnecessary resampling. Librosa: librosa.load(path, sr=16000). Torchaudio: Resample(orig_sr, 16000) only if needed.

Key takeaways

Audio fundamentals are the foundation of speech and sound understanding in AI. Aisha's journey from raw voice notes to structured features (waveforms → spectrograms → mel spectrograms → MFCCs) mirrors the evolution of audio processing.

  • Waveforms capture raw amplitude over time. For visualization and preprocessing only.
  • Sampling rate determines the Nyquist limit (max frequency captured). For equipment with 6 kHz buzzing, use 16+ kHz.
  • Spectrograms show frequency content over time using FFT. Linear bins, not perceptually meaningful.
  • Mel spectrograms compress frequencies logarithmically to match human hearing. Best for deep neural networks. Use mel_spec + CNN.
  • MFCCs further compress via DCT. Decorrelated, compact (13 coefficients). Use for traditional ML, or if you need ultra-fast inference.
  • torchaudio is the production standard. Fast, GPU-compatible, integrates with PyTorch dataloaders.
  • Always normalize features to zero mean, unit variance. Always pad variable-length sequences to fixed shape.

What's next?

You now understand the audio feature pipeline. The next lesson covers audio classification: using CNNs on mel spectrograms to classify equipment issues from voice notes. You'll build a 2D CNN that processes mel spectrograms as images, learning which frequency patterns indicate "battery corrosion" vs "generator clicking" vs "inverter beeping." By end of lesson 26, you'll have a model running on Aisha's field team phones, diagnosing equipment in real time.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to go further?

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →