Aisha is a data scientist at M-Kopa Solar in Nairobi, a company providing pay-as-you-go solar systems to households across East Africa. Field agents (200+ in Kenya, Uganda, Tanzania) record 30-second voice notes on their phones describing equipment problems: "Generator not starting in morning, making clicking sound," "Battery terminal corrosion, not charging," "Inverter beeping three times, no output." Her task: classify equipment issues from voice alone. First step: understand audio fundamentals. She needs to convert raw 16-bit audio files (16 kHz sampling rate, mono) into features that capture frequency content (spectrograms), human perception (mel spectrograms), and pronunciation patterns (MFCCs). Her goal: build a voice-based equipment diagnostics system to reduce technician travel time by 40% (save 15 hours/week across the field team).
🌊 Audio waveforms: amplitude and time
Audio is a continuous signal — sound waves traveling through air. When we record it digitally, we sample the air pressure at regular intervals (e.g., 16,000 times per second). Each sample is a number representing amplitude (loudness). Let's start with the simplest representation: a waveform plot.
Understanding waveforms
A waveform shows amplitude (vertical) vs time (horizontal). The sampling rate (Hz) determines how many samples per second we capture. At 16 kHz, we get 16,000 samples per second. For a 5-second audio file: 16,000 * 5 = 80,000 samples total.
The waveform tells us amplitude at each point in time, but it doesn't tell us what frequencies are present. "Clicking sound" has different frequencies than "beeping." That's where frequency-domain analysis comes in.
🎵 Frequency and the Nyquist theorem
Sound consists of multiple frequencies mixed together. A male voice typically has fundamental frequencies 80-250 Hz. High-frequency noise might be 8,000-10,000 Hz. To capture all frequencies in an audio signal, we need the Nyquist theorem: sample rate must be at least 2x the highest frequency.
Nyquist Theorem: If the highest frequency in your signal is 8 kHz, you must sample at least 16 kHz to avoid aliasing (high frequencies wrapping around to low frequencies, destroying information). Human hearing spans 20 Hz to 20 kHz. Phones/speech typically use 8 kHz (captures up to 4 kHz, enough for speech intelligibility) or 16 kHz (up to 8 kHz, better quality). Video calls: 16 kHz. Music: 44.1 kHz or 48 kHz (captures up to 22 kHz).
Fourier Transform: time to frequency
To see which frequencies are present, we use the Fourier Transform. The Fast Fourier Transform (FFT) converts a time-domain waveform into frequency-domain components.
The frequency spectrum shows energy at 187 Hz (fundamental frequency of voice), 375 Hz (second harmonic), and higher frequencies for consonant sounds. But FFT of the entire file averages out over 5 seconds — we lose temporal information. For dynamic signals like speech, we need time-frequency analysis.
📊 Spectrograms: time and frequency together
A spectrogram shows frequency content over time. We divide the audio into short overlapping windows (e.g., 512 samples = 32ms at 16 kHz), compute FFT for each window, and stack them. Result: a 2D image where x-axis is time, y-axis is frequency, and color is intensity (magnitude).
Spectrogram computation
Standard parameters for 16 kHz audio: window size 512 (32ms), hop length 160 (10ms), so windows overlap by 75%. This gives us about 100 spectrogram frames per second.
This spectrogram is raw frequency content. The x-axis has 501 time frames (5s * 100 frames/sec), y-axis has 257 frequency bins. But human hearing is nonlinear — we hear differences in low frequencies better than high frequencies. That's where mel spectrograms come in.
🎧 Mel spectrograms: human perception
The mel scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. Instead of linear frequency bins (0 Hz, 100 Hz, 200 Hz...), we use logarithmic spacing that matches human hearing. A mel spectrogram applies a set of triangular filters (mel filterbank) to the frequency spectrum, mimicking the nonlinear frequency discrimination of the human ear.
Mel spectrogram computation
Apply 128 mel-spaced triangular filters to the linear spectrogram, then convert to dB scale for better numerical stability during training.
The mel spectrogram (128x501) is a cleaner, more perceptually meaningful representation than the linear spectrogram (257x501). Each row represents a perceptual frequency band. This is often the input to audio neural networks.
🎤 MFCCs: mel-frequency cepstral coefficients
MFCCs further compress mel spectrograms by applying a Discrete Cosine Transform (DCT). While mel spectrograms are good for neural networks, MFCCs are better for traditional ML (SVMs, GMMs) because they decorrelate the features, removing redundancy. Typically we compute 13-40 MFCC coefficients per frame.
MFCC pipeline: Waveform → STFT → Magnitude → Mel filterbank → dB scale → DCT → MFCC coefficients. The DCT keeps only the most important variations in the spectrum (like a lossy compression). MFCCs capture spectral shape compactly, removing fine-grained details that are less important for speech/audio classification.
MFCCs compress spectral information into 13 coefficients per frame. For a 5-second audio file, we get a (13, 501) matrix. This compact representation removes redundancy while preserving speech-relevant information, making it ideal for neural networks.
🔧 PyTorch with torchaudio: production audio loading
In production, you'll use PyTorch's native torchaudio library. It's optimized, supports CUDA, and integrates with neural networks directly.
📋 Feature comparison: which one to use?
| Feature | Dimensions | Time Cost | Best For | Notes |
|---|---|---|---|---|
| Waveform | 80K (5s @ 16kHz) | Fast | Raw time-domain models | High dimensionality, non-stationary |
| Spectrogram | 257 × 501 | Medium | CNNs, visual inspection | Raw frequency content, hard for ML |
| Mel Spectrogram | 128 × 501 | Fast | Deep neural networks | Perceptually relevant, compact. Use this! |
| MFCC | 13 × 501 | Fast | Traditional ML (SVM, GMM) | Ultra-compact, decorrelated. Legacy. |
Interview Q&A: Audio fundamentals in the field
Full audio pipeline: end-to-end
Complete code for Aisha's equipment diagnostics: load voice notes, extract features, prepare for classification.
Quiz: test your understanding
Cheatsheet: audio feature extraction
| Concept | Definition | Code (torchaudio) |
|---|---|---|
| Sampling Rate | Samples per second (Hz). 16 kHz = 16,000 samples/sec | sr = 16000 |
| Nyquist Limit | Max frequency = sr/2. At 16 kHz → max 8 kHz | max_freq = sr / 2 |
| FFT Size (n_fft) | Frequency resolution. 512 = 16000/512 = 31.25 Hz/bin | n_fft=512 |
| Hop Length | Shift between frames (samples). 160 = 10ms at 16 kHz | hop_length=160 |
| Mel Spectro. | 128 perceptual frequency bins, shape (128, T) | T.MelSpectrogram(n_mels=128) |
| MFCC | 13 decorrelated coefficients, shape (13, T) | T.MFCC(n_mfcc=13) |
Common mistakes: what breaks and how to fix it
(mel_spec - mean) / (std + 1e-5). Forgetting this inflates gradients and makes training unstable.librosa.load(path, sr=16000). Torchaudio: Resample(orig_sr, 16000) only if needed.Key takeaways
Audio fundamentals are the foundation of speech and sound understanding in AI. Aisha's journey from raw voice notes to structured features (waveforms → spectrograms → mel spectrograms → MFCCs) mirrors the evolution of audio processing.
- Waveforms capture raw amplitude over time. For visualization and preprocessing only.
- Sampling rate determines the Nyquist limit (max frequency captured). For equipment with 6 kHz buzzing, use 16+ kHz.
- Spectrograms show frequency content over time using FFT. Linear bins, not perceptually meaningful.
- Mel spectrograms compress frequencies logarithmically to match human hearing. Best for deep neural networks. Use mel_spec + CNN.
- MFCCs further compress via DCT. Decorrelated, compact (13 coefficients). Use for traditional ML, or if you need ultra-fast inference.
- torchaudio is the production standard. Fast, GPU-compatible, integrates with PyTorch dataloaders.
- Always normalize features to zero mean, unit variance. Always pad variable-length sequences to fixed shape.
What's next?
You now understand the audio feature pipeline. The next lesson covers audio classification: using CNNs on mel spectrograms to classify equipment issues from voice notes. You'll build a 2D CNN that processes mel spectrograms as images, learning which frequency patterns indicate "battery corrosion" vs "generator clicking" vs "inverter beeping." By end of lesson 26, you'll have a model running on Aisha's field team phones, diagnosing equipment in real time.
Ready to go further?
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →