Kwame is an NLP researcher from Accra building speech-to-text for Twi (Akan language, 9M speakers in Ghana). Traditional speech recognition systems require labeled audio datasets, language-specific acoustic models, and months of engineering. OpenAI's Whisper—pretrained on 680,000 hours of multilingual audio—transcribes Twi with 15% word error rate out-of-the-box. By fine-tuning on 100 hours of Ghanaian Twi speech, Kwame achieves 6% WER in production, enabling voice notes, transcription services, and accessibility tools for Twi speakers.
🔊 What Is Whisper?
Whisper is OpenAI's automatic speech recognition (ASR) model trained on 680,000 hours of multilingual and multitask audio data from the web. Unlike earlier ASR models optimized for English or specific domains (podcasts, phone calls), Whisper is trained on diverse audio—noisy background, accents, music, background speech—making it robust to real-world conditions. It supports 99 languages and performs not just transcription but also language detection, sentence-level timestamps, and translation to English.
Why Whisper Is Revolutionary
- Multilingual from day one: Single model handles 99 languages. No need to train language-specific models or download separate acoustic models.
- Robust to real-world audio: Trained on loud cafes, music, background speech, accents. Works on podcasts, YouTube videos, phone recordings, and conference calls.
- Task-agnostic: Transcription, translation, language identification, and timestamp generation from one model. Condition on task tokens.
- Modest compute requirements: Whisper-small (244M params) runs on CPU. Whisper-base (74M) fits on edge devices. No need for GPUs to transcribe.
- Zero-shot performance: Works well on unseen languages without fine-tuning. Fine-tuning on 100 hours drops WER by 50%.
🏗️ Architecture: Encoder-Decoder with Attention
Whisper uses a standard transformer encoder-decoder architecture, similar to machine translation models like T5 or MarianMT. The encoder processes audio spectrograms; the decoder generates tokens autoregressively.
Encoder (Audio Feature Extraction)
- Input: Raw audio (16kHz sample rate) → mel-spectrogram (80 mel-frequency bins, 50ms frame).
- Sinusoidal positional encoding: Each frame gets a learned position embedding (no absolute time, but relative position).
- Transformer encoder layers: 4-12 layers of self-attention + feed-forward. Each layer attends to all frames in the audio.
- Output: Context vector for each frame (~1500 vectors for 30 seconds of audio).
Decoder (Text Generation)
- Input tokens: Special task tokens (e.g., <|transcribe|>, <|translate|>), language token (e.g., <|tw|> for Twi), and previously generated text tokens.
- Cross-attention: Attends to encoder outputs (audio context) while generating. "hearing" audio features while "writing" text.
- Autoregressive generation: Generates one token at a time. P(token_t | audio, token_0...token_{t-1}).
- Temperature sampling: At inference, use temperature=0 for deterministic output or temperature>0 for diverse outputs.
🌍 Multilingual Training & Language Detection
Whisper was trained on 680K hours of audio from YouTube covering 99 languages. The training mix is imbalanced: English dominates (54%), but languages like Twi, Hausa, Yoruba (African languages) are well-represented due to YouTube's global reach.
Language Token & Task Token Conditioning
At training and inference, Whisper prepends special tokens to the decoder input:
<|transcribe|>or<|translate|>: Task token. Transcribe outputs native language; translate outputs English.<|tw|>,<|fr|>,<|de|>: Language token. Tells decoder which language to expect.<|notimestamps|>: Suppress timestamp tokens. Default includes <|0:00|>, <|0:01|>, etc.
⏱️ Timestamps & Sentence Segmentation
Whisper can generate word-level and sentence-level timestamps without additional training. The decoder produces special tokens like <|0:00|>, <|0:01|> interspersed with text, indicating when each word starts.
Timestamp Generation
During training, Whisper sees (audio, text, timestamps) tuples. It learns to predict timestamp tokens at the right positions. At inference, decode with timestamps enabled to get word-level timing information, useful for subtitles, captions, or audio synchronization.
🎓 Fine-tuning Whisper on Custom Speech Data
Kwame has 100 hours of Ghanaian Twi speech recordings (6,000 utterances, 5-60 seconds each) with manual transcriptions. Fine-tuning Whisper-base on this data reduces WER from 15% to 6% and adapts the model to Ghanaian accents, background noise, and domain-specific vocabulary.
Fine-tuning Workflow
- 1. Prepare dataset: Audio files + transcriptions in JSON/CSV. Load with HuggingFace datasets library.
- 2. Load pretrained Whisper: openai/whisper-base (with language='tw' for Twi).
- 3. Set up trainer: HuggingFace Trainer with speech-specific loss (CTC or Whisper's native cross-entropy).
- 4. Train: 3-5 epochs on 8 GPUs, effective batch size 128. Learning rate: 1e-4. Dropout: 0.1.
- 5. Evaluate: WER on held-out test set. Compare with baseline zero-shot Whisper.
🚀 Inference: Batch Processing & Optimization
For production, Whisper inference should be fast and memory-efficient. Use batching, quantization, and the right model size for your latency budget.
Model Size vs. Latency Trade-off
- Tiny (39M): 0.05s/min audio on CPU. WER ~12% English. Use for edge, real-time chat bots.
- Base (74M): 0.15s/min on CPU, 0.03s/min on GPU. WER ~6% English. Standard choice.
- Large (1.5B): 0.5s/min on GPU. WER ~5% English. Use when accuracy trumps latency.
💼 Interview Q&A
Traditional ASR (Kaldi, Julius) uses separate acoustic models (HMM-GMM, DNN) and language models (n-gram LM). They require language-specific training data, phoneme inventories, and careful feature engineering (MFCCs).
Whisper is end-to-end: raw audio → spectrogram → transformer → text. No separate components. It's multilingual by design, handles noise robustly, and doesn't require phoneme labels. Trade-off: Whisper's 74M-1.5B params vs. Kaldi's smaller models. But Whisper is more practical for deployment.
Whisper was trained on 680K hours of multilingual data with balanced sampling strategies. When fine-tuning on Twi, we use low learning rates (1e-4) and early stopping to avoid overfitting. The pretrained encoder already captures acoustic universals (pitch, formants, vowel spaces), so fine-tuning adapts the decoder more than encoder.
Also, use regularization: LoRA (rank-4 adapters), dropout, and weight decay. Batch size of 8-16 helps; large batches amplify noisy labels.
Hallucination happens when the audio is ambiguous or very noisy, and the language model's prior dominates. For example, if a Twi speaker mumbles "...ba fie...", Whisper might complete it as "Kwame na Ama ba fie" (a common phrase). Or predict common words like "thank you" if silence is mistaken for speech.
Mitigations: (1) Fine-tune on in-domain data to bias toward your domain vocabulary. (2) Use temperature=0 for greedy decoding instead of sampling. (3) Condition on known speaker or context. (4) Post-process with language models or spell-checkers.
Bottlenecks: (1) Acoustic: overlapping speakers, car horns, vendor calls. (2) Language: market slang, code-switching (Twi + English), fast speech. (3) Deployment: mobile phone (CPU only), offline capability, real-time latency.
Solutions: (1) Speech enhancement preprocessing (noise suppression via STFT or neural networks like DEMUCS). (2) Fine-tune Whisper on market recordings with diverse speakers. (3) Add language model decoding with market-vocabulary n-gram LM. (4) Use Whisper-tiny on mobile + speech enhancement on server. (5) Batch requests: collect 30s clips, process batch-wise for 10x speedup.
WER (word error rate) is standard but incomplete. For Twi: (1) Test on held-out speakers not in training (speaker robustness). (2) Test on different noise conditions (market, home, call). (3) Use case-specific accuracy: e.g., if capturing proper nouns (places, names), use N-gram precision. (4) Compare against baseline (zero-shot Whisper, human transcription). (5) Report confidence intervals (bootstrap 95% CI over 10 random seeds).
Also: collect and publish a benchmark dataset (Twi-ASR-test-clean) for reproducibility. Publish pre-trained checkpoints on Hugging Face Hub.
⚡ Challenge: Build a Twi Transcription Service
Scenario: You're launching a voice messaging app for Twi speakers in Ghana. Users send 30-second voice notes that must be transcribed instantly.
Requirements: (1) Transcribe with <1s latency on mobile. (2) Handle background noise from markets, buses, and homes. (3) Support both Twi transcription and English translation. (4) Cost per inference < $0.001.
Your task: Design the system. What model size? How to optimize? Would you fine-tune? What about whisper hallucinations?
Hint 1: Mobile has ~100MB RAM, 1 CPU core. Whisper-tiny (39M) fits; base (74M) is tight.
Hint 2: Speech enhancement (STFT noise reduction) as preprocessing improves WER 10-20% on noisy audio.
Hint 3: Batch transcription on server side is 10x faster. Trade off latency for throughput.
📋 Cheatsheet: Whisper Commands
| Task | Code Snippet | Notes |
|---|---|---|
| Load model | WhisperForConditionalGeneration.from_pretrained("openai/whisper-base") |
tiny, base, small, medium, large |
| Transcribe (auto-detect language) | model.generate(input_features) |
No language token |
| Force language (Twi) | forced_ids = processor.get_decoder_prompt_ids(language="tw") |
Use for known language |
| Translate to English | forced_ids = processor.get_decoder_prompt_ids(language="tw", task="translate") |
Output is English |
| Add timestamps | model.generate(..., return_timestamps=True) |
Includes <|0:00|>, <|0:01|> tokens |
| Batch inference | processor(waveforms_list, return_tensors="pt") |
Pad all to max length |
| Fine-tune | Trainer(..., per_device_train_batch_size=8, learning_rate=1e-4) |
Use HuggingFace Trainer |
| Quantization (4-bit) | model = AutoModelForCausalLM.from_pretrained(..., load_in_4bit=True) |
Reduces size by 4x |
⚠️ Common Mistakes
if sr != 16000: waveform = torchaudio.transforms.Resample(sr, 16000)(waveform).
<|tw|> token in training labels, the model won't learn to expect Twi at inference. Fix: Use processor.get_decoder_prompt_ids(language="tw") and prepend to labels during training.
🎓 Key Takeaways
- Whisper is a game-changer for low-resource ASR: Zero-shot works reasonably well on 99 languages. No need to train from scratch.
- Encoder-decoder design is flexible: Condition on language and task tokens. Supports transcription, translation, and timestamps from one model.
- Fine-tuning on 100 hours of domain data drops WER by ~50%: Adapt to accents, background noise, and vocabulary. Use low learning rates (1e-4) and early stopping.
- Multilingual training is a superpower: Cross-lingual knowledge transfers. Twi benefits from YouTube's Hausa and Yoruba data via shared acoustic/linguistic patterns.
- Timestamps are built-in: No separate post-processing needed. Great for subtitles, captions, and audio sync.
- Hallucinations happen: Ambiguous audio + strong language model prior. Use fine-tuning, temperature=0, and post-processing to mitigate.
- Choose model size by latency budget: Tiny for mobile (<0.1s/min), base for standard (0.15s/min), large for batch (0.5s/min).
- Speech enhancement preprocessing is worth it: Reduces WER 10-20% on noisy market/street audio. Simple STFT filtering or neural denoising.
Ready to Build Multilingual Speech AI?
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →