🎙️
Meet Kwame
NLP Researcher Building Speech Recognition for Twi

Kwame is an NLP researcher from Accra building speech-to-text for Twi (Akan language, 9M speakers in Ghana). Traditional speech recognition systems require labeled audio datasets, language-specific acoustic models, and months of engineering. OpenAI's Whisper—pretrained on 680,000 hours of multilingual audio—transcribes Twi with 15% word error rate out-of-the-box. By fine-tuning on 100 hours of Ghanaian Twi speech, Kwame achieves 6% WER in production, enabling voice notes, transcription services, and accessibility tools for Twi speakers.

680K
Hours trained
6%
WER (fine-tuned)
99
Languages

🔊 What Is Whisper?

Whisper is OpenAI's automatic speech recognition (ASR) model trained on 680,000 hours of multilingual and multitask audio data from the web. Unlike earlier ASR models optimized for English or specific domains (podcasts, phone calls), Whisper is trained on diverse audio—noisy background, accents, music, background speech—making it robust to real-world conditions. It supports 99 languages and performs not just transcription but also language detection, sentence-level timestamps, and translation to English.

Why Whisper Is Revolutionary

  • Multilingual from day one: Single model handles 99 languages. No need to train language-specific models or download separate acoustic models.
  • Robust to real-world audio: Trained on loud cafes, music, background speech, accents. Works on podcasts, YouTube videos, phone recordings, and conference calls.
  • Task-agnostic: Transcription, translation, language identification, and timestamp generation from one model. Condition on task tokens.
  • Modest compute requirements: Whisper-small (244M params) runs on CPU. Whisper-base (74M) fits on edge devices. No need for GPUs to transcribe.
  • Zero-shot performance: Works well on unseen languages without fine-tuning. Fine-tuning on 100 hours drops WER by 50%.
Whisper model sizes: tiny (39M) to large (1.5B params). Trade accuracy for speed and memory.
Whisper model sizes and WER performance comparison
Whisper-large achieves 5% WER on LibriSpeech test-clean (English). Whisper-base: 6.3% WER, 0.3s per minute of audio. Whisper-tiny: 12% WER, 0.05s per minute. Choose tiny for speed/edge, base for balanced accuracy/latency, large for best quality.

🏗️ Architecture: Encoder-Decoder with Attention

Whisper uses a standard transformer encoder-decoder architecture, similar to machine translation models like T5 or MarianMT. The encoder processes audio spectrograms; the decoder generates tokens autoregressively.

Encoder (Audio Feature Extraction)

  • Input: Raw audio (16kHz sample rate) → mel-spectrogram (80 mel-frequency bins, 50ms frame).
  • Sinusoidal positional encoding: Each frame gets a learned position embedding (no absolute time, but relative position).
  • Transformer encoder layers: 4-12 layers of self-attention + feed-forward. Each layer attends to all frames in the audio.
  • Output: Context vector for each frame (~1500 vectors for 30 seconds of audio).

Decoder (Text Generation)

  • Input tokens: Special task tokens (e.g., <|transcribe|>, <|translate|>), language token (e.g., <|tw|> for Twi), and previously generated text tokens.
  • Cross-attention: Attends to encoder outputs (audio context) while generating. "hearing" audio features while "writing" text.
  • Autoregressive generation: Generates one token at a time. P(token_t | audio, token_0...token_{t-1}).
  • Temperature sampling: At inference, use temperature=0 for deterministic output or temperature>0 for diverse outputs.
Whisper architecture (conceptual)

🌍 Multilingual Training & Language Detection

Whisper was trained on 680K hours of audio from YouTube covering 99 languages. The training mix is imbalanced: English dominates (54%), but languages like Twi, Hausa, Yoruba (African languages) are well-represented due to YouTube's global reach.

Language Token & Task Token Conditioning

At training and inference, Whisper prepends special tokens to the decoder input:

  • <|transcribe|> or <|translate|>: Task token. Transcribe outputs native language; translate outputs English.
  • <|tw|>, <|fr|>, <|de|>: Language token. Tells decoder which language to expect.
  • <|notimestamps|>: Suppress timestamp tokens. Default includes <|0:00|>, <|0:01|>, etc.
Whisper training data per language (partial list). English: 54%, Spanish: 4%, Mandarin: 2%, Twi & Yoruba: ~0.3% each.
Whisper training distribution across languages
Despite low-resource African languages being only 0.3% of training data, Whisper generalizes well due to cross-lingual knowledge. Fine-tuning on 100 hours of Twi data (6K utterances) drops WER from 15% to 6%, outperforming English-only models at the same data size.
Language detection & transcription in Twi

⏱️ Timestamps & Sentence Segmentation

Whisper can generate word-level and sentence-level timestamps without additional training. The decoder produces special tokens like <|0:00|>, <|0:01|> interspersed with text, indicating when each word starts.

Timestamp Generation

During training, Whisper sees (audio, text, timestamps) tuples. It learns to predict timestamp tokens at the right positions. At inference, decode with timestamps enabled to get word-level timing information, useful for subtitles, captions, or audio synchronization.

Generate timestamps with Whisper

🎓 Fine-tuning Whisper on Custom Speech Data

Kwame has 100 hours of Ghanaian Twi speech recordings (6,000 utterances, 5-60 seconds each) with manual transcriptions. Fine-tuning Whisper-base on this data reduces WER from 15% to 6% and adapts the model to Ghanaian accents, background noise, and domain-specific vocabulary.

Fine-tuning Workflow

  • 1. Prepare dataset: Audio files + transcriptions in JSON/CSV. Load with HuggingFace datasets library.
  • 2. Load pretrained Whisper: openai/whisper-base (with language='tw' for Twi).
  • 3. Set up trainer: HuggingFace Trainer with speech-specific loss (CTC or Whisper's native cross-entropy).
  • 4. Train: 3-5 epochs on 8 GPUs, effective batch size 128. Learning rate: 1e-4. Dropout: 0.1.
  • 5. Evaluate: WER on held-out test set. Compare with baseline zero-shot Whisper.
Fine-tuning Whisper on Twi speech data
Fine-tuning results on Kwame's 100-hour Twi dataset: WER drops from 15% (zero-shot) to 6% (fine-tuned). Training takes 8 hours on 1 GPU. Freezing the encoder saves 30% compute with only 0.5% WER increase, useful for edge deployment.

🚀 Inference: Batch Processing & Optimization

For production, Whisper inference should be fast and memory-efficient. Use batching, quantization, and the right model size for your latency budget.

Production inference pipeline

Model Size vs. Latency Trade-off

  • Tiny (39M): 0.05s/min audio on CPU. WER ~12% English. Use for edge, real-time chat bots.
  • Base (74M): 0.15s/min on CPU, 0.03s/min on GPU. WER ~6% English. Standard choice.
  • Large (1.5B): 0.5s/min on GPU. WER ~5% English. Use when accuracy trumps latency.

💼 Interview Q&A

Traditional ASR (Kaldi, Julius) uses separate acoustic models (HMM-GMM, DNN) and language models (n-gram LM). They require language-specific training data, phoneme inventories, and careful feature engineering (MFCCs).

Whisper is end-to-end: raw audio → spectrogram → transformer → text. No separate components. It's multilingual by design, handles noise robustly, and doesn't require phoneme labels. Trade-off: Whisper's 74M-1.5B params vs. Kaldi's smaller models. But Whisper is more practical for deployment.

Whisper was trained on 680K hours of multilingual data with balanced sampling strategies. When fine-tuning on Twi, we use low learning rates (1e-4) and early stopping to avoid overfitting. The pretrained encoder already captures acoustic universals (pitch, formants, vowel spaces), so fine-tuning adapts the decoder more than encoder.

Also, use regularization: LoRA (rank-4 adapters), dropout, and weight decay. Batch size of 8-16 helps; large batches amplify noisy labels.

Hallucination happens when the audio is ambiguous or very noisy, and the language model's prior dominates. For example, if a Twi speaker mumbles "...ba fie...", Whisper might complete it as "Kwame na Ama ba fie" (a common phrase). Or predict common words like "thank you" if silence is mistaken for speech.

Mitigations: (1) Fine-tune on in-domain data to bias toward your domain vocabulary. (2) Use temperature=0 for greedy decoding instead of sampling. (3) Condition on known speaker or context. (4) Post-process with language models or spell-checkers.

Bottlenecks: (1) Acoustic: overlapping speakers, car horns, vendor calls. (2) Language: market slang, code-switching (Twi + English), fast speech. (3) Deployment: mobile phone (CPU only), offline capability, real-time latency.

Solutions: (1) Speech enhancement preprocessing (noise suppression via STFT or neural networks like DEMUCS). (2) Fine-tune Whisper on market recordings with diverse speakers. (3) Add language model decoding with market-vocabulary n-gram LM. (4) Use Whisper-tiny on mobile + speech enhancement on server. (5) Batch requests: collect 30s clips, process batch-wise for 10x speedup.

WER (word error rate) is standard but incomplete. For Twi: (1) Test on held-out speakers not in training (speaker robustness). (2) Test on different noise conditions (market, home, call). (3) Use case-specific accuracy: e.g., if capturing proper nouns (places, names), use N-gram precision. (4) Compare against baseline (zero-shot Whisper, human transcription). (5) Report confidence intervals (bootstrap 95% CI over 10 random seeds).

Also: collect and publish a benchmark dataset (Twi-ASR-test-clean) for reproducibility. Publish pre-trained checkpoints on Hugging Face Hub.

⚡ Challenge: Build a Twi Transcription Service

Your Task

Scenario: You're launching a voice messaging app for Twi speakers in Ghana. Users send 30-second voice notes that must be transcribed instantly.

Requirements: (1) Transcribe with <1s latency on mobile. (2) Handle background noise from markets, buses, and homes. (3) Support both Twi transcription and English translation. (4) Cost per inference < $0.001.

Your task: Design the system. What model size? How to optimize? Would you fine-tune? What about whisper hallucinations?

Hint 1: Mobile has ~100MB RAM, 1 CPU core. Whisper-tiny (39M) fits; base (74M) is tight.

Hint 2: Speech enhancement (STFT noise reduction) as preprocessing improves WER 10-20% on noisy audio.

Hint 3: Batch transcription on server side is 10x faster. Trade off latency for throughput.

📋 Cheatsheet: Whisper Commands

Task Code Snippet Notes
Load model WhisperForConditionalGeneration.from_pretrained("openai/whisper-base") tiny, base, small, medium, large
Transcribe (auto-detect language) model.generate(input_features) No language token
Force language (Twi) forced_ids = processor.get_decoder_prompt_ids(language="tw") Use for known language
Translate to English forced_ids = processor.get_decoder_prompt_ids(language="tw", task="translate") Output is English
Add timestamps model.generate(..., return_timestamps=True) Includes <|0:00|>, <|0:01|> tokens
Batch inference processor(waveforms_list, return_tensors="pt") Pad all to max length
Fine-tune Trainer(..., per_device_train_batch_size=8, learning_rate=1e-4) Use HuggingFace Trainer
Quantization (4-bit) model = AutoModelForCausalLM.from_pretrained(..., load_in_4bit=True) Reduces size by 4x

⚠️ Common Mistakes

Mistake #1: Not resampling audio to 16kHz
Whisper expects 16kHz mono. If you feed 48kHz stereo, the spectrogram is malformed. Fix: Always check and resample: if sr != 16000: waveform = torchaudio.transforms.Resample(sr, 16000)(waveform).
Mistake #2: Forgetting to set language token during fine-tuning
If you train on Twi but don't include the <|tw|> token in training labels, the model won't learn to expect Twi at inference. Fix: Use processor.get_decoder_prompt_ids(language="tw") and prepend to labels during training.
Mistake #3: Using too high a learning rate during fine-tuning
Fine-tuning at lr=1e-3 (typical for classification) will diverge or overfit. Whisper is already well-trained; 1e-4 or even 1e-5 is safer. Fix: Start with 1e-4; reduce if loss spikes.
Mistake #4: Inferring on too-long audio at once
Whisper's position embeddings are trained on 30-second clips. Feeding 10-minute audio may cause OOM or degraded accuracy. Fix: Split audio into 30s windows, transcribe each, concatenate results.
Mistake #5: Ignoring data quality during labeling
If your training transcriptions have errors (typos, missing words), the model learns those errors. Fix: Use multiple annotators; compute inter-annotator agreement; remove low-confidence samples.

🎓 Key Takeaways

  • Whisper is a game-changer for low-resource ASR: Zero-shot works reasonably well on 99 languages. No need to train from scratch.
  • Encoder-decoder design is flexible: Condition on language and task tokens. Supports transcription, translation, and timestamps from one model.
  • Fine-tuning on 100 hours of domain data drops WER by ~50%: Adapt to accents, background noise, and vocabulary. Use low learning rates (1e-4) and early stopping.
  • Multilingual training is a superpower: Cross-lingual knowledge transfers. Twi benefits from YouTube's Hausa and Yoruba data via shared acoustic/linguistic patterns.
  • Timestamps are built-in: No separate post-processing needed. Great for subtitles, captions, and audio sync.
  • Hallucinations happen: Ambiguous audio + strong language model prior. Use fine-tuning, temperature=0, and post-processing to mitigate.
  • Choose model size by latency budget: Tiny for mobile (<0.1s/min), base for standard (0.15s/min), large for batch (0.5s/min).
  • Speech enhancement preprocessing is worth it: Reduces WER 10-20% on noisy market/street audio. Simple STFT filtering or neural denoising.
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Ready to Build Multilingual Speech AI?

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →