🌍
Real-world scenario
Kwame in Accra: Building a Twi language sentiment analyzer from raw movie reviews

Kwame is an NLP researcher at a tech lab in Accra. He's building sentiment analysis models for Ghanaian languages—starting with Twi and Akan. His first step: prepare raw text data (English movie reviews, Twi social media posts) for neural networks. This means tokenizing text into words or subwords, cleaning special characters, removing common stopwords, reducing words to their root forms (stemming/lemmatization), building a vocabulary, and padding/truncating sequences to uniform length.

Without preprocessing, neural networks can't learn patterns from raw text. With it—each word becomes a meaningful token, noise is removed, and sequences are shaped to feed into embedding layers, LSTMs, and Transformers.

5,331
Movie reviews (English)
7,769
Vocab tokens
50
Max sequence length

Tokenization fundamentals — breaking text into tokens

Tokenization converts raw text into discrete units (tokens). Three approaches dominate NLP: word tokenization (split on whitespace), subword tokenization (BPE, SentencePiece), and character tokenization. Most modern systems use subword for balance between vocabulary size and representation power.

tokenization_basic.py Word, sentence, and subword tokenization
Output
Sentences (3): 1. This movie is great! 2. The acting was fantastic. 3. I loved it. Word tokens (12): ['This', 'movie', 'is', 'great', '!', 'The', 'acting', 'was', 'fantastic', '.', 'I', 'loved', 'it', '.'] Simple split (11): ['this', 'movie', 'is', 'great!', 'the', 'acting', 'was', 'fantastic.', 'i', 'loved', 'it.'] Subword tokenization (BPE): Word: 'unbelievable' Subwords (example): ['unbe', 'lievable']
Tokenization strategies: word splits on whitespace, subword balances vocab size vs. coverage (e.g., rare words broken into subunits).
Token length distribution showing word vs subword tokenization coverage

Text cleaning pipeline — remove noise, normalize case

Raw reviews contain punctuation, HTML tags, extra whitespace, mixed case, numbers—all noise. A cleaning pipeline removes or normalizes these to focus on meaningful words. Kwame's pipeline: lowercase, remove punctuation, remove extra spaces.

text_cleaning.py Clean movie reviews from NLTK corpus
Output
Raw text: Well the idea behind the movie was clever, but too overdone. I wish... Cleaned text: well the idea behind the movie was clever but too overdone i wish Original length: 500 chars Cleaned length: 445 chars

Stopword removal — filter common, low-signal words

Stopwords (the, is, and, a) appear everywhere but carry little semantic meaning. Removing them reduces vocabulary size and noise. NLTK provides English stopwords; for Akan/Twi, custom lists are needed.

stopword_removal.py Remove stopwords from cleaned reviews
Output
Sample English stopwords: ['ourselves', 'your', 'such', 'a', 'an', 'and', 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'both', 'but', 'by'] Total: 179 stopwords Original tokens: ['This', 'movie', 'is', 'not', 'bad', 'at', 'all', 'I', 'really', 'liked', 'the', 'characters', 'and', 'the', 'plot', 'was', 'engaging'] Filtered tokens: ['movie', 'bad', 'really', 'liked', 'characters', 'plot', 'engaging'] Removed: {'is', 'at', 'all', 'I', 'the', 'and', 'was', 'This'} Akan example: Original: ['na', 'yi', 'de', 'wo', 'a', 'no', 'yi'] Filtered: []

Stemming & lemmatization — reduce words to root forms

Stemming chops suffixes (run, running, runs → run). Lemmatization uses dictionaries to find base forms (running → run, better → good). Lemmatization is slower but more accurate; stemming is faster. Choice depends on your task.

stemming_lemmatization.py Compare stemming and lemmatization
Output
Stemming (Porter): running → run runs → run run → run walked → walk walks → walk greatly → great better → better Lemmatization (WordNet): running → running runs → run run → run walked → walk walks → walk greatly → greatly better → good Review word lemmatization: acting → acting actresses → actress acted → act amazing → amazing amazingly → amazingly

Vocabulary building — map words to indices

Neural networks work with numbers, not text. A vocabulary (word2idx mapping) assigns each unique word an integer. Kwame's movie review corpus has 7,769 unique tokens. During training, text becomes integer sequences; during inference, the same vocab maps new words to indices.

vocabulary_building.py Build and use a word-to-index vocabulary
Output
Total words: 41,234 (first 100 reviews) Unique words: 7,769 Top 10 words: ,: 2341 the: 2056 .: 1843 a: 1512 and: 1301 of: 1145 to: 1087 is: 998 in: 876 it: 825 Vocabulary size: 5000 Sample mappings: 'the' → 2 'movie' → 85 'good' → 412 'bad' → 783 'unknown_word' → 1 Encode 'this movie is good': Indices: [137, 85, 89, 412]
Top 50 most frequent words in movie reviews. Zipf's law: a few words dominate, many appear rarely.
Word frequency distribution (log scale) showing Zipfian pattern

Padding & truncation — uniform sequence length

Neural network layers expect fixed-size inputs. Movie reviews vary in length (10 to 500+ words). Solution: set a max length, truncate long reviews, pad short reviews with special tokens. Kwame uses max_length=50: trim reviews over 50 words, pad shorter ones.

padding_truncation.py Pad and truncate sequences to fixed length
Output
Max length: 10 Sequence 0: Original (4): [2, 85, 89, 412] Padded (10): [2, 85, 89, 412, 0, 0, 0, 0, 0, 0] Sequence 1: Original (12): [2, 102, 145, 67, 234, 56, 78, 90, 11, 23, 34, 45] Padded (10): [2, 102, 145, 67, 234, 56, 78, 90, 11, 23] Sequence 2: Original (2): [412, 567] Padded (10): [412, 567, 0, 0, 0, 0, 0, 0, 0, 0] Shape for embedding layer: (3, 10) Dtype: int64
Class distribution: positive (5-star) vs negative (1-star) reviews in dataset. 50-50 split aids balanced training.
Bar chart showing positive/negative review count

Complete preprocessing pipeline — end-to-end

Kwame's full pipeline: raw review → clean → tokenize → remove stopwords → lemmatize → encode to integers → pad to 50 tokens → ready for embedding layer.

complete_pipeline.py Full text preprocessing pipeline
Output
Max length: 20 Review 1: 'This movie is absolutely amazing! The acting was great!' Processed: [4, 0, 5, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] Review 2: 'Terrible film. Bad plot, bad acting.' Processed: [3, 1, 3, 1, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] Review 3: 'OK' Processed: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

How AI systems use this

Every LLM, chatbot, and sentiment analyzer starts here: raw text → preprocessed sequences → embedding layer → neural network. Transformers (BERT, GPT) use subword tokenization (BPE, WordPiece) to balance vocab coverage and model size. African language models need custom tokenizers trained on Twi, Yoruba, or Swahili corpora—no off-the-shelf solution yet. Kwame's preprocessing pipeline is the foundation for Akan sentiment analysis, machine translation, and named entity recognition.

Text Preprocessing Tokenization NLP Pipelines Multilingual Models

Also used in...

Domain Spotlight

🌍
Multilingual NLP: African language processing
Twi, Yoruba, Swahili models require custom tokenizers and stopword lists tuned to each language's morphology.
💬
Chatbots & LLMs: User input preprocessing
Every user message is tokenized, cleaned, and encoded before passing to the language model.
📰
Sentiment Analysis & Topic Modeling
Movie reviews, social media posts, news articles are preprocessed to extract semantic meaning via embeddings.
🎯 Mini Challenge: Preprocess a Twi movie review

Task: Preprocess this sentence: "Yi film no de ɛnnyɛ. Aktorɔ no yɛ akɔla!" (This film is not good. The actors are bad!) Build a small Akan vocab, tokenize, and encode to integers with padding.

Hint 1: Akan stopwords: {'yi', 'de', 'no', 'yɛ'}

Hint 2: Build word2idx with Akan words: {'PAD': 0, 'film': 1, 'good': 2, 'bad': 3, 'aktor': 4}

Hint 3: Tokenize: split on spaces, lowercase, lemmatize if possible

Hint 4: Encode and pad to max_length=8

Expected: Sequence of 8 integers, padded with 0s

Interview questions: test your understanding

Stemming chops suffixes (running → runn). Lemmatization uses dictionaries to find base forms (running → run, better → good). Lemmatization is more accurate but slower.
Stopwords (the, is, and) appear everywhere but carry little meaning. Removing them reduces vocab size, noise, and improves signal-to-noise ratio in embeddings.
Vocab size: 7,769 unique tokens from 100 movie reviews. Reserved tokens: (index 0) for padding, (index 1) for unknown/out-of-vocabulary words.
Neural networks need fixed input shapes. Batches of padded sequences become 2D tensors [batch_size, max_length], which feed into embedding layers. Variable lengths break tensor operations.
Option 1: Collect Akan text (social media, news, books), train BPE on it. Option 2: Use language-agnostic subword tokenizers (SentencePiece). Option 3: Build rule-based tokenizer for Akan morphology. Kwame would likely use SentencePiece trained on a curated Akan corpus.

Quiz: 5 questions to test your knowledge

Question 1 of 5
What does word tokenization split "I don't know" into?
Question 2 of 5
Which word would NOT be removed by English stopword removal?
Question 3 of 5
Porter Stemmer reduces "running" and "runs" to what?
Question 4 of 5
If max_length=10 and a review has 15 tokens, what happens?
Question 5 of 5
What does the token represent?
0
/ 5
Great job!

Cheatsheet: preprocessing functions & reserved tokens

Operation Function Purpose
Sentence split nltk.sent_tokenize(text) Break text into sentences
Word tokenize nltk.word_tokenize(text) Split into words/punctuation
Remove punctuation text.translate(str.maketrans('', '', string.punctuation)) Strip special characters
Remove stopwords [w for w in tokens if w not in stopwords.words('english')] Filter common words
Lemmatize WordNetLemmatizer().lemmatize(word) Reduce to base form (running→run)
Stem PorterStemmer().stem(word) Chop suffixes (running→runn)
Encode to indices [word2idx.get(w, 1) for w in tokens] Map words to integers
Pad/truncate seq[:max_len] + [0]*(max_len-len(seq)) Uniform sequence length

Common mistakes and how to fix them

Forgetting to lowercase before tokenization
Issue: "Movie" and "movie" become different tokens. Fix: Always lowercase first. text.lower() before splitting.
Not handling out-of-vocabulary (OOV) words
Issue: New words at test time cause crashes. Fix: Reserve index 1 for , use word2idx.get(word, 1).
Removing stopwords blindly from sentiment data
Issue: "not" is a stopword but crucial for negation ("not good" vs "good"). Fix: Use task-specific stopword lists or skip removal entirely.
Not padding/truncating consistently
Issue: Different batch shapes cause tensor errors. Fix: Apply same max_length to all data (train, val, test).
Building vocab on preprocessed data, testing on raw data
Issue: Vocab maps "running" but test data has "Running". Fix: Preprocess consistently everywhere (train + test pipelines).

Key takeaways

1. Tokenization is essential: Text must be split into discrete units (words or subwords) before encoding. Choice of tokenizer affects downstream model.

2. Preprocessing is pipeline: Clean → tokenize → remove stopwords → lemmatize → encode → pad. Each step removes noise and standardizes.

3. Vocabulary maps text to numbers: Neural networks need integers. word2idx is the bridge from language to computation. Handle OOV with .

4. Fixed length enables batching: Padding/truncation standardizes sequence length so batches can be stacked into 2D tensors [batch_size, seq_length].

5. Language-specific considerations: English has ready-made stopwords and lemmatizers. African languages (Twi, Yoruba, Swahili) need custom tokenizers, stopword lists, and morphological rules—this is a frontier in NLP.

Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

Master text preprocessing for African languages.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →