Kwame is an NLP researcher at a tech lab in Accra. He's building sentiment analysis models for Ghanaian languages—starting with Twi and Akan. His first step: prepare raw text data (English movie reviews, Twi social media posts) for neural networks. This means tokenizing text into words or subwords, cleaning special characters, removing common stopwords, reducing words to their root forms (stemming/lemmatization), building a vocabulary, and padding/truncating sequences to uniform length.
Without preprocessing, neural networks can't learn patterns from raw text. With it—each word becomes a meaningful token, noise is removed, and sequences are shaped to feed into embedding layers, LSTMs, and Transformers.
Tokenization fundamentals — breaking text into tokens
Tokenization converts raw text into discrete units (tokens). Three approaches dominate NLP: word tokenization (split on whitespace), subword tokenization (BPE, SentencePiece), and character tokenization. Most modern systems use subword for balance between vocabulary size and representation power.
Text cleaning pipeline — remove noise, normalize case
Raw reviews contain punctuation, HTML tags, extra whitespace, mixed case, numbers—all noise. A cleaning pipeline removes or normalizes these to focus on meaningful words. Kwame's pipeline: lowercase, remove punctuation, remove extra spaces.
Stopword removal — filter common, low-signal words
Stopwords (the, is, and, a) appear everywhere but carry little semantic meaning. Removing them reduces vocabulary size and noise. NLTK provides English stopwords; for Akan/Twi, custom lists are needed.
Stemming & lemmatization — reduce words to root forms
Stemming chops suffixes (run, running, runs → run). Lemmatization uses dictionaries to find base forms (running → run, better → good). Lemmatization is slower but more accurate; stemming is faster. Choice depends on your task.
Vocabulary building — map words to indices
Neural networks work with numbers, not text. A vocabulary (word2idx mapping) assigns each unique word an integer. Kwame's movie review corpus has 7,769 unique tokens. During training, text becomes integer sequences; during inference, the same vocab maps new words to indices.
Padding & truncation — uniform sequence length
Neural network layers expect fixed-size inputs. Movie reviews vary in length (10 to 500+ words). Solution: set a max length, truncate long reviews, pad short reviews with special tokens. Kwame uses max_length=50: trim reviews over 50 words, pad shorter ones.
Complete preprocessing pipeline — end-to-end
Kwame's full pipeline: raw review → clean → tokenize → remove stopwords → lemmatize → encode to integers → pad to 50 tokens → ready for embedding layer.
How AI systems use this
Every LLM, chatbot, and sentiment analyzer starts here: raw text → preprocessed sequences → embedding layer → neural network. Transformers (BERT, GPT) use subword tokenization (BPE, WordPiece) to balance vocab coverage and model size. African language models need custom tokenizers trained on Twi, Yoruba, or Swahili corpora—no off-the-shelf solution yet. Kwame's preprocessing pipeline is the foundation for Akan sentiment analysis, machine translation, and named entity recognition.
Also used in...
Domain Spotlight
Task: Preprocess this sentence: "Yi film no de ɛnnyɛ. Aktorɔ no yɛ akɔla!" (This film is not good. The actors are bad!) Build a small Akan vocab, tokenize, and encode to integers with padding.
Hint 1: Akan stopwords: {'yi', 'de', 'no', 'yɛ'}
Hint 2: Build word2idx with Akan words: {'PAD': 0, 'film': 1, 'good': 2, 'bad': 3, 'aktor': 4}
Hint 3: Tokenize: split on spaces, lowercase, lemmatize if possible
Hint 4: Encode and pad to max_length=8
Expected: Sequence of 8 integers, padded with 0s
Interview questions: test your understanding
Quiz: 5 questions to test your knowledge
Cheatsheet: preprocessing functions & reserved tokens
| Operation | Function | Purpose |
|---|---|---|
| Sentence split | nltk.sent_tokenize(text) |
Break text into sentences |
| Word tokenize | nltk.word_tokenize(text) |
Split into words/punctuation |
| Remove punctuation | text.translate(str.maketrans('', '', string.punctuation)) |
Strip special characters |
| Remove stopwords | [w for w in tokens if w not in stopwords.words('english')] |
Filter common words |
| Lemmatize | WordNetLemmatizer().lemmatize(word) |
Reduce to base form (running→run) |
| Stem | PorterStemmer().stem(word) |
Chop suffixes (running→runn) |
| Encode to indices | [word2idx.get(w, 1) for w in tokens] |
Map words to integers |
| Pad/truncate | seq[:max_len] + [0]*(max_len-len(seq)) |
Uniform sequence length |
Common mistakes and how to fix them
text.lower() before splitting.word2idx.get(word, 1).Key takeaways
1. Tokenization is essential: Text must be split into discrete units (words or subwords) before encoding. Choice of tokenizer affects downstream model.
2. Preprocessing is pipeline: Clean → tokenize → remove stopwords → lemmatize → encode → pad. Each step removes noise and standardizes.
3. Vocabulary maps text to numbers: Neural networks need integers. word2idx is the bridge from language to computation. Handle OOV with
4. Fixed length enables batching: Padding/truncation standardizes sequence length so batches can be stacked into 2D tensors [batch_size, seq_length].
5. Language-specific considerations: English has ready-made stopwords and lemmatizers. African languages (Twi, Yoruba, Swahili) need custom tokenizers, stopword lists, and morphological rules—this is a frontier in NLP.
Master text preprocessing for African languages.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →