Showing all 55 terms
How to use this glossary:
Each term includes a concise definition and a link to the lesson where it was first introduced. Use the search box above to filter by keyword. Terms are grouped alphabetically for browsing.
A-B Activation Function through BPE
Activation Function
A non-linear function applied to a neuron's output to introduce non-linearity into the network. Common activation functions include sigmoid (squashes to 0-1), ReLU (zeroes out negatives), tanh (squashes to -1 to 1), and GELU (smooth approximation of ReLU used in modern Transformers).
Introduced in: Lesson 1 — The Neuron
Attention
A mechanism that allows a model to selectively focus on relevant parts of the input when producing each output. Instead of compressing the entire input into a single fixed vector, attention computes a weighted sum over all input positions, with weights determined by relevance to the current query.
Introduced in: Lesson 9 — Attention Mechanism
Autoregressive
A generation strategy where the model produces tokens one at a time, feeding each predicted token back as input for the next prediction. GPT and all modern LLMs are autoregressive: they predict the next token conditioned on all previous tokens.
Introduced in: Lesson 14 — GPT Architecture
Backpropagation
The algorithm for computing gradients of the loss function with respect to every parameter in the network, using the chain rule applied backward from the output layer to the input layer. Backpropagation enables gradient descent to update all weights efficiently in a single backward pass.
Introduced in: Lesson 3 — Backpropagation
Batch Size
The number of training examples processed together in one forward and backward pass. Larger batch sizes give more stable gradient estimates but require more memory. Mini-batch gradient descent (batch size between 1 and the full dataset) is the standard approach for training neural networks.
Introduced in: Lesson 2 — Neural Network from NumPy
BPE (Byte-Pair Encoding)
A subword tokenization algorithm that iteratively merges the most frequent pair of adjacent tokens in the training corpus. BPE balances between character-level and word-level tokenization, handling rare words by splitting them into known subwords. Used by GPT-2, GPT-3, and most modern LLMs.
Introduced in: Lesson 6 — Tokenization
C-D Causal Masking through Dropout
Causal Masking
A masking technique that prevents each position in the sequence from attending to future positions. Implemented as a lower-triangular matrix applied to attention scores before softmax. Causal masking is what makes GPT autoregressive — each token can only "see" tokens that came before it.
Introduced in: Lesson 14 — GPT Architecture
Chain Rule
The calculus rule for computing the derivative of composed functions: if y = f(g(x)), then dy/dx = f'(g(x)) * g'(x). The chain rule is the mathematical foundation of backpropagation, enabling gradients to flow backward through every layer of a deep network.
Introduced in: Lesson 3 — Backpropagation
CBOW (Continuous Bag of Words)
A Word2Vec variant that predicts the center word from its surrounding context words. CBOW averages the embeddings of context words and uses this average to predict the target word. It is faster to train than Skip-Gram but less effective for rare words.
Introduced in: Lesson 5 — Word Embeddings
Context Vector
In the original seq2seq architecture, the single fixed-length vector produced by the encoder to summarize the entire input sequence. The context vector is the "bottleneck" that motivated the invention of attention — long sequences lose information when compressed into one vector.
Introduced in: Lesson 8 — Seq2Seq
Cross-Attention
An attention mechanism where the queries come from the decoder and the keys and values come from the encoder output. Cross-attention allows the decoder to "look at" the encoder's representations at every step, which is how encoder-decoder Transformers handle translation and summarization.
Introduced in: Lesson 13 — The Full Transformer
Cross-Entropy Loss
A loss function that measures the difference between the predicted probability distribution and the actual (one-hot) distribution over classes. For language modeling, cross-entropy loss compares the model's predicted token probabilities against the true next token. Lower cross-entropy means better predictions.
Introduced in: Lesson 3 — Backpropagation
d_modelThe embedding dimension, also called the model width. Every token is represented as a vector of this size throughout the Transformer. All sub-layers (attention, FFN, layer norm) operate on vectors of dimension d_model. Typical values: 768 (GPT-2 Small), 12288 (GPT-3).
Introduced in: Lesson 12 — Transformer Block
DPO (Direct Preference Optimization)
A simplified alternative to RLHF that skips training a separate reward model. DPO directly optimizes the language model to prefer human-chosen responses over rejected ones using a classification-style loss derived from the RLHF objective. It is simpler to implement and more stable to train than PPO-based RLHF.
Introduced in: Lesson 19 — RLHF
Dropout
A regularization technique that randomly sets a fraction of neuron activations to zero during training. This prevents neurons from co-adapting and forces the network to learn more robust features. Dropout is applied after attention and FFN sub-layers in Transformers. It is disabled during inference.
Introduced in: Lesson 2 — Neural Network from NumPy
E-F Embedding through Fine-Tuning
Embedding
A dense, learned vector representation of a discrete token (word, subword, or character). Embeddings map tokens from a high-dimensional one-hot space into a lower-dimensional continuous space where semantically similar tokens are close together. The embedding layer is typically the first layer of an LLM.
Introduced in: Lesson 5 — Word Embeddings
Encoder-Decoder
An architecture with two distinct stacks: an encoder that reads the input sequence and a decoder that generates the output sequence. The original Transformer uses this design for translation. The encoder processes the source language, and the decoder generates the target language using cross-attention to read the encoder output.
Introduced in: Lesson 13 — The Full Transformer
Epoch
One complete pass through the entire training dataset. Training for multiple epochs means the model sees every example more than once. Too few epochs leads to underfitting; too many can cause overfitting. Modern LLMs often train for less than one epoch over their massive datasets.
Introduced in: Lesson 2 — Neural Network from NumPy
Feed-Forward Network (FFN)
A sub-layer in each Transformer block consisting of two linear transformations with a non-linear activation between them. The FFN expands the representation to 4x d_model, applies GELU or ReLU, then projects back to d_model. This is where much of the "knowledge" in a Transformer is stored.
Introduced in: Lesson 12 — Transformer Block
Fine-Tuning
The process of adapting a pre-trained model to a specific downstream task by continuing training on task-specific data. Fine-tuning adjusts all (or some) of the model's weights, leveraging the knowledge learned during pre-training. It is far cheaper than training from scratch.
Introduced in: Lesson 18 — Supervised Fine-Tuning
G-H GGUF through Hidden State
GGUF
A quantized model file format designed for efficient local inference, popularized by llama.cpp. GGUF files store model weights in reduced precision (INT4, INT8) along with metadata, making it possible to run large models on consumer hardware without a GPU.
Introduced in: Lesson 20 — Evaluation & Deployment
Gradient Clipping
A technique that caps the magnitude of gradients during training to prevent exploding gradients. If the total gradient norm exceeds a threshold, all gradients are scaled down proportionally. Gradient clipping is essential for stable training of deep networks like Transformers.
Introduced in: Lesson 15 — Training Your Mini-GPT
Gradient Descent
An optimization algorithm that updates model parameters by moving in the direction of the negative gradient of the loss function. Each step reduces the loss by a small amount proportional to the learning rate. Variants include SGD, Adam, and AdamW, which are used to train all modern neural networks.
Introduced in: Lesson 3 — Backpropagation
Hidden State
The internal representation maintained by a recurrent neural network (RNN) at each timestep. The hidden state carries information from all previous timesteps forward, acting as the network's "memory." In practice, hidden states struggle to retain information over long sequences due to the vanishing gradient problem.
Introduced in: Lesson 7 — RNN & LSTM
I-K In-Context Learning through KL Divergence
In-Context Learning
The ability of large language models to learn new tasks at inference time from examples provided in the prompt, without any weight updates. The model uses its attention mechanism to identify patterns in the few-shot examples and apply them. This emergent ability appears primarily in large-scale models (100B+ parameters).
Introduced in: Lesson 17 — Scaling Laws
KL Divergence
A measure of how one probability distribution differs from a reference distribution. In RLHF, KL divergence is used as a penalty to prevent the fine-tuned model from deviating too far from the original pre-trained model, maintaining the model's general language capabilities while improving alignment.
Introduced in: Lesson 19 — RLHF
L Layer Normalization through LSTM
Layer Normalization
A normalization technique that normalizes activations across all features within a single layer for each example independently. Unlike batch normalization, layer norm does not depend on batch size, making it suitable for sequence models. Modern Transformers use "pre-norm" (LayerNorm before each sub-layer).
Introduced in: Lesson 12 — Transformer Block
Learning Rate
The step size used for parameter updates during gradient descent. Too large a learning rate causes divergence; too small causes slow convergence. Modern LLM training uses learning rate schedules with warmup (gradually increasing) followed by cosine decay (gradually decreasing).
Introduced in: Lesson 3 — Backpropagation
LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning method that freezes the original model weights and injects small trainable low-rank matrices into each layer. Instead of updating all parameters, LoRA only trains the low-rank decomposition (A and B matrices), reducing trainable parameters by 100-1000x while achieving comparable performance to full fine-tuning.
Introduced in: Lesson 19 — RLHF
Loss Function
A function that measures how wrong the model's predictions are compared to the true targets. The loss function is what the optimizer minimizes during training. Common choices include cross-entropy loss (for classification and language modeling) and mean squared error (for regression).
Introduced in: Lesson 1 — The Neuron
LSTM (Long Short-Term Memory)
A gated variant of the RNN that uses forget, input, and output gates to control information flow. The cell state acts as a "highway" that allows gradients to flow unchanged over long sequences, mitigating the vanishing gradient problem. LSTMs were the dominant sequence model before Transformers.
Introduced in: Lesson 7 — RNN & LSTM
M-N MMLU through Next-Token Prediction
MMLU
Massive Multitask Language Understanding — a benchmark that tests a model's knowledge across 57 academic subjects including STEM, humanities, and social sciences. MMLU scores are widely used to compare LLM capabilities, with human expert performance around 89% and top LLMs now exceeding 90%.
Introduced in: Lesson 20 — Evaluation & Deployment
Multi-Head Attention
A mechanism that runs multiple attention operations in parallel, each with different learned projection matrices. Each "head" can attend to different aspects of the input (e.g., syntactic relationships, semantic similarity). The outputs of all heads are concatenated and projected back to d_model.
Introduced in: Lesson 10 — Multi-Head Attention
Next-Token Prediction
The training objective for causal language models: given a sequence of tokens, predict the next token. Despite its simplicity, this objective forces the model to learn grammar, facts, reasoning, and world knowledge to make accurate predictions. It is the foundation of GPT, Claude, and all autoregressive LLMs.
Introduced in: Lesson 14 — GPT Architecture
P Perplexity through Pre-training
Perplexity
A metric defined as exp(loss) that measures how "surprised" a language model is by the text. Lower perplexity means the model predicts the text more confidently. Perplexity is the standard intrinsic evaluation metric for language models — a perplexity of 20 means the model is as uncertain as choosing uniformly among 20 options.
Introduced in: Lesson 20 — Evaluation & Deployment
Positional Encoding
A technique for injecting sequence position information into the Transformer, which otherwise has no notion of token order. The original Transformer used fixed sinusoidal encodings; modern LLMs use learned positional embeddings or rotary position embeddings (RoPE). Without positional encoding, the model treats input as a bag of tokens.
Introduced in: Lesson 11 — Positional Encoding
PPO (Proximal Policy Optimization)
A reinforcement learning algorithm used in RLHF to fine-tune language models based on reward signals. PPO updates the model's policy (token probability distribution) in small, stable steps by clipping the ratio of new to old probabilities. It was used to train ChatGPT and is the standard RL algorithm for RLHF.
Introduced in: Lesson 19 — RLHF
Pre-training
The initial phase of training a language model on a large, unlabeled corpus using self-supervised objectives (typically next-token prediction). Pre-training teaches the model language structure, world knowledge, and reasoning patterns. It is the most expensive phase, often requiring millions of dollars in compute.
Introduced in: Lesson 15 — Training Your Mini-GPT
Q-R Quantization through RNN
Quantization
The process of reducing the numerical precision of model weights (e.g., FP32 to INT8 or INT4) to decrease model size and speed up inference. Quantization trades a small amount of accuracy for dramatic memory savings — a 70B parameter model at FP32 needs 280GB of RAM, but at INT4 only ~35GB.
Introduced in: Lesson 20 — Evaluation & Deployment
Query, Key, Value (Q, K, V)
The three learned linear projections in the attention mechanism. The query represents "what am I looking for," the key represents "what do I contain," and the value represents "what information do I provide." Attention scores are computed as the dot product of queries and keys, then used to weight the values.
Introduced in: Lesson 9 — Attention Mechanism
Residual Connection
A "shortcut" that adds the input of a sub-layer directly to its output: output = sublayer(x) + x. Residual connections allow gradients to flow directly through the network without degradation, enabling training of very deep models. Every sub-layer in a Transformer uses a residual connection.
Introduced in: Lesson 12 — Transformer Block
Reward Model
A model trained on human preference data to score how "good" a response is. Given two responses to the same prompt, humans label which they prefer, and the reward model learns to assign higher scores to preferred responses. The reward model provides the training signal for PPO during RLHF.
Introduced in: Lesson 19 — RLHF
RLHF (Reinforcement Learning from Human Feedback)
A three-step alignment process: (1) supervised fine-tuning on demonstration data, (2) training a reward model on human preferences, (3) optimizing the LLM with PPO to maximize reward while staying close to the base model. RLHF is what turns a raw language model into a helpful, harmless assistant like ChatGPT or Claude.
Introduced in: Lesson 19 — RLHF
RNN (Recurrent Neural Network)
A neural network architecture designed for sequential data, where the output at each timestep depends on the current input and the hidden state from the previous timestep. RNNs process sequences one token at a time, maintaining a hidden state as "memory." They were largely replaced by Transformers due to the vanishing gradient problem and inability to parallelize.
Introduced in: Lesson 7 — RNN & LSTM
S Scaling Laws through Softmax
Scaling Laws
Empirically observed power-law relationships between model size, dataset size, compute budget, and loss. The Chinchilla scaling laws showed that models and data should be scaled equally — a model with N parameters should be trained on roughly 20N tokens for compute-optimal training.
Introduced in: Lesson 17 — Scaling Laws
Self-Attention
An attention mechanism where the queries, keys, and values all come from the same input sequence. Each position attends to every other position in the sequence, learning contextual relationships. Self-attention is the core operation of the Transformer, replacing the recurrence of RNNs with parallel computation.
Introduced in: Lesson 9 — Attention Mechanism
Seq2Seq (Sequence-to-Sequence)
An architecture that maps an input sequence to an output sequence of potentially different length. The original seq2seq used an RNN encoder and RNN decoder connected by a context vector. It was the foundation of neural machine translation before the Transformer replaced both the encoder and decoder with attention-based stacks.
Introduced in: Lesson 8 — Seq2Seq
SFT (Supervised Fine-Tuning)
The process of fine-tuning a pre-trained language model on high-quality instruction-response pairs written by humans. SFT teaches the model to follow instructions and produce helpful responses in a conversational format. It is the first step in the RLHF pipeline (before reward modeling and PPO).
Introduced in: Lesson 18 — Supervised Fine-Tuning
Skip-Gram
A Word2Vec variant that predicts surrounding context words from the center word. Given the word "cat," Skip-Gram tries to predict nearby words like "the" and "sat." It produces better embeddings for rare words than CBOW and is the more widely used Word2Vec approach.
Introduced in: Lesson 5 — Word Embeddings
Softmax
A function that converts a vector of raw scores (logits) into a probability distribution by exponentiating each score and normalizing so the values sum to 1. In LLMs, softmax is applied to attention scores (to get attention weights) and to the final output logits (to get token probabilities).
Introduced in: Lesson 1 — The Neuron
T Temperature through Transformer
Temperature
A hyperparameter that controls the randomness of token sampling during text generation. Temperature divides the logits before softmax: lower values (e.g., 0.2) make the distribution sharper and outputs more deterministic; higher values (e.g., 1.5) make it flatter and outputs more creative. Temperature 0 is equivalent to greedy decoding.
Introduced in: Lesson 16 — Text Generation
Token
The smallest unit of text processed by the model. Tokens can be words, subwords, or individual characters depending on the tokenizer. Most modern LLMs use subword tokens (via BPE or SentencePiece), where common words are single tokens and rare words are split into multiple tokens.
Introduced in: Lesson 6 — Tokenization
Top-k Sampling
A decoding strategy that restricts sampling to only the k most probable next tokens, setting all other probabilities to zero. Top-k filtering removes low-probability tokens that could produce incoherent text, while still allowing creative variation among the top candidates.
Introduced in: Lesson 16 — Text Generation
Top-p (Nucleus) Sampling
A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds a threshold p (e.g., 0.9). Unlike top-k which uses a fixed number of tokens, top-p adapts dynamically — using fewer tokens when the model is confident and more when it is uncertain.
Introduced in: Lesson 16 — Text Generation
Transformer
The neural network architecture introduced in "Attention Is All You Need" (2017) that replaced recurrence entirely with self-attention. Transformers process all positions in parallel, enabling efficient training on GPUs. Every modern LLM — GPT, Claude, Gemini, Llama — is built on the Transformer architecture.
Introduced in: Lesson 12 — Transformer Block
V-W Vanishing Gradient through Word2Vec
Vanishing Gradient
The problem of gradients shrinking to near-zero as they propagate backward through many layers or timesteps, preventing the network from learning long-range dependencies. This is the fundamental limitation of vanilla RNNs and the reason LSTMs and Transformers were invented.
Introduced in: Lesson 7 — RNN & LSTM
Weight Tying
A technique that shares the same weight matrix between the token embedding layer and the output projection layer. Since both layers map between token IDs and d_model-dimensional vectors, sharing weights reduces the parameter count significantly (by vocab_size * d_model parameters) without hurting performance.
Introduced in: Lesson 14 — GPT Architecture
Word2Vec
A family of algorithms (Skip-Gram and CBOW) for learning dense word embeddings from large text corpora. Introduced by Mikolov et al. in 2013, Word2Vec demonstrated that word embeddings capture semantic relationships (e.g., king - man + woman = queen). It was the precursor to modern contextual embeddings.
Introduced in: Lesson 5 — Word Embeddings
No terms match your search. Try a different keyword.
Africa & India's Best AI Academy
You've learned the vocabulary.
Now build the models yourself.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →