The next-token problem
Your GPT model outputs a vector of logits — one raw score per vocabulary token. Softmax converts these into probabilities. But how do you pick the next token from those probabilities? That single decision determines whether your model writes coherent prose, creative poetry, or incomprehensible garbage.
This lesson covers every major sampling strategy used in production LLMs: greedy decoding, temperature scaling, top-k, top-p (nucleus) sampling, and repetition penalty. These are the knobs you turn when you adjust "temperature" or "top_p" in the ChatGPT or Claude API.
Throughout this lesson, we'll use a tiny vocabulary to make the math visible. Our model has just produced these logits for the next token position:
Logits: [2.1, 1.8, 0.5, 1.2, 0.3, 1.6, 0.1, 0.8, 1.5, 0.9]
Vocab: ["the", "cat", "sat", "on", "mat", "dog", "ran", "big", "a", "is"]
After softmax at T=1.0, "the" has the highest probability at 0.228 — but it's far from certain. How we sample from this distribution determines everything.
Greedy decoding
The simplest strategy: always pick the token with the highest probability. No randomness. Completely deterministic. If you run the same prompt twice, you get the exact same output.
Temperature — controlling randomness
Temperature is the simplest and most important sampling parameter. It scales the logits before softmax, controlling how "peaked" or "flat" the probability distribution becomes.
The formula is: probs = softmax(logits / T)
- T < 1 (e.g., 0.5): Sharpens the distribution. The highest-probability token gets even more likely. More deterministic, more repetitive.
- T = 1.0: Standard softmax. No change to the distribution.
- T > 1 (e.g., 2.0): Flattens the distribution. All tokens become more equally likely. More creative, more chaotic.
- T -> 0: Approaches greedy decoding (argmax).
- T -> infinity: Approaches uniform random sampling.
Top-k sampling
Top-k sampling restricts the candidate pool: only the k most likely tokens are considered, and all others are zeroed out. After filtering, the remaining probabilities are renormalized to sum to 1, and we sample from that reduced distribution.
This prevents the model from ever choosing extremely unlikely tokens (which can cause nonsensical output), while still allowing variety among the top candidates.
Top-p (nucleus) sampling
Top-p sampling (also called nucleus sampling) solves the fixed-k problem by dynamically choosing the number of tokens. Instead of picking a fixed count, it selects the smallest set of tokens whose cumulative probability is at least p.
When the model is confident, the nucleus might contain just 2-3 tokens. When it's uncertain, the nucleus might contain 20+. This adapts to each generation step automatically.
Repetition penalty
Even with temperature and top-p, LLMs can still get stuck in loops: "The cat sat on the mat. The cat sat on the mat. The cat sat on the mat." Repetition penalty discourages the model from repeating tokens it has already generated.
The idea is simple: for any token that has already appeared in the generated text, divide its logit by a penalty factor (typically 1.1-1.3). This reduces its probability without eliminating it entirely.
PyTorch: full generate() function
Here's a complete text generation function that combines all the strategies. This is essentially what happens inside model.generate() in HuggingFace Transformers.
Temperature: creativity vs coherence
Temperature is the single most impactful parameter for controlling generation quality. Here's a practical guide:
- T = 0 (greedy): Code generation, math, factual Q&A. You want the single most likely answer. Used by default in many coding assistants.
- T = 0.3-0.5: Structured writing, summaries, translations. Mostly deterministic with minor variation. Good for business documents.
- T = 0.7-0.9: The sweet spot for most tasks. Creative enough to avoid repetition, coherent enough to make sense. ChatGPT defaults to ~0.7.
- T = 1.0-1.2: Creative writing, brainstorming, poetry. More surprising word choices. Some outputs may be incoherent.
- T > 1.5: Very experimental. Useful for generating diverse candidates to filter later. Most individual outputs will be low quality.
Run it yourself
Try these sampling strategies locally
The NumPy examples run anywhere. For the PyTorch generate() function, you need a trained model:
pip install torch numpy python generate.py
To see sampling in action with a real model, try HuggingFace:
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
input_ids = tokenizer.encode("Once upon a time", return_tensors="pt")
# Try different settings and compare!
for temp in [0.3, 0.7, 1.0, 1.5]:
output = model.generate(input_ids, max_new_tokens=30,
temperature=temp, do_sample=True,
top_p=0.9)
print(f"T={temp}: {tokenizer.decode(output[0])}\n")
On a laptop CPU, GPT-2 generates ~10 tokens/sec. You'll immediately see how temperature changes the output style.
Why this matters for LLMs
Every API call uses these strategies
When you call the ChatGPT or Claude API, the temperature and top_p parameters directly control the sampling strategies you just implemented. The generate() function above is essentially what runs server-side at every inference call.
Understanding these strategies matters because they fundamentally change model behavior without any retraining. The same model with T=0 is a deterministic code generator; with T=1.2, it's a creative writing partner. This is also why "prompt engineering" works — the model's probability distribution over tokens is shaped by the prompt, and then sampling decides which of those tokens gets selected.
Challenge
Beam search is another decoding strategy that keeps the top B candidates (beams) at each step and expands all of them. Implement a beam_search(logits_fn, start_token, beam_width=3, max_len=10) function. Compare its outputs to greedy and sampling — beam search often produces more fluent text than greedy but less diverse than sampling.
Using HuggingFace's GPT-2, generate 10 completions of "The future of artificial intelligence is" with each strategy: greedy, T=0.7, T=1.2, top-k=50, top-p=0.9, and top-p=0.95 with repetition_penalty=1.2. Rate each output on coherence (1-5) and creativity (1-5). Which combination gives the best results?
model.generate() with do_sample=True for sampling-based strategies and do_sample=False for greedy. Set num_return_sequences=10 to get multiple outputs at once. You'll likely find that T=0.7-0.9 with top_p=0.9 gives the best coherence/creativity balance.Quiz
You control how LLMs think.
Now build the career to match.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →