The next-token problem

Your GPT model outputs a vector of logits — one raw score per vocabulary token. Softmax converts these into probabilities. But how do you pick the next token from those probabilities? That single decision determines whether your model writes coherent prose, creative poetry, or incomprehensible garbage.

This lesson covers every major sampling strategy used in production LLMs: greedy decoding, temperature scaling, top-k, top-p (nucleus) sampling, and repetition penalty. These are the knobs you turn when you adjust "temperature" or "top_p" in the ChatGPT or Claude API.

🎲
Running example
A toy vocabulary with 10 tokens

Throughout this lesson, we'll use a tiny vocabulary to make the math visible. Our model has just produced these logits for the next token position:

Logits: [2.1, 1.8, 0.5, 1.2, 0.3, 1.6, 0.1, 0.8, 1.5, 0.9]

Vocab: ["the", "cat", "sat", "on", "mat", "dog", "ran", "big", "a", "is"]

After softmax at T=1.0, "the" has the highest probability at 0.228 — but it's far from certain. How we sample from this distribution determines everything.

Greedy decoding

The simplest strategy: always pick the token with the highest probability. No randomness. Completely deterministic. If you run the same prompt twice, you get the exact same output.

Logits
->
Softmax
->
argmax
->
Token: "the"
greedy_decoding.py Read-only — NumPy
Expected output
Greedy: 'the'
The problem with greedy:
Greedy decoding is fast and coherent, but it produces boring, repetitive text. Given "The cat sat on the", it will always predict "mat" — never "couch" or "windowsill". It also gets stuck in loops: "I think that I think that I think that..." This is why every modern LLM uses some form of sampling.

Temperature — controlling randomness

Temperature is the simplest and most important sampling parameter. It scales the logits before softmax, controlling how "peaked" or "flat" the probability distribution becomes.

The formula is: probs = softmax(logits / T)

  • T < 1 (e.g., 0.5): Sharpens the distribution. The highest-probability token gets even more likely. More deterministic, more repetitive.
  • T = 1.0: Standard softmax. No change to the distribution.
  • T > 1 (e.g., 2.0): Flattens the distribution. All tokens become more equally likely. More creative, more chaotic.
  • T -> 0: Approaches greedy decoding (argmax).
  • T -> infinity: Approaches uniform random sampling.
temperature_scaling.py Read-only — NumPy
Expected output
Temperature effects: T=0.5: Top token: 'the' (0.380) T=1.0: Top token: 'the' (0.228) T=1.5: Top token: 'the' (0.181) T=2.0: Top token: 'the' (0.158)
Probability Distribution at Different Temperatures Same logits, different temperatures — watch the distribution flatten 0.40 0.30 0.20 0.10 0.00 "the" "cat" "dog" "a" "on" T=0.5 (sharp) T=1.0 (normal) T=2.0 (flat)
Key insight:
At T=0.5, "the" dominates with 38% probability — nearly twice its T=1.0 value. At T=2.0, the distribution is nearly uniform — "the" only has 15.8%, barely more than "on" at 10.1%. Temperature doesn't change which token is most likely, only how much more likely it is.

Top-k sampling

Top-k sampling restricts the candidate pool: only the k most likely tokens are considered, and all others are zeroed out. After filtering, the remaining probabilities are renormalized to sum to 1, and we sample from that reduced distribution.

This prevents the model from ever choosing extremely unlikely tokens (which can cause nonsensical output), while still allowing variety among the top candidates.

top_k_sampling.py Read-only — NumPy
Expected output
Top-3 tokens: ['the', 'cat', 'dog'] with probs ['0.228', '0.169', '0.139'] Renormalized: ['0.426', '0.316', '0.258'] Sampled: 'cat'
The problem with fixed k:
A fixed k value doesn't adapt to the shape of the distribution. If the model is very confident (one token has 95% probability), k=50 still considers 49 unlikely tokens. If the model is uncertain (many tokens near 5%), k=5 might cut off perfectly reasonable options. This motivated top-p sampling.

Top-p (nucleus) sampling

Top-p sampling (also called nucleus sampling) solves the fixed-k problem by dynamically choosing the number of tokens. Instead of picking a fixed count, it selects the smallest set of tokens whose cumulative probability is at least p.

When the model is confident, the nucleus might contain just 2-3 tokens. When it's uncertain, the nucleus might contain 20+. This adapts to each generation step automatically.

top_p_sampling.py Read-only — NumPy
Expected output
Top-p (p=0.8): Sorted tokens: ['the', 'cat', 'dog', 'a', 'on', 'is', 'big', 'sat', 'mat', 'ran'] Cumulative probs: ['0.228', '0.398', '0.536', '0.661', '0.754', '0.823'] Nucleus (6 tokens): ['the', 'cat', 'dog', 'a', 'on', 'is'] Sampled: 'dog'
Top-k vs Top-p Selection on Sorted Probabilities Tokens sorted by probability (descending) — same logits at T=1.0 0.25 0.20 0.15 0.10 0.05 0.00 Top-k = 3 Top-p = 0.8 (6 tokens, cumulative = 0.823) .228 the .169 cat .139 dog .125 a .093 on .069 is .062 big .046 sat .038 mat .031 ran Cumulative: 0.228 -> 0.398 -> 0.536 -> 0.661 -> 0.754 -> 0.823 (crosses p=0.8) In top-k (k=3) Extra tokens in top-p (p=0.8) Excluded by both Top-p adapts: 6 tokens here, but could be 2 if the model were more confident
Top-k vs top-p in practice:
Most modern APIs (OpenAI, Anthropic, Google) default to top-p sampling, not top-k. The adaptive nature of top-p makes it more robust — it naturally takes fewer risks when the model is confident, and explores more when it's uncertain. GPT-4 and Claude both use top-p by default.

Repetition penalty

Even with temperature and top-p, LLMs can still get stuck in loops: "The cat sat on the mat. The cat sat on the mat. The cat sat on the mat." Repetition penalty discourages the model from repeating tokens it has already generated.

The idea is simple: for any token that has already appeared in the generated text, divide its logit by a penalty factor (typically 1.1-1.3). This reduces its probability without eliminating it entirely.

repetition_penalty.py Read-only — NumPy
Expected output
Before penalty: the: 0.2284 <-- already generated cat: 0.1692 <-- already generated sat: 0.0461 on: 0.0928 mat: 0.0377 dog: 0.1385 ran: 0.0309 big: 0.0622 a: 0.1253 is: 0.0688 After penalty (factor=1.2): the: 0.1811 <-- penalized cat: 0.1410 <-- penalized sat: 0.0519 on: 0.1045 mat: 0.0425 dog: 0.1559 ran: 0.0348 big: 0.0700 a: 0.1410 is: 0.0774 'the' probability: 0.2284 -> 0.1811 'cat' probability: 0.1692 -> 0.1410
How much penalty?
A penalty of 1.0 means no change. Values of 1.1-1.2 give mild discouragement — good for coherent text that avoids exact repetition. Values of 1.5+ strongly penalize repetition — useful for creative tasks but can cause the model to produce unnatural word choices to avoid repeating anything.

PyTorch: full generate() function

Here's a complete text generation function that combines all the strategies. This is essentially what happens inside model.generate() in HuggingFace Transformers.

generate.py Read-only — PyTorch
Expected output
generate() function defined successfully. Strategies: greedy (T=0), temperature, top-k, top-p, repetition penalty Order of application: rep_penalty -> temperature -> top-k -> top-p -> sample
Order matters!
The standard pipeline is: (1) repetition penalty on raw logits, (2) temperature scaling, (3) top-k filtering, (4) top-p filtering, (5) sample. If you apply temperature after filtering, you'd re-flatten a distribution that was already sharpened — not what you want.

Temperature: creativity vs coherence

Temperature is the single most impactful parameter for controlling generation quality. Here's a practical guide:

🎛
Practical settings
When to use which temperature
  • T = 0 (greedy): Code generation, math, factual Q&A. You want the single most likely answer. Used by default in many coding assistants.
  • T = 0.3-0.5: Structured writing, summaries, translations. Mostly deterministic with minor variation. Good for business documents.
  • T = 0.7-0.9: The sweet spot for most tasks. Creative enough to avoid repetition, coherent enough to make sense. ChatGPT defaults to ~0.7.
  • T = 1.0-1.2: Creative writing, brainstorming, poetry. More surprising word choices. Some outputs may be incoherent.
  • T > 1.5: Very experimental. Useful for generating diverse candidates to filter later. Most individual outputs will be low quality.
Common API defaults:
OpenAI GPT-4: T=1.0, top_p=1.0 (effectively just temperature). Anthropic Claude: T=1.0, top_p=0.999. Both allow T=0 for greedy decoding. When using top-p, it's common to keep temperature at 1.0 and let top-p do the filtering — combining low temperature with low top-p can make the model too conservative.

Run it yourself

Try these sampling strategies locally

The NumPy examples run anywhere. For the PyTorch generate() function, you need a trained model:

pip install torch numpy
python generate.py

To see sampling in action with a real model, try HuggingFace:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

input_ids = tokenizer.encode("Once upon a time", return_tensors="pt")

# Try different settings and compare!
for temp in [0.3, 0.7, 1.0, 1.5]:
    output = model.generate(input_ids, max_new_tokens=30,
                            temperature=temp, do_sample=True,
                            top_p=0.9)
    print(f"T={temp}: {tokenizer.decode(output[0])}\n")

On a laptop CPU, GPT-2 generates ~10 tokens/sec. You'll immediately see how temperature changes the output style.

Why this matters for LLMs

Every API call uses these strategies

When you call the ChatGPT or Claude API, the temperature and top_p parameters directly control the sampling strategies you just implemented. The generate() function above is essentially what runs server-side at every inference call.

Understanding these strategies matters because they fundamentally change model behavior without any retraining. The same model with T=0 is a deterministic code generator; with T=1.2, it's a creative writing partner. This is also why "prompt engineering" works — the model's probability distribution over tokens is shaped by the prompt, and then sampling decides which of those tokens gets selected.

temperature=0 -> deterministic top_p=0.9 -> nucleus sampling repetition_penalty=1.1 GPT-4: T=1.0, top_p=1.0

Challenge

Put it into practice
Challenge 1: Implement beam search

Beam search is another decoding strategy that keeps the top B candidates (beams) at each step and expands all of them. Implement a beam_search(logits_fn, start_token, beam_width=3, max_len=10) function. Compare its outputs to greedy and sampling — beam search often produces more fluent text than greedy but less diverse than sampling.

Maintain a list of (sequence, score) tuples. At each step, expand each beam by all vocab tokens, compute the new scores (sum of log-probabilities), keep only the top beam_width candidates, and repeat. The final output is the highest-scoring complete sequence.
Challenge 2: Compare sampling strategies on GPT-2

Using HuggingFace's GPT-2, generate 10 completions of "The future of artificial intelligence is" with each strategy: greedy, T=0.7, T=1.2, top-k=50, top-p=0.9, and top-p=0.95 with repetition_penalty=1.2. Rate each output on coherence (1-5) and creativity (1-5). Which combination gives the best results?

Use model.generate() with do_sample=True for sampling-based strategies and do_sample=False for greedy. Set num_return_sequences=10 to get multiple outputs at once. You'll likely find that T=0.7-0.9 with top_p=0.9 gives the best coherence/creativity balance.

Quiz

Question 1 of 5
What does temperature T=0 produce in text generation?
Question 2 of 5
How does increasing temperature (e.g., T=2.0) affect the probability distribution?
Question 3 of 5
What is the key advantage of top-p (nucleus) sampling over top-k sampling?
Question 4 of 5
In the standard generation pipeline, what is the correct order of operations?
Question 5 of 5
How does repetition penalty work on the logits?
0
/5
You understand how LLMs choose their words.
Up next

Next: Scaling Laws — What Makes LLMs Work ->

Why does GPT-4 work so much better than GPT-2? Scaling laws reveal the precise relationship between model size, dataset size, compute budget, and performance. The math behind the trillion-dollar bet on bigger models.

Next: Scaling Laws & What Makes LLMs Work ->
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You control how LLMs think.
Now build the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →