Perplexity — how surprised is your model?

You have trained a language model. How do you know if it is any good? The most fundamental metric for language models is perplexity — it measures how "surprised" the model is by new text. Lower perplexity means the model predicts the next token more accurately.

Mathematically, perplexity is the exponential of the cross-entropy loss:

📊
The formula
Perplexity = exp(cross-entropy loss)

If your model has a cross-entropy loss of 1.0 on a validation set, its perplexity is e^1.0 = 2.72. This means the model is, on average, as uncertain as if it were choosing between ~2.72 equally likely tokens at each step.

A perplexity of 1.0 would mean the model perfectly predicts every token. A perplexity equal to the vocabulary size (e.g., 10,000) means the model is completely random. Real models fall somewhere in between.

perplexity.py Read-only — run locally with PyTorch
Expected output
=== Perplexity === Perplexity (lower is better): Mini-GPT (train): loss=1.3215, perplexity=3.75 Mini-GPT (val): loss=1.4872, perplexity=4.42 Random baseline: loss=2.1000, perplexity=8.17 Untrained: loss=3.5000, perplexity=33.12
Perplexity Comparison (lower is better) 0 8 16 24 35 3.75 Train 4.42 Val 8.17 Random 33.12 Untrained Training brings perplexity from 33 (random guessing) down to ~4 (confident predictions)
Interpreting perplexity:
Our trained Mini-GPT achieves a validation perplexity of 4.42 — it narrows its choices to about 4-5 tokens at each position. The untrained model has perplexity 33.12 — essentially guessing among dozens of options. GPT-3 achieves perplexity around 20 on broad web text (but that's a much harder task than our small training set).

Benchmarks — MMLU, HumanEval, HellaSwag

Perplexity tells you how well the model predicts text, but it doesn't tell you how well it reasons, codes, or understands. For that, we use benchmarks — standardized tests that probe specific capabilities.

🏆
The big three
MMLU, HumanEval, and HellaSwag
BenchmarkWhat it measuresFormat
MMLUBroad knowledge across 57 subjects (history, law, math, medicine, etc.)Multiple choice (A/B/C/D)
HumanEvalCode generation ability — can the model write correct Python functions?Function completion, tested against unit tests
HellaSwagCommonsense reasoning — can the model pick the most plausible sentence continuation?Multiple choice (4 options)
benchmark_harness.py Read-only — conceptual overview
Expected output
=== Benchmark Scores (approximate) === Model MMLU HumanEval HellaSwag GPT-2 32% 0% 71% GPT-3 (175B) 43% 0% 79% GPT-3.5 70% 48% 85% GPT-4 86% 67% 95% Claude 3.5 88% 92% ~95% Llama 3 70B 82% 81% 88%
Benchmarks are not perfect:
Models can be "trained on the test" — if benchmark questions leak into training data, scores are inflated. MMLU questions come from real exams, so they may appear in web crawls. The community is moving toward "contamination-free" benchmarks and human evaluations (like Chatbot Arena) to get more honest assessments of model quality.

Quantization — shrinking your model for deployment

A 7-billion-parameter model stored in full FP32 precision requires about 28 GB of memory — more than most consumer GPUs. Quantization reduces the precision of model weights from 32-bit floats to smaller formats (16-bit, 8-bit, even 4-bit), dramatically shrinking memory usage with minimal quality loss.

quantization.py Read-only — run locally with bitsandbytes
Expected output
=== Quantization === Quantization comparison: FP32: 32 bits, ~7.0 GB for 7B model FP16: 16 bits, ~3.5 GB for 7B model INT8: 8 bits, ~1.8 GB for 7B model INT4: 4 bits, ~0.9 GB for 7B model
Quantization — Shrinking a 7B Model FP32 ~7.0 GB (32 bits/param) FP16 ~3.5 GB (16 bits) INT8 ~1.8 GB INT4 ~0.9 GB INT4 quantization reduces model size by ~87% with minimal quality loss
How much quality do you lose?
FP16 is essentially lossless — modern GPUs train in FP16 anyway. INT8 typically loses less than 1% on benchmarks. INT4 can lose 2-5% on harder tasks, but for most applications it's still excellent. The tradeoff is dramatic: a 7B model goes from needing a $1,000 GPU (FP32) to running on a $200 laptop (INT4).

GGUF format and serving with Ollama

GGUF (GPT-Generated Unified Format) is the standard file format for running quantized LLMs on consumer hardware. It's the format used by llama.cpp, the C++ inference engine that made it possible to run LLMs on laptops and phones.

Ollama wraps llama.cpp in a simple, Docker-like interface. You pull models, run them locally, and query them via a REST API — all with one tool.

gguf_conversion.sh Read-only — terminal commands
Expected output
=== GGUF Deployment Pipeline === 1. Train model (PyTorch) 2. Export weights (safetensors / HF format) 3. Convert to GGUF (convert_hf_to_gguf.py) 4. Quantize (llama-quantize -> Q4_K_M) 5. Serve (Ollama / llama.cpp / vLLM) 6. Query via REST API
Deployment Pipeline: Training to Production Trained Model Quantize INT4 / INT8 GGUF Convert Ollama Serve FastAPI REST API From PyTorch weights to a production API in 5 steps
GGUF quantization types:
Q4_K_M is the most popular — it balances size and quality. Q5_K_M gives slightly better quality at ~15% larger size. Q2_K is extremely small but noticeably degrades output. For most deployment scenarios, Q4_K_M is the sweet spot.

Wrapping in a FastAPI endpoint

The last step is exposing your model as an API. FastAPI is the standard Python framework for building ML serving endpoints — it's async, fast, and auto-generates documentation.

serve.py Read-only — run locally with FastAPI + Ollama
Expected output
=== FastAPI Server === Endpoints: POST /generate — Generate text from a prompt GET /health — Check server status GET /docs — Auto-generated Swagger UI Start with: uvicorn serve:app --host 0.0.0.0 --port 8000
Production considerations:
For real deployments, you'd add rate limiting, authentication, streaming responses (Server-Sent Events), request queuing, and monitoring. Tools like vLLM and TGI (Text Generation Inference) handle batching and GPU optimization better than raw Ollama for high-throughput scenarios.

Run it yourself

Deploy a model locally in 3 commands

The fastest way to go from zero to a running LLM API:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a small model
ollama pull llama3.2:1b

# Query it
curl http://localhost:11434/api/generate \
  -d '{"model":"llama3.2:1b","prompt":"Hello!"}'

To add the FastAPI wrapper, install fastapi and uvicorn, save the code above as serve.py, and run uvicorn serve:app --port 8000.

Why this matters for LLMs

You've completed the entire journey — from neurons to deployment

Over 20 lessons, you built every component of a modern LLM from scratch: individual neurons, backpropagation, embeddings, tokenization, attention, the full Transformer, GPT, training, scaling laws, SFT, RLHF, and now evaluation and deployment. You understand not just how to use these models, but how they work — from the math to the metal.

This is the knowledge that separates an AI engineer from someone who calls an API. Every company building or deploying LLMs needs people who understand this full stack — from perplexity metrics to INT4 quantization to serving infrastructure.

Perplexity: 4.42 INT4: 87% smaller GGUF + Ollama FastAPI serving

Challenge

Put it into practice
Challenge 1: Deploy and benchmark

Install Ollama, pull llama3.2:1b, and measure its response time for 10 different prompts. Calculate the average tokens per second. Then pull a larger model (e.g., llama3.2:3b) and compare. How does model size affect throughput?

Use time on each curl request, or parse the eval_duration field from Ollama's JSON response. The 3B model should be roughly 2-3x slower per token than the 1B model on the same hardware, but may produce noticeably better outputs.
Challenge 2: Build a complete API

Extend the FastAPI server with: (1) a /models endpoint that lists available models, (2) a streaming /generate/stream endpoint using Server-Sent Events, and (3) basic API key authentication. Deploy it on a free cloud VM and share the URL.

For streaming, use FastAPI's StreamingResponse and set Ollama's "stream": True. For auth, use FastAPI's Depends with an API key header check. For free hosting, try a Hetzner or Oracle Cloud free-tier VM with at least 8GB RAM for a 1B quantized model.

Quiz

Question 1 of 5
What does a perplexity of 4.42 mean for a language model?
Question 2 of 5
Which benchmark specifically measures code generation ability?
Question 3 of 5
INT4 quantization reduces a 7B model from ~7 GB to approximately how much?
Question 4 of 5
What is the GGUF format used for?
Question 5 of 5
In the deployment pipeline, what role does Ollama play?
0
/5
You're ready to evaluate and deploy LLMs in production.
🎓

Track Complete!

You did it. Over 20 lessons, you built a large language model from scratch — starting with a single neuron and ending with a deployed API. You understand the math, the code, and the engineering behind every modern AI system.

20
Lessons completed
5
Arcs mastered
100
Quiz questions answered

From neurons to transformers. From attention to RLHF. From training to deployment. You built it all.

<- Lesson 19: RLHF Track Complete
Africa & India's Best AI Academy
🤖 AI-first 🎓 Placement Assistance

You built an LLM from scratch.
Now build the career to match.

Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.

Explore The AI-Era Agentic Engineer →