Perplexity — how surprised is your model?
You have trained a language model. How do you know if it is any good? The most fundamental metric for language models is perplexity — it measures how "surprised" the model is by new text. Lower perplexity means the model predicts the next token more accurately.
Mathematically, perplexity is the exponential of the cross-entropy loss:
If your model has a cross-entropy loss of 1.0 on a validation set, its perplexity is e^1.0 = 2.72. This means the model is, on average, as uncertain as if it were choosing between ~2.72 equally likely tokens at each step.
A perplexity of 1.0 would mean the model perfectly predicts every token. A perplexity equal to the vocabulary size (e.g., 10,000) means the model is completely random. Real models fall somewhere in between.
Benchmarks — MMLU, HumanEval, HellaSwag
Perplexity tells you how well the model predicts text, but it doesn't tell you how well it reasons, codes, or understands. For that, we use benchmarks — standardized tests that probe specific capabilities.
| Benchmark | What it measures | Format |
|---|---|---|
| MMLU | Broad knowledge across 57 subjects (history, law, math, medicine, etc.) | Multiple choice (A/B/C/D) |
| HumanEval | Code generation ability — can the model write correct Python functions? | Function completion, tested against unit tests |
| HellaSwag | Commonsense reasoning — can the model pick the most plausible sentence continuation? | Multiple choice (4 options) |
Quantization — shrinking your model for deployment
A 7-billion-parameter model stored in full FP32 precision requires about 28 GB of memory — more than most consumer GPUs. Quantization reduces the precision of model weights from 32-bit floats to smaller formats (16-bit, 8-bit, even 4-bit), dramatically shrinking memory usage with minimal quality loss.
GGUF format and serving with Ollama
GGUF (GPT-Generated Unified Format) is the standard file format for running quantized LLMs on consumer hardware. It's the format used by llama.cpp, the C++ inference engine that made it possible to run LLMs on laptops and phones.
Ollama wraps llama.cpp in a simple, Docker-like interface. You pull models, run them locally, and query them via a REST API — all with one tool.
Wrapping in a FastAPI endpoint
The last step is exposing your model as an API. FastAPI is the standard Python framework for building ML serving endpoints — it's async, fast, and auto-generates documentation.
Run it yourself
Deploy a model locally in 3 commands
The fastest way to go from zero to a running LLM API:
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull a small model
ollama pull llama3.2:1b
# Query it
curl http://localhost:11434/api/generate \
-d '{"model":"llama3.2:1b","prompt":"Hello!"}'
To add the FastAPI wrapper, install fastapi and uvicorn, save the code above as serve.py, and run uvicorn serve:app --port 8000.
Why this matters for LLMs
You've completed the entire journey — from neurons to deployment
Over 20 lessons, you built every component of a modern LLM from scratch: individual neurons, backpropagation, embeddings, tokenization, attention, the full Transformer, GPT, training, scaling laws, SFT, RLHF, and now evaluation and deployment. You understand not just how to use these models, but how they work — from the math to the metal.
This is the knowledge that separates an AI engineer from someone who calls an API. Every company building or deploying LLMs needs people who understand this full stack — from perplexity metrics to INT4 quantization to serving infrastructure.
Challenge
Install Ollama, pull llama3.2:1b, and measure its response time for 10 different prompts. Calculate the average tokens per second. Then pull a larger model (e.g., llama3.2:3b) and compare. How does model size affect throughput?
time on each curl request, or parse the eval_duration field from Ollama's JSON response. The 3B model should be roughly 2-3x slower per token than the 1B model on the same hardware, but may produce noticeably better outputs.Extend the FastAPI server with: (1) a /models endpoint that lists available models, (2) a streaming /generate/stream endpoint using Server-Sent Events, and (3) basic API key authentication. Deploy it on a free cloud VM and share the URL.
StreamingResponse and set Ollama's "stream": True. For auth, use FastAPI's Depends with an API key header check. For free hosting, try a Hetzner or Oracle Cloud free-tier VM with at least 8GB RAM for a 1B quantized model.Quiz
Track Complete!
You did it. Over 20 lessons, you built a large language model from scratch — starting with a single neuron and ending with a deployed API. You understand the math, the code, and the engineering behind every modern AI system.
From neurons to transformers. From attention to RLHF. From training to deployment. You built it all.
You built an LLM from scratch.
Now build the career to match.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →