The problem: each call forgets everything
In Lesson 01, you called the LLM once and got one answer. But real applications need conversations. A user asks a question, the bot answers. The user asks a follow-up based on the answer. The bot should remember what came before.
If you call the LLM function the same way twice, it has no memory of turn 1. It doesn't know the user's account number from 30 seconds ago. It can't refer to the previous exchange.
This is the gap we close in this lesson. Multi-turn conversation requires state—a list of messages that grows with each turn.
How chat APIs actually work: the messages array
Every chat endpoint (Ollama, Gemini, Claude) uses the same pattern: a messages array with roles. Each message has three fields:
A conversation with 3 turns looks like this:
Building conversation state: a ChatSession class
The practical way to build multi-turn chat is a class that maintains the message history. Here's a minimal ChatSession:
The context window problem: token limits
Every model has a maximum context window—the total number of tokens it can read. Common limits:
| Model | Context Window | Real use |
|---|---|---|
| Ollama (Mistral 7B) | 8,000 tokens | ~5 pages of text or 20-turn conversation |
| Ollama (Llama 70B) | 4,096 tokens | ~2 pages or 8-turn conversation |
| Gemini 2.0 | 1,000,000 tokens | Entire books + 500-turn conversation |
If your conversation history exceeds the context window, the model returns an error. For a chatbot that runs for hours, you need a strategy to manage this.
Three strategies: keep, summarize, hybrid
Strategy 1: Sliding Window (keep last N turns)
Discard old messages, keep only the most recent N turns. Simple but loses long-term context.
Strategy 2: Summary Compression (summarize old turns)
Periodically call the LLM to summarize the conversation history into bullet points. Keep that summary and discard old messages.
Strategy 3: Hybrid (use both)
In production, combine sliding window (keep recent turns) with summary compression (preserve key facts from early conversation). This balances recency with long-term context.
Worked example: Jumia Kenya customer support bot
Arjun wants to build a bot for Jumia Kenya customer support. In turn 1, the customer gives their order number. In turn 2, they ask a question. In turn 3, they ask another question—and the bot should still know the order number without the customer repeating it.
Turn 1: "My order is JK-2026-98765."
Turn 2: "Where is my package?"
Turn 3: "Can I get a refund?"
Without multi-turn chat, turn 2 and 3 forget the order number. With our ChatSession, the bot remembers across all turns.
Saving and loading sessions to JSON
Users close apps. Bots need to pause and resume. Save the message history to JSON, then reload it later.
Next: tools + conversation = agents
This is the foundation. In Lesson 05, we give the chat session access to tools. Instead of just talking, the bot can call functions—fetch real order data from Jumia's API, check M-Pesa balance, initiate refunds. A ChatSession that can do things is an agent.
Task: Create a ChatSession for a SouthBank India customer support bot. Run 4 turns:
1. Customer: "My account is SB-5001-Hyderabad."
2. Customer: "What's my recent transaction history?"
3. Customer: "Can I dispute one of those charges?"
4. Customer: "How long will that take?"
Verify that turn 4's response mentions the account number or disputes from earlier turns.
Hint: Use ChatSession from chat_session.py. Create a system prompt for a SouthBank agent, then call session.chat() four times. Print each response.
How AI systems use this
Every chatbot and conversational AI (Siri, ChatGPT, customer service bots) uses the messages array pattern. Larger language models (Claude, GPT-4) have larger context windows, so they can remember longer conversations. But the core pattern is identical: accumulate messages, send full history, extract new message from response, repeat. This is how LLMs "understand" conversation—not through special memory, but by reading the full chat history each time.
Test your knowledge
You've built stateful chat.
Now add tools and launch real agents.
Join the Best AI Program: The AI-Era Agentic Engineer — 5 months, fully online, with placement assistance. Week 1 is free.
Explore The AI-Era Agentic Engineer →