Conversation Memory

What you will be able to do

• ✓Explain why a model does not remember previous calls
• ✓Give an agent memory by keeping and resending the message list
• ✓Explain the context window, and what happens when a conversation outgrows it
• ✓Trim history by whole exchanges while keeping the system message
• ✓Predict what an agent forgets when history is trimmed
• ✓Keep each conversation separate by giving each agent its own list

The idea, in plain English

Each call to the model starts from nothing. Say "My name is Chandu" in one call and ask "What’s my name?" in the next, and the model has no idea - nothing about the first call was kept anywhere. We tried exactly that: "I’m afraid I don’t know your name!"

So "memory" is less magical than it sounds. You keep a running list of every message so far, and send the whole list every time. The model appears to remember because the history is right there in front of it. The application remembers; the model just reads.

That list cannot grow forever. A model can only take in so much text at once - its context window - and every message is processed again on every call. So you trim: keep the system message, which holds the instructions, plus the most recent exchanges, and drop the oldest.

Trimming is forgetting. Whatever you drop is gone as far as the model is concerned - in our runs, after five small questions the agent no longer knew the name it was told at the start. Wrapping the list in a class keeps it tied to one conversation: two ChatAgent objects, two separate memories.

Worked example: A chat agent that remembers your name.

Memory is the message list

The messages list from Lesson 1.1 is the whole mechanism. Append the user’s message, send the list, append the reply. Next turn, the list is two messages longer and the model sees everything that came before.

Nothing is stored on the model side. Two independent calls know nothing about each other, and a brand-new ChatAgent - a new list - does not know your name either. Memory belongs to the list, and the list belongs to the agent instance.

Every call pays for the whole history

The model reads the entire list on every call. In our run the prompt tokens processed per call went 29, 67, 90, 111, 131 - each question more expensive than the last, even though every question was a single short line.

Locally that costs time; on a paid API it costs money per token. Either way, an untrimmed conversation gets steadily slower until it hits a hard limit.

The context window - and silent truncation

The context window is the most text a model can take in at once, measured in tokens. llama3 was trained for 8,192 - but Ollama ran it with 4,096 by default on our machine, as its own log shows.

Exceed the window and nothing errors. We sent a conversation of about 5,700 tokens: a name, then three long pasted texts, then "What’s my name?". Ollama silently dropped the oldest messages to make it fit - only 47 tokens were processed - and the model said it did not have that information.

Raising the window with options={"num_ctx": 8192} let all 5,668 tokens through. The model still could not find the name, buried under pages of filler. A bigger window is not better memory: the model has to notice what matters, and a long history makes that harder.

Watch out: Ollama does not raise an error when a conversation is too long - it drops the oldest messages without telling you. If an agent suddenly forgets something, check the length before you blame the model.

Trimming, done by whole exchanges

The course version keeps the system message plus the last max_history - 1 messages. With max_history=10 that is 9 messages - an odd number, so the oldest one kept is always an assistant reply whose question was dropped. We checked: after trimming, the roles ran system, assistant, user, assistant...

It rarely breaks anything, but it is easy to avoid. Count exchanges instead of messages - a question and its reply - and keep the last few whole ones: history[-2 * max_exchanges:].

Whichever way you count, keep the system message out of the trim. It holds the instructions; lose it and the agent forgets how it is supposed to behave partway through a long chat.

What happens when a call fails

ask() appends the user’s message before calling the model. If that call fails - Ollama not running, a timeout - the question stays in the list with no answer. On the next call the model receives two user messages in a row, the first of which never got a reply. We reproduced exactly that: the roles sent were system, user, user.

Remove the unanswered message when the call fails, and the history stays a clean record of questions and their answers.

This memory ends with the program

self.messages lives in memory. Close the program and the conversation is gone; start again and the agent is a blank slate. That is conversation memory.

Persistent memory - facts that survive a restart - needs a file or a database. POC 2 at the end of this module adds exactly that with a notes.json file, and Module 2 covers summarising old turns instead of simply dropping them.

Step-by-step code

Tip: response["prompt_eval_count"] tells you how many tokens the model actually processed. Print it while you develop - it is the quickest way to see a history growing, or being cut.

Watch out: Trimming is forgetting. Anything the agent must always know - the user’s name, a goal, a constraint - belongs in the system message or in persistent memory, not in an old turn that will be dropped.

Memory at a glance

self.messages.append({...})

self.messages[0]

options={"num_ctx": 8192}

response["prompt_eval_count"]

history[-2 * max_exchanges:]

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

What usually goes wrong

✗ self.messages = self.messages[-10:]

✓ self.messages = [self.messages[0]] + self.messages[1:][-8:]

✗ self.messages[-(self.max_history - 1):]

✓ history[-2 * self.max_exchanges:]

✗ self.messages.append(user_message)
response = ollama.chat(...)

✓ try:
    response = ollama.chat(...)
except Exception:
    self.messages.pop()
    raise

Quick check before you move on

1. 1.

   Where does "memory" actually live in this code?

   Show answer
2. 2.

   Why do we trim self.messages instead of letting it grow forever?

   Show answer
3. 3.

   What would happen if you created a new ChatAgent() before asking "What’s my name?"

   Show answer
4. 4.

   With max_history=10, why is the oldest kept message an assistant reply?

   Show answer
5. 5.

   An agent was told a fact early on and later cannot recall it. Name two possible causes.

   Show answer

Interview questions