Why LLM applications need better memory management

• Context window: Each session retains a rolling buffer of past messages. GPT-4o supports up to 128K tokens, while other models have their own limits (e.g. Claude supports 200K tokens).
• Long-term memory: Some high-level details persist across sessions, but retention is inconsistent.
• System messages: Invisible prompts shape the model’s responses. Long-term memory is often passed into a session this way.
• Execution context: Temporary state, such as Python variables, exists only until the session resets.

Without external memory scaffolding, LLM applications remain stateless. Every API call is independent, meaning prior interactions must be explicitly reloaded for continuity.

Why LLMs are stateless by default

In API-based LLM integrations, models don’t retain any memory between requests. Unless you manually pass prior messages, each prompt is interpreted in isolation. Here’s a simple example of an API call to OpenAI’s GPT-4o:

import { OpenAI } from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: "You are an expert Python developer helping the user debug." },
    { role: "user", content: "Why is my function throwing a TypeError?" },
    { role: "assistant", content: "Can you share the error message and your function code?" },
    { role: "user", content: "Sure, here it is..." },
  ],
});

Each request must explicitly include past messages if context continuity is required. If the conversation history grows too long, you must design a memory system to manage it—or risk responses that truncate key details or cling to outdated context.

This is why memory in LLM applications often feels inconsistent. If past context isn’t reconstructed properly, the model will either cling to irrelevant details or lose critical information.

When LLM applications won’t let go

Some LLM applications have the opposite problem—not forgetting too much, but remembering the wrong things. Have you ever told ChatGPT to “ignore that last part,” only for it to bring it up later anyway? That’s what I call “traumatic memory”—when an LLM stubbornly holds onto outdated or irrelevant details, actively degrading its usefulness.