The Complete Guide to Agent Memory

Most writing about "agent memory" starts with architecture diagrams. This guide starts where I actually started: with a production error, at 11pm, from an agent that was working perfectly.

Part 0 — Start With What Actually Broke

My multi-model research assistant (built on Google's ADK — the same stack from the ADK build guide) threw this mid-session:

litellm.RateLimitError: Request too large for model qwen/qwen3-32b —
Limit 6000, Requested 8938 tokens per minutethe bug

Nothing was wrong with the code. The agent did exactly what it was told: search a few angles, read the pages, save findings. The problem is more fundamental, and understanding it is the key to this entire guide:

An LLM has no memory of its own. Every single call is a brand-new brain that remembers nothing.

When you send a second message in the "same conversation," the model isn't recalling anything — the framework is silently re-typing the entire transcript so far and handing it to a fresh instance of the model as if it just walked into the room. What feels like a conversation is actually: re-read everything, then respond. Every tool call, every search result, every saved finding — retyped, in full, every single turn.

Don't take my word for it — ADK's own source says it outright, in flows/llm_flows/contents.py:

if agent.include_contents == 'default':
    # Include full conversation history
    llm_request.contents = _get_contents(invocation_context.session.events, ...)adk source

Every event, every turn. Nothing is trimmed automatically. My five Tavily searches from earlier in the session didn't disappear once the agent moved on — they were still sitting in the transcript, getting retyped on turn 12, silently costing tokens nobody asked to spend. That's what hit Groq's 6,000 tokens-per-minute ceiling. And here's the part worth sitting with: the agent hit the limit because it was good at its job. It searches a lot; searching generates volume; volume breaks the naive default. Memory problems are a success symptom.

The industry has recently converged on a name for the discipline this guide teaches: context engineering — the craft of deciding what goes into the model's context window, every single call. Memory is the biggest sub-problem of context engineering, and this guide walks every technique that exists for it — from the naive default you have on day one, to what research labs are publishing right now — in the order you'd actually reach for them, with my agent as the running example.

The Three Kinds of Memory Every Framework Has

Before the techniques, the vocabulary. Almost every agent framework — ADK, LangChain/LangGraph, Mem0, Letta — converges on the same three-layer split, because it maps to a real distinction in how long information needs to survive and how expensive it is to keep around:

And because every framework names these differently, the decoder ring:

Here's the thing worth noticing: if you've built any tool-using agent, you almost certainly built layer 2 by instinct. My agent has a save_finding tool because raw search results are noisy and temporary, but the conclusion drawn from them is worth keeping. That instinct — "compress the messy stuff into a clean fact, discard the mess" — is the single idea underneath almost every technique in this guide. Everything from here on is a more disciplined version of save_finding.

Level 1 Full History Replay — The Default

This is ADK's default (include_contents='default'), and it's the default in almost every framework, because it's the easiest to build and the most correct — in the narrow sense that the model genuinely sees everything, so it can never "forget" something you told it five turns ago.

How it works: every turn, replay the whole event log. That's it.

Why it breaks down: token count grows linearly, forever, with no ceiling. Three things degrade as it grows:

• Rate limits — my exact Groq error. Providers cap tokens per minute, not just per request; a fat history burns the budget on every single turn.
• Cost — you pay for the entire history, every turn, including the parts the model no longer needs. A 20-turn session doesn't cost 20 turns; it costs roughly the triangle of 1+2+...+20 turn-sizes.
• Attention quality — the "lost in the middle" research (Liu et al., 2023) showed models are measurably worse at using information buried in the middle of a long context than information near the start or end. A bigger window doesn't mean the model uses it evenly. More context can actively hurt: irrelevant old tool output is a distractor, not an asset.

When it's actually fine: short sessions, cheap or high-limit models, agents that don't call tools much. If my agent ran 2–3 turns with one search each, I'd never have hit the ceiling. Don't engineer past this level until something measurable forces you to — but instrument token counts per turn from day one, so you see the growth curve before your users do.

Level 2 Windowing — Sliding Truncation

The most obvious fix: stop sending everything; send only the last N turns.

# ADK: a before_model_callback that keeps only recent events
def keep_last_n(callback_context, llm_request):
    llm_request.contents = llm_request.contents[-12:]
    return None   # proceed with the trimmed requestpython · adk

(ADK also offers the nuclear version: include_contents='none' — no history at all; the agent becomes stateless and you inject whatever matters through the instruction and state. Surprisingly useful for single-purpose worker agents inside a pipeline.)

Pro: trivial to build; hard, guaranteed ceiling on token growth.

Con: amnesia by age, not by value. Picture my research agent 20 turns in: it confirmed something about Anthropic's safety approach on turn 3, and that event just slid out of the window. Ask a follow-up now and it either re-searches (wasteful) or — worse — doesn't realize it already has the answer sitting in state and guesses instead. Windowing deletes indiscriminately: the crucial early instruction falls out exactly as easily as the useless old search dump.

Level 3 Structured State — The Scratchpad

The first technique that's smart rather than blunt: instead of deciding what to delete based on age, decide what to keep based on value. My save_finding tool is exactly this pattern:

def save_finding(topic: str, summary: str, source: str,
                 confidence: str, tool_context: ToolContext) -> dict:
    findings = tool_context.state.get("findings", [])
    findings.append({"topic": topic, "summary": summary,
                     "source": source, "confidence": confidence})
    tool_context.state["findings"] = findings
    return {"status": "saved", "total": len(findings)}python · adk

A 2,000-token Tavily result gets distilled to a ~30-token structured fact — a 98% compression ratio that is, critically, lossless for the part that matters. You don't need the full page text forever; you need the conclusion, with a source you can revisit if challenged. This is the same move a good analyst makes with raw data, and it's the conceptual heart of every memory system in this guide: distill, then discard.

But here's the gap — and it's why my agent still crashed despite having this tool: the compressed version (state) and the raw version (the original tool-call event in the transcript) both stay in the replayed history. I paid the compression cost and never collected the reward, because nothing told ADK "now that this is saved, stop retyping the raw version." Collecting that reward is Level 4.

Level 4 Pruning Tool Payloads — With the Real Code

This is the fix sized to the bug that opened this guide, and unlike most memory writing, here's the complete working version, not a sketch. The idea: once a web_search or fetch_page result has been distilled into a saved finding, strip the bulky raw payload from what gets replayed — keep the conversation's shape, drop its weight.

# The pruning callback, production version
PRUNABLE    = {"web_search", "fetch_page"}   # bulky, already-distilled tools
KEEP_RECENT = 4                              # never touch the last N contents

def prune_stale_tool_payloads(callback_context, llm_request):
    contents = llm_request.contents or []
    for content in contents[:-KEEP_RECENT]:      # old turns only
        for part in (content.parts or []):
            fr = getattr(part, "function_response", None)
            if fr and fr.name in PRUNABLE:
                part.function_response.response = {
                    "status": "pruned",
                    "note": "Raw result removed after distillation. "
                            "Call get_findings for saved conclusions.",
                }
    return None   # proceed with the lighter request

root_agent = Agent(
    ...,
    tools=[web_search, save_finding, get_findings],
    before_model_callback=prune_stale_tool_payloads,
)python · adk

Three design details that carry the whole trick:

• It's non-destructive. The callback mutates llm_request — this one outgoing call — not the session's stored events. ADK rebuilds the request from session.events fresh each turn, so the raw data still exists on disk; you're editing what gets retyped, not what got recorded. If you ever need the original payload back, it's in the session store.
• The tombstone message matters. The replaced payload doesn't vanish — it becomes a signpost ("pruned — call get_findings"). The model still sees that it searched, roughly when, and where the conclusions live. Pruning to nothing confuses models; pruning to a pointer doesn't.
• The recent window is sacred. The model may be mid-reasoning about the latest search result — never prune what it's actively using. KEEP_RECENT=4 is a decent default; tune to your turn structure.

And the payoff, quantified against my actual failing session:

Level 5 Summarization & Compaction

Pruning handles tool bloat. But once the plain conversation gets long enough on its own — long multi-topic sessions, dozens of user follow-ups — even the dialogue starts to add up. This is where summarization comes in: periodically ask a model to compress older turns into a short paragraph, and replace those turns with the summary.

Turns 1–10 → "User asked me to research AI safety approaches at Anthropic and OpenAI. Anthropic emphasizes interpretability and red-teaming; OpenAI emphasizes RLHF and policy engagement. Both saved with medium confidence — search results were thin."

Variants, in increasing sophistication:

• Rolling summarization: when history crosses a threshold, summarize the oldest chunk, keep recent turns verbatim. Simple, effective, the workhorse.
• Hierarchical / decaying summarization: recent turns verbatim, medium-age turns summarized once, old turns summarized again — compression compounds with age, like a memory that fades but never fully vanishes.
• Compaction at the harness level: this is no longer exotic — it's how coding agents like Claude Code survive day-long sessions. When context fills, the harness summarizes the transcript and continues from the summary plus recent turns. If you've seen an agent "compact" mid-session, you've watched Level 5 run in production.

The real cost, stated honestly: summarization is lossy, and it costs an extra LLM call (latency + money) every time it runs. If the summary says "found thin results with medium confidence," the exact source URL and exact wording are gone. For a research assistant whose entire value is citable, verifiable findings, that's a meaningful trade — which is precisely why Level 4 (prune payloads, keep structured findings) fits my agent better than jumping straight to summarization. Summarize when the thing piling up is conversation; prune when it's tool output you've already distilled. Different bloat, different tool.

Interlude — The Cache Wrinkle Nobody Mentions

Before the cross-session levels, a practical wrinkle that almost every memory guide skips, and that changes the math of everything above: prompt caching.

Providers (Anthropic, OpenAI, Google, and inference stacks generally) cache the processed form of a prompt's prefix. If this turn's context starts with exactly the same bytes as last turn's, the cached prefix is reused — typically ~10x cheaper and much faster than processing those tokens fresh. And here's the catch: full history replay is cache-friendly, because it's append-only. Each turn = previous context + new events; the whole previous context hits the cache.

Now look at what Levels 2–5 do to that property:

• Windowing shifts the start of the context every turn → the prefix changes → cache misses forever.
• Aggressive pruning rewrites old content → everything after the first edited byte is a cache miss.
• Summarization replaces chunks wholesale → same story.

This doesn't make those techniques wrong — my Groq error was a hard TPM limit, and no cache discount fixes "request too large." But it does mean the honest cost equation is: tokens saved by trimming vs cache discount lost by rewriting. Practical guidance:

• Trim rarely and in big steps, not continuously. Prune/summarize when a threshold is crossed (say, every ~20 turns), then leave the context stable so the cache rebuilds — rather than nibbling every turn and never getting a cache hit again.
• Keep the stable stuff first. System instruction, tool definitions, early context — unchanged prefix = maximum cacheable region. Do your editing as late in the context as possible.
• If you're on a provider with explicit cache breakpoints (Anthropic-style), place them at the boundary between "never changes" and "changes per turn."

Level 6 Retrieval-Based Memory — RAG Over History

Everything so far manages one session. Retrieval-based memory answers a different question: "what did we find out last month?" — recall across sessions that have already ended.

How it works, in plain terms:

1. When a session ends, take its content (or better: its saved findings) and convert each chunk into an embedding — a vector that represents its meaning, not its exact words.
2. Store those vectors in a vector database.
3. Next session, embed the new question the same way, and find stored chunks whose vectors are closest — "most similar in meaning."
4. Inject only those top few matches into the prompt — instead of the agent's entire history ever.

If you've read the search engine build-along, you already know this machinery — it's Level 6 of that guide (embeddings + ANN retrieval) pointed at your own transcripts instead of a product catalog. Same math, same infrastructure, same failure modes.

In ADK this is exactly what MemoryService is for — it ships in the box (InMemoryMemoryService for dev, VertexAiRagMemoryService and VertexAiMemoryBankService for production). The pattern: add_session_to_memory(session) when a session finishes, plus a load_memory tool the agent can call to search past sessions on demand.

Why this is the right fix for cross-session recall — and windowing/summarization are the wrong one: those operate on one session's transcript; they have nothing to say about a session that ended yesterday. RAG memory is the layer built specifically to reach across that boundary.

The honest caveat: retrieval is only as good as the query. I learned this failure mode the embarrassing way with a search tool — a badly-phrased query returned nothing even though the information existed. The same risk lives here: ask "what's Anthropic's safety philosophy" next month, and if the memory was embedded around the phrase "red-teaming approach," a weak retrieval setup may not connect them. Chunk size, embedding model choice, hybrid keyword+semantic search — this is active tuning, not a solved checkbox. (Everything in the search build-along about hybrid retrieval applies verbatim.)

Level 7 Temporal Knowledge Graphs

Vector retrieval is excellent at "find the chunk that sounds like this question." It is bad at one specific thing: knowing when a fact has changed.

Say the agent researches a company's funding in March, and again in September, and the number differs. A vector store holds both chunks with no concept of which is still true — ask in October and it may hand back the stale March figure right beside the current one, with no signal about which to trust.

Knowledge graphs fix this by modeling facts as relationships with validity windows, not text blobs: (Anthropic) —[valued at]→ ($X) [valid: Mar–Aug], (Anthropic) —[valued at]→ ($Y) [valid: Sep–now]. The graph knows the first fact expired.

The state-of-the-art open implementation is Graphiti (from Zep). Its key design is bitemporal — every fact carries two timestamps: event time (when it was true in the world) and ingestion time (when the agent learned it). That's what lets it resolve contradictions cleanly instead of accumulating them. On the LongMemEval benchmark, Zep reports 63.8% for this approach vs 49% for a pure-vector baseline with GPT-4o — a measured gap, not a theoretical one.

Notably, the idea is moving into mainstream frameworks: ADK's own VertexAiMemoryBankService config accepts disable_consolidation, revision_expire_time, and revision_ttl — meaning Google's managed memory backend already has built-in notions of merging duplicate memories and expiring outdated ones. Temporal-aware memory is going from research project to checkbox option.

The clear-eyed cost: an extraction step (an LLM call turning text into graph triples), a graph database, and consolidation logic. Real infrastructure. It earns its keep when an agent reasons about changing facts across many entities over a long horizon — a months-long market-intelligence agent, yes; a research session that ends in a report, no.

Level 8 Agentic Memory — The Model Manages Itself

Every technique so far has one thing in common: your code decides what to keep, prune, summarize, or retrieve. The newest direction flips that — give the model the tools to manage its own memory.

Letta (formerly MemGPT — the paper that started this thread) frames it as an operating system analogy:

The key shift: memory management becomes an agentic action, not a framework policy applied uniformly. The model decides "this is worth archiving" or "I need to page that back in."

And here's the satisfying full-circle moment: save_finding was already a tiny, manual version of exactly this. The model choosing, mid-conversation, "this is worth keeping past this moment" — that's an archival write. get_findings — that's paging back in. Letta formalizes and automates the pattern; the instinct was correct all along.

Mem0 takes a related but distinct angle: instead of explicit save/get tools, it watches the conversation and automatically infers ADD / UPDATE / DELETE operations on a memory store — notice the user's favorite language changed, update the old memory instead of appending a contradictory new one. Less agent control, more automatic hygiene.

One more idea from this frontier worth knowing: sleep-time consolidation — running memory maintenance (dedup, summarization, graph consolidation) as a background process between sessions, when no user is waiting, instead of on the critical path of a live turn. The human-memory analogy is irresistible and roughly correct: consolidate while asleep, so recall is fast while awake.

The Landscape Right Now

• Memory has real benchmarks now. LoCoMo and LongMemEval are standardized test sets — multi-session conversations with planted facts, tested for recall weeks or hundreds of turns later. Memory quality used to be an argument; now it's a measured number, the way ImageNet made vision progress measurable.
• No single winner — specialization. Mem0 (vector-based, fast, simple), Zep/Graphiti (temporal graph, best when facts change), Letta (self-directed paging), LangGraph/LangMem (checkpoint-centric persistence). Current head-to-heads show real gaps that flip depending on the task. Choose by problem shape, not by leaderboard.
• Bigger context windows don't retire the problem. Tempting to think million-token contexts end this guide. They don't: lost-in-the-middle means huge contexts get used unevenly; cost still scales with tokens sent; and — as my Groq error demonstrated — rate limits are per-minute regardless of the model's max context. A bigger window raises the ceiling; it doesn't remove the need to be deliberate about what you send.
• Genuinely open problems the field flags right now: identity consistency across months-apart sessions, summarizing time itself at scale (not just content), and detecting when a memory has quietly gone stale. Nobody has a complete answer — you're not behind for not having one.

The Cheat Sheet

For my research agent specifically — and probably yours — in order:

1. Now: the Level 4 pruning callback — directly fixes the TPM error, reuses the scratchpad already built, ~20 lines, no new infrastructure. (Done — the code above is running.)
2. If sessions still grow long: light rolling summarization for the dialogue turns, triggered in batches, cache-consciously.
3. When "remember research from past sessions" becomes real: wire up ADK's MemoryService — an additive capability, not a fix for the original bug.
4. Skip graph memory for now — real cost, and a report-producing research agent doesn't yet have the facts-changing-across-entities problem. Revisit if it becomes a long-lived, multi-project tool.

Sources & Further Reading:
Liu et al. — Lost in the Middle (2023) • Packer et al. — MemGPT (2023) • Zep: Temporal Knowledge Graph Architecture for Agent Memory • Graphiti • Mem0: State of AI Agent Memory 2026 • Graphlit: Survey of Agent Memory Frameworks • Particula: Memory Frameworks Tested • ADK MemoryService docs