Agent Memory That Remembers: Knowledge Graphs and Context Assembly

The Problem: Stateless Models, Stateful Conversations

Let me be direct: a language model has no memory. Every call is a blank slate. The illusion of a system that "remembers you" is entirely engineering that happens outside the model, before the prompt is ever sent.

Get that engineering wrong and you get the two classic failures. Either the agent forgets what the user said three turns ago, or you stuff the entire conversation history into every request and watch your token bill and your latency explode while relevance drops. Real agent memory is the discipline of deciding, per request, exactly what the model needs to see and nothing more.

Memory is not one thing you turn on. It is four different mechanisms with different costs and different jobs, assembled into a single context window at request time.

We run this in production in Exfinity, our multi-tenant AI commerce platform, where eight agents share a memory and knowledge layer across thousands of conversations. This guide is the real architecture, not a toy. For the higher-level view of agentic systems, see our agentic AI systems guide, and for how retrieval fits in, our enterprise RAG systems guide. This is core AI services work.

Who Cares About What

Four Kinds of Memory

Before the architecture, the vocabulary. These four mechanisms do different jobs, and production systems use all of them.

The mistake juniors make is treating memory as a single "chat history" blob. The mistake seniors avoid is paying for context the model does not need this turn.

Context Assembly: The Seven Layers

Every request in Exfinity passes through a context assembly pipeline that builds an ordered array of messages injected before the user's text. Each layer is conditional. This is the heart of the system, so here is the full pipeline.

Two design decisions here matter more than the list itself.

First, not every layer fires. Layer 4 only runs once a tenant's graph has enough nodes to be useful, and Layer 5 only kicks in once a thread is long enough to need summarizing. You do not pay for context you do not need.

Second, and this is the security one, layers that carry data other users produced are never injected as system instructions. Platform rules and the user's own message are the only authoritative voices. Graph insights, compacted history, retrieved products, and retrieved documents are all wrapped as untrusted data and injected at the user role, with the fence delimiters stripped from the content so injected text cannot break out. We come back to why in the guardrails section.

User message
   │
   ▼
[1] Platform rules      (system, always)
[2] Date context        (system, always)
[3] Tenant persona      (system, if configured)
[4] Graph insights      (untrusted-data, if graph is big enough)
[5] Compacted history   (untrusted-data, if thread is long)
[6] Product context     (system, if product-like query)
[7] Knowledge passages  (untrusted-data, if question-like query)
   │
   ▼
Assembled context ──▶ Agent ──▶ Model

Working Memory: The Template the Agent Writes To

Recent messages are expensive to carry forever. Working memory solves this with a structured template the agent reads and writes across turns. Instead of re-reading "I have 200 euro for two adults next Saturday" from the raw history every turn, the agent extracts it once into a template and carries the template.

<user_context>
preferences:
budget:
travel_dates:
group_size:
interests:
viewed_products:
current_intent:
</user_context>

When the user states a budget and dates, the agent writes those values into the fields. On the next turn, the template loads back into context, so the agent remembers without the original messages being present. This is dramatically cheaper than carrying full history, and it survives compaction. Agents that do not need a rigid schema keep a free-form working-memory block instead.

The design rule: put durable, structured facts in working memory, and let the raw message window handle the immediate back-and-forth. Our AI workflow design guide covers how to decide what belongs where.

Semantic Recall Versus the Recent Window

Two mechanisms cover conversation history, tuned per agent.

The recent window is the last N messages, loaded verbatim. Cheap and always relevant to the immediate exchange.

Semantic recall embeds the current message and searches the thread's older history for semantically similar past messages, pulling back a few matches plus the messages around each. This is how an agent recalls something the user said twenty turns ago when it becomes relevant again, without carrying all twenty turns.

Each agent tunes these independently. A booking agent might keep a short recent window and shallow recall because bookings are near-term. A recommendation agent keeps a deeper recall because taste is expressed over a longer arc. The point is that memory settings are per role, not global. Our multi-agent architecture guide covers how specialized agents diverge in configuration.

// Per-agent memory config: recent window + semantic recall
const bookingAgent = {
  memory: {
    lastMessages: 15,                       // verbatim recent window
    semanticRecall: { topK: 3, messageRange: 2 }, // similar past + neighbors
    workingMemory: { enabled: true },
  },
};

const recommendationAgent = {
  memory: {
    lastMessages: 15,
    semanticRecall: { topK: 5, messageRange: 3 }, // deeper recall
    workingMemory: { enabled: true },
  },
};

The Knowledge Graph: Memory That Spans Sessions

Everything above is scoped to one conversation. The knowledge graph is memory that outlives the session and holds patterns across every interaction in a tenant. This is the layer most teams skip, and it is where context stops being "what you just said" and becomes "what tends to be true here."

The graph is per tenant, with typed nodes and typed edges.

Node types include product, query, response, user, document, tool call, booking, and competitor price. Edge types include searched-for, answered-by, booked, viewed, cited, similar-to, asked-about, used-tool, and priced-against.

query ──searched_for──▶ product
query ──answered_by──▶ response ──cited──▶ document
user  ──booked──▶ product
user  ──viewed──▶ product
product ──similar_to──▶ product

Write path. After every agent response, a fire-and-forget job records the interaction: a query node with the personal-data-scrubbed user message, a response node with the scrubbed answer, an answered-by edge weighted by feedback (a thumbs up strengthens it, a thumbs down weakens it), and a product node plus a searched-for edge for each product mentioned. It runs asynchronously so it never adds latency to the response.

Read path. During context assembly (Layer 4), the graph produces insights only once it has enough nodes to be meaningful. Two queries run in parallel: the most-searched products, and recent questions with no good answer. Those become a compact, untrusted-data block that tells the agent what this tenant's users actually care about.

// Read path: only surface insights once the graph is substantial
async function buildGraphInsights(tenantId) {
  const stats = await graph.getStats(tenantId);
  if (stats.totalNodes < GRAPH_MIN_NODES_FOR_INSIGHTS) return null;

  const [popular, unanswered] = await Promise.all([
    graph.getPopularProducts(tenantId, 5),      // top searched
    graph.getUnansweredQuestions(tenantId, 3),  // gaps to fix
  ]);
  return fenceUntrusted(formatInsights(popular, unanswered));
}

Keeping it from growing forever. Two maintenance jobs matter. Edge decay applies age-based exponential decay to edge weights, so old interactions fade in relevance, and it recomputes from the initial weight each run so it is idempotent rather than compounding. Node purge deletes transient nodes past a cutoff and cascades to orphaned edges, while durable nodes like products persist.

weight = initial_weight * exp(-ln(2) / halfLifeDays * ageDays)   # floored at 0.1

Erasure. For a GDPR deletion request, the graph resolves a user's query nodes by thread, follows the answered-by edges to their response nodes, and deletes those nodes plus every edge touching them. Shared product nodes are never erased. Targeted deletion across memory stores is a hard requirement, not a nice-to-have, and we cover the compliance side in our GDPR and AI guide.

Confidence and Auto-Learn: Memory That Improves

Memory that only accumulates is a liability. Memory that improves is an asset. The bridge is a confidence score on every response and a human review loop that turns low-confidence answers into new knowledge.

Confidence is a composite of four factors, weighted.

const composite =
  retrieval * 0.4 + policy * 0.2 + completeness * 0.2 + memory * 0.2;

When a response scores below a per-tenant threshold, it is captured as a learning candidate and classified into a failure class: retrieval miss, policy block, a hallucination signal like a past date presented as future, negative feedback, and so on. A human reviews the candidate in a dashboard. On approval, the answer is formatted into a knowledge document, embedded, and published back into the knowledge base, so the same question retrieves a good answer next time. The threshold even auto-tunes: if reviewers approve almost everything, it lowers to capture more; if they reject a lot, it raises to cut noise.

This is the human-in-the-loop pattern applied to the memory layer itself, and it is what separates a system that degrades from one that compounds. Our AI observability guide covers the metrics you watch to know which it is doing.

Guardrails: Prompt Injection and Untrusted Data

Here is the failure mode that memory introduces: text that one user stored, or that survived compaction, can try to hijack another user's session. If retrieved content or graph insights were injected as system instructions, a malicious document could say "ignore your rules" and be obeyed.

The defense is a strict separation. Only the platform rules and the end user's own request are authoritative. Everything else, retrieved passages, product data, graph insights, compacted history, is declared untrusted data and injected at the user role inside a fence, with the fence delimiters stripped from the content so the injected text cannot close the block early and appear to escape. It is treated as information to consider, never as commands to follow.

This matters more as memory grows, because the surface area of "text other people caused to exist" grows with it. Our AI governance guide and AI failure modes guide go deeper on defending agentic systems.

Cost: Memory Is Not Free

Every layer you add is tokens, and tokens are latency and money. Three controls keep it bounded.

Compaction summarizes older messages once a thread gets long, replacing raw history with a compact structured summary and keeping the token budget flat as conversations grow. Prompt caching serves the static system prefix from the provider cache on repeat calls, so you are not re-paying for the same rules every turn. And hard per-turn ceilings on reasoning steps, tool calls, and output tokens stop a single turn from running away.

We cover cost and latency trade-offs across the stack in our AI latency and accuracy guide.

Getting Started

Adding memory to an agent, in order of return:

1. Recent window first. Carry the last N messages. This alone makes an agent feel coherent.
2. Working memory next. Add a structured template for durable facts. Cheap, high impact.
3. Semantic recall when threads get long. Pull back relevant older messages instead of all of them.
4. Compaction when cost bites. Summarize old history once threads are long.
5. A knowledge graph when you have volume. Only worth it once you have enough interactions for patterns to emerge.

Do not build the graph first. It earns its place only at volume. Our methodology is built around this kind of staged delivery, and our custom software and consulting teams build these layers with clients. Start at contact or get a quote.

Common Ways This Goes Wrong

1. Dumping full history every turn. Cost and latency explode, relevance drops. Assemble, do not dump.
2. Global memory settings. Different agents need different windows and recall. Tune per role.
3. Injecting retrieved text as system instructions. That is a prompt-injection hole. Fence it as untrusted data.
4. A graph with no decay or purge. It grows unbounded and old noise drowns signal. Add maintenance jobs.
5. No erasure path. If you cannot delete a user's memory, you cannot meet a GDPR request. Design deletion in.
6. Accumulating without improving. Add a confidence score and a review loop, or the system just gets bigger, not better.

Who Builds This

Oronts is a founder-led software company in Munich. Refaat Al Ktifan, our founder and solution architect, leads a senior team across backend, data, and AI. The architecture in this guide is the real memory layer behind Exfinity, which we run against ourselves before bringing these patterns to clients. We design the context pipeline, build the memory and graph layers, and hand you a system you can operate and audit. See our services and solutions pages for the full picture.

Takeaways

• A model has no memory. Everything is assembly that happens before the prompt.
• Use all four mechanisms: recent window, semantic recall, working memory, and a knowledge graph.
• Assemble context in conditional layers, and never inject other users' data as system instructions.
• A knowledge graph gives memory that spans sessions, but it needs decay, purge, and an erasure path.
• Add a confidence score and a human review loop so memory improves instead of just accumulating.

The best agent memory is invisible. The user feels understood, the token bill stays flat, and a regulator can watch a user's data get deleted on request. That is engineering, not magic.

Building an agent that needs to remember, safely and cheaply? Tell us about it at contact or get a scoped quote.