Giving an AI Agent a Memory: Patterns, Storage, and What to Forget

You will build a small, production-shaped memory layer for an AI agent, decide where each kind of memory lives, and design the forgetting policy that keeps it useful instead of dangerous. You will learn to separate short-term context from long-term facts, retrieve without flooding the prompt, and delete, expire, and defend memories on purpose.

Most agents fail not because the model is weak, but because everything is treated as the same kind of memory. A current tool output, a user preference, and a half-true inference the model made mid-conversation all end up in one growing prompt — producing rising cost, contradictory instructions, stale facts presented with confidence, and no clean way to remove something on request.

Memory is not a bigger context window. It is a set of decisions about what to keep, where to keep it, how to find it again, and when to let it go.

The kinds of memory you actually need

Borrow the vocabulary from cognitive science, but keep the engineering meaning strict.

Short-term, or working, memory is what the agent is using right now: the current task, recent messages, tool results, and its plan. It lives in the prompt, or in a session object just outside it.

Long-term memory is anything that must survive a session. Split it into two kinds, because they behave and fail differently.

Episodic memory records events: what happened, in what order, in which session. It answers what have we done? Append-heavy and time-ordered, it serves debugging, continuity, and review.

Semantic memory is distilled fact: preferences, entities, and stable knowledge. It answers what is true? Keep it small, curated, and editable — a wrong semantic memory is reused confidently in every future session.

Finally, name procedural memory separately: how the agent does things — system prompts, policies, tool descriptions, skills. Treat it as code, not data. Version it, review changes, and never let a conversation rewrite it silently. An agent that “learns” a procedure from an untrusted conversation has not learned; it has accepted an unauthenticated configuration change.

How memory flows through an agent loop

Design memory as a loop with four explicit stages, not as a database bolted on at the end.

Capture. Decide what is eligible to be remembered. Messages and tool outputs are candidates, not automatic records. Tag each with source, session, and type — a memory without provenance is impossible to audit.

Consolidate. Compress raw events into the form you will reuse: a short summary, a structured fact, or an embedding alongside the original text. This is where you pay an extra model call to save many future ones, and where most quality is won or lost.

Retrieve. Query long-term memory for what is relevant and inject a small, ranked selection into working memory. Every memory injected displaces something else and can distract the model.

Forget. Memories expire, get superseded, get deleted on request, or sink when they stop being retrieved. Give this stage its own rules and tests — do not leave it as a cleanup job for later.

Storage options, side by side

The vector index is not the memory; it is one way to find memory. Keep the canonical record — text, source, timestamps, scope, expiry — in a store you can inspect, filter, and delete from precisely.

Start boring. SQLite handles a single-process agent well, with transactions and exact deletes in one inspectable file. Move to Postgres for concurrent writers or shared access. Add a dedicated vector database when the index outgrows your main store — not before.

Building a small memory layer

This implementation is deliberately small: an interface, an in-memory store for tests, a SQLite store for persistence, and retrieval blending relevance, recency, and importance. It uses only the Python standard library, so you can swap in a real embedding provider later.

Every record field earns its place: without scope you cannot isolate users, without kind you cannot apply different lifetimes, and without source you cannot explain a memory’s origin.

from __future__ import annotations
 
import json
import math
import sqlite3
import time
import uuid
from dataclasses import dataclass, field
 
 
@dataclass
class MemoryRecord:
    text: str
    kind: str  # "episodic" or "semantic"
    scope: str  # e.g. a user id or workspace id
    source: str  # e.g. "user_message", "tool_result", "summary"
    importance: float = 0.5  # 0.0 to 1.0, set at write time
    embedding: list[float] = field(default_factory=list)
    id: str = field(default_factory=lambda: uuid.uuid4().hex)
    created_at: float = field(default_factory=time.time)
    expires_at: float | None = None
    superseded_by: str | None = None

The store interface below has two implementations: in-memory, and SQLite for persistence.

class MemoryStore:
    def add(self, record: MemoryRecord) -> str:
        raise NotImplementedError
 
    def get(self, memory_id: str) -> MemoryRecord | None:
        raise NotImplementedError
 
    def list_for_scope(self, scope: str) -> list[MemoryRecord]:
        raise NotImplementedError
 
    def delete(self, memory_id: str) -> None:
        raise NotImplementedError
 
 
class InMemoryStore(MemoryStore):
    def __init__(self) -> None:
        self._records: dict[str, MemoryRecord] = {}
 
    def add(self, record: MemoryRecord) -> str:
        self._records[record.id] = record
        return record.id
 
    def get(self, memory_id: str) -> MemoryRecord | None:
        return self._records.get(memory_id)
 
    def list_for_scope(self, scope: str) -> list[MemoryRecord]:
        return [r for r in self._records.values() if r.scope == scope]
 
    def delete(self, memory_id: str) -> None:
        self._records.pop(memory_id, None)
 
 
class SQLiteStore(MemoryStore):
    def __init__(self, path: str = "memory.db") -> None:
        self._conn = sqlite3.connect(path)
        self._conn.execute(
            "CREATE TABLE IF NOT EXISTS memories ("
            "id TEXT PRIMARY KEY, text TEXT NOT NULL, "
            "kind TEXT NOT NULL, scope TEXT NOT NULL, "
            "source TEXT NOT NULL, importance REAL NOT NULL, "
            "embedding TEXT NOT NULL, created_at REAL NOT NULL, "
            "expires_at REAL, superseded_by TEXT)"
        )
        self._conn.commit()
 
    def add(self, record: MemoryRecord) -> str:
        self._conn.execute(
            "INSERT OR REPLACE INTO memories VALUES (?,?,?,?,?,?,?,?,?,?)",
            (record.id, record.text, record.kind, record.scope,
             record.source, record.importance,
             json.dumps(record.embedding), record.created_at,
             record.expires_at, record.superseded_by),
        )
        self._conn.commit()
        return record.id
 
    def get(self, memory_id: str) -> MemoryRecord | None:
        row = self._conn.execute(
            "SELECT * FROM memories WHERE id = ?", (memory_id,)
        ).fetchone()
        return self._row_to_record(row) if row else None
 
    def list_for_scope(self, scope: str) -> list[MemoryRecord]:
        rows = self._conn.execute(
            "SELECT * FROM memories WHERE scope = ?", (scope,)
        ).fetchall()
        return [self._row_to_record(row) for row in rows]
 
    def delete(self, memory_id: str) -> None:
        self._conn.execute("DELETE FROM memories WHERE id = ?", (memory_id,))
        self._conn.commit()
 
    @staticmethod
    def _row_to_record(row: tuple) -> MemoryRecord:
        return MemoryRecord(
            id=row[0], text=row[1], kind=row[2], scope=row[3],
            source=row[4], importance=row[5],
            embedding=json.loads(row[6]), created_at=row[7],
            expires_at=row[8], superseded_by=row[9],
        )

Retrieval ranks only records that pass scope, expiry, and supersession filters.

def cosine_similarity(a: list[float], b: list[float]) -> float:
    if not a or not b or len(a) != len(b):
        return 0.0
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if norm_a == 0.0 or norm_b == 0.0:
        return 0.0
    return dot / (norm_a * norm_b)
 
 
def retrieve(
    store: MemoryStore,
    scope: str,
    query_embedding: list[float],
    limit: int = 5,
    now: float | None = None,
) -> list[MemoryRecord]:
    current = now if now is not None else time.time()
    candidates = []
    for record in store.list_for_scope(scope):
        if record.superseded_by is not None:
            continue
        if record.expires_at is not None and record.expires_at <= current:
            continue
        relevance = cosine_similarity(query_embedding, record.embedding)
        age = max(current - record.created_at, 0.0)
        recency = 1.0 / (1.0 + age / 86400.0)  # decays over days
        score = 0.6 * relevance + 0.25 * recency + 0.15 * record.importance
        candidates.append((score, record))
    candidates.sort(key=lambda item: item[0], reverse=True)
    return [record for _, record in candidates[:limit]]

To use it, embed incoming text with whichever model your stack uses, construct a MemoryRecord, and call store.add(record). Keep embedding outside the store — persistence and filtering belong to the store, representation to the model. Mixing them makes tests slow and provider swaps painful.

Retrieval: getting the right memory back

Vector similarity alone is weak. It finds text that sounds like the query, not text useful for the task — a support agent querying refunds will retrieve a policy discussion, a joke about refunds, and another product’s refund flow with equal enthusiasm.

Layer retrieval instead. Apply hard filters first: scope, kind, expiry, supersession. Rank what survives on relevance, recency, and importance. Then add the step most prototypes skip: a relevance check before injection — a rule that a semantic fact must also match on an entity or key, or a small model call asking whether this memory is needed here.

Summarisation is your other retrieval tool, and for episodic memory usually the better one. Periodically consolidate a session into a short summary record and retrieve the summary. You trade detail for density. Keep raw episodes underneath, linked by id, to drill down when a summary is challenged — but do not inject them by default.

Two rules keep retrieval honest. Cap and log what you inject per run, so you can replay a bad answer and see which memories shaped it. And mark retrieved facts as recalled context, not instructions: a preference should inform an answer, never outrank the current user message or the system prompt.

Forgetting on purpose: TTL, decay, and consolidation

Time-to-live (TTL) is the simplest policy and the right starting point. Assign expiry at write time by kind and sensitivity: short for working notes and tool outputs, medium for episodic records (then summarise and archive or delete the originals), and none for semantic facts, which are instead reviewed when superseded. TTL is blunt, but predictable and easy to test.

Decay is softer: a score fades with age or disuse, as in the recency term above. It is good for ranking, never sufficient for deletion — a decayed memory is still stored, still discoverable, and still owed to a user who asks for deletion.

Supersession handles the commonest case: the fact changed. Mark the old record superseded rather than overwriting it. Retrieval skips it, history remains for audit, and a mistaken new fact can be rolled back.

Consolidation is forgetting by compression: many episodes become one summary, many summaries one profile note. Run it as a background job, log what it merged and dropped, and keep it off the request path.

Make user-initiated forgetting first-class too: delete one record, delete a whole scope, export what you hold. Retrofitting exact deletion onto an append-only system is miserable work.

Failure modes: privacy, poisoning, and stale truth

Privacy leakage is the most serious. The scope check must happen in the store query, not in prompt wording — never retrieve across users or workspaces and rely on the model to ignore what does not belong. Assume anything stored can eventually be shown back, in a log or a debug view. Store the minimum needed, redact secrets at capture time, and protect stored memory like any other user data at rest.

Memory poisoning is the attack unique to this architecture. Anything that can write to memory can shape every future answer: a malicious document, crafted tool output, or participant in a shared workspace can plant an instruction disguised as a fact — a fake preference, policy, or prior agreement. Defences are structural. Promote content only from trusted sources, or after extraction that separates facts from embedded instructions. Never let retrieved memory act as a system instruction. Keep procedural memory outside the writable store. Log every promoted fact’s source, so a poisoned memory and everything derived from it can be removed.

Stale and false truth needs no attacker. Users change details, projects get renamed, models extract facts never stated. Store tentative inferences as episodic notes, not semantic facts, confirm high-impact facts occasionally, and treat a user correction as a higher-authority write applied immediately.

Feedback loops are the quiet failure. An agent that summarises its own outputs, then retrieves those summaries as evidence, gradually converts guesses into facts. Tag model-generated content at capture, weight user- and tool-sourced records above agent-sourced ones, and never promote an agent inference without external confirmation.

Verdict: what to build first

Build in this order. First, working memory: a bounded session context with summarisation when it grows, nothing leaking into persistence by accident. Second, episodic storage in SQLite or Postgres behind the store interface, with scope, source, and timestamps on every record. Third, semantic memory as a separate, curated store with supersession and exact deletion. Fourth, vector retrieval on top of those records — as an index, not the memory itself — with hard scope filters, blended ranking, and an injection cap. Fifth, before calling it done, build forgetting: TTLs per kind, user-scoped deletion, and a replayable injection log.

Resist both extremes. A stateless agent is not virtuous if users must re-teach it the same facts every session; an agent that remembers everything forever is not personalised, just unbounded. Agents that earn trust remember deliberately — a little, in the right store, with a source attached — and can show you what they remember, explain why, and forget it the moment you ask.