Ch06 Memory — Agent’s Persistent Information

An Agent without memory forgets every conversation.

You told it your company’s name, business, and competitors last week. This week, it forgot everything and you have to repeat.

This is annoying. Like going to the same coffee shop daily but having to reorder every time.

Memory lets Agents retain information across conversations, understanding you better over time.

6.1 Why Memory is Needed

A Real Dilemma

You ask the Agent to analyze competitors. It does a great job and gives you a detailed report.

The next day, you say “Update yesterday’s competitor analysis with the latest market share data.”

The Agent says: “Which competitors would you like to analyze?”

It doesn’t remember yesterday’s conversation.

This is the daily reality of an Agent without memory. Every conversation starts from zero.

The Nature of Memory

An Agent’s context window is limited. Even the latest models support 1M token contexts, but you can’t stuff all historical information in every time.

Moreover, larger context windows mean slower inference and higher costs.

The essence of memory is: store information that doesn’t need to be seen every time outside, and retrieve it when needed.

This is like the human brain. You don’t keep all your experiences in active memory at once. Most memories are stored in long-term memory; only what’s currently needed gets loaded into working memory.

6.2 Three-Layer Memory Architecture

Working Memory

Working memory is the content within the current context window.

It includes: system instructions, conversation history, retrieved relevant knowledge, current task status. This is the information the Agent can directly “see” at the current step.

Working memory is characterized by limited capacity and fast access. Like human short-term memory—the amount of information you can process simultaneously is limited.

Short-term Memory

Short-term memory is the conversation history within the current session.

What the user said, how the Agent replied, what intermediate results were. This information is typically not retained after the session ends, but is important during the session.

The key to managing short-term memory is compression. When the conversation gets long, summarize earlier content, retain key information, and discard redundant details.

Implementation: LangGraph provides the trim_messages function to automatically trim conversation history, avoiding context window limits. You only need to specify the maximum token count (e.g., 4000) and a trimming strategy (e.g., keep recent messages).

Short-term memory compression approach: When the message token count approaches the limit (e.g., reaches 80%), summarize earlier messages, keeping only recent ones. This preserves key information while controlling context length.

Long-term Memory

Long-term memory is persistent information across sessions.

User preferences, project context, historical decisions, learned experiences. This information is shared between multiple sessions.

Long-term memory is typically stored in external databases. It’s retrieved into context when needed.

6.3 Vector Embeddings: Turning Information into “Mathematics”

An Intuition

“Today’s weather is nice” and “Today’s climate is pleasant” — these two sentences are completely different in text, but their meanings are very close.

How do you let a computer know they’re similar in meaning?

The answer: turn them into numbers.

What are Vector Embeddings

Vector embedding is the technique of converting text into a string of numbers (a vector).

“Today’s weather is nice” gets converted to [0.2, 0.8, 0.1, …]. “Today’s climate is pleasant” gets converted to [0.21, 0.79, 0.11, …]. These two vectors are very close.

“I need to write a financial report” gets converted to [0.9, 0.1, 0.7, …]. This vector is very different from the first two.

The distance between vectors reflects semantic similarity. The closer the distance, the more similar the meaning.

Why Vectors are Needed

Traditional keyword search has limitations. If you search for “weather”, you can only find documents containing those characters. If the document says “climate”, it won’t be found.

Vector search doesn’t have this limitation. It matches based on semantic similarity, not exact keyword matches.

2026 Embedding Models

When choosing an embedding model, consider quality, speed, cost, and language support.

• OpenAI text-embedding-3: Good quality, requires API calls. Suitable for quality-critical scenarios.
• all-MiniLM-L6-v2: Open-source, can run locally, fast. Suitable for cost-sensitive scenarios.
• BGE series: Good Chinese performance, open-source. Suitable for Chinese scenarios.

Implementation: Using OpenAI’s embedding model, call the embeddings.create() method, passing the model name and text to get the vector (e.g., 1536 dimensions).

Vector similarity: Cosine similarity calculates the distance between two vectors. Closer to 1 means more similar. Similar sentences typically have similarity > 0.9; dissimilar ones < 0.5.

2026 Mainstream Embedding Model Comparison:

6.4 RAG’s Underlying Principles

What is RAG

RAG (Retrieval-Augmented Generation) is a technique that lets Agents retrieve information from external knowledge bases, then generate answers based on that information.

Core flow: User question → Retrieve relevant knowledge → Feed knowledge and question to model → Model generates answer based on knowledge.

Why RAG is Needed

A model’s training data has a cutoff date. It doesn’t know yesterday’s news, your company’s internal documents, or the latest product prices.

RAG lets the model “look up information” when answering, rather than relying entirely on knowledge learned during training.

RAG’s Complete Flow

Step 1: Process. Convert raw documents (PDF, web pages, databases) into plain text.

Step 2: Chunk. Split long text into small segments. Typically 200-1000 characters per chunk. Chunks too large → imprecise retrieval; chunks too small → lost context.

Step 3: Embed. Use an embedding model to convert each text chunk into a vector.

Step 4: Store. Store vectors in a vector database.

Step 5: Query. The user’s question is also converted into a vector.

Step 6: Retrieve. Find the text chunks most similar to the question vector in the vector database.

Step 7: Rerank. Re-sort the retrieved results using a more precise model. This step can improve recall by 20-40%.

Step 8: Generate. Feed the reranked text chunks and the user’s question to the model to generate the final answer.

2026 Vector Database Selection

• Pinecone: Fully managed, zero operations. Production RAG首选. Suitable for teams that don’t want to manage their own infrastructure.
• Weaviate: Native hybrid search (vector + BM25 keyword matching). Recall rate 20-40% higher than pure vector search. Suitable for scenarios with high search quality requirements.
• Qdrant: Performance-critical, self-hosting preferred. Suitable for teams with cost and performance requirements.
• pgvector: PostgreSQL ecosystem. If you’re already using PostgreSQL, just add an extension.

Implementation: Using LangChain + Pinecone to build a basic RAG requires just a few lines of code to implement document embedding storage and similarity retrieval.

Complete RAG pipeline: Includes document ingestion (process → chunk → embed → store) and query (embed → retrieve → rerank → generate) two main flows. In actual implementation, pay attention to chunk overlap to maintain context coherence, and use reranking models to improve retrieval precision.

RAG’s Limitations

Retrieval quality is the biggest bottleneck. If retrieved information is irrelevant or incomplete, no matter how smart the model is, it can’t generate good answers. Industry consensus: 80% of RAG system problems come from the retrieval stage, not the model itself.

Chunking strategy has big impact. Chunks too large → imprecise retrieval. Chunks too small → lost context. There’s no universal chunking strategy; it needs adjustment based on specific scenarios.

Knowledge base needs maintenance. When documents update, vectors need to be regenerated. If the knowledge base is outdated, retrieved information may be wrong.

6.5 Self-Evolving Memory: Agent’s “Dreaming” Learning Mechanism

An Interesting Concept

Anthropic proposed an experiment: let the Agent “dream” during session gaps.

After completing a conversation, the Agent launches a background process to review the conversation: what went well? where did it make mistakes? what are the user’s new preferences?

From these reviews, the Agent extracts persistent experiences and stores them in long-term memory.

In the next conversation, the Agent retrieves these experiences to avoid repeating mistakes and leverage accumulated knowledge.

Workflow

Session ends → Background review → Extract patterns → Update memory → Retrieve and use in next session.

This is like how the brain consolidates daytime memories during sleep. Important information gets consolidated; unimportant information gets forgotten.

Intelligent Forgetting

More memory isn’t always better. Outdated, low-value information interferes with the Agent’s judgment.

An intelligent forgetting mechanism uses a scoring model to evaluate each memory’s value: how long since last access? was it helpful when accessed? does it contradict the latest information?

Low-scoring memories get gradually demoted and eventually deleted.

Implementation: Claude Code’s “Dreaming” feature (released 2026) has already implemented this mechanism. The Agent automatically reviews and updates memory between sessions. You don’t need to implement it yourself; using Claude Code gives you this capability automatically.

Self-evolving memory implementation approach: After a session ends, use a model to review the conversation, extract user preferences, successful experiences, error lessons, and new learning, and store them in a vector database. Update access statistics and value scores each time memories are retrieved. Periodically clean up low-value memories (not accessed in 90 days and few access counts).

6.6 Semantic Caching: Saving Money with Memory

A Money-Saving Trick

A user asks a question, the Agent answers. Another user asks a similar question, and the Agent reasons from scratch again.

This is wasteful. If two questions are semantically similar and the answers are about the same, why not cache the first answer and reuse it for the second question?

This is semantic caching.

How it Works

The user’s question is first semantically matched against historical Q&A in the cache. If a historical Q&A with similarity exceeding a threshold is found, the cached answer is returned directly without calling the model again.

This can reduce costs by about 68%. Response speed is also much faster — cache queries take tens of milliseconds; model inference may take seconds.

Implementation: Redis is a commonly used semantic caching solution. In 2026, Bifrost and GPTCache also provide out-of-the-box semantic caching services, with cache hit rates typically between 60-85%.

Chapter Summary

Memory is the Agent’s persistent information. Key points:

1. Three-layer architecture: Working memory (context window), short-term memory (conversation history), long-term memory (external storage).
2. Vector embeddings: Turn text into numbers to enable semantic search.
3. RAG: Retrieve → chunk → embed → store → query → rerank → generate. Retrieval quality is the biggest bottleneck.
4. Vector database selection: Pinecone (managed), Weaviate (hybrid search), Qdrant (self-hosted), pgvector (PostgreSQL).
5. Self-evolving memory: Agents accumulate experiences through review. Claude Code’s Dreaming feature is built-in.
6. Semantic caching: Cache common Q&A to reduce costs by 68%.

The next chapter covers the Harness’s fourth subsystem: Sandbox. How Agents execute code and manipulate files in a secure environment.