Long-term RAG Memory

Qdrant vector store with Voyage embeddings, semantic recall, auto-deduplication, and six memory categories.

The Problem

AI assistants forget everything between sessions. Every conversation starts from zero. Users repeat themselves constantly: their name, their setup, their preferences, the project they have been working on for weeks. Bolting a vector database onto a chatbot helps, but without deduplication, categorization, and layered abstraction, the memory system quickly becomes noisy and unreliable.

How Baker Street Solves It

Baker Street combines two memory systems into a layered architecture. The primary layer is vector memory powered by Qdrant and Voyage AI embeddings. When you send a message, the Brain embeds it and performs a semantic search across your stored memories. Relevant facts appear in the system prompt, giving the agent persistent context about who you are and what you care about.

Memories are automatically deduplicated at 92% cosine similarity, preventing the store from filling up with near-identical entries. Six categories keep things organized: gear, preferences, homelab, personal, work, and general. The agent decides which category fits when storing new facts.

The second layer is observational memory. After each conversation, an observer extracts structured observations -- not just raw facts, but patterns, decisions, and contextual relationships. A reflector periodically compresses these observations into abstract knowledge. The result is a layered understanding: raw memories at the bottom, synthesized knowledge at the top.

Example

// How the Brain searches memory before each response
const query = embedWithVoyage(userMessage);
const memories = await qdrant.search("baker-street-memories", {
  vector: query,
  limit: 10,
  filter: {
    must: [{ key: "user_id", match: { value: userId } }],
  },
});

// Deduplicate before storing new memories
const existing = await qdrant.search("baker-street-memories", {
  vector: newMemoryEmbedding,
  score_threshold: 0.92,
  limit: 1,
});

if (existing.length === 0) {
  await qdrant.upsert("baker-street-memories", {
    id: generateId(),
    vector: newMemoryEmbedding,
    payload: { text, category: "preferences", user_id: userId },
  });
}

Learn More

See the Memory System documentation for configuration options, category management, and tuning similarity thresholds.