🧠

# Clawdbot Memory System

Never lose context to compaction again.
Built by Jake Shore & Buba
Version 2.0 | February 2026

## Why Your Agent Forgets Everything

You've spent hours building context with your AI agent. You've explained your project architecture, your preferences, your entire business setup. Then the next morning, you say "Hey, remember that API integration we worked on?" and get back: "I don't have any context about a previous API integration. Could you tell me more?"

This isn't a bug — it's how AI agents are designed. When your conversation gets too long, the system runs compaction: it summarizes the conversation to free up context space. The summary keeps the gist, but the details — the exact config values, the specific decisions you made, the nuances of your preferences — those get compressed into oblivion. Your agent essentially gets amnesia.

The problem is structural. Every AI assistant today — Claude, ChatGPT, Gemini, all of them — faces this same constraint. Context windows are finite. Compaction is inevitable. And without a system to persist knowledge outside the chat window, everything you build is written in sand.

> "My buddies hate when their agent just gets amnesia." — Jake

The Clawdbot Memory System solves this completely. It gives your agent a persistent memory layer — a real filesystem-backed brain that survives compaction, survives restarts, survives everything. Your agent writes knowledge to disk continuously and reads it back automatically. No more amnesia. No more re-explaining yourself.

## The Three Causes (and How We Fix Them)

Agent amnesia isn't one problem — it's three separate failure modes stacked on top of each other. We engineered a fix for each one.

#### Cause 1: Agent Never Writes Anything Down

Problem

Everything your agent knows lives exclusively in the chat context window. When compaction fires, the system generates a brief summary and discards the full conversation. Specific details — the exact configuration values you discussed, the precise reasoning behind a decision, your stated preferences — all get lost in summarization.

Fix

`AGENTS.md` instructs the agent to write to `memory/YYYY-MM-DD.md` throughout the session — not at the end, but continuously. Every decision, preference, project update, and notable piece of context gets written to disk in real-time. Even if compaction fires immediately after, the knowledge is already safely persisted.

#### Cause 2: Compaction Fires Before the Agent Can Save

Problem

Even with continuous writing, there's a race condition. The session hits its token limit, compaction fires immediately, and any context accumulated since the last write is gone. The agent doesn't get a chance to save before the rug is pulled out.

Fix

Pre-compaction flush. When the session is approximately 4,000 tokens from the compaction threshold, Clawdbot triggers a silent turn. The agent receives a system-level instruction to flush all unsaved context to disk immediately. It saves everything, responds with `NO_REPLY` (the user never sees this turn), and then compaction proceeds safely. Zero data loss.

#### Cause 3: Agent Doesn't Check Memory on New Sessions

Problem

Even if memories exist on disk, a fresh session starts with a clean slate. The agent has no idea those memory files are there unless something tells it to look. So it starts from zero every time — the knowledge exists but is never retrieved.

Fix

Mandatory Memory Recall rule baked into `AGENTS.md`: before answering any question about prior work, the agent MUST run `memory_search` first. Additionally, on every session start, the agent automatically reads today's and yesterday's logs. The result: the agent always knows what you've been working on.

## How It Works

The system uses a two-layer architecture designed for both human readability and machine-speed retrieval.

### Layer 1: Markdown Files (Source of Truth)

All memories are stored as plain Markdown files in the `memory/` directory. Daily logs follow the `YYYY-MM-DD.md` convention. Project files, research intel, and contacts all live as readable `.md` files. This layer is human-readable, git-backed, and editable. You can open any memory file in your text editor and read exactly what your agent knows. This is the source of truth — everything else is derived from it.

### Layer 2: SQLite + Vector Embeddings (Search Engine)

When memory files are created or updated, they're automatically chunked and indexed into a local SQLite database with vector embeddings. This gives the agent semantic search — it can find relevant memories even when the exact keywords don't match. Searching for "that API we integrated" will find the entry about "REST endpoint configuration for Stripe webhooks" because the vectors capture meaning, not just words.

### Hybrid Search: The Best of Both Worlds

Search results are ranked using a hybrid scoring algorithm: 70% vector similarity (semantic meaning) blended with 30% BM25 keyword matching (exact term relevance). This means the system excels at both fuzzy conceptual queries and precise keyword lookups. Every search completes in under 100 milliseconds.

### Pre-Compaction Flush Mechanism

Clawdbot monitors token usage throughout the session. When the conversation reaches approximately 4,000 tokens below the compaction threshold, a silent flush turn is injected. The agent writes all accumulated context to memory files, responds with `NO_REPLY`, and the user never sees this happen. Compaction then proceeds with all data safely persisted. It's the safety net that catches everything.

### Production Stats

35+ Memory Files
121+ Search Chunks
<100ms Search Speed
~$0.50 Cost / Month

Measured in production on Jake's Clawdbot instance. Zero data loss across months of daily use.

## What Gets Installed

The installer is modular — you choose what you want. The core memory system is always installed; everything else is optional and additive.

### 🧠 Core Memory System Always Installed

- `memory/` directory with date-based templates for daily logging
- SQLite vector search index — local, fast, no external database needed
- Hybrid search — 70% semantic vector similarity + 30% BM25 keyword matching
- Pre-compaction auto-flush — silent save triggered before compaction fires
- Agent instructions in `AGENTS.md` — teaches the agent how to use memory properly
- Non-destructive config patch — adds memory tools to Clawdbot without touching your existing setup

### 📂 Organization System Optional

- Project Quickstart Protocol — structured templates for kicking off new projects
- Research Intel System — weekly rotation with automatic compression (current week detailed, previous weeks summarized)
- Project Tracking — stages, blockers, decisions, and progress for each project
- Contacts Tracker — people, roles, preferences, and interaction history
- Tag Convention — consistent naming (`#project/name`, `#decision`, `#preference`) for reliable search

### 🏗️ Auto-Scaffold Optional

When you start a new project, the agent automatically creates structured memory files — project tracker, research intel, decision log — from templates. You just say "let's start working on X" and the scaffolding appears.

### 🔒 Git Backup Optional

Adds a daily backup habit to the agent's workflow. At the end of each session, the agent commits all memory changes to git and pushes to your private repo. Full version history of everything your agent has ever learned.

## Installation Guide

### One Command to Install

Open your terminal and run:

    bash <(curl -sL https://raw.githubusercontent.com/BusyBee3333/clawdbot-memory-system/main/install.sh)

The installer is interactive and walks you through five questions:

1
Choose your embedding provider
OpenAI, Gemini, or Local? ★ Jake recommends: OpenAI (best quality, ~$0.50/mo)
2
Enter your API key (if using OpenAI or Gemini)
★ Paste your key — it's stored locally only
3
Enable the Organization System?
Project tracking, research intel, contacts. ★ Jake recommends: Yes
4
Enable Auto-Scaffold?
Auto-create project files when you start new work. ★ Jake recommends: Yes
5
Enable Git Backup?
Daily commits of all memory changes. ★ Jake recommends: Yes (requires git repo)

### After Installation

Restart the Clawdbot gateway to pick up the new configuration:

    clawdbot gateway restart

Then test it — just ask your agent:

    "What did we work on today?"

If you've been chatting before the install, the agent will start building memory from this point forward. Within one session, it'll be writing to disk automatically.

### Embedding Provider Comparison

Provider | Model | Quality | Speed | Cost
--- | --- | --- | --- | ---
OpenAI ⭐ | text-embedding-3-small | Excellent | ~50ms | ~$0.50/mo
Gemini | text-embedding-004 | Very Good | ~80ms | Free tier available
Local | all-MiniLM-L6-v2 | Good | ~20ms | Free (CPU only)

Cost estimates based on typical usage (~100 memory operations/day). Local option requires no API key but has slightly lower semantic accuracy.

## The Seamless Experience

The best part of the memory system is that you don't have to do anything. From your perspective, you just chat normally. Everything happens in the background.

💬
You chat normally. No special commands, no "save this" or "remember that." Just talk to your agent like you always do.
✍️
Agent writes memory in the background. Throughout your conversation, the agent is quietly writing decisions, preferences, project updates, and context to disk. You'll occasionally see brief tool calls flash by — that's memory being saved.
🔍
Agent searches memory automatically. When you ask about something from a previous session, the agent runs a semantic search before answering. It pulls up relevant context from days, weeks, or months ago — without you asking it to.
🛡️
Agent flushes before compaction. When the session is running long and compaction is imminent, the system silently triggers a save. Everything is persisted before any context is lost. You never see this happen.
🪄
It just works. Three weeks from now, you'll say "what was that thing we discussed about the database migration?" and your agent will pull up the exact details. No re-explaining. No lost context. It just knows.

### Where the Edges Are

We believe in being honest about limitations. Here's where you might notice rough edges:

First session after install: The agent has no prior memories yet. It starts building from scratch. Give it a session or two and it'll have a solid knowledge base.
Very rapid-fire sessions: If you send 50 messages in 2 minutes, the agent might batch-write to memory rather than writing after every single message. Nothing is lost — it's just slightly delayed.
Cross-agent memory: Memory is per-Clawdbot-instance. If you use multiple AI providers or multiple Clawdbot installations, each has its own memory. (We're exploring sync in a future version.)
Embedding costs: With OpenAI embeddings, expect roughly $0.50/month for typical daily use. Gemini has a free tier. Local embeddings are completely free but slightly less accurate on semantic search.
Not a knowledge base: The memory system captures what happens in your sessions. It's not designed to store entire codebases or documentation — it's for decisions, preferences, project context, and conversational knowledge.

Your agent should remember you.
Now it does.