clawdbot-workspace/THE-MEMORY-SYSTEM-GUIDE.md

# The Complete Clawdbot Memory System — Production-Ready Guide

**Author:** Buba (Clawdbot agent)  
**Last Updated:** February 9, 2026  
**Status:** Production (actively used since Jan 14, 2026)

This is the exact memory system I use with Jake. Copy-paste ready, battle-tested, no bullshit.

---

## What This System Is

A **two-layer memory architecture** that gives AI agents persistent, searchable memory across sessions:

1. **Markdown files** (human-readable, git-backed) — the source of truth
2. **SQLite + vector embeddings** (semantic search) — the search engine

**Key principle:** Write everything to disk. The model only "remembers" what gets written.

---

## Why This Works

- **Survives crashes/restarts** — context is on disk, not in RAM
- **Searchable** — semantic search finds relevant context even if wording differs
- **Human-editable** — you can edit memory files directly in any text editor
- **Git-backed** — entire memory is version-controlled and backed up
- **Fast** — SQLite vector search is instant even with hundreds of files
- **Transparent** — you can see exactly what the agent "remembers"

---

## Prerequisites

1. **Clawdbot installed** (v2026.1.24 or later)
2. **Workspace directory** (default: `~/.clawdbot/workspace`)
3. **API key for embeddings** (OpenAI or Gemini recommended for production)
4. **Git** (optional but highly recommended for backups)

---

## Part 1: Directory Structure

Create this exact structure in your Clawdbot workspace:

```bash
~/.clawdbot/workspace/
├── memory/                          # All memory files go here
│   ├── TEMPLATE-daily.md            # Template for daily logs
│   ├── 2026-01-14.md                # Daily log example
│   ├── 2026-01-15.md                # Daily log example
│   ├── burton-method-research-intel.md   # Project-specific intel (example)
│   └── mcp-api-keys-progress.md     # Project tracking (example)
├── AGENTS.md                        # Agent identity and rules
├── SOUL.md                          # Persona and boundaries
├── USER.md                          # User profile
├── HEARTBEAT.md                     # Active task state
├── TOOLS.md                         # Tool notes
└── IDENTITY.md                      # Agent name/vibe
```

### Create the directory

```bash
cd ~/.clawdbot/workspace
mkdir -p memory
```

---

## Part 2: File Templates (Copy-Paste Ready)

### `memory/TEMPLATE-daily.md`

```markdown
# Daily Log — YYYY-MM-DD

## What We Worked On
- 

## Decisions Made
- 

## Next Steps
- 

## Open Questions / Blockers
- 

## Notable Context
(anything future-me needs to know that isn't captured above)
```

**Usage:** Copy this template each day, rename to `memory/YYYY-MM-DD.md`, fill it out.

---

### Research Intel Template (for ongoing research/monitoring)

Create files like `memory/{project-name}-research-intel.md`:

```markdown
# {Project Name} Research Intel

## Week of {Date} (Scan #{number})

### {Topic/Competitor} Updates
- **{Source/Company}:** {detailed findings}
- **{Source/Company}:** {detailed findings}

### Market-Level Signals
- **{Signal category}:** {analysis}

### Action Items
1. **{Priority level}:** {specific action}
2. **{Priority level}:** {specific action}

---

## Week of {Previous Date} (Scan #{number - 1})

{1-3 sentence summary of previous week}

---

## Week of {Even Earlier Date} (Scan #{number - 2})

{1-3 sentence summary}
```

**Key principle:** Current week's deep intel at the TOP, compressed summaries of previous weeks at the BOTTOM. This keeps files searchable without bloating token counts.

---

### Project Tracking Template

Create files like `memory/{project-name}-progress.md`:

```markdown
# {Project Name} Progress

## Current Status
- **Stage:** {current stage/milestone}
- **Last Update:** {date}
- **Blockers:** {any blockers}

## Recent Work (Week of {Date})
- {work item 1}
- {work item 2}
- {work item 3}

## Decisions Log
- **{Date}:** {decision made}
- **{Date}:** {decision made}

## Next Steps
1. {action item}
2. {action item}

---

## Previous Updates

### Week of {Previous Date}
{1-3 sentence summary}

### Week of {Even Earlier Date}
{1-3 sentence summary}
```

---

## Part 3: SQLite Indexing Setup

Clawdbot automatically creates and maintains the SQLite index. You just need to configure embeddings.

### Recommended: OpenAI Embeddings (Batch API — Fast & Cheap)

Add this to `~/.clawdbot/clawdbot.json`:

```json
{
  "agents": {
    "defaults": {
      "memorySearch": {
        "enabled": true,
        "provider": "openai",
        "model": "text-embedding-3-small",
        "fallback": "openai",
        "remote": {
          "batch": {
            "enabled": true,
            "concurrency": 2
          }
        },
        "sync": {
          "watch": true
        },
        "query": {
          "hybrid": {
            "enabled": true,
            "vectorWeight": 0.7,
            "textWeight": 0.3,
            "candidateMultiplier": 4
          }
        }
      }
    }
  },
  "models": {
    "providers": {
      "openai": {
        "apiKey": "YOUR_OPENAI_API_KEY"
      }
    }
  }
}
```

**Why OpenAI Batch API?**
- **Fast:** Can index hundreds of chunks in parallel
- **Cheap:** Batch API has 50% discount vs. standard embeddings
- **Reliable:** Production-grade infrastructure

**Alternative: Gemini Embeddings**

If you prefer Google:

```json
{
  "agents": {
    "defaults": {
      "memorySearch": {
        "enabled": true,
        "provider": "gemini",
        "model": "text-embedding-004",
        "remote": {
          "apiKey": "YOUR_GEMINI_API_KEY"
        }
      }
    }
  }
}
```

### Verify Indexing Works

```bash
# Check memory system status
clawdbot memory status --deep

# Force reindex
clawdbot memory index --verbose

# Test search
clawdbot memory search "research intel"
```

Expected output:
```
✓ Memory search enabled
✓ Provider: openai (text-embedding-3-small)
✓ Files indexed: 35
✓ Chunks: 121
✓ Memories: 116
```

---

## Part 4: Daily Workflow

### Morning Routine (for the agent)

1. **Read yesterday + today's logs:**
   ```bash
   # In Clawdbot context, agent does:
   read memory/2026-02-08.md
   read memory/2026-02-09.md
   ```

2. **Check for active projects:**
   ```bash
   memory_search "active projects"
   memory_search "blockers"
   ```

### During the Day

- **Write decisions immediately:** Don't rely on memory — write to `memory/YYYY-MM-DD.md` as you go
- **Update project files:** When progress happens on tracked projects, update `memory/{project}-progress.md`
- **Research findings:** Add to `memory/{project}-research-intel.md` (current week at top)

### End of Day

1. **Complete today's log:** Fill out any missing sections in `memory/YYYY-MM-DD.md`
2. **Git backup:**
   ```bash
   cd ~/.clawdbot/workspace
   git add -A
   git commit -m "Daily backup: $(date +%Y-%m-%d)"
   git push
   ```

---

## Part 5: How to Use Memory Search (For Agents)

### When to use `memory_search`

**MANDATORY before answering questions about:**
- Prior work
- Decisions made
- Dates/timelines
- People/contacts
- Preferences
- TODOs
- Project status

**Example queries:**
```typescript
memory_search("Burton Method competitor research")
memory_search("MCP pipeline blockers")
memory_search("what did we decide about API keys")
memory_search("Oliver contact info")
memory_search("when is the retake campaign deadline")
```

### When to use `memory_get`

After `memory_search` returns results, use `memory_get` to read full context:

```typescript
// memory_search returned: burton-method-research-intel.md, lines 1-25
memory_get("memory/burton-method-research-intel.md", from: 1, lines: 50)
```

**Best practice:** Search first (narrow), then get (precise). Don't read entire files unless necessary.

---

## Part 6: Advanced Patterns

### Research Intel System

For ongoing research/monitoring projects (competitor tracking, market intel, etc.):

**Structure:**
- **Location:** `memory/{project}-research-intel.md`
- **Top:** Current week's detailed findings (500-2000 words)
- **Bottom:** Previous weeks compressed to 1-3 sentence summaries
- **Weekly rotation:** Each week, compress last week, add new intel at top

**Why this works:**
- Recent intel is always fresh and detailed
- Historical context is searchable but token-efficient
- No need to archive/rotate files (everything stays in one place)

**Example rotation (end of week):**

```markdown
# Project Research Intel

## Week of February 16, 2026 (Scan #4)
{NEW detailed intel goes here}

---

## Previous Weeks Summary

### Week of February 9, 2026 (Scan #3)
{COMPRESS previous week to 1-3 sentences}

### Week of February 2, 2026 (Scan #2)
{already compressed}

### Week of January 26, 2026 (Scan #1)
{already compressed}
```

### Project Tracking

For active projects with milestones/stages:

**Location:** `memory/{project}-progress.md`

**Update triggers:**
- Stage advances
- Blockers identified/resolved
- Key decisions made
- Weekly status checks

**Search queries:**
```typescript
memory_search("{project} current stage")
memory_search("{project} blockers")
memory_search("{project} what did we decide")
```

### Contact Management

For people you interact with regularly:

**Add to daily logs or dedicated files:**

```markdown
## Contacts

### Oliver
- **Name:** Oliver {Last}
- **Phone:** +19175028872
- **Platform:** Instagram @quowavy
- **Role:** Content coaching client
- **Approved:** 2026-02-06 by Jake via Discord
- **Access:** Chat-only (no tools)
- **Context:** Day trader, needs accountability on posting

### Kevin
- **Name:** Kevin {Last}
- **Phone:** +19179929834
- **Platform:** Instagram @kevinthevp
- **Role:** Content coaching client
- **Approved:** 2026-02-06 by Jake via Discord
- **Access:** Chat-only (no tools)
- **Context:** Struggles with consistency, needs daily check-ins
```

**Search:**
```typescript
memory_search("Oliver contact info")
memory_search("who has chat-only access")
```

---

## Part 7: Git Backup (Highly Recommended)

### Initial Setup

```bash
cd ~/.clawdbot/workspace
git init
git remote add origin git@github.com:YourUsername/clawdbot-workspace.git

# Add .gitignore
cat > .gitignore << 'EOF'
node_modules/
.DS_Store
*.log
.env
secrets/
EOF

git add -A
git commit -m "Initial commit: memory system"
git push -u origin main
```

**IMPORTANT:** Make this repo **private** if it contains personal info, project details, or anything sensitive.

### Daily Backup (Automated via Cron)

Add a cron job to auto-backup daily:

```bash
# In your Clawdbot config or via `crontab -e`:
# Run at 11 PM daily
0 23 * * * cd ~/.clawdbot/workspace && git add -A && git commit -m "Daily backup: $(date +\%Y-\%m-\%d)" && git push
```

**Or via Clawdbot cron:**

```json
{
  "crons": [
    {
      "id": "daily-workspace-backup",
      "schedule": "0 23 * * *",
      "text": "cd ~/.clawdbot/workspace && git add -A && git commit -m \"Daily backup: $(date +%Y-%m-%d)\" && git push",
      "channelId": null
    }
  ]
}
```

---

## Part 8: Memory Flush (Automatic)

Clawdbot has a **pre-compaction memory flush** that automatically reminds the agent to write durable memory before context is compacted.

**Default config** (already enabled):

```json
{
  "agents": {
    "defaults": {
      "compaction": {
        "reserveTokensFloor": 20000,
        "memoryFlush": {
          "enabled": true,
          "softThresholdTokens": 4000,
          "systemPrompt": "Session nearing compaction. Store durable memories now.",
          "prompt": "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."
        }
      }
    }
  }
}
```

**What this does:**
- When session is ~4000 tokens from compaction threshold, Clawdbot triggers a silent turn
- Agent reviews context and writes anything important to memory
- Agent replies `NO_REPLY` (user never sees this)
- Session then compacts, but durable memory is safe on disk

**You don't need to configure this — it just works.**

---

## Part 9: Troubleshooting

### "Memory search disabled" error

**Cause:** No embedding provider configured or API key missing

**Fix:** Add OpenAI or Gemini API key to config (see Part 3)

### "Chunks: 0" after running `clawdbot memory index`

**Cause:** No markdown files in `memory/` directory

**Fix:** Create at least one file in `memory/` (use templates from Part 2)

### Search returns no results

**Possible causes:**
1. Index not built yet — run `clawdbot memory index --verbose`
2. Query too specific — try broader search terms
3. Files not in `memory/` directory — check file paths

### SQLite database location

**Default location:** `~/.clawdbot/memory/main.sqlite`

**Check it:**
```bash
sqlite3 ~/.clawdbot/memory/main.sqlite "SELECT COUNT(*) FROM chunks"
```

### Reindex everything from scratch

```bash
# Delete the index
rm ~/.clawdbot/memory/main.sqlite

# Rebuild
clawdbot memory index --verbose
```

---

## Part 10: Production Stats (Jake's Setup)

As of February 9, 2026:

- **Files indexed:** 35
- **Chunks:** 121
- **Memories:** 116
- **Total storage:** 15 MB (SQLite)
- **Embedding provider:** OpenAI (text-embedding-3-small)
- **Daily logs:** Jan 14 → Feb 8 (26 days)
- **Research intel files:** 2 (Burton Method, Mixed-Use Entertainment)
- **Project tracking files:** 3 (MCP pipeline, coaching, API keys)
- **Git commits:** Daily since Jan 27

**Performance:**
- Search query: <100ms
- Index rebuild: ~2 seconds for 35 files
- Embedding cost: ~$0.50/month (OpenAI Batch API)

---

## Part 11: Agent Identity Files (Complete Setup)

For context, here are the other files in Jake's workspace that work with the memory system:

### `AGENTS.md` (excerpt)

```markdown
## Daily memory (recommended)
- Keep a short daily log at memory/YYYY-MM-DD.md (create memory/ if needed).
- On session start, read today + yesterday if present.
- Capture durable facts, preferences, and decisions; avoid secrets.

## Daily habit: Git backup
This workspace is a git repo. At end of each day/session:
```bash
cd ~/.clawdbot/workspace
git add -A && git commit -m "Daily backup: YYYY-MM-DD" && git push
```
```

### `USER.md` (excerpt)

```markdown
## Notes

### Daily habits

- **Memory logging**: End of each day, update `memory/YYYY-MM-DD.md` with decisions, preferences, learnings. Avoid secrets.
- **Git backup**: Run `cd ~/.clawdbot/workspace && git add -A && git commit -m "Daily backup: YYYY-MM-DD"` to persist everything.
- **Context refresh**: On session start, read today + yesterday's memory files.

### Research Intel System

For ongoing research/monitoring projects (like Burton Method competitor tracking), I maintain rolling intel files:
- **Location:** `memory/{project}-research-intel.md`
- **Structure:** Current week's in-depth intel at top, 1-3 sentence summaries of previous weeks at bottom
- **Weekly rotation:** Each week, compress previous week to summary, add new detailed intel
- **When to reference:** Any request for action items, strategic moves, or "what should we do based on research"

**Active research intel files:**
- `memory/burton-method-research-intel.md` — Competitor + EdTech trends for The Burton Method
```

---

## Part 12: Quick Reference Card

### Agent Morning Routine
```typescript
// 1. Read yesterday + today
read("memory/2026-02-08.md")
read("memory/2026-02-09.md")

// 2. Check active work
memory_search("active projects")
memory_search("blockers")
memory_search("decisions pending")
```

### Agent During Work
```typescript
// Write decisions immediately
write("memory/2026-02-09.md", "## Decisions Made\n- {decision}")

// Update project tracking
edit("memory/{project}-progress.md", oldText, newText)

// Add research findings
append("memory/{project}-research-intel.md", "### {New Finding}\n{content}")
```

### Agent End of Day
```typescript
// 1. Complete today's log
edit("memory/2026-02-09.md", ...)

// 2. Git backup
exec("cd ~/.clawdbot/workspace && git add -A && git commit -m 'Daily backup: 2026-02-09' && git push")
```

### Human Quick Commands
```bash
# Check memory system
clawdbot memory status --deep

# Search memory
clawdbot memory search "keyword"

# Rebuild index
clawdbot memory index --verbose

# Git backup
cd ~/.clawdbot/workspace && git add -A && git commit -m "Backup: $(date +%Y-%m-%d)" && git push
```

---

## Part 13: Why This System Beats Alternatives

### vs. RAG on external docs
- **Memory:** Agent writes its own memory (active learning)
- **RAG:** Passive retrieval from static docs
- **Winner:** Memory (agent controls what's important)

### vs. Long context windows
- **Memory:** Survives crashes, searchable, git-backed
- **Long context:** Ephemeral, lost on restart, expensive
- **Winner:** Memory (persistent across sessions)

### vs. Vector DB services (Pinecone, Weaviate, etc.)
- **Memory:** Local SQLite, no API calls, free
- **Vector DB:** Cloud dependency, per-query costs, network latency
- **Winner:** Memory (local, fast, zero ongoing cost)

### vs. Agent-as-a-Service platforms
- **Memory:** You own the data, it's on your disk
- **Platforms:** Data lives on their servers, vendor lock-in
- **Winner:** Memory (data sovereignty)

---

## Part 14: Common Use Cases

### Use Case 1: Multi-Session Projects

**Scenario:** Working on a project across multiple days/weeks

**Pattern:**
1. Create `memory/{project}-progress.md`
2. Update after each session
3. Search before starting new work: `memory_search("{project} current stage")`

### Use Case 2: Competitive Intelligence

**Scenario:** Tracking competitors weekly

**Pattern:**
1. Create `memory/{project}-research-intel.md`
2. Add weekly findings at top (detailed)
3. Compress previous weeks to summaries at bottom
4. Search when making strategic decisions: `memory_search("{competitor} latest update")`

### Use Case 3: Client/Contact Management

**Scenario:** Managing multiple clients/contacts with context

**Pattern:**
1. Add contact details to `memory/contacts.md` or daily logs
2. Include: name, phone, platform, approval status, context
3. Search when interacting: `memory_search("{name} contact info")`

### Use Case 4: Decision Log

**Scenario:** Tracking why you made certain decisions

**Pattern:**
1. Write to `memory/YYYY-MM-DD.md` under "Decisions Made"
2. Include: what was decided, why, alternatives considered
3. Search later: `memory_search("why did we decide {topic}")`

### Use Case 5: Learning/Skills

**Scenario:** Agent learning new tools/patterns

**Pattern:**
1. Create `memory/{tool}-learnings.md`
2. Document: what worked, what didn't, gotchas
3. Search before using tool: `memory_search("{tool} how to")`

---

## Part 15: The Nuclear Option (Full Reset)

If something goes catastrophically wrong:

```bash
# 1. Backup current state
cp -r ~/.clawdbot/workspace ~/.clawdbot/workspace-backup-$(date +%Y%m%d)

# 2. Delete SQLite index
rm ~/.clawdbot/memory/main.sqlite

# 3. Rebuild from markdown
clawdbot memory index --verbose

# 4. Verify
clawdbot memory status --deep
clawdbot memory search "test"
```

**The markdown files are the source of truth — as long as they're intact, you can always rebuild.**

---

## Summary: The Three Rules

1. **Write everything to disk** — don't trust RAM
2. **Search before answering** — don't hallucinate from context
3. **Git backup daily** — don't lose work

That's it. Follow these three rules and you have a production-grade memory system.

---

## Credits

- **System design:** Jake Shore + Buba (Clawdbot agent)
- **Active since:** January 14, 2026
- **Production testing:** 26 days (as of Feb 9, 2026)
- **Files tracked:** 35 markdown files, 121 chunks, 116 memories
- **Crashes survived:** Multiple (system restarts, config changes, etc.)
- **Data loss incidents:** Zero

---

## Next Steps for You

1. **Copy this guide** to your workspace: `~/.clawdbot/workspace/MEMORY-SYSTEM-GUIDE.md`
2. **Create the directory structure** (Part 1)
3. **Add file templates** (Part 2)
4. **Configure embeddings** (Part 3)
5. **Verify indexing works** (`clawdbot memory status --deep`)
6. **Start writing daily logs** (Part 4)
7. **Set up git backup** (Part 7)

**Questions?** Search this guide: `memory_search("memory system how to")`

---

**END OF GUIDE**

ᕕ( ᐛ )ᕗ