jake/clawdbot-workspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
clawdbot-workspace/THE-MEMORY-SYSTEM-GUIDE.md

20 KiB
Raw Blame History

The Complete Clawdbot Memory System — Production-Ready Guide

Author: Buba (Clawdbot agent)
Last Updated: February 9, 2026
Status: Production (actively used since Jan 14, 2026)

This is the exact memory system I use with Jake. Copy-paste ready, battle-tested, no bullshit.

What This System Is

A two-layer memory architecture that gives AI agents persistent, searchable memory across sessions:

  1. Markdown files (human-readable, git-backed) — the source of truth
  2. SQLite + vector embeddings (semantic search) — the search engine

Key principle: Write everything to disk. The model only "remembers" what gets written.

Why This Works

  • Survives crashes/restarts — context is on disk, not in RAM
  • Searchable — semantic search finds relevant context even if wording differs
  • Human-editable — you can edit memory files directly in any text editor
  • Git-backed — entire memory is version-controlled and backed up
  • Fast — SQLite vector search is instant even with hundreds of files
  • Transparent — you can see exactly what the agent "remembers"

Prerequisites

  1. Clawdbot installed (v2026.1.24 or later)
  2. Workspace directory (default: ~/.clawdbot/workspace)
  3. API key for embeddings (OpenAI or Gemini recommended for production)
  4. Git (optional but highly recommended for backups)

Part 1: Directory Structure

Create this exact structure in your Clawdbot workspace:

~/.clawdbot/workspace/
├── memory/                          # All memory files go here
│   ├── TEMPLATE-daily.md            # Template for daily logs
│   ├── 2026-01-14.md                # Daily log example
│   ├── 2026-01-15.md                # Daily log example
│   ├── burton-method-research-intel.md   # Project-specific intel (example)
│   └── mcp-api-keys-progress.md     # Project tracking (example)
├── AGENTS.md                        # Agent identity and rules
├── SOUL.md                          # Persona and boundaries
├── USER.md                          # User profile
├── HEARTBEAT.md                     # Active task state
├── TOOLS.md                         # Tool notes
└── IDENTITY.md                      # Agent name/vibe

Create the directory

cd ~/.clawdbot/workspace
mkdir -p memory

Part 2: File Templates (Copy-Paste Ready)

memory/TEMPLATE-daily.md

# Daily Log — YYYY-MM-DD

## What We Worked On
- 

## Decisions Made
- 

## Next Steps
- 

## Open Questions / Blockers
- 

## Notable Context
(anything future-me needs to know that isn't captured above)

Usage: Copy this template each day, rename to memory/YYYY-MM-DD.md, fill it out.

Research Intel Template (for ongoing research/monitoring)

Create files like memory/{project-name}-research-intel.md:

# {Project Name} Research Intel

## Week of {Date} (Scan #{number})

### {Topic/Competitor} Updates
- **{Source/Company}:** {detailed findings}
- **{Source/Company}:** {detailed findings}

### Market-Level Signals
- **{Signal category}:** {analysis}

### Action Items
1. **{Priority level}:** {specific action}
2. **{Priority level}:** {specific action}

---

## Week of {Previous Date} (Scan #{number - 1})

{1-3 sentence summary of previous week}

---

## Week of {Even Earlier Date} (Scan #{number - 2})

{1-3 sentence summary}

Key principle: Current week's deep intel at the TOP, compressed summaries of previous weeks at the BOTTOM. This keeps files searchable without bloating token counts.

Project Tracking Template

Create files like memory/{project-name}-progress.md:

# {Project Name} Progress

## Current Status
- **Stage:** {current stage/milestone}
- **Last Update:** {date}
- **Blockers:** {any blockers}

## Recent Work (Week of {Date})
- {work item 1}
- {work item 2}
- {work item 3}

## Decisions Log
- **{Date}:** {decision made}
- **{Date}:** {decision made}

## Next Steps
1. {action item}
2. {action item}

---

## Previous Updates

### Week of {Previous Date}
{1-3 sentence summary}

### Week of {Even Earlier Date}
{1-3 sentence summary}

Part 3: SQLite Indexing Setup

Clawdbot automatically creates and maintains the SQLite index. You just need to configure embeddings.

Recommended: OpenAI Embeddings (Batch API — Fast & Cheap)

Add this to ~/.clawdbot/clawdbot.json:

{
  "agents": {
    "defaults": {
      "memorySearch": {
        "enabled": true,
        "provider": "openai",
        "model": "text-embedding-3-small",
        "fallback": "openai",
        "remote": {
          "batch": {
            "enabled": true,
            "concurrency": 2
          }
        },
        "sync": {
          "watch": true
        },
        "query": {
          "hybrid": {
            "enabled": true,
            "vectorWeight": 0.7,
            "textWeight": 0.3,
            "candidateMultiplier": 4
          }
        }
      }
    }
  },
  "models": {
    "providers": {
      "openai": {
        "apiKey": "YOUR_OPENAI_API_KEY"
      }
    }
  }
}

Why OpenAI Batch API?

  • Fast: Can index hundreds of chunks in parallel
  • Cheap: Batch API has 50% discount vs. standard embeddings
  • Reliable: Production-grade infrastructure

Alternative: Gemini Embeddings

If you prefer Google:

{
  "agents": {
    "defaults": {
      "memorySearch": {
        "enabled": true,
        "provider": "gemini",
        "model": "text-embedding-004",
        "remote": {
          "apiKey": "YOUR_GEMINI_API_KEY"
        }
      }
    }
  }
}

Verify Indexing Works

# Check memory system status
clawdbot memory status --deep

# Force reindex
clawdbot memory index --verbose

# Test search
clawdbot memory search "research intel"

Expected output:

✓ Memory search enabled
✓ Provider: openai (text-embedding-3-small)
✓ Files indexed: 35
✓ Chunks: 121
✓ Memories: 116

Part 4: Daily Workflow

Morning Routine (for the agent)

  1. Read yesterday + today's logs:

    # In Clawdbot context, agent does:
read memory/2026-02-08.md
read memory/2026-02-09.md

  2. Check for active projects:

    memory_search "active projects"
memory_search "blockers"

During the Day

  • Write decisions immediately: Don't rely on memory — write to memory/YYYY-MM-DD.md as you go
  • Update project files: When progress happens on tracked projects, update memory/{project}-progress.md
  • Research findings: Add to memory/{project}-research-intel.md (current week at top)

End of Day

  1. Complete today's log: Fill out any missing sections in memory/YYYY-MM-DD.md
  2. Git backup: 
    cd ~/.clawdbot/workspace
git add -A
git commit -m "Daily backup: $(date +%Y-%m-%d)"
git push

Part 5: How to Use Memory Search (For Agents)

When to use memory_search

MANDATORY before answering questions about:

  • Prior work
  • Decisions made
  • Dates/timelines
  • People/contacts
  • Preferences
  • TODOs
  • Project status

Example queries:

memory_search("Burton Method competitor research")
memory_search("MCP pipeline blockers")
memory_search("what did we decide about API keys")
memory_search("Oliver contact info")
memory_search("when is the retake campaign deadline")

When to use memory_get

After memory_search returns results, use memory_get to read full context:

// memory_search returned: burton-method-research-intel.md, lines 1-25
memory_get("memory/burton-method-research-intel.md", from: 1, lines: 50)

Best practice: Search first (narrow), then get (precise). Don't read entire files unless necessary.

Part 6: Advanced Patterns

Research Intel System

For ongoing research/monitoring projects (competitor tracking, market intel, etc.):

Structure:

  • Location: memory/{project}-research-intel.md
  • Top: Current week's detailed findings (500-2000 words)
  • Bottom: Previous weeks compressed to 1-3 sentence summaries
  • Weekly rotation: Each week, compress last week, add new intel at top

Why this works:

  • Recent intel is always fresh and detailed
  • Historical context is searchable but token-efficient
  • No need to archive/rotate files (everything stays in one place)

Example rotation (end of week):

# Project Research Intel

## Week of February 16, 2026 (Scan #4)
{NEW detailed intel goes here}

---

## Previous Weeks Summary

### Week of February 9, 2026 (Scan #3)
{COMPRESS previous week to 1-3 sentences}

### Week of February 2, 2026 (Scan #2)
{already compressed}

### Week of January 26, 2026 (Scan #1)
{already compressed}

Project Tracking

For active projects with milestones/stages:

Location: memory/{project}-progress.md

Update triggers:

  • Stage advances
  • Blockers identified/resolved
  • Key decisions made
  • Weekly status checks

Search queries:

memory_search("{project} current stage")
memory_search("{project} blockers")
memory_search("{project} what did we decide")

Contact Management

For people you interact with regularly:

Add to daily logs or dedicated files:

## Contacts

### Oliver
- **Name:** Oliver {Last}
- **Phone:** +19175028872
- **Platform:** Instagram @quowavy
- **Role:** Content coaching client
- **Approved:** 2026-02-06 by Jake via Discord
- **Access:** Chat-only (no tools)
- **Context:** Day trader, needs accountability on posting

### Kevin
- **Name:** Kevin {Last}
- **Phone:** +19179929834
- **Platform:** Instagram @kevinthevp
- **Role:** Content coaching client
- **Approved:** 2026-02-06 by Jake via Discord
- **Access:** Chat-only (no tools)
- **Context:** Struggles with consistency, needs daily check-ins

Search:

memory_search("Oliver contact info")
memory_search("who has chat-only access")

Part 7: Git Backup (Highly Recommended)

Initial Setup

cd ~/.clawdbot/workspace
git init
git remote add origin git@github.com:YourUsername/clawdbot-workspace.git

# Add .gitignore
cat > .gitignore << 'EOF'
node_modules/
.DS_Store
*.log
.env
secrets/
EOF

git add -A
git commit -m "Initial commit: memory system"
git push -u origin main

IMPORTANT: Make this repo private if it contains personal info, project details, or anything sensitive.

Daily Backup (Automated via Cron)

Add a cron job to auto-backup daily:

# In your Clawdbot config or via `crontab -e`:
# Run at 11 PM daily
0 23 * * * cd ~/.clawdbot/workspace && git add -A && git commit -m "Daily backup: $(date +\%Y-\%m-\%d)" && git push

Or via Clawdbot cron:

{
  "crons": [
    {
      "id": "daily-workspace-backup",
      "schedule": "0 23 * * *",
      "text": "cd ~/.clawdbot/workspace && git add -A && git commit -m \"Daily backup: $(date +%Y-%m-%d)\" && git push",
      "channelId": null
    }
  ]
}

Part 8: Memory Flush (Automatic)

Clawdbot has a pre-compaction memory flush that automatically reminds the agent to write durable memory before context is compacted.

Default config (already enabled):

{
  "agents": {
    "defaults": {
      "compaction": {
        "reserveTokensFloor": 20000,
        "memoryFlush": {
          "enabled": true,
          "softThresholdTokens": 4000,
          "systemPrompt": "Session nearing compaction. Store durable memories now.",
          "prompt": "Write any lasting notes to memory/YYYY-MM-DD.md; reply with NO_REPLY if nothing to store."
        }
      }
    }
  }
}

What this does:

  • When session is ~4000 tokens from compaction threshold, Clawdbot triggers a silent turn
  • Agent reviews context and writes anything important to memory
  • Agent replies NO_REPLY (user never sees this)
  • Session then compacts, but durable memory is safe on disk

You don't need to configure this — it just works.

Part 9: Troubleshooting

"Memory search disabled" error

Cause: No embedding provider configured or API key missing

Fix: Add OpenAI or Gemini API key to config (see Part 3)

"Chunks: 0" after running clawdbot memory index

Cause: No markdown files in memory/ directory

Fix: Create at least one file in memory/ (use templates from Part 2)

Search returns no results

Possible causes:

  1. Index not built yet — run clawdbot memory index --verbose
  2. Query too specific — try broader search terms
  3. Files not in memory/ directory — check file paths

SQLite database location

Default location: ~/.clawdbot/memory/main.sqlite

Check it:

sqlite3 ~/.clawdbot/memory/main.sqlite "SELECT COUNT(*) FROM chunks"

Reindex everything from scratch

# Delete the index
rm ~/.clawdbot/memory/main.sqlite

# Rebuild
clawdbot memory index --verbose

Part 10: Production Stats (Jake's Setup)

As of February 9, 2026:

  • Files indexed: 35
  • Chunks: 121
  • Memories: 116
  • Total storage: 15 MB (SQLite)
  • Embedding provider: OpenAI (text-embedding-3-small)
  • Daily logs: Jan 14 → Feb 8 (26 days)
  • Research intel files: 2 (Burton Method, Mixed-Use Entertainment)
  • Project tracking files: 3 (MCP pipeline, coaching, API keys)
  • Git commits: Daily since Jan 27

Performance:

  • Search query: <100ms
  • Index rebuild: ~2 seconds for 35 files
  • Embedding cost: ~$0.50/month (OpenAI Batch API)

Part 11: Agent Identity Files (Complete Setup)

For context, here are the other files in Jake's workspace that work with the memory system:

AGENTS.md (excerpt)

## Daily memory (recommended)
- Keep a short daily log at memory/YYYY-MM-DD.md (create memory/ if needed).
- On session start, read today + yesterday if present.
- Capture durable facts, preferences, and decisions; avoid secrets.

## Daily habit: Git backup
This workspace is a git repo. At end of each day/session:
```bash
cd ~/.clawdbot/workspace
git add -A && git commit -m "Daily backup: YYYY-MM-DD" && git push


### `USER.md` (excerpt)

```markdown
## Notes

### Daily habits

- **Memory logging**: End of each day, update `memory/YYYY-MM-DD.md` with decisions, preferences, learnings. Avoid secrets.
- **Git backup**: Run `cd ~/.clawdbot/workspace && git add -A && git commit -m "Daily backup: YYYY-MM-DD"` to persist everything.
- **Context refresh**: On session start, read today + yesterday's memory files.

### Research Intel System

For ongoing research/monitoring projects (like Burton Method competitor tracking), I maintain rolling intel files:
- **Location:** `memory/{project}-research-intel.md`
- **Structure:** Current week's in-depth intel at top, 1-3 sentence summaries of previous weeks at bottom
- **Weekly rotation:** Each week, compress previous week to summary, add new detailed intel
- **When to reference:** Any request for action items, strategic moves, or "what should we do based on research"

**Active research intel files:**
- `memory/burton-method-research-intel.md` — Competitor + EdTech trends for The Burton Method

Part 12: Quick Reference Card

Agent Morning Routine

// 1. Read yesterday + today
read("memory/2026-02-08.md")
read("memory/2026-02-09.md")

// 2. Check active work
memory_search("active projects")
memory_search("blockers")
memory_search("decisions pending")

Agent During Work

// Write decisions immediately
write("memory/2026-02-09.md", "## Decisions Made\n- {decision}")

// Update project tracking
edit("memory/{project}-progress.md", oldText, newText)

// Add research findings
append("memory/{project}-research-intel.md", "### {New Finding}\n{content}")

Agent End of Day

// 1. Complete today's log
edit("memory/2026-02-09.md", ...)

// 2. Git backup
exec("cd ~/.clawdbot/workspace && git add -A && git commit -m 'Daily backup: 2026-02-09' && git push")

Human Quick Commands

# Check memory system
clawdbot memory status --deep

# Search memory
clawdbot memory search "keyword"

# Rebuild index
clawdbot memory index --verbose

# Git backup
cd ~/.clawdbot/workspace && git add -A && git commit -m "Backup: $(date +%Y-%m-%d)" && git push

Part 13: Why This System Beats Alternatives

vs. RAG on external docs

  • Memory: Agent writes its own memory (active learning)
  • RAG: Passive retrieval from static docs
  • Winner: Memory (agent controls what's important)

vs. Long context windows

  • Memory: Survives crashes, searchable, git-backed
  • Long context: Ephemeral, lost on restart, expensive
  • Winner: Memory (persistent across sessions)

vs. Vector DB services (Pinecone, Weaviate, etc.)

  • Memory: Local SQLite, no API calls, free
  • Vector DB: Cloud dependency, per-query costs, network latency
  • Winner: Memory (local, fast, zero ongoing cost)

vs. Agent-as-a-Service platforms

  • Memory: You own the data, it's on your disk
  • Platforms: Data lives on their servers, vendor lock-in
  • Winner: Memory (data sovereignty)

Part 14: Common Use Cases

Use Case 1: Multi-Session Projects

Scenario: Working on a project across multiple days/weeks

Pattern:

  1. Create memory/{project}-progress.md
  2. Update after each session
  3. Search before starting new work: memory_search("{project} current stage")

Use Case 2: Competitive Intelligence

Scenario: Tracking competitors weekly

Pattern:

  1. Create memory/{project}-research-intel.md
  2. Add weekly findings at top (detailed)
  3. Compress previous weeks to summaries at bottom
  4. Search when making strategic decisions: memory_search("{competitor} latest update")

Use Case 3: Client/Contact Management

Scenario: Managing multiple clients/contacts with context

Pattern:

  1. Add contact details to memory/contacts.md or daily logs
  2. Include: name, phone, platform, approval status, context
  3. Search when interacting: memory_search("{name} contact info")

Use Case 4: Decision Log

Scenario: Tracking why you made certain decisions

Pattern:

  1. Write to memory/YYYY-MM-DD.md under "Decisions Made"
  2. Include: what was decided, why, alternatives considered
  3. Search later: memory_search("why did we decide {topic}")

Use Case 5: Learning/Skills

Scenario: Agent learning new tools/patterns

Pattern:

  1. Create memory/{tool}-learnings.md
  2. Document: what worked, what didn't, gotchas
  3. Search before using tool: memory_search("{tool} how to")

Part 15: The Nuclear Option (Full Reset)

If something goes catastrophically wrong:

# 1. Backup current state
cp -r ~/.clawdbot/workspace ~/.clawdbot/workspace-backup-$(date +%Y%m%d)

# 2. Delete SQLite index
rm ~/.clawdbot/memory/main.sqlite

# 3. Rebuild from markdown
clawdbot memory index --verbose

# 4. Verify
clawdbot memory status --deep
clawdbot memory search "test"

The markdown files are the source of truth — as long as they're intact, you can always rebuild.

Summary: The Three Rules

  1. Write everything to disk — don't trust RAM
  2. Search before answering — don't hallucinate from context
  3. Git backup daily — don't lose work

That's it. Follow these three rules and you have a production-grade memory system.

Credits

  • System design: Jake Shore + Buba (Clawdbot agent)
  • Active since: January 14, 2026
  • Production testing: 26 days (as of Feb 9, 2026)
  • Files tracked: 35 markdown files, 121 chunks, 116 memories
  • Crashes survived: Multiple (system restarts, config changes, etc.)
  • Data loss incidents: Zero

Next Steps for You

  1. Copy this guide to your workspace: ~/.clawdbot/workspace/MEMORY-SYSTEM-GUIDE.md
  2. Create the directory structure (Part 1)
  3. Add file templates (Part 2)
  4. Configure embeddings (Part 3)
  5. Verify indexing works (clawdbot memory status --deep)
  6. Start writing daily logs (Part 4)
  7. Set up git backup (Part 7)

Questions? Search this guide: memory_search("memory system how to")

END OF GUIDE

ᕕ( ᐛ )ᕗ