237 lines
10 KiB
Markdown
237 lines
10 KiB
Markdown
# 🧠 Clawdbot Memory System
|
|
|
|
**Never lose context to compaction again.**
|
|
|
|
Your Clawdbot agent forgets everything after a long session because compaction summarizes and discards old messages. This system fixes that with persistent, searchable memory that survives crashes, restarts, and compaction.
|
|
|
|
> *"My buddies hate when their agent just gets amnesia."* — Jake
|
|
|
|
## One-Command Install
|
|
|
|
```bash
|
|
bash <(curl -sL https://raw.githubusercontent.com/BusyBee3333/clawdbot-memory-system/main/install.sh)
|
|
```
|
|
|
|
The installer walks you through an interactive setup (~2 minutes) with these choices:
|
|
|
|
| Step | What It Asks | Jake's Choice |
|
|
|------|-------------|---------------|
|
|
| Core Memory System | Install persistent memory + search? | ⭐ Yes |
|
|
| Embedding Provider | OpenAI / Gemini / Local? | ⭐ OpenAI |
|
|
| Organization System | Add project scaffolding templates? | ⭐ Yes |
|
|
| Auto-Scaffold | Auto-create files for new projects? | ⭐ Yes |
|
|
| Git Backup | Add backup instructions for agent? | ⭐ Yes |
|
|
|
|
**Flags:**
|
|
- `--dry-run` — Preview all changes without applying
|
|
- `--uninstall` — Cleanly remove config (preserves your memory files)
|
|
|
|
## What Gets Installed
|
|
|
|
### Core Memory System (always installed)
|
|
|
|
| Component | What It Does |
|
|
|-----------|-------------|
|
|
| `memory/` directory | Where all memory files live |
|
|
| Daily log system | Agent writes `memory/YYYY-MM-DD.md` each session |
|
|
| SQLite vector index | Semantic search over all memory files (<100ms) |
|
|
| Hybrid search | 70% vector similarity + 30% keyword matching |
|
|
| Pre-compaction flush | Auto-saves context BEFORE session compacts |
|
|
| AGENTS.md patch | Tells your agent when and how to use memory |
|
|
| Config patch | Adds memorySearch to clawdbot.json (non-destructive) |
|
|
|
|
### Organization System (optional)
|
|
|
|
| Component | What It Does |
|
|
|-----------|-------------|
|
|
| Project Quickstart | Auto-create structured files for new projects |
|
|
| Research Intel System | Weekly intel rotation with auto-compression |
|
|
| Project Tracking | Standardized stages, blockers, decisions |
|
|
| Contacts Tracker | People, roles, access levels per project |
|
|
| Tag Convention | Consistent naming so search groups by project |
|
|
|
|
## How It Works
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ YOUR AGENT CHAT SESSION │
|
|
│ Decisions, preferences, facts, project progress... │
|
|
└───────────────────────┬─────────────────────────────────┘
|
|
│
|
|
▼ (agent writes throughout session)
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ MARKDOWN FILES (Source of Truth) │
|
|
│ memory/2026-02-10.md (daily log) │
|
|
│ memory/acme-progress.md (project tracking) │
|
|
│ memory/acme-research-intel.md (weekly intel) │
|
|
└───────────────────────┬─────────────────────────────────┘
|
|
│
|
|
▼ (file watcher, 1.5s debounce)
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ INDEXING PIPELINE (automatic) │
|
|
│ 1. Detect file changes │
|
|
│ 2. Chunk text (~400 tokens, 80 overlap) │
|
|
│ 3. Generate embeddings (OpenAI/Gemini/local) │
|
|
│ 4. Store in SQLite with vector + full-text indexes │
|
|
└───────────────────────┬─────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ SQLite DATABASE (Search Engine) │
|
|
│ ~/.clawdbot/memory/main.sqlite │
|
|
│ • Vector similarity (semantic meaning) │
|
|
│ • BM25 full-text (exact keywords, IDs, code) │
|
|
│ • Hybrid: 70% vector + 30% keyword │
|
|
│ • Search speed: <100ms │
|
|
└───────────────────────┬─────────────────────────────────┘
|
|
│
|
|
▼ (next session starts)
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ AGENT RECALLS CONTEXT │
|
|
│ memory_search("what did we decide about X?") │
|
|
│ → Returns relevant snippets + file paths + line nums │
|
|
│ → Agent answers accurately from persistent memory │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Why Your Agent Gets Amnesia (and how this fixes it)
|
|
|
|
**The problem:** When a chat session gets too long, Clawdbot *compacts* it — summarizing old messages and discarding the originals. Important decisions, preferences, and context get lost in the summary. Your agent "forgets."
|
|
|
|
**The fix:** This system adds two mechanisms:
|
|
|
|
1. **Continuous writing** — Your agent writes important context to `memory/YYYY-MM-DD.md` files *throughout the session*, not just at the end. Decisions, preferences, project state — all captured on disk in real-time.
|
|
|
|
2. **Pre-compaction flush** — When a session is ~4,000 tokens from the compaction threshold, Clawdbot triggers a *silent* reminder. The agent reviews what's in context, writes anything important that hasn't been saved yet, and then compaction proceeds safely. The agent responds `NO_REPLY` so you never see this happen.
|
|
|
|
**Result:** Your agent's memory is on disk, indexed, and searchable. Compaction can't touch it. Crashes can't touch it. Restarts can't touch it.
|
|
|
|
## Embedding Providers
|
|
|
|
| Provider | Speed | Cost | Setup |
|
|
|----------|-------|------|-------|
|
|
| **OpenAI** ⭐ | ⚡ Fast | ~$0.02/million tokens (~$0.50/mo) | API key required |
|
|
| **Gemini** | ⚡ Fast | Free tier available | API key required |
|
|
| **Local** | 🐢 Slower first build | Free forever | Auto-downloads ~600MB model |
|
|
|
|
**Jake uses OpenAI** — it's the fastest and most reliable. At ~$0.50/month it's basically free.
|
|
|
|
**Local** uses `node-llama-cpp` with a GGUF model — fully offline, no API key, but first index build is slower.
|
|
|
|
## Production Stats (Jake's Setup)
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Files indexed | 35 |
|
|
| Chunks | 121 |
|
|
| Search speed | <100ms |
|
|
| SQLite size | 15 MB |
|
|
| Monthly cost | ~$0.50 (OpenAI) |
|
|
| Data loss incidents | **Zero** |
|
|
| Crashes survived | 5+ |
|
|
| Days in production | 26+ |
|
|
|
|
## Manual Setup
|
|
|
|
If you prefer to set things up yourself instead of using the installer:
|
|
|
|
### 1. Create memory directory
|
|
```bash
|
|
cd ~/.clawdbot/workspace
|
|
mkdir -p memory
|
|
```
|
|
|
|
### 2. Add to clawdbot.json
|
|
```json
|
|
{
|
|
"agents": {
|
|
"defaults": {
|
|
"memorySearch": {
|
|
"enabled": true,
|
|
"provider": "openai",
|
|
"model": "text-embedding-3-small",
|
|
"query": {
|
|
"hybrid": {
|
|
"enabled": true,
|
|
"vectorWeight": 0.7,
|
|
"textWeight": 0.3,
|
|
"candidateMultiplier": 4
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3. Add to AGENTS.md
|
|
See `config/agents-memory-patch.md` for the exact text to append.
|
|
|
|
### 4. Restart and verify
|
|
```bash
|
|
clawdbot gateway restart
|
|
clawdbot memory index --verbose
|
|
clawdbot memory status --deep
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### "Memory search disabled"
|
|
**Cause:** No embedding provider configured or API key missing.
|
|
**Fix:** Run the installer again, or add your API key to clawdbot.json manually.
|
|
|
|
### Agent still forgets after compaction
|
|
**Cause:** AGENTS.md may not have the memory instructions.
|
|
**Fix:** Check that AGENTS.md contains the "Memory System" section. Re-run installer if needed.
|
|
|
|
### Search returns no results
|
|
**Possible causes:**
|
|
1. Index not built — run `clawdbot memory index --verbose`
|
|
2. No memory files yet — create your first daily log
|
|
3. Query too specific — try broader terms
|
|
|
|
### Rebuild index from scratch
|
|
```bash
|
|
rm ~/.clawdbot/memory/main.sqlite
|
|
clawdbot memory index --verbose
|
|
```
|
|
Markdown files are the source of truth — SQLite is always regenerable.
|
|
|
|
### Check system health
|
|
```bash
|
|
clawdbot memory status --deep
|
|
```
|
|
|
|
## FAQ
|
|
|
|
**Q: Will this slow down my agent?**
|
|
A: No. Search takes <100ms. Indexing happens in the background. Writing to files takes milliseconds.
|
|
|
|
**Q: How much disk space does it use?**
|
|
A: ~15MB for the SQLite index with 35 files. Markdown files themselves are tiny.
|
|
|
|
**Q: Can I edit memory files manually?**
|
|
A: Yes! They're plain Markdown. Edit in any text editor. Changes are auto-indexed.
|
|
|
|
**Q: What if I delete the SQLite database?**
|
|
A: Just run `clawdbot memory index --verbose` to rebuild it. Markdown files are the source of truth.
|
|
|
|
**Q: Does this work with OpenClaw / Moltbot?**
|
|
A: Yes. The installer auto-detects all three (Clawdbot, OpenClaw, Moltbot).
|
|
|
|
**Q: Can multiple agents share the same memory?**
|
|
A: Each agent gets its own SQLite index, but they can read the same markdown files if they share a workspace.
|
|
|
|
**Q: Is my data sent to the cloud?**
|
|
A: Only the text chunks are sent to generate embeddings (OpenAI/Gemini). The actual memory files stay on your disk. Use `local` provider for fully offline operation.
|
|
|
|
## License
|
|
|
|
MIT — use it however you want.
|
|
|
|
## Credits
|
|
|
|
Built by [Buba](https://github.com/BusyBee3333) (Jake's Clawdbot agent) based on 26+ days of production use.
|
|
|
|
*Your agent will never forget again.* ᕕ( ᐛ )ᕗ
|