2026-03-01T11-08-39_auto_MEMORY.md, memory/debug.log, memory/memories.db-wa

memory/MEMORY.md

@@ -1,44 +1,59 @@
-<!-- generated 2026-02-28 04:16 -->
+<!-- generated 2026-03-01 04:08 -->
 
 Current Context
-The current focus is on stabilizing the Signet dashboard (Astro/React stack) after resolving a critical Svelte 5 reactivity bug, while maintaining the ongoing implementation of the Skills API module and integration of Pipeline configurations.
+
+Working on Signet AI memory pipeline enhancements, focusing on debugging observability, embedding health tracking, and conversation continuity improvements.
 
 Active Projects
 
-1.  Signet Dashboard Bug Fixes
-       Location: `dashboard/` (specifically `SettingsTab.svelte`)
-       Status: Completed & Verified.
-       Details: Resolved a `effect_update_depth_exceeded` error that prevented collapsible sections in the Settings tab from being clickable. The issue was a reactivity cycle where the `$effect` tracked a reactive `configFiles` prop while mutating `$state`.
-       Next Steps: Ensure the fix is stable in production and monitor for similar reactivity issues in other Svelte components.
+1. Signet Debug Harness (HIGH PRIORITY - LONG-TERM)
+Status: Planning/Initial Scaffolding
+Location: New package to be added to workspace
+Context: Building a conversational interface for full pipeline observability, inspired by pi-mono's event-driven architecture and extension system
+Next Steps: 
+- Scaffolding `packages/harness` package
+- Building daemon client for real-time pipeline monitoring
+- Implementing observer mode and multiple visualization modes (inline, hidden, split pane)
+- Integrating pi-mono's extension API to mirror Signet's hook types
 
-2.  Skills Module Implementation
-       Location: `packages/daemon/src/routes/skills.ts`
-       Status: Designed/Architected.
-       Details: Developing RESTful HTTP API endpoints for skills management (GET, POST, DELETE). The module is designed with zero database dependencies.
-       Next Steps: Implement the route handlers and test daemon change verification steps.
+2. Incremental Embedding Refresh Tracker (HIGH PRIORITY - ACTIVE)
+Status: Implementation in progress
+Location: `packages/core/src/embedding-tracker.ts`, daemon integration in `packages/daemon/src/daemon.ts`
+Context: Background polling loop to detect stale/missing embeddings when content changes or model switches
+Next Steps: 
+- Add `PipelineEmbeddingTrackerConfig` type and parsing
+- Create tracker module with `trackStaleEmbeddings()` function
+- Wire tracker into daemon lifecycle (start after pipeline, stop in cleanup)
+- Enhance `/status` endpoint to report tracker status
 
-3.  Pipeline Configuration Integration
-       Location: `isaac-research-chain` (destination repo)
-       Status: Pending.
-       Details: Integrating the PipelineV2Config, which currently uses a flat structure, into the target repository.
-       Next Steps: Execute the merge process for the PipelineV2Config.
+3. Pre-Compaction Capture (MEDIUM PRIORITY - ACTIVE)
+Status: Phase 2 in progress
+Location: `packages/session-manager/src/continuity-state.ts`, `packages/session-manager/src/session-checkpoints.ts`
+Context: Improving conversation continuity at compaction boundaries
+Next Steps: 
+- Add `pendingPromptSnippets` to continuity state
+- Modify `recordPrompt()` to store keyword-only snippets
+- Update `consumeState()` to snapshot and reset snippets
+- Add session_end checkpoint trigger and digest formatters with snippet enrichment
 
 Recent Work
 
-   Svelte 5 Reactivity Fix: Identified and patched the reactivity bug in `SettingsTab.svelte`. The fix involves wrapping the `st.init(configFiles)` call in `untrack()` to prevent the `$effect` from creating a dependency loop that triggered infinite re-initialization of the store. The dashboard and daemon have been restarted and verified to be functioning correctly with Playwright.
-   Daemon Management: Performed daemon restarts and testing to validate the Settings tab fix. Ensured `bun.lockb` and environment variables are correctly configured.
-   Agent Configuration: Clarified the path for agent profile storage (`~/.agents/`) to ensure correct file system interactions.
+- Debug Harness Planning: User requested observability tool; began architectural planning inspired by pi-mono's approach
+- Embedding Tracker: Added types to `core/src/types.ts`, created `embedding-tracker.ts` module, updated `memory-config.ts` for tracker configuration, started daemon wiring
+- Pre-Compaction: Enhanced checkpoint system to capture periodic snapshots; now adding passive capture for compaction boundaries with prompt snippet enrichment
 
 Technical Notes
 
-   Stack: The application frontend uses Astro for the static site shell and React components for dynamic UI elements. The core daemon logic is written in TypeScript.
-   Deployment Target: The static site builds are intended for deployment at `https://www.signetai.sh`.
-   Agent Config Path: All agent profile configurations must be stored in `~/.agents/` and sourced from `AGENTS.md`, not `~/.signet/`.
-   UI Development Constraints: Strict rule—UI work (Svelte, React, Tailwind) must not be delegated to subagents. Must be handled directly by the assistant with direct reference passing.
+- Project Structure: Monorepo with `packages/core`, `packages/daemon`, `packages/session-manager`
+- Key Files Modified: `daemon.ts`, `memory-config.ts`, `continuity-state.ts`, `session-checkpoints.ts`
+- Integration Points: Pipeline hooks, connector pre-compaction, status endpoints
+- Models: Using local LLMs; embedding health critical for RAG performance
+- Configuration: Pipeline configuration loaded via `loadPipelineConfig()` with support for multiple feature flags
 
 Rules & Warnings
 
-   UI Work Prohibition: Never delegate UI debugging or coding tasks (Svelte/React components, styling) to subagents. The assistant must execute these tasks manually.
-   Daemon Verification: Before deploying daemon changes, ensure full verification is performed to prevent runtime errors.
-   Path Correctness: Always use `~/.agents/` for agent configuration files.
-- Svelte 5 Awareness: Be vigilant about Svelte 5 reactivity features. Issues often arise when `$state` is initialized inside closures or when `$effect` creates circular dependencies with reactive props. Use `untrack()` when necessary to break those loops.
+- Pipeline Hooks: Claude Code connector lacks PreCompaction hook; this is the #1 continuity pain point
+- Checkpoint Data: Phase 1 used keyword-only data; Phase 2 adds prompt snippets for richer context
+- File Paths: Working in `packages/` directory structure; verify exports from core barrel before use
+- Daemon Lifecycle: Trackers must start after pipeline init and stop in cleanup
+- Checkpoint Triggers: Ensure `"session_end"` added to CheckpointTrigger union for proper formatting

memory/debug.log

@@ -740,3 +740,5 @@ We are synthesizing a WORKING MEMORY document for Nicholai. Focus is on CURRENT
 2026-03-01T04:08:06.703539 [regenerate] starting regeneration
 2026-03-01T04:08:10.297136 [regenerate] found 654 transcripts, 50 memories
 2026-03-01T04:08:10.297250 [regenerate] trying model: glm-4.7-flash
+2026-03-01T04:08:30.861258 [regenerate] success with glm-4.7-flash (3308 chars)
+2026-03-01T04:08:30.861479 [regenerate] wrote 3345 chars to MEMORY.md