Nicholai 8b34becbeb
feat(agent): AI agent harness with memory, GitHub, audio & feedback (#37)
* feat(agent): replace ElizaOS with AI SDK v6 harness

Replace custom ElizaOS sidecar proxy with Vercel AI SDK v6 +
OpenRouter provider for a proper agentic harness with multi-step
tool loops, streaming, and D1 conversation persistence.

- Add AI SDK agent library (provider, tools, system prompt, catalog)
- Rewrite API route to use streamText with 10-step tool loop
- Add server actions for conversation save/load/delete
- Migrate chat-panel and dashboard-chat to useChat hook
- Add action handler dispatch for navigate/toast/render tools
- Use qwen/qwen3-coder-next via OpenRouter (fallbacks disabled)
- Delete src/lib/eliza/ (replaced entirely)
- Exclude references/ from tsconfig build

* fix(chat): improve dashboard chat scroll and text size

- Rewrite auto-scroll: pin user message 75% out of
  frame after send, then follow bottom during streaming
- Use useEffect for scroll timing (DOM guaranteed ready)
  instead of rAF which fired before React commit
- Add user scroll detection to disengage auto-scroll
- Bump assistant text from 13px back to 14px (text-sm)
- Tighten prose spacing for headings and lists

* chore: installing new components

* refactor(chat): unify into one component, two presentations

Extract duplicated chat logic into shared ChatProvider context
and useCompassChat hook. Single ChatView component renders as
full-page hero on /dashboard or sidebar panel elsewhere. Chat
state persists across navigation.

New: chat-provider, chat-view, chat-panel-shell, use-compass-chat
Delete: agent-provider, chat-panel, dashboard-chat, 8 deprecated UI files
Fix: AI component import paths (~/  -> @/), shadcn component updates

* fix(lint): resolve eslint errors in AI components

- escape unescaped entities in demo JSX (actions, artifact,
  branch, reasoning, schema-display, task)
- add eslint-disable for @ts-nocheck in vendor components
  (file-tree, terminal, persona)
- remove unused imports in chat-view (ArrowUp, Square,
  useChatPanel)

* feat(agent): rename AI to Slab, add proactive help

rename assistant from Compass to Slab and add first
interaction guidance so it proactively offers
context-aware help based on the user's current page.

* fix(build): use HTML entity for strict string children

ReasoningContent expects children: string, so JSX
expression {"'"} splits into string[] causing type error.
Use ' HTML entity instead.

* feat(agent): add memory, github, audio, feedback

- persistent memory system (remember/recall across sessions)
- github integration (commits, PRs, issues, contributors)
- audio transcription via Whisper API
- UX feedback interview flow with auto-issue creation
- memories management table in settings
- audio waveform visualization component
- new schema tables: slab_memories, feedback_interviews
- enhanced system prompt with proactive tool usage

* feat(agent): unify chat into single morphing instance

Replaces two separate ChatView instances (page + panel) with
one layout-level component that transitions between full-page
and sidebar modes. Navigation now actually works via proper
AI SDK v6 part structure detection, with view transitions for
smooth crossfades, route validation to prevent 404s, and
auto-opening the panel when leaving dashboard.

Also fixes dark mode contrast, user bubble visibility, tool
display names, input focus ring, and system prompt accuracy.

* refactor(agent): rewrite waveform as time-series viz

Replace real-time frequency equalizer with amplitude
history that fills left-to-right as user speaks.
Bars auto-calculated from container width, with
non-linear boost and scroll when full.

* (feat): implemented architecture for plugins and skills, laying a foundation for future implementations of packages separate from the core application

* feat(agent): add skills.sh integration for slab

Skills client fetches SKILL.md from GitHub, parses
YAML frontmatter, and stores content in plugin DB.
Registry injects skill content into system prompt.
Agent tools and settings UI for skill management.

* feat(agent): add interactive UI action bridge

Wire agent-generated UIs to real server actions via
an action bridge API route. Forms submit, checkboxes
persist, and DataTable rows support CRUD operations.

- action-registry.ts: maps 19 dotted action names to
  server actions with zod validation + permissions
- /api/agent/action: POST route with auth, permission
  checks, schema validation, and action execution
- schema-agent.ts: agent_items table for user-scoped
  todos, notes, and checklists
- agent-items.ts: CRUD + toggle actions for agent items
- form-context.ts: FormIdProvider for input namespacing
- catalog.ts: Form component, value/onChangeAction props,
  DataTable rowActions, mutate/confirmDelete actions
- registry.tsx: useDataBinding on all form inputs, Form
  component, DataTable row action buttons, inline
  Checkbox/Switch mutations
- actions.ts: mutate + confirmDelete handlers that call
  the action bridge, formSubmit now collects + submits
- system-prompt.ts: interactive UI patterns section
- render/route.ts: interactive pattern custom rules

* docs: reorganize into topic subdirectories

Move docs into auth/, chat/, openclaw-principles/,
and ui/ subdirectories. Add openclaw architecture
and system prompt documentation.

* feat(agent): add commit diff support to github tools

Add fetchCommitDiff to github client with raw diff
fallback for missing patches. Wire commit_diff query
type into agent github tools.

* fix(ci): guard wrangler proxy init for dev only

initOpenNextCloudflareForDev() was running unconditionally
in next.config.ts, causing CI build and lint to fail with
"You must be logged in to use wrangler dev in remote mode".
Only init the proxy when NODE_ENV is development.

---------

Co-authored-by: Nicholai <nicholaivogelfilms@gmail.com>
2026-02-06 17:04:04 -07:00

426 lines
11 KiB
TypeScript
Executable File

import { tool } from "ai"
import { z } from "zod/v4"
import { getCloudflareContext } from "@opennextjs/cloudflare"
import { getDb } from "@/db"
import { getCurrentUser } from "@/lib/auth"
import { saveMemory, searchMemories } from "@/lib/agent/memory"
import {
installSkill as installSkillAction,
uninstallSkill as uninstallSkillAction,
toggleSkill as toggleSkillAction,
getInstalledSkills as getInstalledSkillsAction,
} from "@/app/actions/plugins"
const queryDataInputSchema = z.object({
queryType: z.enum([
"customers",
"vendors",
"projects",
"invoices",
"vendor_bills",
"schedule_tasks",
"project_detail",
"customer_detail",
"vendor_detail",
]),
id: z
.string()
.optional()
.describe("Record ID for detail queries"),
search: z
.string()
.optional()
.describe("Search term to filter results"),
limit: z
.number()
.optional()
.describe("Max results to return (default 20)"),
})
type QueryDataInput = z.infer<typeof queryDataInputSchema>
const VALID_ROUTES: ReadonlyArray<RegExp> = [
/^\/dashboard$/,
/^\/dashboard\/customers$/,
/^\/dashboard\/vendors$/,
/^\/dashboard\/projects$/,
/^\/dashboard\/projects\/[^/]+$/,
/^\/dashboard\/projects\/[^/]+\/schedule$/,
/^\/dashboard\/financials$/,
/^\/dashboard\/people$/,
/^\/dashboard\/files$/,
/^\/dashboard\/files\/.+$/,
]
function isValidRoute(path: string): boolean {
return VALID_ROUTES.some((r) => r.test(path))
}
const navigateInputSchema = z.object({
path: z
.string()
.describe("The URL path to navigate to"),
reason: z
.string()
.optional()
.describe("Brief explanation of why navigating"),
})
type NavigateInput = z.infer<typeof navigateInputSchema>
const notificationInputSchema = z.object({
message: z.string().describe("The notification message"),
type: z
.enum(["default", "success", "error"])
.optional()
.describe("Notification style"),
})
type NotificationInput = z.infer<
typeof notificationInputSchema
>
const generateUIInputSchema = z.object({
description: z.string().describe(
"Layout and content description for the " +
"dashboard to generate. Be specific about " +
"what components and data to display."
),
dataContext: z
.record(z.string(), z.unknown())
.optional()
.describe(
"Data to include in the rendered UI. " +
"Pass query results here."
),
})
type GenerateUIInput = z.infer<
typeof generateUIInputSchema
>
const rememberInputSchema = z.object({
content: z.string().describe(
"What to remember (a preference, decision, fact, or workflow)"
),
memoryType: z.enum([
"preference",
"workflow",
"fact",
"decision",
]).describe("Category of memory"),
tags: z
.string()
.optional()
.describe("Comma-separated tags for categorization"),
importance: z
.number()
.min(0.3)
.max(1.0)
.optional()
.describe("Importance weight 0.3-1.0 (default 0.7)"),
})
type RememberInput = z.infer<typeof rememberInputSchema>
const recallInputSchema = z.object({
query: z
.string()
.describe("What to search for in memories"),
limit: z
.number()
.optional()
.describe("Max results (default 5)"),
})
type RecallInput = z.infer<typeof recallInputSchema>
async function executeQueryData(input: QueryDataInput) {
const { env } = await getCloudflareContext()
const db = getDb(env.DB)
const cap = input.limit ?? 20
switch (input.queryType) {
case "customers": {
const rows = await db.query.customers.findMany({
limit: cap,
...(input.search
? {
where: (c, { like }) =>
like(c.name, `%${input.search}%`),
}
: {}),
})
return { data: rows, count: rows.length }
}
case "vendors": {
const rows = await db.query.vendors.findMany({
limit: cap,
...(input.search
? {
where: (v, { like }) =>
like(v.name, `%${input.search}%`),
}
: {}),
})
return { data: rows, count: rows.length }
}
case "projects": {
const rows = await db.query.projects.findMany({
limit: cap,
...(input.search
? {
where: (p, { like }) =>
like(p.name, `%${input.search}%`),
}
: {}),
})
return { data: rows, count: rows.length }
}
case "invoices": {
const rows = await db.query.invoices.findMany({
limit: cap,
})
return { data: rows, count: rows.length }
}
case "vendor_bills": {
const rows = await db.query.vendorBills.findMany({
limit: cap,
})
return { data: rows, count: rows.length }
}
case "schedule_tasks": {
const rows = await db.query.scheduleTasks.findMany({
limit: cap,
...(input.search
? {
where: (t, { like }) =>
like(t.title, `%${input.search}%`),
}
: {}),
})
return { data: rows, count: rows.length }
}
case "project_detail": {
if (!input.id) {
return { error: "id required for detail query" }
}
const row = await db.query.projects.findFirst({
where: (p, { eq }) => eq(p.id, input.id!),
})
return row ? { data: row } : { error: "not found" }
}
case "customer_detail": {
if (!input.id) {
return { error: "id required for detail query" }
}
const row = await db.query.customers.findFirst({
where: (c, { eq }) => eq(c.id, input.id!),
})
return row ? { data: row } : { error: "not found" }
}
case "vendor_detail": {
if (!input.id) {
return { error: "id required for detail query" }
}
const row = await db.query.vendors.findFirst({
where: (v, { eq }) => eq(v.id, input.id!),
})
return row ? { data: row } : { error: "not found" }
}
default:
return { error: "unknown query type" }
}
}
export const agentTools = {
queryData: tool({
description:
"Query the application database. Describe what data " +
"you need in natural language and provide a query type.",
inputSchema: queryDataInputSchema,
execute: async (input: QueryDataInput) =>
executeQueryData(input),
}),
navigateTo: tool({
description:
"Navigate the user to a page in the application. " +
"Returns navigation data for the client to execute.",
inputSchema: navigateInputSchema,
execute: async (input: NavigateInput) => {
if (!isValidRoute(input.path)) {
return {
error:
`"${input.path}" is not a valid page. ` +
"Valid: /dashboard, /dashboard/projects, " +
"/dashboard/projects/{id}, " +
"/dashboard/projects/{id}/schedule, " +
"/dashboard/customers, /dashboard/vendors, " +
"/dashboard/financials, /dashboard/people, " +
"/dashboard/files",
}
}
return {
action: "navigate" as const,
path: input.path,
reason: input.reason ?? null,
}
},
}),
showNotification: tool({
description:
"Show a toast notification to the user. Use for " +
"confirmations or important alerts.",
inputSchema: notificationInputSchema,
execute: async (input: NotificationInput) => ({
action: "toast" as const,
message: input.message,
type: input.type ?? "default",
}),
}),
generateUI: tool({
description:
"Generate a rich interactive UI dashboard. " +
"Use when the user wants to see structured " +
"data (tables, charts, stats, forms). Always " +
"fetch data with queryData first, then pass " +
"it here as dataContext.",
inputSchema: generateUIInputSchema,
execute: async (input: GenerateUIInput) => ({
action: "generateUI" as const,
renderPrompt: input.description,
dataContext: input.dataContext ?? {},
}),
}),
rememberContext: tool({
description:
"Save something to persistent memory. Use when the " +
"user shares a preference, makes a decision, or " +
"mentions a fact worth remembering across sessions.",
inputSchema: rememberInputSchema,
execute: async (input: RememberInput) => {
const { env } = await getCloudflareContext()
const db = getDb(env.DB)
const user = await getCurrentUser()
if (!user) return { error: "not authenticated" }
const id = await saveMemory(
db,
user.id,
input.content,
input.memoryType,
input.tags,
input.importance,
)
return {
action: "memory_saved" as const,
id,
content: input.content,
memoryType: input.memoryType,
}
},
}),
recallMemory: tool({
description:
"Search persistent memories for this user. Use when " +
"the user asks if you remember something or when you " +
"need to look up a past preference or decision.",
inputSchema: recallInputSchema,
execute: async (input: RecallInput) => {
const { env } = await getCloudflareContext()
const db = getDb(env.DB)
const user = await getCurrentUser()
if (!user) return { error: "not authenticated" }
const results = await searchMemories(
db,
user.id,
input.query,
input.limit,
)
return {
action: "memory_recall" as const,
results,
count: results.length,
}
},
}),
installSkill: tool({
description:
"Install a skill from GitHub (skills.sh format). " +
"Source format: owner/repo or owner/repo/skill-name. " +
"Requires admin role. Always confirm with the user " +
"what skill they want before installing.",
inputSchema: z.object({
source: z.string().describe(
"GitHub source path, e.g. " +
"'cloudflare/skills/wrangler'",
),
}),
execute: async (input: { source: string }) => {
const user = await getCurrentUser()
if (!user || user.role !== "admin") {
return { error: "admin role required to install skills" }
}
return installSkillAction(input.source)
},
}),
listInstalledSkills: tool({
description:
"List all installed agent skills with their status.",
inputSchema: z.object({}),
execute: async () => getInstalledSkillsAction(),
}),
toggleInstalledSkill: tool({
description:
"Enable or disable an installed skill.",
inputSchema: z.object({
pluginId: z.string().describe("The plugin ID of the skill"),
enabled: z.boolean().describe("true to enable, false to disable"),
}),
execute: async (input: {
pluginId: string
enabled: boolean
}) => {
const user = await getCurrentUser()
if (!user || user.role !== "admin") {
return { error: "admin role required" }
}
return toggleSkillAction(input.pluginId, input.enabled)
},
}),
uninstallSkill: tool({
description:
"Remove an installed skill permanently. " +
"Requires admin role. Always confirm before uninstalling.",
inputSchema: z.object({
pluginId: z.string().describe("The plugin ID of the skill"),
}),
execute: async (input: { pluginId: string }) => {
const user = await getCurrentUser()
if (!user || user.role !== "admin") {
return { error: "admin role required" }
}
return uninstallSkillAction(input.pluginId)
},
}),
}