clawdbot-workspace/reonomy-api/ARCHITECTURE.md

Reonomy Scraper Platform — Architecture

Overview

Full-stack Reonomy property data extraction platform with:

• REST API server (Express.js)
• MCP Server (TypeScript, stdio transport)
• MCP App(s) with React UI
• Integration with LocalBosses web app

Components

1. API Server (/api)

Express.js REST API that orchestrates scraping.

Endpoints:

• POST /api/scrape — Start a scrape job with search + output config
• GET /api/scrape/:jobId — Check job status
• GET /api/scrape/:jobId/results — Get results (JSON)
• GET /api/filters — List all available Reonomy filter options
• GET /api/exports/:jobId?format=csv|json — Export results

Auth: API key header (X-API-Key)

2. Scraper Engine (/engine)

Core scraping logic (evolved from v13). Modular extraction.

Modules:

• auth.js — Login, session management, state save/load
• search-builder.js — Translates filter config → Reonomy UI actions
• extractor.js — Modular tab extraction (only grabs what user requested)
• anti-detection.js — Random delays, humanization, daily limits
• queue.js — Job queue with rate limiting

Extraction modules (per tab):

• extract-building.js — Building & Lot data
• extract-owner.js — Owner names, phones, emails (CURRENT v13 logic)
• extract-sales.js — Sale history
• extract-debt.js — Mortgage/lender info
• extract-tax.js — Tax assessed values

3. MCP Server (/mcp-server)

TypeScript MCP server exposing Reonomy tools.

Tools:

• reonomy_search — Configure search filters, returns search ID + count
• reonomy_scrape — Start extraction from a search (with output config)
• reonomy_get_results — Fetch results for a job
• reonomy_get_filters — List all available filter options
• reonomy_export — Export results as CSV/JSON

Resources:

• reonomy://app/search — Search configuration UI
• reonomy://app/results — Results viewer UI
• reonomy://app/dashboard — Dashboard with stats

4. MCP App(s) (/mcp-app)

React + Vite, bundled to single HTML files per app.

Apps:

• Search Builder App — Visual filter configuration

  • Location picker, property type checkboxes
  • Owner filters (phone/email toggles)
  • Building & Lot ranges
  • Output field selection
  • "Start Scrape" button

• Results Viewer App — Table/card view of scraped leads

  • Sortable/filterable data table
  • Expandable owner cards with contact info
  • Export to CSV button
  • Job status indicator

• Dashboard App — Scrape stats overview

  • Total leads, daily usage, job history
  • Properties by type chart

5. LocalBosses Integration

• Add Reonomy channel to toolbar
• Wire MCP apps into iframe system
• API endpoints accessible from app

Tech Stack

• Runtime: Node.js 22
• API: Express.js
• MCP Server: @modelcontextprotocol/sdk (TypeScript)
• MCP App: React 18 + Vite + Tailwind
• Browser Automation: agent-browser CLI
• Queue: In-memory (Bull optional for production)
• Storage: SQLite (better-sqlite3) for results + job tracking
• Rate Limiting: Built-in daily caps + per-request delays

File Structure

reonomy-api/
├── ARCHITECTURE.md
├── package.json
├── tsconfig.json
├── src/
│   ├── server.ts           # Express API entry
│   ├── routes/
│   │   ├── scrape.ts       # /api/scrape endpoints
│   │   ├── filters.ts      # /api/filters
│   │   └── exports.ts      # /api/exports
│   ├── engine/
│   │   ├── auth.ts         # Reonomy auth
│   │   ├── search-builder.ts
│   │   ├── extractor.ts    # Orchestrates tab extraction
│   │   ├── anti-detection.ts
│   │   ├── queue.ts
│   │   └── extractors/
│   │       ├── building.ts
│   │       ├── owner.ts
│   │       ├── sales.ts
│   │       ├── debt.ts
│   │       └── tax.ts
│   ├── mcp/
│   │   ├── server.ts       # MCP server entry
│   │   ├── tools.ts        # Tool definitions
│   │   └── resources.ts    # Resource definitions
│   ├── db/
│   │   ├── schema.ts       # SQLite schema
│   │   └── queries.ts      # DB operations
│   └── types.ts            # Shared types
├── mcp-app/
│   ├── package.json
│   ├── vite.config.ts
│   ├── src/
│   │   ├── apps/
│   │   │   ├── SearchBuilder.tsx
│   │   │   ├── ResultsViewer.tsx
│   │   │   └── Dashboard.tsx
│   │   ├── components/
│   │   │   ├── FilterPanel.tsx
│   │   │   ├── PropertyCard.tsx
│   │   │   ├── DataTable.tsx
│   │   │   └── ExportButton.tsx
│   │   └── lib/
│   │       └── mcp-app-sdk.ts
│   └── dist/               # Built HTML files
└── data/
    └── reonomy.db          # SQLite database

Search Config Schema

interface SearchConfig {
  location: string;  // "Miami-Dade, FL"
  propertyTypes?: string[];  // ["Multifamily", "Multi Family (General)"]
  building?: {
    yearBuiltFrom?: number;
    yearBuiltUntil?: number;
    yearRenovatedFrom?: number;
    yearRenovatedUntil?: number;
    zoning?: string;
    lotSizeSfMin?: number;
    lotSizeSfMax?: number;
    lotSizeAcresMin?: number;
    lotSizeAcresMax?: number;
    opportunityZone?: boolean;
    totalUnitsMin?: number;
    totalUnitsMax?: number;
    buildingAreaMin?: number;
    buildingAreaMax?: number;
  };
  owner?: {
    nameOrCompany?: string;
    ownerType?: "Company" | "Person";
    includesPhone?: boolean;
    includesEmail?: boolean;
    includesMailingAddress?: boolean;
    portfolioMin?: number;
    portfolioMax?: number;
    ownerOccupied?: boolean;
    inStateOwner?: boolean;
    portfolioValueMin?: number;
    portfolioValueMax?: number;
    reportedOwner?: string;
    mailingAddress?: string;
  };
  occupants?: {
    name?: string;
    naicsSic?: string;
    website?: string;
  };
  sales?: {
    dateRange?: string;
    multiParcel?: boolean;
    priceMin?: number;
    priceMax?: number;
    pricePerSfMin?: number;
    pricePerSfMax?: number;
    likelyToSell?: boolean;
  };
  debt?: {
    amountMin?: number;
    amountMax?: number;
    originationFrom?: string;
    originationUntil?: string;
    maturityFrom?: string;
    maturityUntil?: string;
    lenderName?: string;
    cmbsLoan?: boolean;
  };
  distressed?: {
    auctionDateFrom?: string;
    auctionDateUntil?: string;
    preForeclosureCategory?: string;
    cmbsWatchlist?: boolean;
  };
}

interface OutputConfig {
  propertyInfo?: ("address" | "type" | "units" | "sqft" | "yearBuilt" | "lotSize" | "zoning" | "opportunityZone" | "apn" | "legal")[];
  ownerInfo?: ("name" | "company" | "portfolioSize" | "portfolioValue" | "ownerType")[];
  contactInfo?: ("phones" | "emails")[];
  salesInfo?: ("lastSaleDate" | "lastSalePrice" | "buyer" | "seller" | "deedType")[];
  debtInfo?: ("lender" | "loanType" | "mortgageAmount" | "maturityDate" | "interestType")[];
  taxInfo?: ("assessedValue" | "taxAmount")[];
}