Peekaboo & macOS UI Automation Learnings

Key Principles (from 2026-02-04 session)

1. Element-Based Clicking > Coordinate-Based

DON'T: Calculate coordinates from screenshots and click blindly
DO: Use peekaboo click "Element Text" or peekaboo click --on elem_XX
WHY: Coordinates are unreliable across different window positions/scaling

2. Monitor Info

Jake's setup: 45-inch monitor at 2560x1080
Peekaboo uses logical coordinates (not physical pixels)
No scaling math needed - click at the coordinates you see

3. 1Password CLI Authorization Dialogs

The "1Password Access Requested" dialog is a special system dialog that:

Does NOT appear in peekaboo's element detection
Cannot be clicked by text query ("Authorize")
SOLUTION: Use keyboard input!

# Authorize 1Password CLI dialog with Enter key
osascript -e 'tell application "1Password" to activate'
sleep 0.3
osascript -e 'tell application "System Events" to key code 36'  # Enter key

4. Workflow for 1Password CLI Setup

Enable "Integrate with 1Password CLI" in 1Password > Settings > Developer
Run op signin in tmux session
Wait ~2 seconds for dialog
Activate 1Password and press Enter to authorize
Verify with op whoami

5. Peekaboo Best Practices

Always use peekaboo see --annotate first to get element IDs
Click by element: peekaboo click --on elem_XX (most reliable)
Click by text: peekaboo click "Button Text" (good fallback)
Use --app "AppName" to target specific apps
Use --window-id XXXXX for specific windows

6. System Dialogs & Sheets

macOS system-level dialogs (like 1Password authorization) may not expose accessibility elements. Use keyboard shortcuts:

Enter (key code 36) - confirm/default button
Escape (key code 53) - cancel
Tab (key code 48) - move focus
Space (key code 49) - activate focused button

Last updated: 2026-02-04 23:05 EST

1.9 KiB Raw Blame History