52 lines
1.9 KiB
Markdown
52 lines
1.9 KiB
Markdown
# Peekaboo & macOS UI Automation Learnings
|
|
|
|
## Key Principles (from 2026-02-04 session)
|
|
|
|
### 1. Element-Based Clicking > Coordinate-Based
|
|
- **DON'T**: Calculate coordinates from screenshots and click blindly
|
|
- **DO**: Use `peekaboo click "Element Text"` or `peekaboo click --on elem_XX`
|
|
- **WHY**: Coordinates are unreliable across different window positions/scaling
|
|
|
|
### 2. Monitor Info
|
|
- Jake's setup: 45-inch monitor at 2560x1080
|
|
- Peekaboo uses logical coordinates (not physical pixels)
|
|
- No scaling math needed - click at the coordinates you see
|
|
|
|
### 3. 1Password CLI Authorization Dialogs
|
|
The "1Password Access Requested" dialog is a **special system dialog** that:
|
|
- Does NOT appear in peekaboo's element detection
|
|
- Cannot be clicked by text query ("Authorize")
|
|
- **SOLUTION**: Use keyboard input!
|
|
|
|
```bash
|
|
# Authorize 1Password CLI dialog with Enter key
|
|
osascript -e 'tell application "1Password" to activate'
|
|
sleep 0.3
|
|
osascript -e 'tell application "System Events" to key code 36' # Enter key
|
|
```
|
|
|
|
### 4. Workflow for 1Password CLI Setup
|
|
1. Enable "Integrate with 1Password CLI" in 1Password > Settings > Developer
|
|
2. Run `op signin` in tmux session
|
|
3. Wait ~2 seconds for dialog
|
|
4. Activate 1Password and press Enter to authorize
|
|
5. Verify with `op whoami`
|
|
|
|
### 5. Peekaboo Best Practices
|
|
- Always use `peekaboo see --annotate` first to get element IDs
|
|
- Click by element: `peekaboo click --on elem_XX` (most reliable)
|
|
- Click by text: `peekaboo click "Button Text"` (good fallback)
|
|
- Use `--app "AppName"` to target specific apps
|
|
- Use `--window-id XXXXX` for specific windows
|
|
|
|
### 6. System Dialogs & Sheets
|
|
macOS system-level dialogs (like 1Password authorization) may not expose
|
|
accessibility elements. Use keyboard shortcuts:
|
|
- Enter (key code 36) - confirm/default button
|
|
- Escape (key code 53) - cancel
|
|
- Tab (key code 48) - move focus
|
|
- Space (key code 49) - activate focused button
|
|
|
|
---
|
|
*Last updated: 2026-02-04 23:05 EST*
|