Let Claude Use Your Computer: CLI Computer Use on macOS
Traditional AI development tools operate strictly inside text terminals or DOM trees. However, modern software engineering workflows extend across GUI applications: testing desktop apps in native windows, interacting with database GUIs (Postico, TablePlus), configuring third-party SaaS portals, or inspecting iOS Simulators. Anthropic's Computer Use API enables Claude Code to take desktop screenshots, calculate pixel coordinate targets, and synthesize native mouse and keyboard events on macOS.
In this article, we explore how to configure and control CLI Computer Use on macOS. We trace a hands-on scenario — deploying Claude Code to automate Xcode build verification and inspect a local desktop application UI — while setting strict screen boundary permissions. For related CLI features and agent architecture, see How Claude Code Works and Claude Code CLI Reference.
Computer Use Architecture on macOS
Computer Use operates as an iterative See-Plan-Act-Observe loop. Rather than interacting with internal application APIs, the agent interacts directly with the operating system display plane.
1. See: The harness takes a display screenshot via macOS CoreGraphics APIs.
2. Plan: Claude analyzes the screenshot image payload, identifies target GUI elements, and calculates target (x, y) display coordinates.
3. Act: The harness dispatches OS-level mouse movement, click, drag, or keystroke events.
4. Observe: The agent captures a post-action screenshot to confirm UI state transition.
| Layer | API Integration | DOM Browser Automation | GUI Computer Use |
|---|---|---|---|
| Target Surface | HTTP REST / gRPC endpoints | HTML DOM elements | OS Screen Pixels & Mouse/Keyboard |
| App Compatibility | Requires exposed backend API | Web browsers only | Any native desktop application |
| Setup Overhead | High (API keys, OpenAPI specs) | Medium (CSS selectors) | Zero (Operates on visual display) |
| Brittle Threshold | Low (Stable schema contracts) | Medium (DOM shifts) | Medium (Requires visual re-grounding) |
Quick reference
- CoreGraphics APIs capture display frames without needing third-party screen recording tools.
- Target coordinates are calculated dynamically based on relative display dimensions.
- Visual feedback loops ensure that misclicks are detected and retried automatically.
Remember this
Computer Use allows Claude Code to automate any native desktop application by operating directly on OS display pixels.
Configuring macOS System Preferences & Accessibility Permissions
On macOS, executing OS-level mouse clicks and keyboard events requires explicit system permission. Before launching Computer Use in Claude Code, grant Accessibility and Screen Recording permissions to your terminal application (Terminal, iTerm2, or VS Code integrated terminal).
Navigate to System Settings → Privacy & Security: - Accessibility: Enable access for your terminal app to synthesize input events. - Screen Recording: Enable access for display frame capture.
Quick reference
- Without Accessibility permissions, synthetic mouse clicks will be silently dropped by macOS Security.
- Screen Recording permissions are required for the harness to capture desktop display frames.
- Permissions can be revoked at any time via macOS System Settings.
Remember this
Properly configuring macOS System Settings ensures safe, authorized terminal control over desktop inputs.
Screen Boundaries & User Intercept Safety
Allowing an AI model to control mouse and keyboard inputs requires clear security boundaries. Claude Code enforces interactive confirmation prompts before executing Computer Use actions outside pre-authorized application windows.
Developers can specify allowed coordinate boundaries or application window handles in .claude/settings.json to prevent accidental clicks on system settings or external browser windows.
Quick reference
- Bounding boxes restrict mouse clicks to designated desktop screen regions.
- Emergency stop key combinations (
Ctrl+Cor mouse corner hotkeys) halt computer use instantly. - All visual action histories are saved to local session transcripts for auditing.
Remember this
Bounding boxes and emergency stop hotkeys keep desktop automation safe and predictable.
Practical Enterprise Use Cases for Computer Use
Computer Use unlocks powerful cross-application workflows that were previously impossible to automate:
1. Native App Testing: Verify Xcode simulator builds, Electron apps, or desktop GUIs hands-free. 2. Legacy Software Integration: Interface with legacy desktop applications lacking API endpoints. 3. SaaS Portal Setup: Automate multi-step admin configuration forms across web and desktop tools. 4. Visual Bug Verification: Inspect layout glitches across native desktop screen resolutions.
Quick reference
- Automating native app testing eliminates manual QA click-through routines.
- Visual bug verification catches rendering issues across different display scalings.
- Legacy desktop integration bridges modern AI agents with old enterprise tools.
Remember this
Computer Use bridges the gap between text-based AI agents and native desktop GUI applications.
Hands-on Practice: Automate a Desktop Calculator App
Practice native Computer Use automation with this guided exercise:
1. Grant Permissions: Ensure your terminal app has Accessibility and Screen Recording permissions enabled in macOS System Settings.
2. Open Application: Launch the native macOS Calculator.app on your desktop.
3. Prompt Agent: Start Claude Code and issue: "Use computer control to click 45 + 55 = on the Calculator app and tell me the result".
4. Observe Execution: Watch the agent take a screenshot, calculate coordinate targets, click the Calculator buttons, and report the answer 100.
Quick reference
- Verify that the terminal app displays mouse movement across the Calculator window.
- Confirm that post-action screenshots verify the display output shows 100.
- Ensure the session completes cleanly without taking actions outside the Calculator window.
Remember this
Automating a desktop calculator proves how visual grounding enables efficient native GUI control.
Key takeaway
CLI Computer Use on macOS transforms Claude Code into a complete, cross-application engineering assistant. By Grounding LLM reasoning directly in OS screen pixels and synthesized input events, software teams can automate desktop application testing, legacy software integration, and complex multi-tool workflows while enforcing strict security boundaries.
Practice (20 min): Open Calculator.app on macOS. Ensure Accessibility permissions are granted, then instruct Claude Code to visually click buttons to solve an equation. Verify that visual grounding and click synthesis complete accurately.
Related Articles
Explore this topic