Computer-Use and Browser Agents: How Screen-Controlling AI Works

A computer-use or browser agent perceives the interface the way a human does: a rendered screenshot, often paired with the page's DOM tree or the OS/browser accessibility tree — the structured role-and-name tree screen readers already rely on. It acts through the same input surface a person has: mouse clicks, keystrokes, scrolling, drag gestures. Nothing about that action space is privileged. There is no bookFlight() function hiding underneath; there is only the page, exactly as rendered, exactly as fragile as it is for a human clicking too fast. This is what separates it from an agentic AI system built around typed tool calls: the tool is the screen.

Computer use exists because most software a person interacts with does not expose an API for what that person can do in the UI — legacy internal admin panels, one-off partner portals, and plenty of consumer sites with no booking, search, or checkout endpoint anyone will hand you a key for. Take the running example for this guide: task BOOK-SEA-AUS — search a one-way Seattle-to-Austin flight, select the cheapest nonstop, and stop before payment for a human to confirm. There is no airline API in scope, so every step — opening the site, filling the origin and destination fields, reading the results, selecting a fare — has to happen through the same rendered page a customer would use.

Follow BOOK-SEA-AUS through one iteration. The agent takes a screenshot of the airline's homepage. A vision-capable model reasons over that screenshot plus the task and history, and produces an intent in symbolic terms — something like { action: "click", target: "Search flights button" }. Notice what that intent is not: it is not yet a coordinate, and it is not yet anything the operating system can act on.

Grounding is the step that turns that symbolic intent into a real location. There are two common strategies. Coordinate-based grounding has the model predict raw x, y pixels directly from the screenshot; it works on anything with pixels, including canvas UIs and native apps with no DOM at all, but it is brittle to scroll position, zoom level, and viewport size. Structure-based grounding instead resolves the intent against the DOM or accessibility tree by role and name — for example, Playwright's page.getByRole("button", { name: "Search flights" }) — and returns a bounding box guaranteed to match a real, currently-rendered element, or fails cleanly if it does not exist.

Once grounded, an executor dispatches a real mouse-move-and-click or keystroke at that resolved location, and the loop re-observes: a fresh screenshot becomes the input to the next planning step. This loop is a close cousin of the general reason-then-act pattern used across tool-calling agents — the difference is that here the observation is a picture of a screen, and the action has to survive a grounding step before it can execute at all.

Every action in the perceive-plan-ground-act loop round-trips through rendering, screenshot capture, a full vision-model inference pass, and dispatch — typically multiple seconds per step. An API-calling agent pays only the API's response time, usually well under a second, and gets back a structured, typed result instead of a picture it has to reinterpret from scratch. That gap compounds: a ten-step browser task can easily cost an order of magnitude more wall-clock time and inference spend than the equivalent ten typed function calls.

The failure surface is different in kind, not just slower. Visual ambiguity — two near-identical buttons, a promotional "Search deals" banner sitting next to the real "Search flights" button — is a category of mistake a JSON schema simply cannot produce. Layout drift between the planning screenshot and the dispatched click (async content, an A/B test, a responsive reflow) is the single most common root cause of a misclick, covered in the failure story next. Interstitials — cookie-consent banners, login walls, CAPTCHAs — exist specifically to interrupt automated and human traffic alike, and a code or API agent never encounters any of them because it never renders a page.

None of this makes computer-use agents a lesser tool; it makes them a specific tool for a specific gap. The decision rule follows directly from the trade-off table below: reach for a computer-use agent only when no API exists for the task, and re-check that assumption before committing engineering effort to make the screen-driving path robust.

Mid-run on BOOK-SEA-AUS, the search-results page loads and the agent screenshots it: a 6:40am nonstop is visible with a Select flight button at a known pixel position. Roughly 400ms later — after that screenshot but before the click dispatches — the site's cookie-consent banner slides in and shifts the layout, pushing Select flight down by several dozen pixels. The click, aimed at the pre-banner coordinates, lands on the banner instead. Because the click missed the banner's actual Accept hit target by a few pixels too (icon padding is a classic offender), the banner never dismisses, the next screenshot looks almost identical to the last one, and the agent replans the same click at the same coordinates. The run is now stuck clicking a cookie banner in a loop that never advances and never raises an error.

Root mechanism: grounding was performed once, against a screenshot taken before the banner rendered, and then reused across a click dispatched after the page had already changed — exactly the stale-coordinate failure the previous section's code sample was built to prevent. The loop had no mechanism to notice it kept dispatching an identical action with zero effect on page state.

Diagnostic: replay the action log. The same (action, target) tuple appearing three or more times in a row, with an unchanged DOM state or screenshot hash between attempts, is the signature — that pattern means grounding is broken, not that the site needs one more retry.