Computer-Use Agent Architectures

Topics We'll Cover

• The Computer-Use Idea
• Anthropic Computer Use — Screen + Keyboard + Mouse
• OpenAI Operator
• Gemini Mariner
• Action Schemas — Click x,y vs Accessibility-Tree vs CSS Selectors
• Screenshots vs DOM vs Accessibility Tree
• Containers & VMs for Sandboxed Computer-Use
• Eval Benchmarks — OSWorld, WebVoyager, AgentBench
• What to Take Away

The Computer-Use Idea

Traditional tool-calling hands the model a curated JSON schema for each capability. Computer-use takes a different bet: give the model a pixel-level view of any GUI and the ability to click, type, and scroll — then it can operate any application without bespoke integration work.

The premise is compelling but deceptively hard. A naive screenshot-and-click loop needs to solve: grounding (mapping natural-language intent to exact screen coordinates), sequencing (multi-step tasks with intermediate state), error recovery (unexpected dialogs, slow loads, stale screenshots), and safety (preventing the model from being hijacked by adversarial page content).

Anthropic Computer Use — Screen + Keyboard + Mouse

Anthropic released computer-use in public beta on 22 October 2024 alongside Claude 3.5 Sonnet (claude-3-5-sonnet-20241022). The API surface is a single built-in tool block with "type": "computer_20241022". A revised version computer_20250124 shipped with Claude 3.5 Sonnet v2 and Claude 3.7 Sonnet in early 2025.

import anthropic

client = anthropic.Anthropic()
response = client.beta.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    tools=[{
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px":  1280,
        "display_height_px": 800,
        "display_number":    1,       # X11 DISPLAY index
    }],
    messages=[{"role": "user", "content": "Open Firefox and navigate to news.ycombinator.com"}],
    betas=["computer-use-2024-10-22"],
)

The 11 action types

OpenAI Operator

OpenAI launched Operator in January 2025, powered by the Computer-Using Agent (CUA) model — a variant of GPT-4o fine-tuned specifically for GUI grounding. Unlike Anthropic's raw-pixel approach, Operator wraps a managed browser environment with an agentic scaffolding layer that handles retries, auth flows, and confirmation prompts semi-autonomously.

from openai import OpenAI
import base64, PIL.ImageGrab

client = OpenAI()

# Initial request — agent returns an action block, not a final answer
resp = client.responses.create(
    model="computer-use-preview",
    tools=[{"type": "computer_use_preview",
             "display_width": 1280, "display_height": 800,
             "environment": "browser"}],
    input=[{"role": "user",
            "content": "Go to hacker news and tell me the top story"}],
    truncation="auto",
)

# Subsequent turns: inject screenshot as computer_call_output
while resp.output[-1].type == "computer_call":
    action = resp.output[-1].action
    screenshot_b64 = take_screenshot()   # your capture fn
    apply_action(action)                 # your actuator fn
    resp = client.responses.create(
        model="computer-use-preview",
        previous_response_id=resp.id,
        tools=[{"type": "computer_use_preview",
                 "display_width": 1280, "display_height": 800}],
        input=[{"type": "computer_call_output",
                "call_id": resp.output[-1].call_id,
                "output": {"type": "image_url",
                             "image_url": f"data:image/png;base64,{screenshot_b64}"}}],
        truncation="auto",
    )

Gemini Mariner

Mariner is Google DeepMind's computer-use research project, first demonstrated in December 2024 alongside Gemini 2.0 Flash. It targets browser-only automation rather than full-desktop control, leaning on Gemini's native multimodal context window (up to 1M tokens) to keep long navigation histories visible to the model.

Technical distinguishing features

Action Schemas — Click x,y vs Accessibility-Tree vs CSS Selectors

The choice of how the model identifies a UI target is the central engineering decision in any computer-use system. Three families dominate:

# computer_20250124 adds two new actions:

# 1. left_click now accepts an optional "element" descriptor
{
  "type": "left_click",
  "coordinate": [452, 213],       # still required as anchor
  "element_description": "Submit button in the checkout form"
}

# 2. New first-class action: get_accessibility_tree
# Returns a partial AXTree around a region — model can use it to
# plan clicks without a full screenshot roundtrip.
{
  "type": "get_accessibility_tree"
}

Screenshots vs DOM vs Accessibility Tree

The perception layer — what the model actually sees — matters as much as the action schema. The three main channels have different fidelity/cost trade-offs:

Containers & VMs for Sandboxed Computer-Use

A computer-use agent has broad write access: it can install software, delete files, exfiltrate data, and be manipulated via adversarial web content. Sandboxing is not optional. The two main isolation strategies are Docker containers with a virtual X server, and full VM snapshots.

# Pull and run the reference sandbox (ARM or x86_64)
docker pull ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest

docker run \
  -e  ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
  -v  "$HOME/.anthropic:/home/computeruse/.anthropic" \
  -p  5900:5900  \   # VNC
  -p  8501:8501  \   # Streamlit control UI
  -p  6080:6080  \   # noVNC web client
  -p  8080:8080  \   # API passthrough
  --rm \
  ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest

Eval Benchmarks — OSWorld, WebVoyager, AgentBench

Computer-use agents need evals that actually execute tasks in live environments — a screenshot match or perplexity score is useless here. Three benchmarks dominate the field:

OSWorld task distribution

What to Take Away

• Computer-use agents trade integration effort for fragility. They work on any GUI without a bespoke API, but pixel co-ordinates shift with layout changes and screenshot-driven loops are 10× slower than API calls.
• Three competing implementations, three different bets. Anthropic bets on raw pixels + optional a11y tree (computer_20250124); OpenAI bets on a managed browser + server-side threading (Responses API); Google bets on long-context history + Chrome extension access (Mariner).
• Action schema matters as much as the model. Set-of-marks prompting and AXTree-based grounding push accuracy up 20–30% over raw pixel-co-ordinate clicking.
• Screenshot ≠ the only perception channel. AXTree gives you semantics at ~230 tokens vs ~765 for a screenshot. Hybrid (AXTree + screenshot fallback) is the practical optimum for web tasks.
• Sandboxing is mandatory, not optional. Prompt injection via malicious web content is a real, demonstrated attack. Run every agent in an ephemeral container or VM snapshot; restrict egress; add human confirmation gates for destructive actions.
• OSWorld is the honest eval. Live VM + functional checks; scores in the 22–50% range as of early 2025. Human performance is ~72%. There is a large gap to close.