Browser Agent Patterns

Topics We'll Cover

• Browser-Use, Playwright Agents, Steel
• DOM Extraction Strategies — Accessibility Tree, Interactive-Element Detection
• Screen Capture vs DOM-Only — When Each Wins
• WebArena and Mind2Web Evals
• Captcha, Anti-Bot, Ethics
• Stateful Sessions — Cookies, Auth, Persistence
• Pricing — Minutes Per Task, Infrastructure
• Failure Modes & Recovery
• What to Take Away

Browser-Use, Playwright Agents, Steel

Three tiers of abstraction have emerged for browser-based LLM agents. They differ in where the DOM/screenshot extraction happens, which models they support, and how much infrastructure they manage for you.

DOM Extraction Strategies — Accessibility Tree, Interactive-Element Detection

The quality of the agent's world-model depends entirely on how well the page is reduced to a form the LLM can plan over. Four strategies are in common use, from least to most sophisticated:

Strategy 1 — Full inner-HTML dump

Naive. Grabs document.body.innerHTML or the Playwright content() call. Typical page: 200–800 KB raw, 30–150 KB stripped. Token count after stripping scripts and styles: 8 000–60 000. Almost always too large for a single context window. Only viable for very small pages or if you're summarising the page separately.

Strategy 2 — Interactive-element list

async def get_interactive_elements(page):
    return await page.evaluate("""() => {
        const selectors = [
          'a[href]', 'button', 'input', 'select', 'textarea',
          '[role=button]', '[role=link]', '[role=checkbox]',
          '[role=combobox]', '[tabindex]:not([tabindex="-1"])'
        ];
        return selectors
          .flatMap(s => [...document.querySelectorAll(s)])
          .filter(el => {
              const r = el.getBoundingClientRect();
              return r.width > 0 && r.height > 0;  // visible only
          })
          .slice(0, 150)  // hard cap
          .map((el, i) => ({
              index: i,
              tag: el.tagName.toLowerCase(),
              type: el.type || null,
              text: (el.innerText || el.value || el.placeholder || '').trim().slice(0, 120),
              href: el.href || null,
              rect: el.getBoundingClientRect()
          }));
    }""")

Strategy 3 — CDP Accessibility tree

cdp = await page.context.new_cdp_session(page)
tree = await cdp.send("Accessibility.getFullAXTree")
# tree['nodes'] — list of dicts with role, name, nodeId, backendDOMNodeId
interactive_roles = {
    "button", "link", "textbox", "combobox", "checkbox",
    "radio", "menuitem", "tab", "listbox", "searchbox"
}
filtered = [
    n for n in tree["nodes"]
    if n.get("role", {}).get("value") in interactive_roles
    and not n.get("ignored", False)
]
# Typical result: 20–100 nodes, ~1.5 KB JSON — fits easily in context

Strategy 4 — Set-of-Marks (SOM) overlay

Draw numbered bounding boxes on the screenshot (using Canvas API or Pillow post-capture) for each interactive element. Pass the annotated image + the index→element mapping. Model says click(mark=12). Avoids raw co-ordinate hallucination. Adds ~200 ms overhead for overlay generation.

Screen Capture vs DOM-Only — When Each Wins

No single perception strategy dominates across all web tasks. The right choice depends on the page's structure and the nature of the task.

WebArena and Mind2Web Evals

Browser-focused evals differ from the full-desktop OSWorld in that they run against controlled web environments. This makes ground-truth verification easier but raises contamination concerns for any model trained on public web data.

Captcha, Anti-Bot, Ethics

Browser agents inevitably collide with anti-bot defences. Understanding the technical layers helps both in mitigating them legitimately and in knowing when to refuse.

The anti-bot stack

from playwright.async_api import async_playwright
from playwright_stealth import stealth_async

async with async_playwright() as p:
    browser = await p.chromium.launch(
        headless=True,
        args=["--disable-blink-features=AutomationControlled"]
    )
    page = await browser.new_page()
    await stealth_async(page)   # patches navigator.webdriver, plugins, CDP artefacts

    # Add mouse-movement noise before clicks
    await page.mouse.move(x + random.gauss(0, 3),
                          y + random.gauss(0, 3))
    await asyncio.sleep(random.uniform(0.08, 0.35))
    await page.mouse.click(x, y)

Stateful Sessions — Cookies, Auth, Persistence

Most non-trivial web tasks require authentication. The three main patterns for session persistence differ in security, portability, and implementation complexity.

# --- Save after manual login ---
context = await browser.new_context()
page = await context.new_page()
await page.goto("https://example.com/login")
# ... perform login manually or via the agent ...
await context.storage_state(path="example_session.json")
await context.close()

# --- Restore on next run ---
context = await browser.new_context(
    storage_state="example_session.json"
)
page = await context.new_page()
await page.goto("https://example.com/dashboard")
# Already authenticated; no login page shown

Pricing — Minutes Per Task, Infrastructure

Browser-agent costs have three components: LLM inference (per token), browser infrastructure (per minute), and operator overhead (retries, human-in-the-loop interrupts). Understanding each matters for cost-effective deployment.

Failure Modes & Recovery

Browser agents fail in characteristic ways. Understanding the failure taxonomy is necessary before designing a recovery strategy.

async def run_with_recovery(task, max_steps=40, max_retries=3):
    checkpoint = None   # URL + storage state at last verified good state
    for attempt in range(max_retries):
        try:
            context = await make_context(checkpoint)
            result = await run_agent(task, context, max_steps)
            if result.success:
                return result
            # Save checkpoint at last successful sub-goal
            checkpoint = await save_checkpoint(context)
        except PlaywrightTimeoutError:
            await asyncio.sleep(2 ** attempt)   # back-off
        finally:
            await context.close()
    raise AgentExhausted(f"Task failed after {max_retries} attempts")

What to Take Away

• Three-tier tooling. Browser-Use for fast iteration; raw Playwright for full CDP control; Steel for managed infrastructure with residential proxies.
• AXTree + element list is the efficient perception default. ~230 tokens per step vs ~765 for a screenshot. Fall back to screenshot for canvas apps and legacy portals.
• Set-of-Marks gives 20–30% grounding improvement. Number bounding boxes on the screenshot; the model references marks by index, not raw co-ordinates.
• WebArena (live, ~36% SOTA) is the honest browser eval. Mind2Web (static replay, ~40%) is useful for development but doesn't test async page handling.
• Anti-bot defences are multi-layer. playwright-stealth handles layers 1–2. Cloudflare Turnstile needs attestation or a managed proxy. Never bypass ToS protections.
• Credentials belong outside the LLM context. Use Playwright storage-state files or Steel persistent profiles; never pass passwords in messages.
• Cost: inference dominates for short tasks; browser time dominates for long ones. 30-step task on Steel + Claude 3.5 Sonnet: ~$2–$4. Self-hosted: ~$1–$2.
• Recovery: checkpoint sub-goals; verify state after write actions; never blindly retry POSTs; detect session-expiry by monitoring URL patterns.