Browser Automation with TypeSafe Jev: Sub-Second Action Selection in Browser Agents

Many autonomous web agents rely on multimodal Vision-Language Models (VLMs) to control web browsers. In vision-centric loops, the agent captures a full-screen screenshot, sends high-resolution image tokens to the model, and waits multiple seconds per step for an autoregressive tool call to execute a single click or keystroke.

The open-source browser-use/jev-ultrafast project explores an alternative architecture: pairing DOM extraction with fast decision evaluation via TypeSafe Jev. Rather than processing screenshots autoregressively, Jev evaluates an indexed interactive DOM table, selecting actions and target elements within TypeSafe's fast evaluation envelope while reserving generative LLMs for tasks that require drafting freeform text.

The Architectural Separation: Perception, Decision, and Execution

Jev accepts text only; it cannot consume image pixels, directly manipulate DOM nodes, or generate arbitrary search strings. The jev-ultrafast approach separates the browser control loop into distinct functional stages:

1. Perception (Playwright / CDP):
   Extract interactive DOM elements ──> Build indexed interactive element table

2. Decision (TypeSafe Jev):
   Indexed Table + User Goal ──> Parallel Pass (evaluated over state) ──> Action: CLICK, Target: Element [14]

3. Text Generation (Generative LLM - Conditional):
   Only if Action == TYPE_TEXT ──> LLM drafts text string ──> Pass to input field

4. Execution (Playwright):
   Dispatch click or keypress event to element locator ──> Wait for DOM update

Pipeline Stage	Technology Responsible	Operational Role
DOM Extraction	Playwright / Chrome DevTools Protocol	Prunes non-interactive nodes; generates indexed element table
Action & Target Selection	TypeSafe Jev (`jev-latest`)	Evaluates action type and target element concurrently over extracted state
Text Drafting (Conditional)	Generative LLM	Only invoked when an action requires drafting unconstrained text
Browser Execution & Wait	Playwright	Dispatches synthetic browser events and listens for navigation updates

How the DOM State Is Indexed for Jev

Instead of submitting raw HTML with thousands of lines of CSS, headers, and metadata, the agent extracts only interactive elements into a clean, indexed tabular state:

{
  "state": {
    "user_goal": "Book one-way flight from SFO to JFK on October 12",
    "current_url": "https://flights.example.com",
    "interactive_elements": [
      {"id": 0, "tag": "input", "label": "Departure airport", "value": "SFO"},
      {"id": 1, "tag": "input", "label": "Destination airport", "value": ""},
      {"id": 2, "tag": "input", "label": "Departure date", "value": "2026-10-12"},
      {"id": 3, "tag": "button", "label": "Search flights", "state": "enabled"},
      {"id": 4, "tag": "a", "label": "Manage booking", "state": "visible"}
    ]
  }
}

This compact representation consumes only 150 to 300 tokens, well within Jev's context budget, and avoids the 1,500+ token penalty of raw screenshots.

The Jev Decision Contract

In each step, the browser loop sends the indexed state to Jev, evaluating action type and element target concurrently:

from typesafe_sdk import TypeSafeClient, Choice, Noul

client = TypeSafeClient()

# Illustrative decision schema representing dual-head action and target selection
response = client.system_one(
    model="jev-latest",
    state=indexed_browser_state,
    questions={
        "action_type": Choice(
            instructions="Select the next browser action required to make progress toward 'user_goal'.",
            criteria={
                "click": "Click an interactive button, link, or option element",
                "type_text": "Focus and type text into an input or textarea",
                "scroll_down": "Relevant elements are not visible; scroll viewport down",
                "wait": "Page is currently loading or updating network requests",
                "goal_achieved": "The user's goal is fully satisfied on the current screen"
            }
        ),
        "target_element": Choice(
            instructions="Select the ID of the interactive element to interact with.",
            criteria={
                "0": "Departure airport input",
                "1": "Destination airport input",
                "2": "Departure date input",
                "3": "Search flights button",
                "none": "No interactive element target needed for this action"
            }
        ),
        "task_complete": Noul(
            instructions="Does the current page state prove that the flight search is complete and results are displayed?"
        )
    }
)

action = response.answers["action_type"].choice
target = response.answers["target_element"].choice
is_done = response.answers["task_complete"].noul > 0.90

print(f"Action: {action} on Target {target} (Done: {is_done})")
# Example illustrative output: Action: type_text on Target 1 (Done: False)

Where a Generative Model Still Sits

Because Jev is strictly a decision model, it cannot generate freeform string content. When Jev selects "type_text" on target "1" ("Destination airport"), the agent routes to a secondary generative model:

if action == "type_text":
    # Call a generative LLM only when freeform text must be drafted
    text_to_type = call_generative_model(
        prompt=f"Goal: {user_goal}. What exact value should be typed into '{target_element_label}'?"
    )
    # text_to_type returns "JFK"
    page.locator(f"[data-browser-id='{target}']").fill(text_to_type)
elif action == "click":
    page.locator(f"[data-browser-id='{target}']").click()

By isolating text generation to steps that genuinely require drafting strings, clicks, navigations, and status checks proceed through Jev's fast evaluation layer without invoking an autoregressive LLM.

Limitations in Browser Automation

While Jev handles structured DOM navigation effectively, it has jagged edges that developers must handle in application code:

Canvas and WebGL Applications: Jev cannot evaluate canvas elements (such as Google Maps or Figma) where controls do not exist as standard DOM nodes. Such applications still require vision-based computer use models.
Dynamic ID Shifts: If DOM mutations re-index elements between steps, the agent must re-extract the interactive element table before sending state to Jev.
Complex Form Validation Errors: If a page displays an ambiguous inline validation message, Jev's accuracy depends on that error string being explicitly captured in the state dictionary.

Related Context

Agent & Tool Routing with TypeSafe Jev

workflow

Structuring State for TypeSafe Jev

implementation

Jev vs. Structured Outputs Comparison

comparison