Browser Automation with TypeSafe Jev

Browser agents that send screenshots to a vision model can spend seconds per step deciding what to click. A DOM-based agent skips the screenshot. It extracts the page's interactive elements into an indexed table and asks TypeSafe Jev to pick the next operation and target element. A generative model only runs when a field needs typed text. Browser Use's open-source browser-use/jev-ultrafast is the most complete public example.

Browser Control Loop Architecture

Jev accepts text only; it cannot consume image pixels, directly manipulate DOM nodes, or generate arbitrary search strings. A DOM-based loop splits the work into separate stages. The version below uses Playwright for illustration. jev-ultrafast itself drives Chrome through Browser Harness and asks for the operation and its target in a single request:

1. Perception (Playwright / CDP):
   Extract interactive DOM elements ──> Build indexed interactive element table

2. Decision (TypeSafe Jev):
   Indexed Table + User Goal ──> Parallel Pass (evaluated over state) ──> Action: CLICK, Target: Element [14]

3. Text Generation (Generative LLM - Conditional):
   Only if Action == TYPE_TEXT ──> LLM drafts text string ──> Pass to input field

4. Execution (Playwright):
   Dispatch click or keypress event to element locator ──> Wait for DOM update

---

How the DOM State Is Indexed for Jev

Instead of submitting raw HTML with thousands of lines of CSS, headers, and metadata, the agent extracts only interactive elements into a clean, indexed tabular state:

{
  "state": {
    "user_goal": "Book one-way flight from SFO to JFK on October 12",
    "current_url": "https://flights.example.com",
    "interactive_elements": [
      {"id": 0, "tag": "input", "label": "Departure airport", "value": "SFO"},
      {"id": 1, "tag": "input", "label": "Destination airport", "value": ""},
      {"id": 2, "tag": "input", "label": "Departure date", "value": "2026-10-12"},
      {"id": 3, "tag": "button", "label": "Search flights", "state": "enabled"},
      {"id": 4, "tag": "a", "label": "Manage booking", "state": "visible"}
    ]
  }
}

This compact representation consumes only 150 to 300 tokens, well within Jev's context budget, and avoids the 1,500+ token penalty of raw screenshots.

---

The Jev Decision Contract

In each step, the browser loop sends the indexed state to Jev, evaluating action type and element target concurrently:

from typesafe_sdk import TypeSafeClient, Choice, Noul

client = TypeSafeClient()

# Illustrative decision schema representing dual-head action and target selection
response = client.system_one(
    model="jev-latest",
    state=indexed_browser_state,
    questions={
        "action_type": Choice(
            instructions="Select the next browser action required to make progress toward 'user_goal'.",
            criteria={
                "click": "Click an interactive button, link, or option element",
                "type_text": "Focus and type text into an input or textarea",
                "scroll_down": "Relevant elements are not visible; scroll viewport down",
                "wait": "Page is currently loading or updating network requests",
                "goal_achieved": "The user's goal is fully satisfied on the current screen"
            }
        ),
        "target_element": Choice(
            instructions="Select the ID of the interactive element to interact with.",
            criteria={
                "0": "Departure airport input",
                "1": "Destination airport input",
                "2": "Departure date input",
                "3": "Search flights button",
                "none": "No interactive element target needed for this action"
            }
        ),
        "task_complete": Noul(
            instructions="Does the current page state prove that the flight search is complete and results are displayed?"
        )
    }
)

action = response.answers["action_type"].choice
target = response.answers["target_element"].choice
is_done = response.answers["task_complete"].noul > 0.90

print(f"Action: {action} on Target {target} (Done: {is_done})")
# Example illustrative output: Action: type_text on Target 1 (Done: False)

---

Where a Generative Model Still Sits

Because Jev is strictly a decision model, it cannot generate freeform string content. When Jev selects "type_text" on target "1" ("Destination airport"), the agent routes to a secondary generative model:

if action == "type_text":
    # Call a generative LLM only when freeform text must be drafted
    text_to_type = call_generative_model(
        prompt=f"Goal: {user_goal}. What exact value should be typed into '{target_element_label}'?"
    )
    # text_to_type returns "JFK"
    page.locator(f"[data-browser-id='{target}']").fill(text_to_type)
elif action == "click":
    page.locator(f"[data-browser-id='{target}']").click()

By isolating text generation to steps that genuinely require drafting strings, clicks, navigations, and status checks proceed through Jev's fast evaluation layer without invoking an autoregressive LLM.

---

Limitations in Browser Automation

While Jev handles structured DOM navigation effectively, it has jagged edges that developers must handle in application code:

1. Canvas and WebGL Applications: Jev cannot evaluate canvas elements (such as Google Maps or Figma) where controls do not exist as standard DOM nodes. Such applications still require vision-based computer use models.
2. Dynamic ID Shifts: If DOM mutations re-index elements between steps, the agent must re-extract the interactive element table before sending state to Jev.
3. Complex Form Validation Errors: If a page displays an ambiguous inline validation message, Jev's accuracy depends on that error string being explicitly captured in the state dictionary.

Related Context

How to Use Jev: Structuring State and Designing Answer Spaces

implementation↗

Jev vs. Structured Outputs Comparison: Latency and Accuracy

comparison↗

Sources & Documentation

1. Browser Use: jev-ultrafast Repository — Experimental open-source project demonstrating DOM-based browser action selection with TypeSafe Jev.
2. Browser Use: jev-ultrafast performance notes — Source of the 178ms median Jev latency (17 requests, one recorded Google Flights run). Checked September 26, 2026.
3. TypeSafe AI Documentation: State Concepts — Official documentation on structuring JSON objects and context for System One evaluation.