Vision Simulation

If your agent lives on a web page, the conversation is only half the test. The customer also signs in, opens the chat, presses the buttons under the agent's reply, and fills in the form the agent asked for. Vision Simulation plays that customer in a real browser, turn by turn, and scores the whole flow.

You write three things. The Target is your page, written as settings: its address, the secrets the Driver types, and where the browser runs. The Driver is the customer: its prompt holds every step on the page, the sign-in and the way to the chat included, then who the customer is and what they want. The Scenario rows hold one conversation each. Okareo drives the browser: it reads each screen, hands it to the Driver, and makes the Driver's moves on the page.

You make the Target in the Okareo app, or in Python with okareo-vision. The browser runs in one of two places: a browser Okareo hosts, or the Chrome on your machine through the relay, which the package runs for you.

The package is on PyPI. This page gets a first Run going; Vision Targets is the reference for every setting.

tip

New to simulations? See the Simulation Overview.

What Vision Simulation Does

What the customer does	What the simulated customer does	What you find out
Signs in and opens the chat	Follows the steps in the Driver's prompt, one screen at a time: fills in the sign-in form, presses Log In, presses the chat button	Where the sign-in or the way to the chat breaks
Asks the chat for a product and adds it to the cart	Sends the request, reads the reply, presses Add to Cart on the item the chat suggested	The agent suggests something the customer can act on, and the cart shows it
Fills in a form the agent asked for	Fills in each field the screen shows, then presses the button that sends the form	The form takes what the agent told the customer to enter
Picks from buttons or suggestions under the agent's reply	Presses the one that fits its goal	The suggestions lead where they say they do
Picks from a dropdown	Names the option it wants, and Okareo picks it from the list	The option the agent named is there to pick
Waits for a slow reply	Waits, reads the screen again, and decides what to do next	Where the page or the agent stalls

How a Conversation Goes

Each Scenario row is one conversation, and each conversation is one flow, from the first press to the last reply:

1. Okareo opens the Target's page, with no cookies or sign-in left from another conversation, and fires the Target's page events, if it has any.
2. The Driver reads the first screen. There is no setup stage before it.
3. The Driver's first turns are the steps at the top of its prompt: the sign-in, if the page has one, and the way to the chat.
4. The Driver sends its first message, and the conversation goes on, a move each turn.
5. The conversation ends when the Driver is done or when it reaches the Run's max_turns. Okareo opens the Target's sign_out_url, when it has one, and closes the page.

Each turn, the Driver makes one move, and Okareo makes it on the page, waits for the page to finish, and reads the screen again:

The Driver's move	What Okareo does on the page
Press	Presses the control the Driver names by its label, as the screen shows it: a button, a link, a suggestion, a menu entry.
Fill	Enters a value into each field the Driver names, picks from a list where the field is one, then presses the button that sends them, when the Driver names one.
Send a message	Types the message into the chat's message box and sends it.
Wait	Reads the screen again without acting, for a reply that is still coming.

Every turn is part of the conversation, the sign-in's included. The app shows each one, the Checks read all of them, and each counts toward max_turns.

Install

A Target made in the Okareo app needs nothing installed. The rest of this section is for Python.

pip install okareo-vision

Python 3.10 to 3.12. The package installs the Okareo SDK (okareo). A Target on a browser Okareo hosts needs nothing more on your machine. A Target on the Chrome on your machine needs okareo 0.0.163 or newer and Google Chrome or Chromium installed; Where the Browser Runs compares the two.

A Run needs an Okareo account and its API key:

export OKAREO_API_KEY="<your-token>"

Your First Vision Simulation

You write three things: the Target, the Driver, and the Scenario rows. The example is a store whose shopping helper opens from a button on the page.

The Target

The Target is the page and nothing else: where it is, and where its browser runs. Here, a browser Okareo hosts opens a public page. Vision Targets covers every setting.

Okareo UI

1. Go to Targets and click New Target.

2. Switch the type selector to Vision, and fill in:

   • Name: the name Okareo lists the Target under, such as Store shopper.
   • URL: the page each conversation opens, here https://shop.example.com/.

   Leave Sign-out URL, Max Parallel Requests and Secrets empty. A public page needs none of them.

3. Click Create. The Target shows under Targets with the type Vision. Pick it on the simulation page with the Driver and the Scenario below, as in Running from the UI.

A page the internet cannot reach runs on the Chrome on your machine through the relay, which Python sets up; see Where the Browser Runs.

Python

from okareo_vision import BrowserConfig, VisionTarget

store = VisionTarget(
    url="https://shop.example.com/",
    browser_config=BrowserConfig(host="hosted"),
)

A page the internet cannot reach, such as one on your own network or on your laptop, runs on the Chrome on your machine through the relay instead: BrowserConfig(host="client_relay").

The Driver

A Driver is a prompt you write and register with the Okareo SDK. In a vision Run, the prompt opens with the steps that get the customer to the chat, in plain words, one screen at a time. The steps end on where the first message goes. The rest of the prompt is the customer: who they are, what they want, and when they are done. Okareo gives every Driver in a vision Run what it needs to read a screen and make moves, so your text is the steps and the customer only. {scenario_input.KEY} is filled from the row's input. Save it as driver.md:

## Getting to the chat

The conversation starts on the store's page, with its shopping helper closed. Open it first, one screen at a time:

1. If a notice about cookies covers the page, close it with its X, accepting nothing.
2. Press Ask Shopwise, or the round chat button in the corner when there is no Ask Shopwise.

Your first message goes in the box named Ask anything; until that box shows, press or wait instead.

## Scenario Details

What you want: {scenario_input.task}
Budget: {scenario_input.budget}
Notes: {scenario_input.notes}

## Objectives

1. Open by telling the helper what you are shopping for, in one short sentence in your own words.
2. Answer the helper's questions from the facts above, and pick from what it shows you.
3. Stop once the helper has answered everything your request asks for. A question from the helper is something to answer, not the end.

## Ending the conversation

- Never end before your first message.
- End only when the helper's own message on screen answers everything your request asks for.

A page with a sign-in puts it first, as steps like any other. A password is typed by name, as {secret.NAME}, and the Driver never sees its value; see Secrets.

info

Configuring LLMs to role-play as a user can be challenging. See Creating Drivers.

The Scenario

Each line of scenarios.jsonl is one conversation: an input with the facts the Driver uses, and a result saying what a finished conversation looks like.

{"input": {"task": "Find out which bag sizes the lump charcoal comes in.", "budget": "under $40", "notes": "Ask for lump charcoal first, then press the 'Show bag sizes' suggestion under Sprig's reply."}, "result": "Sprig suggests lump charcoal under $40 and says its bags come in 15 lb and 20 lb."}

Run It

import json
import os

from okareo import Okareo
from okareo.model_under_test import Driver, Target
from okareo_api_client.models import ScenarioSetCreate

okareo = Okareo(api_key=os.environ["OKAREO_API_KEY"])
rows = [json.loads(line) for line in open("scenarios.jsonl") if line.strip()]
scenario = okareo.create_scenario_set(ScenarioSetCreate(
    name="Store shopper", seed_data=Okareo.seed_data_from_list(rows)))
driver = Driver(name="Store shopper", prompt_template=open("driver.md").read())

run = okareo.run_simulation(
    name="Store shopper",
    scenario=scenario,
    target=Target(name="Store shopper", target=store),
    driver=driver,
    checks=["result_completed"],
    max_turns=20,
    first_turn="driver",
)
print(run.app_link)

run_simulation registers the Target, starts the Run, and waits for it to finish. first_turn="driver" has the Driver make the first move, on the page as it opened. max_turns counts every move, the steps to the chat included, so a long sign-in may need it raised. On the relay, this process drives the Chrome on your machine for Okareo until the Run finishes, so keep it running. Running Simulations covers how many conversations a Run holds.

In a pipeline

submit=True returns as soon as the Run starts, and okareo.wait_for_test_run(run.id) waits for it to finish. It raises when the Run fails, so the script's exit code says how the Run went. wait_for_test_run needs okareo 0.0.163 or newer.

Reading the Results

run.app_link opens the Run in the Okareo app. Each conversation shows every turn, from the Driver's first press to the agent's last reply: what the Driver pressed, filled in or sent, beside the screen that move produced. The steps to the chat are there too, so a conversation that never reached the chat shows the screen where it stopped. Above the transcript, a player steps through the conversation screen by screen, so when something goes wrong you see exactly what the customer saw. Checks such as result_completed score each conversation against its row's result.

Where to Go Next

• Vision Targets: every setting, where the browser runs, secrets, and fixes for common problems.
• Creating Drivers: writing the customer the simulation plays.
• Running Simulations: Scenarios, turns, and how many conversations a Run holds.
• Checks: scoring each conversation.