evOo-local/docs/agent-verification.md

Agent verification

evOo separates deterministic execution regressions from real-model task quality. A scripted response can prove that a tool call executes correctly; only a real model can demonstrate that it chooses a useful action for a natural request.

Layer	Real components	Stand-ins	Command
Existing fast tests	Product functions and contracts	Scripted agent, services, sandbox	pnpm test
Pi protocol regressions	Pi agent loop, HTTP/SSE parsing, tool dispatch	Loopback model endpoint, tool effects	pnpm test:pi
Pi product journey	API, saved model connection, Postgres, executor, Pi	Model endpoint, sandbox, connectors	pnpm test:integration
Computer replay	Pi, browser tool handlers, page state	Model endpoint, browser and sandbox	pnpm test
Docker computer replay	Pi, supervisor, Chromium, page helper, downloads and files	Model endpoint, local fixture website	pnpm test:computer-replay
Agent quality	Product API, Postgres, executor, Pi, real model	Sandbox and connected services	pnpm test:evals --live ...
Vision acceptance	Product API, Pi, real vision model, Box or E2B desktop	Fixture website	pnpm test:computer

Default and PR tests never require paid inference. Nightly runs only the web tests with emulated providers. Docker topology and browser replay have a manual workflow; hosted sandboxes and real-model quality runs require an explicit local command. Nightly verification never starts computer sandboxes or requests model or sandbox credentials. Missing live credentials mean not run, not a passing model evaluation.

Deterministic Pi tests

packages/testkit/src/model-emulator.ts serves a loopback OpenAI-compatible stream through evOo's existing generic connection. It does not replace Pi. Each step validates the actual request before streaming a response, and tests must assert that all expected steps were consumed without unexpected requests. The next request must contain the tool result from real execution.

Coverage includes fragmented tool arguments, tool failures, rejected model requests, interrupted streams, cancellation of a quiet stream, and concurrent connections. The Postgres journey also verifies the persisted run, message and file through the product boundary.

Computer replay

The synthetic contacts-export scenario navigates a fixture page, observes the computer, uses fresh element references to open an export dialog and download a CSV, then reads the artifact. Independent checks require the exact CSV and one export. Invalid or stale clicks and cancellation cannot create the artifact.

The emulator models page state; it does not advance just because another tool was called. The Docker lane executes the same scenario against real Chromium and the production supervisor/page-browser helper. It checks a real screenshot and browser-created file. The generic compatible model fixture is text-only; Pi's image-omission behavior is checked explicitly. Vision interpretation is covered by the separate real-model acceptance test.

pnpm sandbox:build
pnpm test:computer-replay
# Or use an already built image:
pnpm test:computer-replay --image=evoo/computer:local
# If Docker has exhausted its automatic address pools, choose an unused subnet:
pnpm test:computer-replay --subnet=<unused-private-cidr>

Docker replay does not open Electron windows, use model credentials, or attach to an existing browser. It owns and cleans up its test resources. It exercises the runtime and computer boundary; it does not claim UI, approval, or database executor coverage.

The suite includes authored failure scenarios and a contacts-export recording captured with Luna through OpenRouter against real Docker Chromium. Replay uses the HTTP model emulator with real Pi and resolves fresh page references. Tests cover cancellation, a transient download failure, workspace restoration, and an interrupted model stream after export without duplicating the export. Separate Postgres integration tests verify that computer actions wait for approval, execute the approved payload once, and have no effects after denial.

To capture another successful run manually:

# Requires OPENROUTER_API_KEY; incurs inference usage, with a local Docker sandbox.
pnpm test:computer-replay --image=evoo/computer:local \
  --live --record=new-fixture.json

The recorder accepts only this synthetic contacts scenario. Its closed vocabulary retains action names, known button labels and success/error outcomes; it excludes model prose, raw responses, URLs, credentials, IDs, element refs and screenshots. A new fixture is written only after both live execution and an immediate offline replay produce the exact CSV and exactly one export. Failed attempts print only the sanitized decisions and do not publish a fixture.

This records semantic browser actions, not coordinate or vision decisions. Box currently lacks the page-browser capability used here, so this capture uses local Docker. Box remains the default for the separate screenshot/coordinate acceptance journey. One successful captured run demonstrates replay coverage; it does not measure the live model's reliability.

Live computer acceptance

Run a vision-capable model through OpenRouter against a real Box desktop:

COMPUTER_E2E_MODEL=openai/gpt-5.6-luna pnpm test:computer
# Run the same checks against E2B when provider-specific verification is needed:
COMPUTER_E2E_MODEL=openai/gpt-5.6-luna pnpm test:computer --sandbox e2b

Box is the default regardless of the application's sandbox setting. This opt-in test requires OPENROUTER_API_KEY and the selected sandbox's credential (BOX_API_KEY or E2B_API_KEY) and incurs inference and sandbox usage. It checks visual observation, a real browser click, terminal access and exact file contents. It then destroys the sandbox outside the app and calls computer/recover, requiring a new sandbox with the saved file restored. computer/boot returns the stored running state and does not request recovery from an externally deleted sandbox.

Both providers run the same assertions. This journey does not cover PTY sessions or multiple screens, which Box does not support. Provider conformance tests remain separate. Live runs retain error logs to diagnose provider failures.

Real-model quality

List the cases without inference:

pnpm test:evals --list

Run through a normal provider connection, referring to an existing credential variable rather than placing a key on the command line:

pnpm test:evals --live --provider openrouter --model <model-id> \
  --api-key-env OPENROUTER_API_KEY --trials 3

--connection <private-json-file> also accepts the shared models/connect shape, including a generic compatible endpoint. Never commit that file. Use --case <case-id> to select a regression. Each run provisions isolated Postgres and synthetic services; it does not use production application data. Each trial uses fresh accounts and services. The runner disables its routines and cancels its work before proceeding; cleanup failure leaves remaining trials not run.

Live evaluations run only through an explicit local CLI invocation. They are not scheduled by the nightly workflow.

The September 2026 Luna/OpenRouter baseline passed all 15 cases over three trials each (45/45) after correcting the harness. The initial run passed 41/45: one shell command was falsely reported successful by the fake sandbox, and three correct outcomes failed overly narrow wording checks. Regression tests cover those fixes; the original report remains separate from the corrected run.

The suite covers artifacts and calculations, inbox grounding and injected instructions, precise and read-only CRM operations, approval payloads, uncertain writes, durable preferences, workspace memory isolation, saved taught playbooks, GitHub release monitoring, and a Slack-to-Salesforce-and-Zendesk customer update. The recorded baseline above covers the original 15 cases; the current 16-case suite also verifies that a real model selects matching records across two local service emulators and posts a grounded reply to the originating Slack-like DM. It does not evaluate visual teaching or native mobile recording.

Keep compact scenario seeds, generators and grading criteria in this repository so contract changes are reviewed together and failures reproduce from one commit. Fixtures must be synthetic. If a future corpus is too large for the repository, publish it as a versioned artifact pinned by immutable digest and retain a small offline conformance fixture here. Do not import third-party eval data unless its redistribution terms are explicit and compatible.

The eval sandbox executes file tools but returns an explicit error for model shell commands instead of pretending to run them. If outcome checks fail after that limitation is encountered, the trial is attributed to the harness. Confirm shell-dependent outcomes in the real Docker or live computer lane. Release monitoring checks a daily schedule and a newly introduced release; unchanged-release notification deduplication is not covered by this suite.

Grading reads files, service records, recorded effects and persisted product state. A model's claim of completion is not sufficient. Cases do not receive repair prompts or coaching after a failure. Multi-turn setup and fresh-context checks are explicit parts of the scenario.

Reports under test-report/evals/ contain per-case success counts, first trial success, autonomous success rate, latency, tool counts, criteria, redacted traces, artifacts and redacted memory evidence. Tool and usage totals retain records across conversation clearing; clearing history cannot reset a trial’s tool budget. Unavailable token or cost measurements remain null. Failures distinguish agent outcomes, product errors, provider failures, harness failures and incomplete runs. Read the category and evidence before attributing a red run to a prompt change. Several trials establish an initial baseline, not a statistically precise reliability estimate.

The injection case checks the requested artifact and forbidden service effects. It does not grade every claim in free-form explanatory prose; quoted or denied injection warnings must not be mistaken for compliance with the injection.

Keep functional criteria deterministic. Add a model judge only for a quality that cannot be graded directly, with a versioned rubric and human calibration. Do not let a judge override forbidden effects or missing artifacts.