Compare commits

...

94 Commits

Author SHA1 Message Date
Nikhil Sonti
dd73a4f5e1 fix(eval): address executor backend review comments 2026-04-29 17:41:05 -07:00
Nikhil Sonti
f2f8f5cbd2 refactor(eval): split orchestrated executor backends 2026-04-29 17:28:47 -07:00
Nikhil
84a79ba0a1 feat: refactor eval pipeline workflow (#875)
* feat(eval): add suite variant config bridge

* feat(eval): add stable run artifacts

* refactor(eval): add shared grader contract

* feat(eval): persist grader artifacts

* refactor(eval): rename runner layers

* refactor(eval): add executor backend boundary

* refactor(eval): split clado backend

* feat(eval): add workflow compatible cli

* feat(eval): add r2 publisher module

* ci(eval): migrate weekly workflow to eval cli

* docs(eval): document suite pipeline

* chore(eval): verify pipeline refactor

* fix: address review feedback for PR #875

* docs(eval): add env example

* docs(eval): explain suites and variants

* chore(eval): organize config layouts

* chore(eval): colocate grader python evaluators
2026-04-29 17:21:02 -07:00
Nikhil
6e3306f5e5 fix: make R2 uploads retryable (#874)
* fix: make R2 uploads retryable

* fix: address review feedback for PR #874
2026-04-29 16:43:33 -07:00
Nikhil
c244462b29 fix: use Node 24 GitHub actions (#872) 2026-04-29 15:31:23 -07:00
Nikhil
ebf97f74f6 fix: bound VM agent cache smoke test (#870)
* fix: bound VM agent cache smoke test

* fix: address review feedback for PR #870
2026-04-29 13:43:37 -07:00
Nikhil
561f2baf97 fix(eval): split AGISDK smoke and full configs (#871)
* fix(eval): split agisdk smoke and full configs

* fix(eval): default agisdk smoke to openrouter
2026-04-29 13:38:55 -07:00
shivammittal274
df0f45dd29 Feat: eval debug dev ci (#869)
* chore(eval): instrument server startup to root-cause dev CI health-check timeouts

Three diagnostics + one config swap to investigate why the eval-weekly
workflow has been failing on dev since 2026-04-25 with "Server health
check timed out" (every worker, every retry).

Background:
- Last successful weekly eval on dev: 2026-04-18 (sha f5a2b73)
- Since then, ~30 server commits landed including Lima/VM runtime,
  OpenClaw service, ACL system, ACP SDK — 108 server files changed,
  ~13K LOC added.
- Server process spawns cleanly in CI (PID logged) but never binds
  /health within the 30s eval-side timeout. Static analysis finds no
  obvious blocker; we need runtime evidence.

Changes:

1. apps/server/package.json — add `start:ci` script (no `--watch`).
   The default `start` uses `bun --watch` which forks a child process
   that watches every file in the import graph. Dev's graph is ~108
   files larger than main's; on a cold CI runner the watcher setup is a
   plausible source of multi-second startup overhead.

2. apps/eval/src/runner/browseros-app-manager.ts:
   - Use `start:ci` when `process.env.CI` is set (true on
     GitHub-hosted runners by default), else `start`.
   - Capture per-worker server stderr to /tmp/browseros-server-logs/
     instead of ignoring it. Without this we have no visibility into
     why the server is hung pre-/health.
   - Bump SERVER_HEALTH_TIMEOUT_MS 30s -> 90s. Dev's larger module
     graph may simply need more cold-start time on CI.

3. .github/workflows/eval-weekly.yml — upload the server logs dir as a
   workflow artifact (always, not just on success) so we can post-mortem
   any startup failure on the next run.

4. configs/agisdk-real-smoke.json — swap K2.5 from OpenRouter ->
   Fireworks (bypasses the OpenRouter per-key spend cap that has been
   eating recent runs) and drop num_workers 10 -> 4 (well below the
   Fireworks per-account TPM threshold that overwhelmed the original
   2026-04-23 run).

Plan: trigger the eval-weekly workflow on this branch with the agisdk
config and observe (a) whether it gets past server startup, and
(b) if it doesn't, what the captured server stderr says.

* fix(eval): capture stdout too — pino logger writes to stdout, not stderr

Previous diagnostic patch only redirected stderr; the captured per-worker
log files came back as 0 bytes because the server uses pino which writes
all log output to stdout (fd 1), not stderr (fd 2). Capture both into
the same file.

* fix(server): catch sync throw from OpenClaw constructor on Linux

The container runtime constructor in OpenClawService throws synchronously
on non-darwin platforms, e.g. GitHub Actions Linux runners. The existing
.catch() on tryAutoStart() only handles async throws inside auto-start —
the sync throw from configureOpenClawService(...) itself propagates up
through Application.start() and crashes the process via index.ts:48
(process.exit(EXIT_CODES.GENERAL_ERROR)).

This is what's been killing dev's eval-weekly CI: the server crashes in
milliseconds, the eval client polls /health, gets nothing, times out.

Fix: wrap the configureOpenClawService call in try/catch matching the
existing .catch() intent (best-effort, don't crash). Server continues
without OpenClaw on platforms where it can't initialize.

Verified by reading captured server stdout from run 25123195126:
  Failed to start server: error: browseros-vm currently supports macOS only
    at buildContainerRuntime (container-runtime-factory.ts:54:11)
    at new OpenClawService (openclaw-service.ts:652:15)
    at configureOpenClawService (openclaw-service.ts:1527:19)
    at start (main.ts:127:5)

* fix(server): defer OpenClaw chat client port lookup to request time

apps/server/src/api/server.ts:149 was calling getOpenClawService().getPort()
synchronously when constructing the OpenClawGatewayChatClient inside the
createHttpServer object literal. On non-darwin platforms this throws via
the OpenClawService constructor → buildContainerRuntime, escaping the
try/catch added in 5cf7b765 (which only protected the configureOpenClawService
call further down in main.ts).

Every other getOpenClawService() reference in server.ts is already wrapped
in an arrow function. This was the lone holdout. Make it lazy too: change
the chat client constructor to take getHostPort: () => number instead of
hostPort: number, evaluate it inside streamTurn at request time. Behavior
on darwin is unchanged.

This unblocks dev's eval-weekly CI on Linux runners where OpenClaw isn't
available — the chat endpoint isn't exercised by the eval, so a deferred
throw is acceptable.

* fix(server): allow Linux to skip OpenClaw via BROWSEROS_SKIP_OPENCLAW=1

Earlier surgical fixes (try/catch in main.ts, lazy chat client port) didn't
unblock dev's Linux CI — same throw kept reproducing. Whether this is bun
caching stale stack frames or a missed eager call site, the safer move is
to fix it at the root: make buildContainerRuntime never throw on Linux
when the runner has explicitly opted out.

Adds BROWSEROS_SKIP_OPENCLAW env check alongside the existing NODE_ENV=test
escape hatch in container-runtime-factory.ts. When set, returns the existing
UnsupportedPlatformTestRuntime stub — server boots normally, /health binds,
any actual OpenClaw API call still fails loudly at request time.

eval-weekly.yml sets the flag for the Linux runner. Darwin behavior and
non-CI Linux behavior unchanged (without the flag they still throw).

* feat(eval): align Clado action executor with new endpoint contract

David Shan shared the updated Clado BrowserOS Action Model spec.
Changes to match it:

- Bump endpoint URL + model id to the 000159-merged checkpoint
  (clado-ai--clado-browseros-action-000159-merged-actionmod-f4a6ef)
  in browseros-oe-clado-weekly.json and the README example.
- CLADO_REQUEST_TIMEOUT_MS 120s → 360s. Cold start can take ~5 min;
  the 2-min ceiling was failing every cold-start request.
- Treat HTTP 200 with action=null / parse_error as an INVALID step
  instead of aborting the executor loop. The model can self-correct
  on the next call. Cap consecutive parse failures at 3 to avoid
  infinite loops.
- Capture final_answer from end actions. Surface it in the observation
  back to the orchestrator so its task answer can use the model's
  declared result.
- Add macOS Cmd-* key mappings (M-a, M-c, M-v, M-x → Meta+A/C/V/X).
- Switch screenshot format from webp → png to match the documented
  "PNG or JPEG" contract.

* chore(eval): refresh test-clado-api script for new Clado contract

Updated the local smoke-test to match the new Clado endpoint and
response contract:

- New action + health URLs (000159-merged checkpoint).
- Drop the grounding-model branch (orchestrator-executor doesn't
  use it; the README David shared only documents the action model).
- Health-check waits up to 6 minutes for cold start with a 30s
  warning so the operator knows it's spinning up.
- Print every documented response field (action, x/y, text, key,
  direction, amount, drag start/end, time, final_answer, thinking,
  parse_error, inference_time_seconds).
- Three-step run that exercises a click, a typing continuation
  with formatted history, and an end+final_answer probe.

* chore(eval): point clado weekly config at agisdk-real

Switches the orchestrator-executor + Clado weekly config to run on the
AGI SDK / REAL Bench task set with the deterministic agisdk_state_diff
grader. Matches the orchestrator-executor smoke target (Fireworks K2.5
orchestrator + Clado action executor) we want to track week-over-week.

* chore(eval): run clado weekly headless

Default to headless so the weekly job (and local repros) don't pop ten
visible Chrome windows. Set headless=false locally if you need to watch
a worker.

* fix(eval): address Greptile P1+P2 on server log fd handling

P1: openSync was outside the mkdirSync try/catch, so a swallowed mkdir
failure (e.g. unwritable custom BROWSEROS_SERVER_LOG_DIR) would leave the
log directory missing and crash the server spawn with ENOENT. Move openSync
into the same try block; fall back to /dev/null so spawn always succeeds.

P2: the log fd was opened on every server start but never closed. Each
restart attempt leaked one fd across all workers — over a long eval run
that could exhaust the process fd limit. Track the fd on the manager and
closeSync it in killApp() right after the server process exits (the child's
dup keeps the file open until it exits, so we don't truncate output).
2026-04-30 01:33:49 +05:30
Nikhil
edfc5c751c fix: align OpenClaw gateway image with VM cache (#868)
* fix: load OpenClaw gateway image from VM cache

* fix: use container port for OpenClaw ACP bridge

* fix: address review feedback for PR #868
2026-04-29 12:11:00 -07:00
Nikhil
471256f31c fix: stop passing native permission flags to ACP adapters (#867) 2026-04-29 11:07:51 -07:00
Nikhil
4c90ca696b fix(agents): connect OpenClaw ACP inside gateway container (#866) 2026-04-29 11:07:29 -07:00
Nikhil
f2ac87d7c3 feat: show created agents in sidepanel (#865)
* feat(agent): list created agents in sidepanel target catalog

* feat(agent): show created agents in sidepanel selector

* feat(server): add sidepanel chat route for created agents

* feat(agent): route sidepanel agent sends by agent id

* chore(agent): retire virtual sidepanel acp targets

* fix: address review feedback for PR #865
2026-04-29 10:15:58 -07:00
shivammittal274
231bd6821d fix(eval): pin agisdk version + exclude 4 invalid tasks (Phase 2 dataset hygiene) (#844)
* chore(eval): pin agisdk version to prevent silent dataset drift

`pip install agisdk` previously fetched whatever version pip resolved at
CI time. If agisdk publishes a new version with changed task definitions
or grader behavior, the weekly eval silently shifts under our feet —
making "did the score move because of code or data?" unanswerable.

Pin to agisdk==0.3.5 (the version we currently develop against). Bump
intentionally with a documented re-baseline run.

* fix(eval): exclude 4 more tasks identified by 8-trial never-passing audit

After 8 trials across K2.5 + Opus 4.6 (Phase 1 and Phase 2), 5 tasks
never passed. Per-task root-cause investigation via parallel deep-dive
subagents flagged 4 of them as fundamentally unfixable in the eval
pipeline as it stands; the 5th (`dashdish-5`) is a prompt-rule fix
that stays in.

- gocalendar-7: goal/grader contradiction. Goal says "move event to
  July 19, 10 AM"; grader expects `eventsDiff.updated.*.start ==
  "2024-07-18T17:00Z"` (= July 18, 10 AM PDT — same day, 1 hour shift).
  Even after the Phase 2 HTML5 dnd dispatch fix correctly populates
  `eventsDiff.updated`, the values are July 19 (matching the goal),
  which the grader rejects.

- staynb-5: grader hardcodes literal `'Oct 13 2025'` and `'Oct 23 2025'`
  year strings. The staynb date picker interprets bare "Oct 13" as the
  most-recent-past instance (currently 2024 since today is 2026), not
  2025. No agent can produce a persisted date string containing 2025.

- staynb-9: under-specified task. Goal says "maximum number of guests
  supported"; grader requires the very specific string "32 Guests, 16
  Infants" — encoding UI knowledge (Adults+Children=Guests display,
  Infants render separately, per-category cap=16, Pets excluded) that
  isn't in the prompt. Even Opus 4.6 stopped at 16 across 3 trials.

- opendining-3: grader requires `contains(booking.date, '2024-07-20')`
  but the React-controlled date textbox flakily no-ops on `fill`. 3/8
  trial pass rate is essentially coin-flip noise driven by tool-fidelity
  variance rather than agent capability. Removing to reduce score noise;
  Phase 2 fill post-validate warning helps when it does work, but the
  task's signal-to-noise is too low for the eval set.

Dataset goes from 40 -> 36 tasks. Total EXCLUDED_TASKS now 11 entries.

Validated by 8-trial pass-record audit; deep-dive notes saved to
plans/audits/.
2026-04-29 22:07:53 +05:30
Dani Akash
a228c278c6 feat(agents): background-resilient chat — turns survive tab disconnect (#863)
* feat(agents): decouple chat turn lifecycle from SSE response

Introduce a per-process ActiveTurnRegistry that owns each agent turn's
lifecycle and a ring-buffered event stream, so chat tabs that close,
refresh, or navigate away no longer cancel the in-flight turn. New
endpoints:

  POST   /agents/:id/chat          starts a turn (now returns 409 when
                                   one is already running, with the
                                   active turnId for attaching)
  GET    /agents/:id/chat/active   reports the running turn for a UI
                                   that just mounted
  GET    /agents/:id/chat/stream   subscribes to a turn; supports
                                   Last-Event-ID resume via per-event
                                   seq ids
  POST   /agents/:id/chat/cancel   explicit cancel — fetch abort no
                                   longer affects the underlying turn

The chat hook now captures X-Turn-Id, tracks lastSeq from SSE id lines,
re-attaches on mount when the server still has an active turn, and
routes Stop through the cancel endpoint. The runtime call uses the
registry's per-turn AbortController instead of the HTTP request signal,
which is the core decoupling that lets turns outlive their initiator.

* feat(agents): add ActiveTurnRegistry primitive backing the new chat lifecycle

The previous commit referenced these files in tests and the harness
service but global gitignore swallowed them on the first add.

The registry owns the per-turn ring buffer (drop-oldest, terminal frame
preserved), the per-turn AbortController, and subscriber fan-out used
by /chat/stream resume.
2026-04-29 21:01:06 +05:30
Dani Akash
e2ec1991cf feat(agents): redesign the agent command center for multi-adapter use (#861)
* feat(agents): redesign agent rail to match the rest of the app

Reshape the `/agents` page so it reads as a sibling of `/scheduled`
and `/soul` and adapts to the multi-adapter world (OpenClaw, Claude
Code, Codex). Visual scaffolding only in this commit — per-agent
liveness state ships as `unknown` until the server-side activity
tracker lands.

  - New `AgentsHeader` mirrors `SoulHeader`/`ScheduledTasksHeader`:
    accent bot tile, title, descriptive subtitle, "+ New Agent"
    button. Replaces the loose top toolbar that mixed page-level and
    OpenClaw-lifecycle controls.
  - New `GatewayStatusBar` collects the OpenClaw lifecycle pills
    (running, control plane connected) plus the Terminal/Refresh
    affordances into a single labeled bar that only renders when the
    gateway is running AND there is at least one OpenClaw agent in
    the merged list.
  - New `AgentRowCard` per agent: adapter tile with liveness dot,
    name + status badge, adapter/model/reasoning chips, last-used
    relative time + truncated workspace path, primary "Chat" button,
    overflow menu (Copy id / Rename* / Reset history* / Delete).
    Rename + Reset are disabled with "coming soon" tooltips until
    the corresponding endpoints ship; Delete is hidden for the
    protected `main` agent.
  - New `AgentsEmptyState` mirrors the scheduled-tasks empty card.
  - New `AdapterIcon` + `LivenessDot` + `agent-display.helpers.ts`
    keep the row card focused on layout; helpers cover display name
    fallbacks for legacy `oc-<uuid>` titles, workspace label rules,
    and a tiny relative-time formatter.
  - `AgentList` now sorts by `lastUsedAt` desc with `null`s falling
    to the bottom; the gateway's `main` agent is pinned to the top
    only while it has zero turns so a fresh install has an obvious
    starting point. The list also threads a per-agent activity map
    so future commits can light up working/idle/asleep without
    reshuffling the API.
  - `AgentsPage` swaps to the standard `fade-in slide-in-from-bottom-5
    animate-in space-y-6 duration-500` shell and threads a
    `harnessAgentLookup` Map down to the row card so adapter chips
    and reasoning effort render correctly without a re-fetch.

* feat(agents): wire per-agent liveness end-to-end into the rail

Closes the placeholder `unknown` dot from the redesign's first
commit. The rail now shows real working / idle / asleep / error
states per agent, with `lastUsedAt` driving the recency sort.

Server side:
  - `AgentHarnessService` keeps an in-memory activity tracker keyed
    by agentId. `notifyTurnStarted` flips an entry to `working`,
    `notifyTurnEnded({ok})` either drops it (success) or pins it to
    `error` (failure / error event).
  - `send()` wraps the runtime stream so the lifecycle hook fires
    exactly once on natural close, error event, downstream cancel,
    or thrown setup. The runtime itself stays unchanged — fork is
    contained at the harness layer.
  - New `listAgentsWithActivity()` method enriches every agent with
    `{ status, lastUsedAt }`. lastUsedAt is read from the acpx
    session record's last persisted item via `runtime.getHistory`,
    so it survives server restart even though the activity map
    doesn't.
  - Status derivation: `working`/`error` take precedence; otherwise
    timestamp-based — `idle` until 15 min of silence, then `asleep`.
    Never-used agents resolve to `idle` (asleep implies "was active,
    went quiet").
  - `GET /agents` returns the enriched shape.

Client side:
  - `HarnessAgent` UI type extended with optional `status` +
    `lastUsedAt` so older deployments still typecheck.
  - `useHarnessAgents` flips on `refetchInterval: 5_000` (with
    `refetchIntervalInBackground: false` so hidden tabs go quiet)
    so the per-row dots and last-used copy stay fresh without a
    websocket.
  - `AgentsPage` builds an activity map from the harness listing
    response and threads it into `AgentList` → `AgentRowCard`. The
    sort by `lastUsedAt` desc (already in the row card) now has
    real data to operate on.

Tests:
  - New `marks an agent working while a turn streams and idle once
    it ends` exercises the wrap; uses a held upstream stream so
    the in-flight `working` state is observable.
  - New `flips to error when a turn emits an error event`.

* fix(agents): dedupe agent rail when /claw/agents and /agents share an id

The agents page was rendering every OpenClaw agent twice — once from
the legacy `/claw/agents` listing (`useOpenClawAgents`) and once from
the harness `/agents` listing (`useHarnessAgents`). Post Step 9
backfill the harness store contains every gateway agent, so the
overlap is the rule, not the exception.

Mirror the dedup the chat-panel layout already does: when a gateway
agent's id appears in the harness listing, drop the legacy entry and
keep the harness one (it has adapter/model/reasoning/status/lastUsedAt
the chat path actually consumes).

* feat(agents): swap GatewayStatusBar refresh icon for a Restart Gateway button + tooltips

The manual refresh became redundant once `useHarnessAgents` and
`useOpenClawStatus` started polling on a 5s interval — every visible
field self-refreshes within seconds. The previous AgentsPageHeader
had a real Restart action that the redesign dropped; reinstate it on
the bar so a wedged gateway is one click away again.

  - GatewayStatusBar: dropped the `RotateCcw` refresh icon and the
    `onRefresh` prop. Added `onRestart` + `actionInProgress` props;
    the button shows a spinner while a gateway lifecycle mutation is
    in flight.
  - Both Terminal and Restart Gateway buttons get tooltips explaining
    what they do — Terminal as a power-user shell escape hatch,
    Restart for unsticking a wedged gateway or after manual config
    edits.
  - AgentsPage: drop the now-unused `refreshAll` helper and the
    `refetchStatus`/`refetchAdapters`/`refetchOpenClawAgents`
    destructures it depended on. Wire `restartOpenClaw` (already
    pulled from `useOpenClawMutations`) through
    `runWithPageErrorHandling` like the legacy header did.

* feat(agents): consolidate gateway status into the /agents listing

Folds the gateway lifecycle snapshot into the harness listing so the
agents page polls one endpoint instead of two. Drops the dead
`/claw/status` call from the command center while keeping every UI
affordance the page already shipped (Running / Control plane
connected pills, GatewayStateCards setup/start prompts,
ControlPlaneAlert for degraded states).

Server side:
  - `OpenClawProvisioner.getStatus()` (optional) — when wired, returns
    the same `GatewayStatusSnapshot` shape `/claw/status` does.
  - `AgentHarnessService.getGatewayStatus()` — best-effort wrapper
    around the provisioner method; logs and swallows errors so a
    transient gateway issue doesn't 500 the listing endpoint.
  - `GET /agents` now returns `{agents, gateway}` in a single
    `Promise.all`. Both fields are independent — agents enrichment
    succeeds even if the gateway snapshot is null.
  - `server.ts` wires `getOpenClawService().getStatus()` into the
    provisioner accessor object alongside `createAgent` /
    `removeAgent` / `listAgents`.

Client side:
  - `useHarnessAgents` returns `{harnessAgents, gateway}` (plus the
    legacy `agents` mapping). Same 5s `refetchInterval` as before —
    one round-trip drives the per-row liveness AND the gateway pills.
  - `AgentsPage` drops `useOpenClawStatus` entirely; `status` comes
    from the harness query. Loader + error/lifecycle plumbing
    rewired around the harness query's loading/error.
  - `agents-page-utils.getInlineError` and `getAgentsLoading` lose
    the now-redundant `statusError` / `statusLoading` /
    `openClawAgentsEnabled` params.

The chat-panel layout (`agent-command-layout.tsx`) still consumes
`useOpenClawStatus(5000)` for now — left intact per the user's "only
the command center" scope. Folding that one in is a separate,
smaller pass once we're sure no regression slipped here.

* test(agents): teach the route fake service about the new listing shape

PR #861 CI surfaced two failures in tests/api/routes/agents.test.ts:
both call \`GET /agents\` and the route handler now invokes
\`service.listAgentsWithActivity()\` + \`service.getGatewayStatus()\`
which the fake created here didn't implement. Add both methods to
the fake (returning idle / null) and update the empty-list assertion
to expect the new \`{agents, gateway}\` envelope.
2026-04-29 19:03:29 +05:30
Dani Akash
0c84547e8f feat(agents): migrate OpenClaw chat onto the unified harness/ACP path (#859)
* chore(acp): smoke-test ACP capabilities against running gateway

Adds apps/server/scripts/acp-smoke.ts which spawns `openclaw acp`
inside the gateway container and exercises every method we plan to
depend on: initialize, newSession, prompt (text + image), cancel,
listSessions, loadSession.

SDK pinned to 0.19.1 (Bun's minimum-release-age policy blocks 0.20+
which were released < 7 days ago).

Findings (full notes in plan outcomes):
- promptCapabilities advertises image:true but the model does NOT see
  image bytes — silently dropped at the bridge.
- sessionCapabilities advertises {list:{}} but session/list throws
  "Method not found": stale capability advertising.
- loadSession works; replays user/assistant/thought text and
  session_info/usage/commands updates. No tool_call replay, as
  documented.
- cancel works end-to-end: stopReason=cancelled.
- closeSession/resumeSession are not on ClientSideConnection in
  0.19.1; kill child to close, use loadSession for rebind.

Plan revisions triggered by spike are recorded in
plans/browseros-ai/BrowserOS/features/2026-04-28-2310-claude-code-acp-implementation-roadmap.md.

* chore(acp): re-run smoke on SDK 0.21.0 and add mode/config/auth scenarios

After bypassing Bun's minimum-release-age and upgrading the SDK to
0.21.0, restore the previously-skipped resume/close paths and add
three new scenarios: mode (setSessionMode), config (setSessionConfigOption,
correct configId field), and auth (authenticate noop).

Findings, all bridge-side (independent of SDK):
- session/list, session/resume, session/close all throw -32601 on
  OpenClaw 2026.4.12 — capability advertising is stale.
- Image content blocks silently dropped; model never sees the bytes.
- setSessionMode and setSessionConfigOption work; latter requires
  `configId` (not `optionId`) per the schema.
- loadSession replays user/assistant/thought text + session_info +
  usage + available_commands; no tool_call replay (documented).
- authenticate is a noop on OpenClaw (no authMethods advertised).

Plan outcomes updated with full method-support matrix.

* chore(deps): promote @agentclientprotocol/sdk to a runtime dependency

The smoke script in apps/server/scripts/acp-smoke.ts used the SDK as
devDependency. The upcoming ACP bridge (apps/server/src/api/services/acp/)
needs it at runtime, not just for tooling. Move the entry from
devDependencies to dependencies, alphabetically first under @a*.

Pinned to 0.21.0 — same version the smoke script validated against.
README gains a small Dependencies note pointing at the future bridge
location.

No code changes yet. The bridge wiring lands in subsequent commits.

* fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config

When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.

Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
  side (useOpenClaw.ts) and the route body schema + SetupInput +
  createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
  LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
  `input: ['text', 'image']` on the model entry when the flag is
  truthy; otherwise emits the explicit `['text']` so the value is
  always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
  bootstrap batch so the gateway's image-understanding pipeline is
  always wired up — per-model `input` still gates which models see
  images, this just enables the global path.

ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.

Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.

* feat(agents): add OpenClaw to AgentAdapter union and catalog

Extends AgentAdapter to 'claude' | 'codex' | 'openclaw' and adds the
OpenClaw entry to AGENT_ADAPTER_CATALOG. The new entry has:

- defaultModelId: 'default' — OpenClaw's ACP bridge does not surface
  per-session model selection (verified during the ACP spike), so
  models live in the OpenClawService config, not in the adapter
  catalog. AgentDefinition.modelId carries the gateway-side model
  name for display only.
- models: [] — empty list signals "no per-session model picker" in
  the UI; isSupportedAgentModel('openclaw', undefined|'default')
  returns true via the existing fallback path.
- reasoningEfforts mirror OpenClaw's session-level `thought_level`
  config option (off / minimal / low / medium / high / adaptive).

Also extends:
- isAgentAdapter type guard recognizes 'openclaw'
- HarnessAgentAdapter union on the extension side
- agents.test.ts createAgent fake type
- agent-catalog.test.ts asserts on the new entry, empty model list
  passthrough behavior, and OpenClaw's reasoning effort set

Lockfile delta is the workspace SDK pin reconciling 0.20.0 (taken
from dev's lock) up to our package.json's 0.21.0 (added in
c1d987ea). acpx still uses 0.20.0 transitively — both are present.

No runtime wiring yet — the registry override and AcpxRuntime
plumbing land in subsequent commits.

* feat(agents): plumb OpenClaw gateway accessors into AcpxRuntime

Adds an optional `openclawGateway` accessor to AcpxRuntime so the
upcoming registry override (Step 4) can spawn `openclaw acp` inside
the gateway container with the right port, token, and container/VM
identity. All accessors are getter-shaped so values stay live across
gateway restarts (port can change, token can rotate).

The accessor is threaded:
  server.ts → createAgentRoutes → AgentHarnessService → AcpxRuntime
                            ↘ sidepanel lazy AcpxRuntime

Also adds OpenClawService.getGatewayToken() returning the in-memory
token string. We pass it via OPENCLAW_GATEWAY_TOKEN env var on the
spawn (per OpenClaw's documented env-var precedence) instead of via
`--token` flag (which leaks to ps aux) or `--token-file` path (no
discrete token file lives inside the container — the token is nested
inside openclaw.json).

Wiring is dormant — the registry override that consumes these
accessors lands in Step 4. Typecheck + existing acpx/harness/routes
tests pass unchanged.

* refactor(agents): scrub local plan-step references from code comments

Replaces forward-looking comments that referenced internal plan
steps (e.g. "Step 4 wires this into…") with comments that justify
the code on its own merits. Plan files live locally on the
contributor's machine, so cross-references are noise to the rest of
the project.

No behavior change.

* feat(agents): spawn openclaw ACP adapter inside the gateway container

When the harness resolves the `openclaw` adapter, it now returns a
command that runs `openclaw acp` inside the bundled gateway container
via `limactl shell <vm> -- nerdctl exec -i ... openclaw acp --url
ws://127.0.0.1:<port>`. This reuses the openclaw binary already
installed alongside the gateway — no host-side openclaw install is
required.

Auth: the gateway token is injected via OPENCLAW_GATEWAY_TOKEN on
the container exec rather than `--token` on the openclaw CLI, so
the secret never appears in `ps aux`.

Banner output: OPENCLAW_HIDE_BANNER=1 and OPENCLAW_SUPPRESS_NOTES=1
keep stdout JSON-RPC-clean.

LIMA_HOME: prefixed via `env LIMA_HOME=<path>` on the resolved
command so the spawned limactl finds the BrowserOS-owned VM (the
server doesn't set LIMA_HOME on its own process env).

When the gateway accessor is absent, falls through to acpx's
built-in openclaw adapter which assumes a host-side install — that
branch will fail at spawn time with a descriptive error.

Verified end-to-end via the existing acp-smoke script during the
Step 0 spike.

* feat(agents): dual-create OpenClaw harness agents on the gateway

When the harness creates an `openclaw` adapter agent, it now also
provisions a matching agent on the OpenClaw gateway via the existing
CLI path (OpenClawService.createAgent). Symmetric on delete: gateway
removeAgent runs alongside the harness-store delete.

- Adds an OpenClawProvisioner interface (decoupled from OpenClawService
  for testability) and injects it through AgentHarnessService.
- createAgent rolls back the harness record if gateway provisioning
  fails; deleteAgent tolerates gateway-side failures so harness
  identity stays consistent with the user-facing UI.
- New OpenClawProvisionerUnavailableError surfaces as a 503 when an
  openclaw create request lands on a harness with no provisioner
  wired in (instead of a generic 500).
- FileAgentStore mints openclaw agent ids with an 'oc-' prefix so
  the id satisfies the gateway's `^[a-z][a-z0-9-]*$` agent name
  pattern. Other adapters keep raw UUIDs to preserve compatibility.
- POST /agents body schema accepts providerType / providerName /
  baseUrl / apiKey / supportsImages, forwarded to the provisioner
  when adapter='openclaw'.

The agents-page UI still routes openclaw create through the legacy
/claw/agents flow; switching that path to the harness is a separate
UI cutover.

Tests cover: gateway dual-create on success, rollback on gateway
failure, 503 when provisioner is missing, and tolerant delete on
gateway-side failure.

* fix(agents): skip catalog model validation for OpenClaw adapter

OpenClaw agents resolve their model from the gateway-side provider
config (set at agent-create time via OpenClawService) rather than
from the harness catalog, which has an empty `models: []` entry by
design. Without this carve-out, every OpenClaw create body fails
parsing with "Invalid modelId" because no concrete model id can
satisfy isSupportedAgentModel('openclaw', ...).

The reasoning-effort check still runs against the catalog (those
values map directly to OpenClaw's session `thought_level` config
option).

* fix(agents): pass --session to openclaw bridge so newSession routes correctly

acpx's AcpClient.createSession calls connection.newSession with cwd
and mcpServers but never forwards the sessionKey. Without it, the
openclaw bridge falls back to a synthetic acp:<uuid> session that
doesn't resolve to any provisioned gateway agent — every harness
chat returns a generic "Internal error" from -32603.

Fix: bake `--session <key>` into the resolved spawn command. The
bridge then uses that as the default session key for any newSession
the bridge receives, routing the turn to the matching gateway agent.

Per-session keying means each openclaw agent gets its own
AcpxCoreRuntime instance (cached by sessionKey on top of the
existing cwd/permissionMode key). This adds one extra runtime per
active openclaw session — claude/codex are unaffected.

Test asserts the resolved command includes the right --session arg.

* fix(agents): suppress BrowserOS MCP for openclaw bridge

The openclaw ACP bridge rejects newSession when mcpServers is non-empty
because its provider tooling comes from the gateway, not from ACP-side
MCP servers. Forwarding the BrowserOS HTTP MCP made every harness chat
fail with a JSON-RPC -32603 "Internal error" before the session was even
opened. Claude/codex still need the BrowserOS MCP for browser tooling,
so the carve-out is keyed off whether the runtime is for an openclaw
session.

* feat(agents): route OpenClaw chat through the harness behind a flag

Adds the `feature.useAcpxForOpenClaw` extension storage flag. When on,
OpenClaw agents in the agent-command chat panel use the harness
/agents/<id>/chat SSE and harness history hook instead of the legacy
/claw/agents/<id>/chat. When off, behavior is unchanged.

Also dedupes the agent rail when the same id appears in both stores
(dual-created agents from /claw/agents and /agents) by preferring the
harness entry — without this, every dual-created OpenClaw agent shows
up twice after Step 5.

Image attachments are temporarily disabled when the harness path is
active; the carve-out lands in the next commit.

* fix(agents): keep legacy OpenClaw agents on ClawChat

The previous commit's flag-gated branch routed every `source='openclaw'`
agent through `/agents/<id>/chat` when the flag was on, but the layout
dedup means the only agents that ever reach that branch are legacy
gateway-only entries (`main`, orphan agents from rolled-back creates) —
which by definition have no harness record, so the harness path 404s
and chat is unusable. Source is the only routing signal again: harness
agents go through the harness, legacy agents stay on ClawChat. The
storage flag stays for Step 9/10's migration story.

* feat(agents): expose OpenClaw in sidepanel and route through gateway main

`buildSidepanelChatTargets` now emits a single default ACP target for
adapters with no per-session model picker (OpenClaw, whose model is
configured on the gateway-side agent). Without this, OpenClaw never
appeared in the sidepanel target picker because the catalog entry has
`models: []`.

Sidepanel sessions don't have a dedicated provisioned gateway agent.
The openclaw bridge `--session` flag previously got the raw sidepanel
key (`sidepanel:<convId>:openclaw:...`), which doesn't match any
gateway agent — newSession was accepted but every prompt hung
forever. The bridge command now rewrites non-harness session keys
onto the always-present `main` gateway agent, encoding the original
key as a channel suffix to keep state segregated per conversation.
Verified end-to-end via curl: sidepanel openclaw chat streams
`text-delta` + `finish: stop`.

* feat(agents): backfill harness records for legacy gateway agents

Reframes Step 9 of the OpenClaw-on-acpx migration. The plan's literal
Step 9 (route OpenClaw history through the harness when the flag is on)
was already a no-op after the Step 6 walkback — history is routed by
source today. The actual blocker for Steps 10–13 was that legacy
gateway-only agents (e.g. `main`, orphans from rolled-back creates) had
no harness record, so they could never migrate to the harness path
without breaking chat.

`AgentHarnessService.reconcileWithGateway()` now lists every gateway
agent and upserts a matching harness record for any that are missing.
The pass runs lazily on first `listAgents()` call (memoized on success,
retried on failure so a gateway-down boot doesn't permanently disable
backfill). Verified end-to-end: the legacy `agent` agent now streams
`text_delta` + `done(end_turn)` through `/agents/agent/chat`, with the
bridge resolving to the gateway's `agent` record via the existing
`agent:<name>:main` session-key format.

After this, every OpenClaw agent surfaces as `source='agent-harness'`
post-dedup, the legacy `useClawChatHistory` hook becomes unreachable
for OpenClaw, and Steps 11–13 (delete legacy chat/history paths) are
unblocked.

* fix(agents): drop duplicate OpenClaw entry from NewAgentDialog adapter list

The adapter Select hardcoded an `<SelectItem value="openclaw">OpenClaw</SelectItem>`
on top of iterating `adapters`, which now includes OpenClaw post the
catalog change. The dropdown rendered "OpenClaw" twice — once at the
top, once at the bottom of the list. The literal was a pre-catalog
artifact; removing it leaves a single OpenClaw entry sourced from the
catalog. Routing into `handleOpenClawCreate` is unchanged because
the value (`'openclaw'`) is identical either way.

* fix(agents): always reconcile harness with gateway on list, just dedupe concurrent calls

Memoizing the first successful reconcile meant new gateway agents (created
via the legacy /claw/agents path or out-of-band CLI) never appeared in the
harness until server restart. The Promise now serves as a concurrent-call
dedupe only — cleared on settle — so every listAgents call picks up the
current gateway state. Reconcile is one cheap idempotent CLI call.

* chore(agents): remove dormant useAcpxForOpenClaw flag

The flag was scaffolded in Step 6 but its routing effect was walked
back the same day after it broke chat for legacy gateway-only agents.
After the Step 9 backfill, every OpenClaw agent has a harness record
and routes through the harness path purely from `source='agent-harness'`
— no flag is consulted anywhere. Remove the dead storage item, hook,
and stale comment.

* refactor(agents): drop legacy /claw/agents/:id/history endpoint

The harness /agents/:id/sessions/main/history endpoint replaced this
once every OpenClaw agent got a harness record (Step 9 backfill).
Routing is fully source-driven now, so the UI's useClawChatHistory
hook is never enabled today — verified live: legacy URL returns 404,
harness history hydrates correctly for the same agent.

Removes the GET /claw/agents/:id/history route, OpenClawService's
getAgentHistoryPage method plus its cursor/limit helpers and the
history-only types it owned (BrowserOSOpenClawHistoryPageResponse,
HistoryPageInput, normalizeHistoryLimit, encodeHistoryCursor,
decodeHistoryCursor, jsonlEventsToHistoryItems), and the route +
service tests that covered the dropped endpoint.

OpenClawJsonlReader stays alive — still feeds /claw/dashboard,
/claw/agents/:id/sessions, and the boot-time clawSession seed.
Removing those is its own follow-up since the dashboard would need
a harness-side replacement first.

* feat(agents): wire image attachments through the harness ACP path

Composer attachments now flow into the ACP `prompt` request as
spec-compliant `image` content blocks alongside the user's text. End
to end:

  composer → chatWithHarnessAgent({attachments}) →
  POST /agents/:id/chat {message, attachments} →
  parseChatBody decodes data: URLs to {mediaType, base64} →
  AgentHarnessService.send forwards →
  AcpxRuntime.send forwards →
  acpx startTurn({attachments}) → ACP image blocks

UI no longer disables the attach button on harness agents — the
gating was just a placeholder before this commit landed. Verified
end to end with a 1×1 red PNG against a Claude harness agent: model
replies "Red." correctly.

OpenClaw's `acp` bridge still drops image content blocks upstream
(verified by the same probe — Kimi-k2p5 reports "I don't see an
image"). That's an upstream openclaw limitation, not a harness-side
gap; Claude/Codex agents work as advertised today.

* chore(openclaw): delete OpenClawJsonlReader and JSONL-backed routes

* chore(openclaw): remove legacy /claw/agents/:id/chat and /queue routes

* chore(agents): collapse chat panel to harness-only path

* feat(agents): route OpenClaw image turns through the gateway HTTP client

The OpenClaw `acp` bridge silently drops ACP `image` content blocks
(verified during dogfood — model says "I don't see an image"). When
the user attaches images to an OpenClaw agent, the harness now diverts
that turn to the gateway's HTTP `/v1/chat/completions` endpoint, which
accepts OpenAI-style `image_url` parts and forwards them natively to
the provider.

  - New `OpenClawGatewayChatClient` translates an OpenAI streaming
    response into the same `AgentStreamEvent` shape the rest of the
    harness already consumes, so the chat panel renders identically
    whether a turn went through ACP or the gateway carve-out.
  - `AcpxRuntime.send` forks at the top: openclaw + any image
    attachment + a wired gateway client → `sendOpenclawViaGateway`.
    Other turns (text-only openclaw, claude, codex) take the existing
    ACP path unchanged.
  - The diverted path reads the prior turn history from the acpx
    session record so context is preserved, builds the OpenAI
    multimodal user message with text + image_url parts, and pumps
    the gateway SSE back to the caller through a tee that accumulates
    the assistant text. On natural completion, persists a synthetic
    user+assistant message pair to the acpx session record so reload
    shows the image turn in history.
  - Wired `OpenClawGatewayChatClient` into `AgentHarnessService` via
    `server.ts` (gateway port + token accessor, just like the existing
    `openclawGateway`).

Persistence note: the acpx record requires User messages to carry an
`id` and Agent messages to carry `tool_results` — without them the
record fails to round-trip through `parseSessionRecord`. The persist
helper now sets both.

Limitation by design: image recognition only works if the OpenClaw
agent's provider supports vision (e.g. Claude-via-OpenClaw, GPT-4o).
The pipeline routes images correctly to the provider regardless;
text-only providers like Kimi-k2p5 will reply "I don't see an image"
because the model itself has no vision capability — that's a provider
config issue, not a routing bug. The unit test asserts the image_url
part is present in the OpenAI request the gateway client sends.

The wider plan (background-resilient chat, queue, replay) remains in
`plans/.../2026-04-29-1527-...-background-resilient-chat-and-image-uploads.md`
as Phases 3–12; this commit ships only Phases 1–2.

* feat(agents): validate inbound image attachments on /agents/:id/chat

The harness chat body parser was accepting any mediaType and any
dataUrl length. The composer enforces these caps client-side but the
endpoint also serves direct curl/script callers, so the server has to
defend itself.

Restores the same caps the legacy /claw/agents/:id/chat parser had
before it was deleted in the migration:

  - 10 attachments per message
  - 5 MB raw image bytes (≈ 6.7 MB once base64-encoded plus prefix)
  - PNG / JPEG / WebP / GIF only
  - Must start with `data:`

Each violation returns 400 with a specific error message instead of
silently dropping or forwarding the payload.
2026-04-29 16:37:03 +05:30
Nikhil
2ff5c12840 feat: add sidepanel ACP chat targets (#857)
* feat(agent): add sidepanel chat target catalog

* feat(agent): show acp models in sidepanel selector

* feat(server): adapt acp events to ui message streams

* feat(server): add sidepanel acp chat route

* feat(agent): route sidepanel chat through acp targets

* chore: self-review fixes

* fix: address review feedback for PR #857
2026-04-28 18:23:38 -07:00
Nikhil
d87422eea1 fix: hide BrowserOS ACP wrapper in history (#856) 2026-04-28 17:31:11 -07:00
Nikhil
1946ca0cf8 chore: clean up unused agent sdk (#855) 2026-04-28 17:21:46 -07:00
Nikhil
754f7d0e1d test: cover terminal limactl resolver errors (#854) 2026-04-28 17:12:08 -07:00
Nikhil
85bb3f7b42 fix: avoid eager limactl resolution in server tests (#853) 2026-04-28 16:56:41 -07:00
Nikhil
cb32b8191d fix: show rich ACP harness history from ACPX (#852)
* fix: load ACP harness history from ACPX

* fix: address ACP history review comments
2026-04-28 16:40:22 -07:00
Nikhil
7a92654abc feat: add BrowserOS MCP to ACP agents (#851)
* feat: add BrowserOS MCP to ACP agents

* fix: bypass ACP agent permissions

* fix: address review feedback for PR #851
2026-04-28 16:30:20 -07:00
Nikhil
91d3285aa0 feat: add ACP agent harness (#849)
* feat: add acp agent runtime spike

* feat: add agent harness catalog

* feat: persist harness agents in json

* feat: persist agent transcripts

* feat: route harness service through agent records

* feat: expose generic agent harness routes

* feat: add harness agent frontend api

* feat: create harness agents from agents page

* feat: chat with persisted harness agents

* chore: remove obsolete agent profile spike

* chore: self-review fixes

* fix: combine openclaw and harness agents UI

* refactor: split agents page components

* fix: hide persisted harness turns
2026-04-28 15:29:38 -07:00
Nikhil
7bb6dac949 fix(dogfood): copy extension state into dev profile (#850)
* fix(dogfood): copy extension state into dev profile

* fix(dogfood): address profile import review feedback

* fix(dogfood): clarify refresh profile in-use error
2026-04-28 15:25:38 -07:00
shivammittal274
d9c254053e refactor(eval): drop unused agents/graders, collapse registries (#847)
* refactor(eval): drop unused agents/graders, collapse registries

Sweep of dead code in the eval app: deleted gemini-computer-use and
yutori-navigator agents, fara/webvoyager/mind2web graders, eight
debug/analyze/test scripts, three stale planning docs, and the orphaned
eval-targets/coordinate-click testbed.

With two agents and three graders left, the Map-backed plugin registries
were over-engineered — collapsed both into plain switches. Removed the
now-dead GraderOptions plumbing (no remaining grader takes API keys),
dropped grader_api_key_env/grader_base_url/grader_model from the schema
and configs, and de-duped PASS_FAIL_GRADER_ORDER (was defined in three
places). Replaced the URL-parsing extractCdpPort hack in single-agent
and orchestrator-executor with workerIndex passed cleanly through
AgentContext.

README and --help text rewritten to match reality. Renamed
configs/test_*.json to test-*.json for kebab-case consistency.

Net: ~10,460 LOC removed across 60 files. Typecheck clean, all tests
pass.

* ci(eval): pull BrowserOS from rolling stable CDN URL

The pinned v0.44.0.1 .deb on GitHub releases regressed on Linux —
servers start but never become healthy. Switch to the canonical rolling
URL at cdn.browseros.com/download/BrowserOS.deb so CI tracks the same
stable channel users get from the marketing site.
2026-04-29 02:14:47 +05:30
Nikhil
6b9945f933 feat(dev): use dev dock icon for browser launches (#848) 2026-04-28 13:28:19 -07:00
Dani Akash
6a5a7775a9 fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config (#846)
When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.

Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
  side (useOpenClaw.ts) and the route body schema + SetupInput +
  createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
  LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
  `input: ['text', 'image']` on the model entry when the flag is
  truthy; otherwise emits the explicit `['text']` so the value is
  always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
  bootstrap batch so the gateway's image-understanding pipeline is
  always wired up — per-model `input` still gates which models see
  images, this just enables the global path.

ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.

Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.
2026-04-29 00:23:45 +05:30
shivammittal274
af48a2110c feat(eval): Phase 1 — exclude broken tasks, freshen card dates, add grader leniency (#841)
* fix(eval): exclude broken tasks + freshen expired card dates

Two AGISDK tasks are unsolvable today for non-model reasons:

- topwork-1: evals-topwork.vercel.app throws Minified React error #185
  ("Maximum update depth exceeded") on every form submit. The page renders
  "Application error: a client-side exception has occurred" instead of saving.
  Whole-task failure, every model affected.

- fly-unified-2: hardcodes Exp: 12/25 in both the goal text AND a jmespath
  grader criterion. Today is 2026-04, so the eval-site rejects the card.
  Freshening the goal alone leaves the grader expecting the original value;
  freshening both would require monkey-patching agisdk's TaskConfig at
  runtime — too fragile to maintain.

Adds these to a new EXCLUDED_TASKS set alongside the existing
EXCLUDED_WEBSITES (omnizon).

Also adds freshen_goal_dates(): for AGISDK fly-unified tasks whose goal
contains an `Exp: MM/YY` within 6 months of today (or past), rewrites it
to a far-future date (12/30). This rescues fly-unified-5 (had Exp 12/25,
no card-exp grader criterion) and protects fly-unified-4 (had Exp 06/26,
2 months from expiring) from the next eval run hitting the same trap.

Dataset goes from 47 -> 45 tasks; 2 freshened.

* feat(eval): add lenient-strings grader softening

The agisdk grader compares jmespath-extracted values via strict equality.
For tasks where the model adds harmless decoration to a free-text field
(e.g. topwork-3 expects title "Full-Stack Developer" but model produces
"Full-Stack Developer - Enterprise Microservices Platform"), this fails
every other criterion would pass.

Adds a substring fallback in the wrapper: a failed criterion is re-marked
as a softened pass when both actual_value and expected_value are strings
and the (stripped, lower-cased) expected_value is contained in the
actual_value. Numbers/bools/dates/None stay strict.

- Default-on. Set AGISDK_STRICT_STRINGS=1 to recover the strict score.
- Softened criteria are tagged with `softened: true` in per_criterion
  output for transparency in run manifests.
- Aggregate `pass`/`reward` are recomputed after softening.

Expected to rescue 4 tasks in our 45-set: topwork-3, topwork-4 (both pure
title-decoration), gomail-8 (grader contradicts goal), and networkin-6
(grader hardcodes profile id).

* fix(eval): exclude 5 more tasks where pipeline (not agent) fails

Extends EXCLUDED_TASKS to 7 entries based on the K2.5 + Opus 4.6
head-to-head deep-dive on the 2026-04-28 runs. The exclusion rule:
remove a task only if it is unsolvable for any agent — either the task
data is invalid, the eval site is broken, or the grader penalizes
correct work. Tasks that fail because of our agent's tool fidelity
(drag, custom-widget fill, click on React submit, etc.) STAY in — those
are real capability gaps the team should see in the score.

New exclusions:

- fly-unified-9: goal references "Dec 18 2024 at 10:00" but the live
  eval site has only 2025 inventory and no 10:00 slot. Both models
  successfully booked the closest available flight and were penalized
  on a grader expectation that can never be met.

- fly-unified-4: eval site stores wall-clock flight times as bare UTC
  (T08:00:00.000Z) while the grader expects them shifted by 8h
  (T16:00:00.000Z = 8 AM PST). Opus 4.6 completed the entire booking
  correctly. Eval-site TZ-storage bug.

- gomail-8: goal says "Clear all emails from GitHub in the inbox", but
  criterion 3 expects exactly 1 email updated. Both K2.5 and Opus
  correctly cleared all 4 GitHub emails. Grader contradicts goal.

- networkin-6: goal says "Choose a random person you haven't connected
  with"; grader hardcodes profilesDiff.updated."4".connectionGrade.
  Both models randomized correctly and missed id 4. Grader contradicts
  goal.

- networkin-9: eval site's searchHistoryDiff doesn't record queries
  submitted via the autocomplete + Enter path. Opus 4.6 completed the
  task end-to-end (Stanford alum, connection request, message); only
  failed because the search-history criterion was never written
  server-side. Eval-site bug.

Dataset goes from 45 -> 40 tasks. Score impact (same K2.5/Opus runs,
recomputed against the cleaned 40-task denominator):

  K2.5:     21/45 (46.7%) -> 21/40 (52.5%)
  Opus 4.6: 28/45 (62.2%) -> 28/40 (70.0%)
  Δ:        15.6 pp -> 17.5 pp (real model gap, less pipeline noise)
2026-04-28 23:19:31 +05:30
Nikhil
c5ff8d75bc fix(dogfood): clarify init prompts (#839) 2026-04-28 07:48:42 -07:00
Nikhil
445a6a6c45 fix(dogfood): use alpha dock icon (#837) 2026-04-27 21:47:10 -07:00
Nikhil
72d39b9a0f docs(dogfood): simplify alpha workflow readme (#838) 2026-04-27 21:44:03 -07:00
Nikhil
3b47f330f5 fix(dogfood): separate BrowserOS state root (#836) 2026-04-27 17:38:15 -07:00
Nikhil
15a82ff9cb feat: add dogfood background daemon mode (#833) 2026-04-27 17:15:50 -07:00
Nikhil
427549f081 feat: Add BrowserOS Dock icon variants (#835) 2026-04-27 17:10:36 -07:00
Nikhil
a11f9caa64 fix(dogfood): colorize cli output (#834)
* fix(dogfood): colorize cli output

* fix: address dogfood cli review comments
2026-04-27 16:29:25 -07:00
Nikhil
da1397900b refactor: rename internal BrowserOS CLIs (#832)
* refactor: rename internal BrowserOS CLIs

* fix: update dogfood binary gitignore
2026-04-27 16:18:45 -07:00
Nikhil
368c7dcfe8 fix(alpha): write balpha process logs (#830)
* fix(alpha): write balpha process logs

* fix(alpha): address log review feedback
2026-04-27 15:48:40 -07:00
Nikhil
599f8b6b9c fix: address balpha CLI dogfooding feedback (#831) 2026-04-27 15:43:22 -07:00
Nikhil
27834b1d31 fix: udpate readme (#829) 2026-04-27 15:27:16 -07:00
Nikhil
aa30eb3aaa feat: add balpha dogfooding CLI (#828)
* feat(alpha): scaffold balpha cli

* fix(alpha): address scaffold review

* feat(alpha): add balpha config

* feat(alpha): parse browseros profiles

* feat(alpha): import browseros profile

* feat(alpha): add browser launch helpers

* feat(alpha): add repo build and env pipeline

* feat(alpha): add process supervision

* feat(alpha): add balpha commands

* docs(alpha): document balpha setup

* fix(alpha): reuse dev setup script

* fix(alpha): address review feedback

* fix(alpha): normalize imported browser profile

* fix(alpha): use generic profile fixture names
2026-04-27 15:03:37 -07:00
shivammittal274
e045e34b73 fix(eval): switch weekly eval configs from Fireworks to OpenRouter (#827)
The 2026-04-23 weekly run had 42% of AGISDK and 46% of Infinity tasks
fail with `AI_RetryError: ... the service is overloaded` from Fireworks
(20 concurrent kimi-k2p5 streams across both runs at 10 workers each).

Switching to OpenRouter (which fronts the same Moonshot K2.5 weights
and falls back across providers) for the three weekly configs:
- browseros-agent-weekly.json
- agisdk-real-smoke.json
- infinity-hard-50.json

Model accounts/fireworks/models/kimi-k2p5 -> moonshotai/kimi-k2.5
(same weights, same 262K context). API key env var, base URL updated.

OPENROUTER_API_KEY is already wired into .github/workflows/eval-weekly.yml
and present in repo secrets — no GH config changes needed.

Orchestrator-executor configs and test_webvoyager left on Fireworks
intentionally; can switch later if needed.
2026-04-27 21:52:26 +05:30
shivammittal274
01d649da9a feat(eval): bring deterministic graders to dev + drop omnizon (#824)
* feat: deterministic eval graders (AGI SDK + WebArena-Infinity) (#664)

* feat: add deterministic eval graders (AGI SDK + WebArena-Infinity)

Two new benchmark integrations with programmatic grading — no LLM judge.

AGI SDK / REAL Bench (52 tasks):
- 11 React/Next.js clones of consumer apps (DoorDash, Amazon, Gmail, etc.)
- Grader navigates browser to /finish, extracts state diff from <pre> tag
- Python verifier checks exact values via jmespath queries

WebArena-Infinity (50 hard tasks):
- 13 LLM-generated SaaS clones (Gmail, GitLab, Linear, Figma, etc.)
- InfinityAppManager starts fresh app server per task per worker
- Python verifier calls /api/state and asserts on JSON state

Infrastructure:
- GraderInput extended with mcpUrl + infinityAppUrl for parallel workers
- Each worker gets isolated ports (no cross-worker state contamination)
- CI workflow: pip install agisdk, clone webarena-infinity repo

* chore: switch eval configs back to kimi-k2p5

* fix: register deterministic graders in pass rate calculation

Add agisdk_state_diff and infinity_state to PASS_FAIL_GRADER_ORDER
in both runner types and weekly report script, so scores show correctly
in the dashboard.

* chore: temp switch to opus 4.6 for eval run

* chore: restore kimi-k2p5 as default eval config

* ci: add timeout and continue-on-error for trend report step

* fix(eval): drop omnizon from AGISDK dataset (DMCA takedown)

evals-omnizon.vercel.app returns HTTP 451 ("This content has been
blocked for legal reasons / DMCA_TAKEDOWN"). All 5 omnizon-* tasks
fail grading with "Failed to fetch /finish endpoint: JSON Parse error".

Adds an EXCLUDED_WEBSITES set to the dataset builder and regenerates
agisdk-real.jsonl (52 → 47 tasks).

* fix(eval): correct Infinity port-assignment bugs

Two related bugs in the Infinity eval runner that cause silent port
collisions / fallbacks under parallel execution:

1. build-infinity-dataset.py emitted "app_port" but task-executor and
   the committed JSONL both read "app_base_port". Re-running the build
   script would silently make every task fall back to the 8000 default,
   ignoring per-app port assignments. Renamed the key to match.

2. task-executor derived workerIndex as `base_server_port - 9110`, but
   parallel-executor doesn't override base_server_port per worker —
   only server_url. Every worker computed workerIndex = 0, causing all
   parallel workers to spawn Infinity app servers on the same port.
   Threading workerIndex explicitly through TaskExecutor instead.

Also drops an unused app_name parameter from load_tasks().
2026-04-27 21:35:43 +05:30
Dani Akash
ddbb2cf492 feat(agent): composer attachments + server-side outbound message queue (#826)
* feat(agent): attach images and text files to chat messages

Adds end-to-end support for image and text file attachments in the chat
composer, with the staged files round-tripping through the OpenClaw
gateway as OpenAI-compatible content blocks and persisting in the JSONL
so they show up in the historical view.

Server
- HTTP client: new OpenClawChatContentPart union and a buildUserContent
  helper that emits multimodal content arrays when messageParts is
  supplied, falls back to the legacy string content otherwise.
- Service: chatStream takes an optional messageParts array and forwards
  it; BrowserOSChatHistoryItem gains an attachments field.
- JSONL reader: PiContentBlock learns the OpenAI image_url and Anthropic
  image source/data shapes; user messages now emit user.attachment
  events that the history mapper accumulates onto the next user item.
- Route: validates an inbound attachments[] (kind/mime/size/count),
  inlines text-shaped files as <attachment> blocks in the message body,
  attaches images via image_url parts. Replaces the immediate 409 on
  active monitoring session with a 30s waitForSessionFree(agentId) wait
  (registry now exposes onSessionEnd) so cron/hook contention does not
  reject a user-chat send outright. Returns 503 if the wait times out.

Client
- New lib/attachments.ts: validateAttachment / compressImageIfNeeded
  (canvas downscale to 2048px long edge, JPEG 0.85 re-encode for >1.5
  MB inputs) / stageAttachment / stageAttachments that produces the
  staged-attachment shape the composer renders and the payload the
  server accepts.
- ConversationInput: drag-and-drop, paperclip button, clipboard paste,
  staged attachment chip strip with thumbnails for images and a
  paperclip+name chip for text files. Send button enables on either
  text or attachments. Drop-zone overlay during drag.
- chatWithAgent forwards attachments[]; useAgentConversation.send
  accepts a SendInput shape and renders user attachments on the
  optimistic streaming turn via MessageAttachments / MessageAttachment.
- ClawChatMessage groups historical attachment parts into a single
  MessageAttachments strip, ordered before reasoning/tools/text.
- claw-chat-types adds an attachment ClawChatMessagePart variant; the
  history mapper emits attachment parts first and skips the text part
  when the user only sent media.
- AgentCommandHome forwards the new SendInput shape — home composer
  drops attachments at the boundary in v1 (the conversation page is
  where staging is most useful; carrying bytes through the URL bar
  is not sensible).

Limits: 10 attachments per message, 5 MB per image (post compression),
1 MB per text file, mime types png/jpeg/webp/gif and text/* +
application/json. PDFs and other binaries are deferred to v2.

* feat(agent): outbound message queue for chats while agent is mid-turn

Lets users keep typing and submitting messages while the agent is still
streaming a previous turn. Each press is appended to a single-flight
queue and dispatched as soon as `streaming` flips false; the queued
state renders as a strip above the composer so the user sees what's
pending vs. what's already sending.

- New `useOutboundQueue` hook owns the queue, the worker effect, and
  cancel/retry actions. Single-flight by design — a re-entrancy ref
  guard prevents two simultaneous dispatches when `streaming` flickers.
- Composer (`ConversationInput`) accepts optional `outboundQueue`,
  `onCancelQueued`, `onRetryQueued` props. When the queue is provided
  the send-button gate stops blocking on `streaming`; the spinner stays
  as the visual cue that the agent is still busy. Legacy direct-send
  callers keep the old streaming-blocks-send semantic.
- Renders an OutboundQueueStrip above the staged-attachment strip with
  per-item status (queued / sending / failed), a cancel button on
  queued items, and retry + discard on failed items.
- AgentCommandConversation wires `onSend` to `queue.enqueue` and routes
  the home composer's `?q=` initial-message handoff through the queue
  too, so it inherits the same single-flight serialization.

The server-side `waitForSessionFree` (added with attachments) and this
client-side queue together cover both contention sources: cron / hook
turns and back-to-back user sends. Persistence across reloads is
intentionally out of scope for v1 — losing the queue on extension
reload is documented as a known limitation.

* feat(server): server-side outbound message queue

Replaces the client-only React-state queue from 123ef21d with a
proper server-owned queue. Closing the tab is now safe — the server
holds queued messages and dispatches them through the existing
chatStream path the moment the agent's ClawSession status flips to
idle.

Server
- New OutboundQueueService (apps/server/src/api/services/queue) — per
  agent FIFO, in-memory. Subscribes to ClawSession.onStateChange
  through OpenClawService.onAgentStatusChange, and dispatches via
  OpenClawService.chatStream so attachments / history / monitoring
  all behave identically to the existing /chat route. The worker
  drains the SSE response server-side so the gateway run finalizes
  cleanly even with no client connected.
- Four new routes under /claw/agents/:id/queue:
  POST   /queue            enqueue
  DELETE /queue/:itemId    cancel a queued item
  POST   /queue/:itemId/retry  re-queue a failed item
  GET    /queue/stream     SSE feed of the per-agent queue state.
  Validation reuses validateChatAttachments and
  buildMessagePartsFromAttachments from the existing chat route.
- Singleton wired in apps/server/src/main.ts; shutdown on SIGTERM.
- New OpenClawService.getAgentState getter for the queue worker's
  pre-dispatch sanity check.

Client
- useOutboundQueue rewritten as an SSE-backed projection over server
  state. Public API unchanged so the composer still works.
- enqueue POSTs to /queue and shows an optimistic local entry until
  the server's SSE snapshot reflects it; local-only entries get a
  `local-` id prefix so cancel can short-circuit them without
  hitting the server.
- AgentCommandConversation watches the queue for sending items
  dropping out and refetches history so the new assistant turn shows
  up in the conversation view (the server worker streams the
  dispatched turn into OpenClaw without exposing per-turn SSE to
  the client).

Out of scope (documented in the plan as v2 follow-ups): disk
persistence (server restart loses queue), per-turn live streaming
of queued sends in the conversation view, and switching the
underlying dispatch from /v1/chat/completions to the chat.send RPC
(which would also fix the multimodal attachment routing problem).

* fix(server): outbound queue must reuse existing session, not spawn UUIDs

The queue worker was generating a fresh randomUUID() as the sessionKey
when the queued item didn't carry one — and the client wasn't sending
one. Result: every queued message kicked off a brand-new OpenClaw
session, orphaning the user's active conversation behind the new
"most recent" entry in sessions.json. The history endpoint then
resolved to the orphan and the chat appeared to disappear.

Fix is layered:
- Client (useOutboundQueue): forward the current resolvedSessionKey
  in the POST /queue body so every queued message targets the same
  conversation the user is viewing. AgentCommandConversation passes
  resolvedSessionKey into the hook.
- Server (OutboundQueueService): the worker now resolves to the
  agent's existing user-chat session when no sessionKey is provided
  on the queued item, via OpenClawService.resolveAgentSession. UUID
  fallback is now reserved for the first-ever message on a brand
  new agent — same semantic the existing /chat route has implicitly
  through the catalog of historical sessions.

No JSONL data was lost by the original bug (the prior conversations
are intact on disk); the orphan sessions just shadowed the original
in sessions.json.

* fix(agent,server): address PR review feedback for chat queue

- Tighten image data URL cap to base64-aware ~6.7 MB (was ~7.5 MB
  through `MAX_IMAGE_BYTES * 2`).
- Forward chat history from useOutboundQueue.enqueue so queued sends
  preserve conversation context like direct sends do.
- Match local attachment previews to server snapshots by id (not by
  message text), and prune the preview map as items drain.
- Pass an AbortSignal into chatStream so a queue shutdown cancels the
  initial OpenClaw handshake, not just the SSE drain loop.
- Track previously gitignored apps/agent/lib/attachments.ts (was caught
  by global lib/ ignore) so CI typecheck can resolve @/lib/attachments.
- Update server-api openclaw route tests to the new chatStream signature
  and the waitForSessionFree-based busy-agent path.

* fix(agent): dedupe optimistic queue entries for text-only sends

The localId↔serverId map was only populated when the message had
attachments, so plain-text sends left the optimistic local entry in
place after the server snapshot arrived — the user saw the same
message rendered twice in the queue strip.

* fix(agent): prune optimistic queue entry on POST ack, not just SSE

The server broadcasts the new queue snapshot before its POST response
returns, so the SSE handler often runs first — at that point the
localId↔serverId map has no entry for the new server id yet, so the
SSE-based dedupe path can't drop the optimistic local entry. Pruning
on POST success closes the race deterministically.

* fix(agent): hand off optimistic queue entry without a render gap

Pruning the local entry on POST success only worked when the SSE
snapshot had already overwritten it; if the POST response landed
first, the optimistic row disappeared for a frame before the SSE
snapshot brought back the server-keyed row, producing a visible
flicker. Gate the POST-side prune on the SSE snapshot already
carrying the server id, and rely on the SSE-based dedupe (now
guaranteed to find the localId↔serverId link in the map) to clean
up when SSE arrives later.

* fix(agent,server): client-generated queue id eliminates render flicker

The server used to assign its own UUID when an item was enqueued, so
the optimistic client row carried a `local-` id while the SSE snapshot
carried a server UUID — the client had to wait for the POST response
to learn the mapping before it could dedupe, and during that window
both rows rendered.

Now the browser generates the id, sends it in the POST body, and the
server uses it verbatim (falling back to a fresh UUID only if the id
collides with an existing item). The client collapses to a single
id-keyed list, so the optimistic row and the SSE row reconcile on the
same key from the very first render.
2026-04-27 21:31:03 +05:30
Dani Akash
711934555d feat(agent): enrich chat UI with tool activity, reasoning duration, and cost (#825)
* feat: pass per-turn cost and token data through chat history items

- Add costUsd, tokensIn, tokensOut to BrowserOSChatHistoryItem (server)
- Pass through from JSONL agent.message events in jsonlEventsToHistoryItems()
- Add same fields to client-side BrowserOSChatHistoryItem and ClawChatMessage
- Map cost/token data in mapHistoryItemToClawMessage()

Data flows: JSONL message.usage → server history item → API response →
client ClawChatMessage. Available for rendering in ClawChatMessage
component (message toolbar, cost badges).

* feat: add message toolbar with copy button and per-turn cost display

Add MessageToolbar to historical assistant messages in ClawChatMessage:
- Copy button copies message text to clipboard via MessageAction
- Per-turn token count (22.7K → 238) and cost ($0.003) shown as muted
  tabular-nums text on the right side of the toolbar
- Toolbar appears on hover (opacity transition via group-hover)
- Only shown when the message has text content
- Cost/token display only shown when data is available from JSONL

* fix: toolbar only on assistant messages, always visible, cost only

- Only render toolbar on assistant messages (not user messages)
- Remove hover-only opacity — toolbar is always visible
- Remove token counts (22.7K → 238 is meaningless to users)
- Show only cost as a budget signal ($0.003)

* feat: group all tool activity into single Task collapsible per turn

Replace flat tool rows with a single ai-elements Task collapsible per
assistant turn that lists every tool/MCP call in sequence.

Live streaming (ConversationMessage):
- Aggregate all tool-batch parts into one Task
- Title: "Working… (N actions)" while running, "Agent activity (N actions)" when done
- Default open while turn is in progress
- Wrench icon in trigger

Historical (ClawChatMessage):
- Group all tool-call parts into one Task
- Title includes failed count if any tools errored
- Default collapsed — expandable on click
- Tool name + status icon + error text per row

Both views show one clean collapsible per turn instead of N individual
tool cards. Collapsed reads "5 actions"; expanded shows the timeline.

* feat: include tool calls in chat history responses

Server: jsonlEventsToHistoryItems() now walks ALL events (not just
messages) and pairs agent.tool_use with agent.tool_result by toolCallId.
The resulting tool call list is attached to the next assistant text
message as toolCalls[]. Each entry includes status, input arguments,
output text, error string, and duration computed from event timestamps.

Client:
- BrowserOSChatHistoryItem gets optional toolCalls field
- Tool-call message part type gets durationMs field
- mapHistoryItemToClawMessage() emits tool-call parts BEFORE the text
  part (the order the agent produced them)
- ClawChatMessage Task view now shows tool duration in seconds

Result: historical messages now display the full tool activity
timeline grouped into the single Task collapsible per turn (designed
in step 3), instead of showing only the final text response.

* feat: render activity rows as human verbs sourced from tool registry

Tool calls in the chat activity view now read as sentences:
"Opened tab · news.ycombinator.com" instead of "browseros__new_page".

Server (tool-label-registry.ts):
- Curated verb override map for ~70 BrowserOS first-party tools
- Per-tool subject extractors that pull the meaningful argument from
  input (URL → host, query → quoted, element → ID, etc.)
- Generic fallback humanizes snake_case for any unmapped tool
- Strips MCP namespace prefixes (browseros__, mcp_)

Server (openclaw-service.ts):
- jsonlEventsToHistoryItems calls buildToolLabel for each tool_use,
  attaches label and subject to the BrowserOSChatHistoryToolCall

Client:
- Mirrored label module at lib/tool-labels.ts
- useAgentConversation tool-start handler computes label/subject
  from the SSE tool args
- ClawChatMessage and ConversationMessage render label · subject
  with foreground/muted styling, no font-mono
- ToolEntry, BrowserOSChatHistoryToolCall, and tool-call message
  part types all carry label and optional subject

* fix: drop meaningless tab N subject from page-read tool rows

Page IDs are internal numbers, not URLs. 'Took screenshot · tab 4'
tells the user nothing. Removed subject extractors for take_snapshot,
take_enhanced_snapshot, get_page_content, get_page_links, get_dom,
and take_screenshot. The verb alone is the right signal.

* fix: gate initial loading on historyQuery.isFetched not isLoading

The session and history queries are sequential: the history query is
disabled until session resolves. After session resolves, there's a render
frame where historyQuery.isLoading is still false (the query hasn't
been kicked off yet). isInitialLoading flipped to false during that
window, exposing an empty chat shell with just Task collapsibles and
copy buttons before the messages filled in.

Switching the guard to isFetched closes that window — the loading state
stays true until the first history fetch actually completes.

* fix: render historical messages immediately instead of through Streamdown's idle-callback debounce

Streamdown defaults to mode="streaming" which uses requestIdleCallback (300ms
debounce, 500ms idle timeout) and lazy/Suspense to optimize for token-by-token
live streams. For finalized historical messages this caused tool collapsibles
and copy buttons to paint while text bodies stayed blank for ~300-500ms after
load. Pass mode="static" + parseIncompleteMarkdown=false on the historical
MessageResponse so completed text paints in the same frame as the surrounding
chrome. Live streaming turns still use the default streaming mode.

Also collapse the redundant /agents/:id/session round-trip into the existing
/history endpoint (server already resolves the most recent user-chat session
when sessionKey is omitted) and tighten the initial-loading gate to stay true
across the render frame where the query is enabled but hasn't started fetching.

* feat: surface thinking duration on historical reasoning collapsibles

Server accumulates agent.thinking events per turn from JSONL and attaches a
single reasoning block (joined text + durationMs from first thinking event
to the closing agent.message) on each assistant history item. Reasoning
buffer resets on user.message alongside the tool-call buffer.

Client mirrors the type, emits the reasoning part before tool calls in
mapHistoryItemToClawMessage (chronological: think → act → answer), and
passes duration in seconds to <Reasoning> so the trigger reads "Thought
for N seconds" instead of just "Thinking" on collapsed historical turns.

* fix: read thinking blocks from the correct JSONL field name

OpenClaw stores reasoning blocks as {type:'thinking', thinking:'...'} but
the JSONL parser was reading block.text, so every thinking event was
silently dropped before it ever reached jsonlEventsToHistoryItems. As a
result the reasoning field on history items was always empty even though
the new accumulator was wired up correctly.

Also guard the client mapping: when durationMs is 0 (think + answer
emitted in the same JSONL line, no real elapsed wall-clock) pass
undefined to <Reasoning> so it renders the static "Thinking" trigger
instead of the streaming shimmer / "Thought for 0 seconds".

* fix: reset reasoning buffer on discarded turns and drop dead session hook

Two cleanups from PR review:

1. jsonlEventsToHistoryItems: when an agent.message is discarded (the
   "[Chat messages since your last reply" wrapper without a current-message
   marker) the tool buffers were already reset but the reasoning buffer
   was not. Accumulated thinking from the discarded turn would bleed onto
   the next assistant message. Reset pendingReasoningTexts and
   pendingReasoningFirstAt alongside the tool buffers.

2. useClawAgentSession, the AgentSessionResponse type, and the unused
   session entry in CLAW_CHAT_QUERY_KEYS became dead code after the
   session round-trip was folded into the history endpoint. Removed.
2026-04-27 18:29:15 +05:30
Nikhil
5125dffbf3 fix: sign limactl with VZ entitlement (#822) 2026-04-26 13:30:09 -07:00
Dani Akash
0035893f33 feat: dashboard API, JSONL reader, and OpenClaw observer for enriched home page (#810)
* feat: draft agent chat ui exploration

* feat: refine agent chat ui draft

* feat: remove outer frame from agent chat workspace

* fix: offset agent chat for app sidebar

* fix: simplify agent conversation shell

* fix: remove redundant chat header actions

* fix: unify agent conversation headers

* fix: tighten agent chat spacing

* fix: bound agent chat composer height

* fix: remove agent chat page inset

* fix: align agent header height with sidepanel

* fix: center agent composer resting state

* fix: anchor multiline composer controls

* fix: remove focus grid from agent home

* fix: remove redundant agent home header

* fix: constrain home agent composer

* fix: match home composer default posture

* feat: add openclaw chat history APIs

* feat: add claw chat history hydration

* fix: stabilize claw chat viewport layout

* fix: use conversation scroll base for claw chat

* refactor: split claw chat controller responsibilities

* fix: keep active agent turns in memory

* fix: normalize openclaw chat sessions

* refactor: use HTTP client for agent history instead of CLI client

Replace the CLI-based getChatHistory() call in getAgentHistoryPage()
with the HTTP client's getSessionHistory() from PR #795. This uses
the direct HTTP transport to OpenClaw's /sessions/<key>/history
endpoint instead of shelling out through the CLI.

- Add filterHttpSessionHistoryMessages() for flat-string content format
- Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape
- Update getAgentHistoryPage() to call getSessionHistory() via httpClient
- Remove unused getChatHistory(), filterOpenClawSystemMessages(),
  normalizeChatHistoryMessages(), and getTextContent()
- Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory
- Update MutableOpenClawService type: chatClient -> httpClient

* fix: fetch all session messages by iterating OpenClaw pagination

OpenClaw's HTTP history endpoint returns a limited page by default.
When called without a limit, only the first ~27 messages were returned,
causing all newer conversation messages to be silently dropped.

Add fetchAllSessionMessages() that iterates through OpenClaw's cursor-
based pagination (200 messages per page) until hasMore is false, then
feeds the complete message list into the existing BrowserOS normalization
and in-memory pagination layer.

* refactor: migrate chat history from HTTP gateway to direct JSONL file reads

Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw
gateway /sessions/:key/history pagination loop) with direct JSONL file reads
from the host filesystem via Lima's virtiofs mount.

- Add OpenClawJsonlReader that reads session JSONL files directly from
  ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/
- Replace fetchAllSessionMessages() HTTP pagination with single file read
- Replace CLI-based listSessions() with sessions.json file reads
- Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous
- Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages,
  normalizeHttpHistoryMessages helpers
- Update route handlers to drop unnecessary async/await
- Update tests to use temp JSONL files instead of mocked HTTP/CLI clients

* fix: restore async route handlers for test compatibility with mocked service

* fix: address review feedback — path traversal guard, lazy reader, exists flag

- Add safePath() to OpenClawJsonlReader that validates resolved paths stay
  within stateRoot, preventing path traversal via crafted agentId values
- Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients)
  instead of creating a new instance per property access
- Return exists: false from resolveSpecificAgentSession when no session
  matches instead of fabricating a ghost session with sessionId: ''

* feat: add dashboard API and enrich home page agent cards

Server:
- Add summarizeToolActivity() that converts tool events into natural
  language descriptions ("Browsed 3 pages, took 2 screenshots")
- Add getDashboard() to OpenClawService that aggregates per-agent stats
  from JSONL: latest message, activity summary, cost, session count
- Add GET /claw/dashboard endpoint

Client:
- Add useAgentDashboard() React Query hook (10s refetch, 5s stale)
- Rewrite useAgentCardData from async IndexedDB hook to pure
  buildAgentCardData() function merging agent entries with dashboard data
- Add activity summary and cost to AgentCardExpanded footer
- Add activitySummary and costUsd fields to AgentCardData type
- Remove IndexedDB dependency from the home page

* feat: add OpenClawObserver for real-time per-agent status via gateway WS

- Add OpenClawObserver that connects to the OpenClaw gateway WebSocket
  control plane and subscribes to chat broadcast events
- Track per-agent status in real time: working (streaming), idle (turn
  complete), error (run failed), with current tool name
- Auto-connect when gateway control plane becomes available, auto-
  reconnect on disconnect with 5s backoff
- Disconnect observer on stop/shutdown
- Wire live status + currentTool into getDashboard() response
- Update client: AgentOverview includes status + currentTool, card shows
  spinning loader + tool name when agent is working
- Status resolution: per-agent WS status takes precedence over gateway-
  level status for working/error states

* feat: add SSE dashboard stream for real-time agent status on home page

Server:
- Add GET /claw/dashboard/stream SSE endpoint that sends an initial
  snapshot then pushes per-agent status events as they arrive from
  the OpenClaw observer
- Add onAgentStatusChange() to OpenClawService exposing the observer's
  listener for the route layer
- Heartbeat every 15s to keep connections alive

Client:
- useAgentDashboard() now subscribes to EventSource at /claw/dashboard/stream
- SSE snapshot event hydrates the React Query cache immediately
- SSE status events patch individual agent status + currentTool in the
  cache without refetching — agent cards update instantly
- Polling fallback raised to 30s since SSE handles real-time

* fix: observer WS handshake — wait for challenge before sending connect

The OpenClaw gateway sends a connect.challenge event before accepting
the connect request. The observer was sending the connect request on
ws.open which raced with the challenge. Now waits for the challenge
event before sending the handshake.

Also add dangerouslyDisableDeviceAuth to the gateway setup config
batch so the observer can connect without device identity on new
installs.

* fix: JSONL reader falls back to most recent file when sessions.json is stale

OpenClaw's sessions.json can record a Pi session ID that doesn't match
the actual JSONL filename on disk. This happens after context compaction
or session restart — the JSONL file gets a new UUID but sessions.json
keeps the old one.

Previously this caused history to silently disappear (the reader tried
to open a non-existent file and returned empty). Now resolveJsonlPath()
checks if the mapped file exists and, when it doesn't, scans the
sessions directory for the most recently modified .jsonl file as a
fallback.

* feat: add ClawSession state machine for reliable per-agent status

The OpenClawObserver only knows about status changes it witnesses via
WS events. If an agent was already running when the observer connected,
or after a reconnect, statuses were stuck at "unknown".

ClawSession is an in-memory state machine that solves this:

1. Seeds from JSONL on first control plane call — reads the latest
   events for each agent and infers working/idle. A session is "working"
   if the last event is a user.message with no subsequent agent.message,
   or an agent.tool_use with no matching agent.tool_result.

2. Receives live transitions from the WS observer — the observer now
   delegates all state management to ClawSession instead of maintaining
   its own status map.

3. Applies a 5-minute staleness threshold — if the last JSONL event
   is older than 5 minutes, assume idle (handles agent crashes).

Consumers (SSE stream, dashboard endpoint) read from ClawSession and
get correct state from the first call — no "unknown" period.

* fix: remove staleTime so dashboard refetches on every mount

* fix: reset stale working status on WS disconnect, eliminate redundant JSONL reads

- Observer resets all "working" agents to "unknown" when the WS closes,
  preventing agents from appearing stuck as Working indefinitely after
  a gateway restart. ClawSession re-seeds correct state on reconnect.

- getDashboard() now derives latestAgentMessage and cost from the
  already-loaded events array for the latest session instead of calling
  latestAgentMessage() and getSessionStats() which each re-read the
  same JSONL file. Reduces file reads from 3x to 1x per agent.
2026-04-25 19:03:03 +05:30
Neel Gupta
4284e88625 feat: Implement lazy LLM judge for passive monitoring (#777)
* fix: double close on stream controller

* feat: initial lazy llm judge impl

* feat: added regex-based matching to insert button context

* fix: tests & bugfix

fix: redundant truthiness check

* fix(tests): stabilize server suites on dev
2026-04-25 12:52:41 +01:00
Nikhil
0b91c735ab chore: bump server version, offset and patch for release (#814) 2026-04-24 12:05:47 -07:00
Nikhil
d189b50b03 fix: package bundled Lima guest agent (#813)
* fix(build): upload Lima runtime files

* fix(build): stage Lima prefix resources

* fix(vm): resolve bundled Lima prefix

* docs(build): document Lima runtime packaging

* chore: self-review fixes

* fix: address review feedback for PR #813
2026-04-24 12:03:26 -07:00
Nikhil
a407e48209 Prefetch runtime VM cache (#811)
* feat: add runtime vm cache sync

* feat: configure runtime vm cache sync

* feat: prefetch vm cache on startup

* feat: await vm cache before vm startup

* fix: recheck vm cache after prefetch wait

* fix: address vm cache review feedback

* build(server): require VM cache manifest env
2026-04-24 10:41:20 -07:00
shivammittal274
1f75b91fba feat(openclaw): add Claude CLI as a CLI-backed provider (#791)
* feat(openclaw): add Claude CLI as a CLI-backed provider

Extensible registry of "OpenClaw CLI-backed providers" — tools that run
as subprocesses inside the gateway container rather than via an API key.
Claude CLI is the first entry; Gemini CLI / Codex CLI / etc. are
one-line additions in the same shape.

Backend:
- New openclaw-cli-providers/ module: types, registry, claude-cli entry.
- OpenClawService: generic ensureAllCliProvidersInstalled() (runs on
  setup/start/restart/auto-start) and getCliProviderAuthStatus(provider).
- Provider dispatch: resolveProviderForAgent() short-circuits CLI
  providers (no env var, no custom-provider merge) before falling
  through to the API-key resolver. No changes to openclaw-provider-map.
- Container runtime: PATH + NPM_CONFIG_PREFIX env so tools installed
  under /home/node/.npm-global/bin (mounted) are discoverable by
  OpenClaw's child-process spawns and persist across restarts.
- New route: GET /claw/providers/:providerId/auth-status returns
  installed / loggedIn / account / plan / error.

Frontend:
- New openclaw-cli-providers.tsx: mirrors backend registry (id, models,
  authLoginCommand), useOpenClawCliProviderAuthStatus hook (2-s poll
  while enabled), OpenClawCliProviderStatusPanel component.
- AgentsPage: synthesized CLI-provider options merged into the Create
  Agent dropdown, inline status panel, auth modal mounting the existing
  AgentTerminal with provider.authLoginCommand, auto-close on loggedIn.
- AgentTerminal: new optional initialCommand + onSessionExit props
  (ref-based so parent re-renders don't rebuild the PTY).

No global ProviderType changes. No custom container image — runtime
install into the mounted home dir persists across restarts.

* fix(openclaw): address review comments for claude-cli provider

- Drop redundant providerId field from OpenClawCliProviderOption (type
  already carries the same value).
- Reuse SetupInput type in resolveProviderForAgent instead of inlining.
- Split ensureCliProviderInstalled into probe + install so logs
  distinguish "already present" from "freshly installed".
- Narrow union in handleCreate via explicit LlmProviderConfig cast; the
  'in'-based narrowing stopped working once the two option shapes
  overlapped on required fields.

* fix: green up server-api tests after claude-cli additions

- Update container-runtime.test.ts snapshot to include the new
  PATH + NPM_CONFIG_PREFIX env args.
- Add a defensive guard in ensureAllCliProvidersInstalled so test
  mocks that swap runtime for a partial stub without execInContainer
  simply skip the install step; production runtime always provides it.

No production behavior change.

* fix(openclaw): use claude /login for auth flow and render terminal full-page

`claude auth login` in 2.1.x silently discards stdin, so the pasted OAuth
code never reaches claude. Switch to the REPL's `/login` slash command,
which does accept a pasted token. Also render the auth terminal
full-page instead of inside a Radix Dialog — the focus trap was hiding
keyboard events from xterm's helper textarea. Finally, guard the async
WebSocket in AgentTerminal against React 18 StrictMode's double-invoke
so the first mount's orphaned WS doesn't leak a second live session.

- terminal-session: pass PATH on podman exec so user-installed CLIs
  resolve in interactive sessions without manual re-exports.
- claude-cli parseAuthStatus: treat exit-code-1 as a valid "not logged
  in" JSON payload instead of a hard error.

* fix(openclaw): drop unnecessary PATH override on podman exec

`podman exec` inherits the container's run-time env (PATH includes
/home/node/.npm-global/bin via `podman run -e PATH=…`), so the extra
`-e PATH` on the exec call was redundant. Reverts the export of
GATEWAY_PATH and the exec flag added in the previous commit.

* feat(openclaw): show CLI-backed providers in Set Up dialog

The Set Up OpenClaw dialog previously listed only API-key LLM
providers. Add the CLI-backed ones (currently just Claude CLI) so
users can bootstrap the gateway with a Claude.ai-subscription-backed
agent without round-tripping through the Create Agent flow first.

When the user picks a CLI provider at setup, skip the apiKey/baseUrl
fields and open the auth terminal immediately after the gateway comes
up, so /login runs in one click.

* fix(openclaw): robust claude auth-status parsing and cleaner CLI UX

parseClaudeAuthStatus was doing JSON.parse on the entire stdout, which
fails when Lima/nerdctl appends a stderr line like `level=fatal
msg="exec failed with exit code 1"` whenever the inner command exits
non-zero (claude auth status exits 1 when not logged in). The panel
then surfaced the raw output as an error. Switch to a line-by-line
scan that picks the first parseable JSON object — handles trailing
noise and nested JSON fields cleanly.

UI polish around the Setup dialog:
- Hide the "uses your API key" hint when the selected provider is
  CLI-backed — it is inaccurate and confusing.
- When a CLI provider is picked in Setup, show a short helper line
  instead of the status panel (the /auth-status poll would be
  pre-gateway and would always fail). Set Up & Start boots the
  gateway and then auto-opens the auth terminal in one click.
- Track the active CLI provider across both Setup and Create dialogs
  so the auth terminal opens for the right provider regardless of
  which dialog triggered it.

* feat(terminal): make selection + copy work under TUI mouse tracking

Interactive TUIs like `claude /login` enable xterm mouse-tracking,
which forwards every click to the app and disables click-drag text
selection. Our terminal had no escape hatch, so users couldn't grab
the OAuth URL.

Three general-purpose fixes (none CLI-specific):
- macOptionClickForcesSelection: Opt+drag always selects on Mac,
  regardless of what the running program does with mouse events.
- Cmd/Ctrl+A and Cmd/Ctrl+C custom key handler: select-all and copy
  to clipboard via navigator.clipboard, even when the TUI would
  swallow the keys.
- Copy button in the terminal header: writes the current selection
  to the clipboard, or the full visible viewport if nothing is
  selected. One-click escape hatch that works in every state.

Applies to any interactive CLI in our terminal (sudo, vim, claude,
gh auth, etc.), not just the claude login flow.

* fix(terminal): make xterm selection actually visible

Selection was registering internally (xterm-selection layer had
correct width/height rects), but the rectangles rendered in
rgb(252,252,251) — practically invisible against the white
background — so users concluded selection was broken.

Root cause: the theme derived selectionBackground from
`withAlpha(resolveCssColor('--accent-orange'), 0.2)`. When the CSS
var failed to resolve it fell back near-white, and the alpha
compositing against the page background made the result
indistinguishable from the background.

Switch to solid terminal-standard selection colors (VSCode-like
light-blue / dark-indigo). Also set selectionInactiveBackground so
the selection persists when focus moves away (useful while copying).
Drop the now-unused withAlpha helper.

* fix(openclaw): handle pretty-printed JSON in claude auth status parser

claude auth status --json emits multi-line pretty-printed JSON. The previous line-by-line parser never matched, so the UI treated every response as an error and surfaced the raw JSON — even when loggedIn was true. Replace with a brace-matching JSON extractor (string- and escape-aware) that tolerates multi-line JSON, leading banners, trailing lima/nerdctl stderr, and nested objects.

* refactor(openclaw): separate exec streams, argv installs, cleaner async cleanup

Audit-driven cleanup. Net -42 lines, four concrete issues fixed:

1. ContainerRuntime.runInContainer() exposes {exitCode, stdout, stderr}
   from the nerdctl exec (ContainerCli.runCommand already tracked them
   separately; we were just throwing stderr into the same string). The
   40-line hand-rolled brace-matching JSON extractor in claude-cli.ts
   existed only because the prior merged-stream output had lima/
   nerdctl's 'level=fatal' line fused with claude's JSON. parser is
   now JSON.parse(stdout.trim()).

2. Replace shell-based 'sh -lc "npm install -g ${pkg}@latest"' with
   argv: execInContainer(['npm','install','-g','${pkg}@${version}']).
   Registry values no longer flow through a shell (removes injection
   surface from future CLI providers). Pinned version instead of
   @latest (adds npmPackageVersion to the provider type).

3. AgentTerminal: replace the 'let cancelled' + out-of-effect
   disposeSocketBindings pattern with an AbortController scoped to
   the effect and a cleanups[] array. Matches the canonical React 18
   async-effect pattern — no partial-cleanup race if StrictMode
   unmounts between the async await and the resolve.

4. AgentTerminal: drop the full-buffer fallback in the Copy button
   (was copying all 8000 scrollback lines when nothing selected —
   surprising). Button now only copies the actual xterm selection,
   or no-ops silently. Users who want everything can Cmd+A first.
2026-04-24 20:13:18 +05:30
Dani Akash
752f42d1fe refactor: migrate chat history to direct JSONL file reads via Lima filesystem (#808)
* feat: draft agent chat ui exploration

* feat: refine agent chat ui draft

* feat: remove outer frame from agent chat workspace

* fix: offset agent chat for app sidebar

* fix: simplify agent conversation shell

* fix: remove redundant chat header actions

* fix: unify agent conversation headers

* fix: tighten agent chat spacing

* fix: bound agent chat composer height

* fix: remove agent chat page inset

* fix: align agent header height with sidepanel

* fix: center agent composer resting state

* fix: anchor multiline composer controls

* fix: remove focus grid from agent home

* fix: remove redundant agent home header

* fix: constrain home agent composer

* fix: match home composer default posture

* feat: add openclaw chat history APIs

* feat: add claw chat history hydration

* fix: stabilize claw chat viewport layout

* fix: use conversation scroll base for claw chat

* refactor: split claw chat controller responsibilities

* fix: keep active agent turns in memory

* fix: normalize openclaw chat sessions

* refactor: use HTTP client for agent history instead of CLI client

Replace the CLI-based getChatHistory() call in getAgentHistoryPage()
with the HTTP client's getSessionHistory() from PR #795. This uses
the direct HTTP transport to OpenClaw's /sessions/<key>/history
endpoint instead of shelling out through the CLI.

- Add filterHttpSessionHistoryMessages() for flat-string content format
- Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape
- Update getAgentHistoryPage() to call getSessionHistory() via httpClient
- Remove unused getChatHistory(), filterOpenClawSystemMessages(),
  normalizeChatHistoryMessages(), and getTextContent()
- Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory
- Update MutableOpenClawService type: chatClient -> httpClient

* fix: fetch all session messages by iterating OpenClaw pagination

OpenClaw's HTTP history endpoint returns a limited page by default.
When called without a limit, only the first ~27 messages were returned,
causing all newer conversation messages to be silently dropped.

Add fetchAllSessionMessages() that iterates through OpenClaw's cursor-
based pagination (200 messages per page) until hasMore is false, then
feeds the complete message list into the existing BrowserOS normalization
and in-memory pagination layer.

* refactor: migrate chat history from HTTP gateway to direct JSONL file reads

Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw
gateway /sessions/:key/history pagination loop) with direct JSONL file reads
from the host filesystem via Lima's virtiofs mount.

- Add OpenClawJsonlReader that reads session JSONL files directly from
  ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/
- Replace fetchAllSessionMessages() HTTP pagination with single file read
- Replace CLI-based listSessions() with sessions.json file reads
- Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous
- Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages,
  normalizeHttpHistoryMessages helpers
- Update route handlers to drop unnecessary async/await
- Update tests to use temp JSONL files instead of mocked HTTP/CLI clients

* fix: restore async route handlers for test compatibility with mocked service

* fix: address review feedback — path traversal guard, lazy reader, exists flag

- Add safePath() to OpenClawJsonlReader that validates resolved paths stay
  within stateRoot, preventing path traversal via crafted agentId values
- Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients)
  instead of creating a new instance per property access
- Return exists: false from resolveSpecificAgentSession when no session
  matches instead of fabricating a ghost session with sessionId: ''
2026-04-24 13:19:46 +05:30
Nikhil
2f8e36546f fix: resize BrowserOS VM resources (#807) 2026-04-23 18:24:49 -07:00
Nikhil
461dcd29e8 fix: upload Lima resources under vendor prefix (#805) 2026-04-23 17:19:45 -07:00
Nikhil
c6c902a4ab feat: improve dev watch Lima preflights (#802)
* feat: improve dev watch lima preflights

* fix: note vm cache sync duration

* fix: address review feedback for PR #802
2026-04-23 17:16:50 -07:00
Nikhil
6e37742a5a feat: reuse agent command chat for agents page (#803) 2026-04-23 17:09:49 -07:00
Nikhil
1186c2c0d7 merge: feat/new-lima-vm
feat: new vm integration
2026-04-23 16:41:14 -07:00
Nikhil
0288cc040d feat: use rootless nerdctl in BrowserOS VM (#800)
* feat: use rootless nerdctl in BrowserOS VM

* fix: validate openclaw gateway auth before reuse

* fix: forward rootless containerd socket

* fix: address VM review comments
2026-04-23 16:36:51 -07:00
Nikhil
07b7bf5977 feat(build-tools): seed dev agent tarballs (#799)
* feat(build-tools): seed dev agent tarballs

* fix: address review comments for 0423-build_agent_tarball_dev_sync

* chore(build-tools): remove dev cache sync alias
2026-04-23 15:47:00 -07:00
Nikhil
d1a3d67e29 chore(dev): add VM cache setup flow (#798) 2026-04-23 15:47:00 -07:00
Nikhil
35134518f0 fix(vm): use system nerdctl in Lima runtime (#797) 2026-04-23 15:47:00 -07:00
Nikhil Sonti
4083155e81 feat(container): migrate container runtime to nerdctl over Lima VM
Replace the podman-based runtime with nerdctl running inside the Lima
VM introduced in the previous commit. OpenClaw is cut over to the new
VM-backed container runtime; legacy podman code paths are removed.

- New container CLI (lib/container): nerdctl ContainerCli, ImageLoader
  with cache-tarball fallback, shared types
- OpenClaw: container-runtime-factory orchestrates VM lifecycle + gateway
  startup; container-runtime.ts rewritten to speak nerdctl; Linux test
  startup kept disabled behind the factory
- Terminal: session + routes moved onto Lima shell transport; server
  wires the VM-backed runtime via main.ts
- Agent UI: simplify AgentsPage/useOpenClaw after route consolidation
- Remove podman-runtime, podman-overrides, and their tests
- Tests: container-cli, image-loader, container-runtime-factory, and
  updated openclaw/terminal/main suites
2026-04-23 15:46:50 -07:00
Nikhil Sonti
72ef4f068e feat(vm): add Lima-based BrowserOS VM runtime
Introduce a new VM runtime layer using Lima for running containerised
workloads on macOS. Lifecycle covers decompress/create/start/stop with
stubs for upgrade/reset plus version-mismatch warnings.

- Foundation modules: paths, errors, manifest, telemetry
- lima.yaml generator + typed limactl wrapper with structured debug logging
- ssh ControlMaster transport for fast in-VM commands
- Ubuntu 24.04 minimal template, containerd default, 30GiB overlay disk
- browseros-dir helpers (getLimaHomeDir, getVmStateDir, getVmDisksDir);
  OpenClaw dir moves into VM state dir
- Test helpers (fake-limactl, fake-ssh, test-env), vm-smoke integration
  coverage, NODE_ENV propagation for spawned server test groups
2026-04-23 15:46:25 -07:00
Nikhil
6b6ed1582c feat(openclaw): HTTP session history endpoint (JSON + SSE) (#795)
* refactor(openclaw): rename http chat client to http client

Session history is about to land on the same HTTP client. 'Chat client'
will no longer describe it, so rename the class, file, and service field
up front. No behavior change.

* feat(openclaw): add session history fetch + sse stream to http client

Adds getSessionHistory (JSON) and streamSessionHistory (SSE) to the
OpenClaw HTTP client. Both target GET /sessions/<key>/history on the
loopback gateway, reusing the same bearer-token auth as streamChat.

- 404 from the gateway surfaces as OpenClawSessionNotFoundError so
  callers can map it to a typed HTTP status.
- The SSE path parses named 'history', 'message', and 'error' events
  into a typed OpenClawSessionHistoryEvent union.
- AbortSignal propagates to fetch and cancels the reader mid-stream.

* feat(openclaw): expose session history over GET /claw/session/:key/history

Wire the new getSessionHistory / streamSessionHistory service methods
through a route that defaults to JSON and upgrades to SSE when the
client sends Accept: text/event-stream.

- OpenClawSessionNotFoundError lives in errors.ts alongside the other
  OpenClaw errors so routes can import it from one place.
- The route propagates c.req.raw.signal into streamSessionHistory so
  client disconnects cancel the upstream fetch.
- Route tests cover the JSON path (with query param forwarding), the
  404 path, and the SSE framing.

* chore(openclaw): drop NaN from session history route limit param
2026-04-23 11:19:16 -07:00
Nikhil
a3764e7599 feat(build-tools): add cache:sync:dev for local tarball seeding (#794)
Seeds ~/.browseros-dev/cache/vm/ from ./dist/ without touching R2, so
devs can test the server against a freshly-built tarball before anything
is published to cdn.browseros.com. Hardcodes arm64 since all devs are on
Apple Silicon; refuses to run unless NODE_ENV=development; idempotent
(skips copy on sha256 match).

Also fixes the R2_BUCKET default in .env.sample from browseros-artifacts
to browseros to match the actual bucket.
2026-04-23 10:33:51 -07:00
Nikhil
c656f6236c feat: ship Lima template for BrowserOS VM (#787)
* feat(build-tools): add Lima template for BrowserOS VM

* feat(build-tools): remove build-disk pipeline and recipe directory

Task 2 verification removed the scripts, recipe directory, workflow, and package scripts. Typecheck remains green here because manifest disk fields are removed in the next task, so the plan's expected missing-import failure does not apply yet.

* feat(build-tools): rename VmManifest to AgentManifest, drop disk fields

* feat(build): stage Lima template into server resources

Verified local-resource staging with: bun scripts/build/server.ts --target=darwin-arm64 --ci. The template was copied to dist/prod/server/darwin-arm64/resources/vm/browseros-vm.yaml and included in the zip. bun run build:server:test still fails on the pre-existing R2 limactl resource with: The specified key does not exist.

* docs(build-tools): Lima template dev loop + record D9

Updated the build-tools README in this worktree. Also recorded D9 in the canonical external spec file at /Users/shadowfax/llm/code/browseros-project/grove-ref/browseros-main/specs/decisions.md, which is outside this git checkout.

* chore(build-tools): sweep orphaned references to retired disk pipeline

* chore: self-review fixes
2026-04-22 17:17:12 -07:00
Nikhil
4d660874ad feat: consolidate build tools package (#785)
* feat(build-tools): scaffold package + cache dir helpers

* feat(build-tools): manifest types + R2 helper

* feat(build-tools): build-disk script with virt-customize + zstd

* feat(build-tools): build-tarball script

* feat(build-tools): emit-manifest + cache:sync

* ci(build-tools): independent build-vm + build-agent workflows

* chore: remove legacy container packages + workflows

* fix: address review feedback for PR #785

* fix: stabilize VM build DNS in CI

* fix: prioritize arm64 build workflows

* fix: keep arm64 VM recipe simple

* fix: set VM build DNS in apt command

* fix: avoid guest DNS for VM package install

* fix: limit VM PR checks to build-tools validation
2026-04-22 16:23:11 -07:00
Nikhil
819887a2c5 feat(vm-container): WS1 VM disk image pipeline (#783)
* feat(vm-container): ship the WS1 VM disk image pipeline

New Bun/TS workspace package @browseros/vm-container that produces a
reproducible, versioned Debian 12 + Podman qcow2 disk image for arm64 and
x64, and publishes it to Cloudflare R2 under vm/<version>/ with a per-
version manifest.json and a latest.json pointer.

- virt-customize-driven build with a git-tracked recipe DSL.
- zstd-compressed artifacts; sha256 sidecars for compressed + uncompressed.
- Public surface at @browseros/vm-container/schema exposes zod-validated
  VmManifest + R2 key helpers for WS4 to import; /download is a stub
  landing pad for WS4 to fill in.
- Rollback on partial upload failure: any exception after the first
  successful put deletes all previously uploaded keys for that version.
- GHA workflow build-vm-container.yml runs a matrix build per arch on
  native runners, an x64 Lima boot smoke test, and a gated publish job.
- Full unit coverage for arch, r2-keys, manifest, recipe parser, and
  publish (rollback + happy path via aws-sdk-client-mock).

* fix(vm-container): address review comments

- Split buildDisk into prepareCustomizedDisk + finalizeArtifacts for
  testability.
- Replace resolvePinnedSha's sentinel-prefix check with a positive
  sha256-hex regex test, switch base-image.ts placeholder to empty string.
- Drop unused R2_VM_PREFIX from .env.example; document CDN_BASE_URL
  override precedence in README.
- Replace SSH host-key explicit list in recipe with `ssh_host_*` glob so
  .pub keys and future key types are also removed.
- lima-boot: introduce BunRequestInit type for the unix fetch option and
  reject empty limactlPath loudly.
- Extend publish test suite: mid-manifest-upload failure path verifies
  both arches' qcow+sha are rolled back and latest.json is never written.
- Add missing tests: parseArch('ARM64') case-sensitivity rejection,
  composeVirtCustomizeArgv unresolved-substitution pass-through.

* fix(vm-container): pin a real Debian snapshot, switch verify to SHA-512, streaming download

- Pin Debian base to bookworm/20260413-2447 with real SHA-512 values
  from upstream SHA512SUMS (the sentinel placeholder never corresponded
  to a real build). Debian cloud images only publish SHA512SUMS today,
  so switch base-image verification to SHA-512 throughout: rename
  BaseImage.sha256 → sha512, manifest field base_image_sha256 →
  base_image_sha512, base_image.sha256_url → sha512_url,
  debianSha256SumsUrl → debianSha512SumsUrl. Our own artifact hashes
  (compressed_sha256, uncompressed_sha256, recipe_sha256) stay SHA-256.
- Fix downloadTo: previous Bun.write(dest, response) buffered the
  entire 300 MB response before writing (100% CPU, empty dir). Replace
  with a getReader() loop that streams chunks through Bun.file().writer().
- build CLI now auto-derives --version from today's date when omitted
  (defaults to YYYY.MM.DD-dev1); explicit --version still overrides.
  Broaden CALVER_REGEX to accept alphanumeric suffixes so -dev1/-rc1
  tags are valid. New todayCalver() helper.
- Update GHA workflow fallback to github.run_number (shorter) instead
  of run_id.

* fix(vm-container): resolve copy-in paths against recipeDir after substitution

The copy-in path resolver checked op.src.startsWith('/') before running
the {placeholder} substitution, so an absolute-after-substitution path
like {manifest_tmp} → /tmp/vm-dist/manifest-stub-arm64.json was treated
as relative and joined against recipeDir, producing a nonexistent path.
Check the *substituted* value for absoluteness via path.isAbsolute.

* fix: address review comments for 0422-ws1_vm_disk_pipeline

* fix(ci): repair vm-container workflow

* fix(ci): expose vm build logs on failure

* fix(vm-container): expose base_image_sha256 in manifest per PRD

The published manifest contract (consumed by WS4) now uses base_image_sha256
as the PRD specified. Internally the build still verifies the downloaded
Debian base against the pinned sha512 (that's what Debian actually signs in
SHA512SUMS) — then hashes the same bytes as sha256 and records that in the
manifest. One extra digest pass of a ~300 MB file; negligible.

- manifest.json: base_image_sha256 replaces base_image_sha512; sha512_url
  removed (not needed — sha256 is the consumer-facing hash).
- CLI: --base-image-sha256 override validates against the locally-computed
  sha256 after download.
- BuildResult.baseImage gains sha256 alongside sha512.
- Tests updated to the new field.

The auth.json bug (reviewer #2) is resolved: the source file is
recipe/auth.json and the recipe emits `copy-in auth.json:/etc/containers/`
so libguestfs writes /etc/containers/auth.json.

* ci(vm-container): fix supermin kernel-read + rename sha512 inputs to sha256

- Ubuntu 24.04 GHA runners ship /boot/vmlinuz-* as mode 0600, which blocks
  libguestfs's supermin appliance builder when virt-customize runs as a
  non-root user. Chmod 0644 before the build — canonical CI workaround.
- Rename workflow_dispatch input base_image_sha512 → base_image_sha256
  and CLI flag --base-image-sha512 → --base-image-sha256 to match the
  orchestrator's renamed override.

* ci(vm-container): give runner KVM access + install passt for libguestfs

The supermin fix got us past appliance-build, but virt-customize then hit
"passt exited with status 1". The passt networking helper misbehaves when
libguestfs falls back to TCG emulation, which happens because the runner
user isn't in the kvm group even though /dev/kvm exists on the GHA host.

- chmod 0666 /dev/kvm → libguestfs uses hardware acceleration, avoids TCG.
- install passt explicitly so the networking helper is present and current.

* ci(vm-container): disable passt to force libguestfs slirp fallback

libguestfs 1.54+ prefers passt for guest networking, but the passt binary
on GHA ubuntu-24.04 exits with status 1 when invoked from the appliance
— an AppArmor/capability issue that doesn't surface a useful diagnostic.
The reliable workaround is to remove passt so libguestfs picks QEMU's
built-in user-mode SLIRP as the network backend. SLIRP is slower but
functional and doesn't require escalated privileges.
2026-04-22 14:04:00 -07:00
Nikhil
114d5e3a9f feat: add agent container tarball pipeline (#782)
* feat: add agent container tarball pipeline

* docs: add agent-container env sample

* refactor: simplify agent container pipeline

* fix: address review feedback for PR #782

* fix: emit clean matrix JSON in CI

* fix: align agent container artifact paths
2026-04-22 13:14:27 -07:00
Nikhil
ecba7de221 fix: address PR review comments for 0422-ws3_lima_resources (#780)
- Guard uploaded_keys append with !dry_run so the rollback list
  never contains keys for objects that were never written.
- Prefer GITHUB_ACTOR over local OS username for manifest.uploaded_by;
  manifest.json is CDN-fronted so leaking a developer's login is
  unnecessary (falls back to 'local').
- Extend test_windows_has_no_stale_third_party to cover bun.exe/rg.exe
  too, matching the macOS forbidden-set pattern.
2026-04-22 10:47:25 -07:00
Nikhil
123a13fe62 feat(build): swap podman server resources for Lima (WS3) (#778)
* feat(build): swap podman server resources for Lima (WS3)

- Upload limactl (arm64 + x64) to R2 via new 'browseros upload lima' CLI.
- Rewrite scripts/build/config/server-prod-resources.json: 2 Lima entries,
  12 podman-family entries removed.
- Update codesign metadata (server_binaries.py) to add limactl, drop podman
  family. Sign modules need no edits (data-driven).
- Delete orphaned podman-{vfkit,krunkit} entitlement plists.
- Release-gating note in browseros-agent/CLAUDE.md: don't cut releases off
  dev between this commit and WS6 landing (OpenClaw still invokes podman).

* fix: address review comments for 0422-ws3_lima_resources

- Tighten _find_limactl_member to match exactly .../bin/limactl via
  Path.parts, avoiding incidental matches like 'xbin/limactl'.
- Fall back USER -> USERNAME -> 'unknown' for uploaded_by so Windows
  shells don't all record 'unknown'.
- Comment the broad except in upload_lima to explain why rollback
  must fire for any mid-loop failure.

* chore: drop bun + rg from Windows sign list

These executables are already absent from server-prod-resources.json (no
Windows entries shipped); keeping them in the sign list produces
"Binary not found" warnings on every Windows build.
2026-04-22 10:40:53 -07:00
Nikhil
5ccdbaf87f feat(openclaw): lifecycle progress banner + live podman readiness (#772)
* fix(openclaw): serialize lifecycle operations

* feat(openclaw): lifecycle progress banner and live podman readiness check

* fix: address review comments for openclaw-lifecycle-progress
2026-04-21 07:59:33 -07:00
Nikhil
0650f21c80 fix(openclaw): allocate gateway host port dynamically + name the two ports distinctly (#771)
* feat(openclaw): dynamically allocate and persist gateway host port

The gateway container always listens on OPENCLAW_GATEWAY_CONTAINER_PORT
(18789) internally, but that port may be taken on the user's host. Allocate
a free host port on each lifecycle transition, persist it to
~/.browseros/openclaw/.openclaw/runtime-state.json, and prefer the
persisted value on subsequent starts so the mapping is stable.

Split the naming so the two sides of the -p mapping are no longer
ambiguous: the shared constant becomes OPENCLAW_GATEWAY_CONTAINER_PORT
and the service/spec/chat-client/runtime probes all use hostPort for
the mapped host-side port.

* fix(openclaw): remove duplicate Podman overrides card from status panels
2026-04-20 17:32:10 -07:00
Dani Akash
e80ec467f4 feat: wire lazy monitoring to OpenClaw chat handoff (#768)
* feat: add lazy monitoring substrate

* feat: wire lazy monitoring to openclaw chat handoff

* test: cover openclaw chat history handoff

* fix: reject concurrent monitored chats
2026-04-20 21:52:03 +05:30
Dani Akash
41374439c4 feat: add passive lazy monitoring substrate for MCP tool calls (#766)
* feat: add lazy monitoring substrate

* fix: validate monitoring run ids

* fix: harden monitoring storage recovery
2026-04-20 21:10:09 +05:30
Dani Akash
ad99cd6cc1 fix: restore openai-compatible OpenClaw providers (#767)
* fix(openclaw): restore openai-compatible providers

* fix(openclaw): preserve custom provider model lists
2026-04-20 20:25:37 +05:30
Nikhil
47fc9e1292 feat(openclaw): user-supplied Podman binary path override (#759)
* feat(openclaw): user-supplied Podman binary path override

Expose the existing `configurePodmanRuntime({ podmanPath })` knob as a UI
input on the Agents page so users blocked by the bundled gvproxy helper
discovery bug can install their own Podman (e.g. `brew install podman`)
and point BrowserOS at it.

- podman-overrides.ts: persist {podmanPath} at ~/.browseros/.openclaw/
- openclaw-service: applyPodmanOverrides/getPodmanOverrides, rebuilds
  ContainerRuntime + CLI clients in place (no server restart needed)
- routes: GET/POST /claw/podman-overrides with absolute-path + existsSync
  validation
- main: load override on boot, pass resourcesDir into the service so
  clearing the override restores bundled fallback
- AgentsPage: PodmanOverridesCard rendered inline in the degraded /
  uninitialized / error cards and as a collapsible standalone section

Dev mode is unchanged; prod gets the same lever dev has had all along.

* refactor(openclaw): address review comments for podman-path override

- extract getPodmanOverrideValidationError() to mirror the existing
  getCreateAgentValidationError() pattern in openclaw.ts
- extract rebuildRuntimeClients() so applyPodmanOverrides doesn't
  re-spell the three-step runtime/CLI-client reinit
- rename shadowing local path -> overridesPath in loadPodmanOverrides

* fix(openclaw): clear gateway log tail before swapping runtime

rebuildRuntimeClients replaces this.runtime but the cached stopLogTail
still closes over the old runtime's log-tail process. The existing
guard in startGatewayLogTail (if (this.stopLogTail) return) would then
short-circuit the next restart and leave the new runtime without a
tail. Clear it inside the helper so the rebuild is self-consistent
regardless of caller order.

* fix(openclaw): check podmanPath executability and note singleton mutation

- validator: after existsSync, accessSync(X_OK) so a non-executable file
  fails fast at save time with a clear 400 instead of a cryptic spawn
  error later. Added a matching route test.
- applyPodmanOverrides: one-line comment flagging the intentional
  module-level PodmanRuntime singleton mutation so future readers know
  this is by design, not an accident.
2026-04-18 17:27:25 -07:00
Nikhil
2a61dcbc58 fix: remove podman compose from OpenClaw runtime (#758)
* refactor: rename OpenClaw runtime away from compose semantics

* feat: run OpenClaw containers with direct podman commands

* test: assert exact podman run args

* fix: stage direct runtime container migration safely

* refactor: switch OpenClaw service to direct podman runtime

* test: cover direct-runtime lifecycle paths in openclaw service

* fix: handle legacy openclaw gateway container during runtime cutover

* chore: remove OpenClaw compose resources from server build

* refactor: drop obsolete setup-command overload

* fix: remove dead OpenClaw runtime env file flow

* fix: restore scoped OpenClaw gateway container name

* test: assert scoped OpenClaw terminal container name

* fix: make OpenClaw gateway removal idempotent

* fix: harden OpenClaw setup container lifecycle
2026-04-18 13:53:18 -07:00
Nikhil
f5a2b7315c fix: run all browseros-agent tests from root (#750)
* fix: run full browseros-agent test suite

* fix: stabilize server test reporting in CI

* fix: address PR review feedback

* refactor: extract server core test runner

* refactor: group server tests by filesystem

* fix: align CI suites with server test groups

* fix: provision server env for all CI suites

* fix: stabilize ci checks

* fix: report real test counts in ci
2026-04-17 17:26:44 -07:00
Nikhil
6de3b3422c fix: package OpenClaw compose resource (#749)
* fix: package openclaw compose resource

* fix: address PR review comments for docker-compose-missing
2026-04-17 15:01:59 -07:00
Nikhil
224b6cd3a8 chore: remove bun and ripgrep prod resources (#748) 2026-04-17 13:03:42 -07:00
Nikhil
7baee8d57e chore: release server alpha - 0.0.88 (#747) 2026-04-17 12:44:41 -07:00
Nikhil
e8e8c36fdb fix: pin OpenClaw image to 2026.4.12 (#746)
* fix: pin OpenClaw image to 2026.4.12

* fix: address PR review comments for 0417-openclaw-image-pin
2026-04-17 12:14:37 -07:00
Nikhil
3810005457 refactor: stabilize local OpenClaw integration (#741)
* feat(openclaw): add CLI client

* fix(openclaw): swap service to cli client

* fix(openclaw): restore mixed json parsing

* fix(openclaw): validate agent list payloads

* fix(openclaw): simplify cli client boundary

* fix(openclaw): simplify cli client boundary

* fix(openclaw): prefer outer config json payloads

* fix(openclaw): ignore trailing config log payloads

* refactor(openclaw): bootstrap config through cli

* fix(openclaw): narrow bootstrap ownership

* fix(openclaw): avoid noop key restarts

* fix(openclaw): enforce supported provider sync

* refactor(openclaw): remove agent role contract

* fix(openclaw): migrate legacy state and apply model updates

* fix(openclaw): migrate legacy agent state

* fix(openclaw): harden state updates

* refactor: stabilize local OpenClaw bootstrap and chat auth

* fix(openclaw): propagate container env and drop legacy paths

Compose now loads provider creds from .openclaw/.env and passes the
gateway token through, so in-container CLI commands (tui, doctor,
config) authenticate correctly and the gateway process sees
OPENROUTER_API_KEY. Service ensures the state env file exists and
rewrites the compose env with the token before composeUp in setup,
start, and tryAutoStart. Podman machine gets larger defaults and the
container enables NODE_COMPILE_CACHE + OPENCLAW_NO_RESPAWN. Legacy
state migration, the unused WebSocket gateway-client, memorySearch,
and thinking defaults are removed.
2026-04-17 11:00:07 -07:00
Nikhil
688f7962cb fix: rerun dev port cleanup before server restarts (#745) 2026-04-17 08:13:22 -07:00
Felarof
526d784d82 chore: add .auctor entries to gitignore (#739)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-16 18:00:24 -07:00
Nikhil
331fec07e6 fix: use separate BrowserOS dir in development (#736) 2026-04-16 16:42:06 -07:00
Nikhil
0652ee8ca8 feat: better hidden windows (#730)
* feat: better hidden windows

* fix: addressing review comments
2026-04-16 16:33:12 -07:00
Nikhil
156f5dbc5d feat: redesign OpenClaw control plane around CLI and HTTP (#735)
* feat: move OpenClaw control plane to CLI and HTTP

* fix: address PR review comments for 0416-openclaw_cli_http_redesign
2026-04-16 16:29:26 -07:00
Nikhil
ebd3200cfe feat(build): add arm64-only macOS release config (#728)
Introduces release.macos.arm64.yaml for single-architecture arm64
macOS release builds. Mirrors the windows/linux single-arch pattern
(configure -> compile -> sign_macos -> package_macos -> upload),
skipping the universal_build module to avoid the x64 cross-compile
and lipo merge. Reuses the sparkle_setup step and the same
notarization env vars as the universal macOS config.
2026-04-16 13:09:46 -07:00
Nikhil
4172daa130 chore: bump PATCH and OFFSET (#727) 2026-04-16 13:05:01 -07:00
Nikhil
c1b1e53a86 feat(ota): bundle full server resources tree in Sparkle payload (#726)
* feat(ota): bundle full server resources tree (server + third_party bins)

The OTA Sparkle payload now ships the complete resources/ tree the agent
build produced, not just browseros_server. Every third-party binary (bun,
ripgrep, podman, gvproxy, vfkit, krunkit, podman-mac-helper, win-sshproxy)
flows to OTA-updated installs so podman integration works for users on the
OTA channel, matching fresh Chromium-build installs.

Extract the per-binary sign table into build/common/server_binaries.py so
the Chromium-build sign path (modules/sign/) and OTA sign path (modules/ota/)
share a single source of truth. Adding a new third-party dep is now a
one-file edit that both paths pick up automatically; unknown executables
under resources/bin/ are a hard error at release time.

* fix(ota): address review comments on bundle signing flow

- Avoid double-zipping during notarization: add notarize_macos_zip for
  pre-built Sparkle bundles so notarytool submits the zip directly
  instead of re-wrapping it through ditto --keepParent (Apple's service
  does not descend into nested archives). Keep notarize_macos_binary for
  single-binary callers. Share credential setup + submit logic via
  internal helpers.
- Fail fast on unknown executables in sign_server_bundle_macos: collect
  the unknown-files list before any codesign call so a missing shared-
  table entry aborts in seconds, not after a full signing round.
- Drop dead get_entitlements_path helper (no callers remain after the
  bundle refactor).

* fix(ota): address PR review comments (greptile + claude)

- sign_server_bundle_macos filters to executables only (p.is_file() +
  not p.is_symlink() + os.access X_OK) before applying the unknown-file
  guard. Non-Mach-O files (configs, dylibs, etc.) under resources/bin/
  no longer cause misleading 'unknown executable' hard failures.
- sign_server_bundle_windows now hard-errors on a missing expected
  binary instead of silently skipping it. Symmetric with the macOS
  guard — an incomplete bundle must not publish.
- ServerOTAModule.execute() uses tempfile.TemporaryDirectory context
  managers for both the download and staging roots so they are cleaned
  up on every path, including failures.
- Per-platform sign/notarize/Sparkle-sign failures now raise RuntimeError
  instead of silently skipping the platform — a release pipeline can no
  longer omit a target while reporting success.
- Move import os and import shutil to the top of ota/sign_binary.py.
- Drop unused log_error import from ota/server.py.

* chore: bump server
2026-04-16 12:59:49 -07:00
Nikhil
d653883e99 fix(ci): add PR comment with test summary (#724)
* fix(ci): add PR comment with test summary and block on failure

Add a `comment` job to the test workflow that parses JUnit XML artifacts
and posts a sticky PR comment showing pass/fail counts per suite, with
failed test names listed in a collapsible section and a link to the run.

Guards against fork PRs (read-only token) and stale overlapping runs
(skips comment if PR head has moved past our SHA).

* fix(ci): use payload SHA for staleness check, handle missing artifacts

- Replace context.sha (merge commit SHA) with
  context.payload.pull_request.head.sha so the staleness guard
  compares the correct values and the comment actually gets posted
- Add continue-on-error to download-artifact so cancelled runs
  gracefully fall through to the "no test results" message

* fix(ci): show warning icon for zero-test suites instead of failure
2026-04-15 21:35:58 -07:00
542 changed files with 47302 additions and 22474 deletions

View File

@@ -14,7 +14,7 @@ on:
config:
description: 'Eval config file (relative to apps/eval/)'
required: false
default: 'configs/browseros-agent-weekly.json'
default: 'configs/legacy/browseros-agent-weekly.json'
permissions:
contents: read
@@ -30,8 +30,9 @@ jobs:
- name: Install BrowserOS
run: |
wget -q https://github.com/browseros-ai/BrowserOS/releases/download/v0.44.0.1/BrowserOS_v0.44.0.1_amd64.deb
sudo dpkg -i BrowserOS_v0.44.0.1_amd64.deb
# Rolling stable channel — see https://cdn.browseros.com/download/BrowserOS.deb
wget -q -O BrowserOS.deb https://cdn.browseros.com/download/BrowserOS.deb
sudo dpkg -i BrowserOS.deb
browseros --version || echo "BrowserOS installed at $(which browseros)"
- name: Install Bun
@@ -41,7 +42,15 @@ jobs:
- name: Install dependencies
working-directory: packages/browseros-agent
run: bun install --ignore-scripts && bun run build:agent-sdk
run: bun install --ignore-scripts
- name: Install Python eval dependencies
# agisdk pinned so silent upstream releases can't shift task definitions
# or grader behavior. Bump intentionally with a documented re-baseline.
run: pip install agisdk==0.3.5 requests
- name: Clone WebArena-Infinity
run: git clone --depth 1 https://github.com/web-arena-x/webarena-infinity.git /tmp/webarena-infinity
- name: Install xvfb
run: sudo apt-get update && sudo apt-get install -y xvfb
@@ -53,34 +62,32 @@ jobs:
curl -sL -o /tmp/nopecha.zip https://github.com/NopeCHALLC/nopecha-extension/releases/latest/download/chromium_automation.zip
unzip -qo /tmp/nopecha.zip -d extensions/nopecha
- name: Run eval
- name: Run eval and publish to R2
working-directory: packages/browseros-agent/apps/eval
env:
FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
NOPECHA_API_KEY: ${{ secrets.NOPECHA_API_KEY }}
BROWSEROS_BINARY: /usr/bin/browseros
EVAL_CONFIG: ${{ github.event.inputs.config || 'configs/browseros-agent-weekly.json' }}
run: |
echo "Running eval with config: $EVAL_CONFIG"
xvfb-run --auto-servernum --server-args="-screen 0 1440x900x24" bun run src/index.ts -c "$EVAL_CONFIG"
- name: Upload runs to R2
if: success()
working-directory: packages/browseros-agent/apps/eval
env:
EVAL_R2_ACCOUNT_ID: ${{ secrets.EVAL_R2_ACCOUNT_ID }}
EVAL_R2_ACCESS_KEY_ID: ${{ secrets.EVAL_R2_ACCESS_KEY_ID }}
EVAL_R2_SECRET_ACCESS_KEY: ${{ secrets.EVAL_R2_SECRET_ACCESS_KEY }}
EVAL_R2_BUCKET: ${{ secrets.EVAL_R2_BUCKET }}
EVAL_R2_CDN_BASE_URL: ${{ secrets.EVAL_R2_CDN_BASE_URL }}
EVAL_CONFIG: ${{ github.event.inputs.config || 'configs/browseros-agent-weekly.json' }}
BROWSEROS_BINARY: /usr/bin/browseros
WEBARENA_INFINITY_DIR: /tmp/webarena-infinity
# OpenClaw container runtime is macOS-only; opt the Linux runner
# into the no-op stub so the server can boot and the eval can run.
BROWSEROS_SKIP_OPENCLAW: '1'
EVAL_CONFIG: ${{ github.event.inputs.config || 'configs/legacy/browseros-agent-weekly.json' }}
run: |
CONFIG_NAME=$(basename "$EVAL_CONFIG" .json)
bun scripts/upload-run.ts "results/$CONFIG_NAME"
echo "Running eval with config: $EVAL_CONFIG"
xvfb-run --auto-servernum --server-args="-screen 0 1440x900x24" bun run src/index.ts suite --config "$EVAL_CONFIG" --publish r2
- name: Generate trend report
if: success()
timeout-minutes: 5
continue-on-error: true
working-directory: packages/browseros-agent
env:
EVAL_R2_ACCOUNT_ID: ${{ secrets.EVAL_R2_ACCOUNT_ID }}
@@ -96,3 +103,11 @@ jobs:
with:
name: eval-report-${{ github.run_id }}
path: /tmp/eval-report.html
- name: Upload server stderr logs (for post-mortem on startup failures)
if: always()
uses: actions/upload-artifact@v4
with:
name: browseros-server-logs-${{ github.run_id }}
path: /tmp/browseros-server-logs/
if-no-files-found: ignore

View File

@@ -0,0 +1,176 @@
name: Publish VM Agent Cache
on:
workflow_dispatch:
inputs:
agent:
description: "Agent name from bundle.json"
required: true
type: string
default: openclaw
publish:
description: "Upload to R2 and merge manifest slice"
required: false
default: false
type: boolean
pull_request:
paths:
- "packages/browseros-agent/packages/build-tools/**"
- ".github/workflows/publish-vm-agent-cache.yml"
env:
BUN_VERSION: "1.3.6"
PKG_DIR: packages/browseros-agent/packages/build-tools
permissions:
contents: read
jobs:
check:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@v2
with:
bun-version: ${{ env.BUN_VERSION }}
- working-directory: packages/browseros-agent
run: bun install --frozen-lockfile
- working-directory: packages/browseros-agent
run: bun run --filter @browseros/build-tools typecheck
- working-directory: packages/browseros-agent
run: bun run --filter @browseros/build-tools test
build:
needs: check
strategy:
fail-fast: false
matrix:
include:
- arch: arm64
runner: ubuntu-24.04-arm
- arch: x64
runner: ubuntu-24.04
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@v2
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install podman
run: |
sudo apt-get update
sudo apt-get install -y podman
- working-directory: packages/browseros-agent
run: bun install --frozen-lockfile
- name: Build tarball
working-directory: ${{ env.PKG_DIR }}
env:
AGENT: ${{ inputs.agent || 'openclaw' }}
OUT: ${{ github.workspace }}/dist/images
run: bun run build:tarball -- --agent "$AGENT" --arch "${{ matrix.arch }}" --output-dir "$OUT"
- uses: actions/upload-artifact@v7
with:
name: tarball-${{ inputs.agent || 'openclaw' }}-${{ matrix.arch }}
path: dist/images/
retention-days: 7
smoke:
needs: build
strategy:
fail-fast: false
matrix:
include:
- arch: arm64
runner: ubuntu-24.04-arm
- arch: x64
runner: ubuntu-24.04
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@v2
with:
bun-version: ${{ env.BUN_VERSION }}
- uses: actions/download-artifact@v8
with:
name: tarball-${{ inputs.agent || 'openclaw' }}-${{ matrix.arch }}
path: dist/images
- name: Install podman
run: |
sudo apt-get update
sudo apt-get install -y podman
- working-directory: packages/browseros-agent
run: bun install --frozen-lockfile
- name: Smoke test tarball
timeout-minutes: 10
working-directory: ${{ env.PKG_DIR }}
env:
AGENT: ${{ inputs.agent || 'openclaw' }}
run: |
set -euo pipefail
tarball="$(find "$GITHUB_WORKSPACE/dist/images" -name "${AGENT}-*-${{ matrix.arch }}.tar.gz" -print -quit)"
if [ -z "$tarball" ]; then
echo "missing ${{ matrix.arch }} tarball artifact for ${AGENT}" >&2
exit 1
fi
checksum="${tarball}.sha256"
if [ ! -f "$checksum" ]; then
echo "missing checksum sidecar: $checksum" >&2
exit 1
fi
echo "smoke-testing $tarball"
ls -lh "$tarball" "$checksum"
(cd "$(dirname "$tarball")" && sha256sum -c "$(basename "$checksum")")
timeout --verbose --kill-after=30s 8m bun run smoke:tarball -- --agent "$AGENT" --arch "${{ matrix.arch }}" --tarball "$tarball"
publish:
needs: [build, smoke]
if: ${{ github.event_name == 'workflow_dispatch' && inputs.publish == true }}
runs-on: ubuntu-24.04
environment: release
concurrency:
group: r2-manifest-publish
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
- uses: oven-sh/setup-bun@v2
with:
bun-version: ${{ env.BUN_VERSION }}
- uses: actions/download-artifact@v8
with:
pattern: tarball-*
path: dist/images
merge-multiple: true
- working-directory: packages/browseros-agent
run: bun install --frozen-lockfile
- name: Upload tarballs to R2
working-directory: ${{ env.PKG_DIR }}
env:
R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
run: |
set -euo pipefail
for file in "$GITHUB_WORKSPACE"/dist/images/*.tar.gz; do
base="$(basename "$file")"
bun run upload -- --file "$file" --key "vm/images/$base" --content-type "application/gzip" --sidecar-sha
done
- name: Merge agent slice into manifest
working-directory: ${{ env.PKG_DIR }}
env:
AGENT: ${{ inputs.agent || 'openclaw' }}
R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
run: |
set -euo pipefail
mkdir -p dist/images
cp -R "$GITHUB_WORKSPACE"/dist/images/* dist/images/
bun run download -- --key vm/manifest.json --out dist/baseline-manifest.json
bun run emit-manifest -- \
--slice "agents:${AGENT}" \
--dist-dir dist \
--merge-from dist/baseline-manifest.json \
--out dist/manifest.json
bun run upload -- --file dist/manifest.json --key vm/manifest.json --content-type "application/json"

View File

@@ -1,168 +1,11 @@
name: Release BrowserOS Agent SDK
name: Release BrowserOS Agent SDK (disabled)
on:
workflow_dispatch:
concurrency:
group: release-agent-sdk
cancel-in-progress: false
jobs:
publish:
if: github.ref == 'refs/heads/main'
disabled:
if: ${{ false }}
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
defaults:
run:
working-directory: packages/browseros-agent/packages/agent-sdk
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- uses: oven-sh/setup-bun@v2
- uses: actions/setup-node@v6
with:
node-version: "20"
registry-url: "https://registry.npmjs.org"
- name: Install dependencies
run: bun ci
working-directory: packages/browseros-agent
- name: Build
run: bun run build
- name: Test
run: bun test
- name: Get version
id: version
run: |
echo "version=$(node -p "require('./package.json').version")" >> "$GITHUB_OUTPUT"
echo "release_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
- name: Generate release notes
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
SDK_PATH="packages/browseros-agent/packages/agent-sdk"
CURRENT_TAG="agent-sdk-v${{ steps.version.outputs.version }}"
# Find the previous tag, excluding the current version's tag
# (which may already exist from a prior failed run)
PREV_TAG=$(git tag -l "agent-sdk-v*" --sort=-v:refname | grep -v "^${CURRENT_TAG}$" | head -n 1)
if [ -z "$PREV_TAG" ]; then
echo "Initial release" > /tmp/release-notes.md
else
# Get commits scoped to the SDK directory
COMMITS=$(git log "$PREV_TAG"..HEAD --pretty=format:"%H" -- "$SDK_PATH")
if [ -z "$COMMITS" ]; then
echo "No notable changes." > /tmp/release-notes.md
else
echo "## What's Changed" > /tmp/release-notes.md
echo "" >> /tmp/release-notes.md
# For each commit, find the associated PR and format with author
CONTRIBUTORS=""
while IFS= read -r SHA; do
# Get commit subject and author
SUBJECT=$(git log -1 --pretty=format:"%s" "$SHA")
AUTHOR=$(git log -1 --pretty=format:"%an" "$SHA")
GITHUB_USER=$(gh api "/repos/${{ github.repository }}/commits/${SHA}" --jq '.author.login // empty' 2>/dev/null)
# Find associated PR number
PR_NUM=$(gh api "/repos/${{ github.repository }}/commits/${SHA}/pulls" --jq '.[0].number // empty' 2>/dev/null)
# Format line: skip PR number if already in the commit subject
# (squash merges include "(#123)" in the subject automatically)
if [ -n "$PR_NUM" ] && ! echo "$SUBJECT" | grep -qF "(#${PR_NUM})"; then
echo "- ${SUBJECT} (#${PR_NUM})" >> /tmp/release-notes.md
else
echo "- ${SUBJECT}" >> /tmp/release-notes.md
fi
done <<< "$COMMITS"
fi
fi
working-directory: ${{ github.workspace }}
- name: Publish
run: npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Create GitHub release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
TAG="agent-sdk-v${{ steps.version.outputs.version }}"
RELEASE_SHA="${{ steps.version.outputs.release_sha }}"
TITLE="BrowserOS Agent SDK - v${{ steps.version.outputs.version }}"
# Create or reuse tag (idempotent for re-runs)
if git rev-parse "$TAG" >/dev/null 2>&1; then
echo "Tag $TAG already exists, skipping tag creation"
else
git tag "$TAG" "$RELEASE_SHA"
fi
# Push tag (skip if already on remote)
if git ls-remote --tags origin "$TAG" | grep -q "$TAG"; then
echo "Tag $TAG already on remote, skipping push"
else
git push origin "$TAG"
fi
# Create or update release
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists, updating"
gh release edit "$TAG" --title "$TITLE" --notes-file /tmp/release-notes.md
else
gh release create "$TAG" --title "$TITLE" --notes-file /tmp/release-notes.md
fi
working-directory: ${{ github.workspace }}
- name: Update CHANGELOG.md via PR
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
VERSION="${{ steps.version.outputs.version }}"
DATE=$(date -u +"%Y-%m-%d")
BRANCH="docs/agent-sdk-changelog-v${VERSION}"
CHANGELOG="packages/browseros-agent/packages/agent-sdk/CHANGELOG.md"
# Return to main before branching
git checkout main
# Use head/tail to safely insert without sed quoting issues
{
head -n 1 "$CHANGELOG"
echo ""
echo "## v${VERSION} (${DATE})"
echo ""
cat /tmp/release-notes.md
echo ""
tail -n +2 "$CHANGELOG"
} > /tmp/new-changelog.md
mv /tmp/new-changelog.md "$CHANGELOG"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git checkout -b "$BRANCH"
git add "$CHANGELOG"
git commit -m "docs: update agent-sdk changelog for v${VERSION}"
git push origin "$BRANCH"
gh pr create \
--title "docs: update agent-sdk changelog for v${VERSION}" \
--body "Auto-generated changelog update for BrowserOS Agent SDK v${VERSION}." \
--base main \
--head "$BRANCH"
gh pr merge "$BRANCH" --squash --auto || true
working-directory: ${{ github.workspace }}
- run: echo "Agent SDK publishing is disabled."

View File

@@ -30,12 +30,50 @@ jobs:
fail-fast: false
matrix:
include:
- suite: tools
test_path: tests/tools
junit_path: test-results/tools.xml
- suite: integration
test_path: tests/server.integration.test.ts
junit_path: test-results/integration.xml
- suite: server-agent
command: (cd apps/server && bun run test:agent)
junit_path: test-results/server-agent.xml
needs_browser: false
- suite: server-api
command: (cd apps/server && bun run test:api)
junit_path: test-results/server-api.xml
needs_browser: false
- suite: server-skills
command: (cd apps/server && bun run test:skills)
junit_path: test-results/server-skills.xml
needs_browser: false
- suite: server-tools
command: (cd apps/server && bun run test:tools)
junit_path: test-results/server-tools.xml
needs_browser: true
- suite: server-browser
command: (cd apps/server && bun run test:browser)
junit_path: test-results/server-browser.xml
needs_browser: false
- suite: server-integration
command: (cd apps/server && bun run test:integration)
junit_path: test-results/server-integration.xml
needs_browser: true
- suite: server-lib
command: (cd apps/server && bun run test:lib)
junit_path: test-results/server-lib.xml
needs_browser: false
- suite: server-root
command: (cd apps/server && bun run test:root)
junit_path: test-results/server-root.xml
needs_browser: false
- suite: agent
command: bun run test:agent
junit_path: test-results/agent.xml
needs_browser: false
- suite: eval
command: bun run test:eval
junit_path: test-results/eval.xml
needs_browser: false
- suite: build
command: bun run test:build
junit_path: test-results/build.xml
needs_browser: false
steps:
- name: Checkout code
@@ -48,6 +86,7 @@ jobs:
run: bun ci
- name: Resolve BrowserOS cache key
if: matrix.needs_browser == true
id: browseros-cache-key
run: |
set -euo pipefail
@@ -62,6 +101,7 @@ jobs:
echo "key=browseros-appimage-${{ runner.os }}-$cache_key" >> "$GITHUB_OUTPUT"
- name: Restore BrowserOS cache
if: matrix.needs_browser == true
id: browseros-cache
uses: actions/cache@v4
with:
@@ -69,13 +109,14 @@ jobs:
key: ${{ steps.browseros-cache-key.outputs.key }}
- name: Download BrowserOS
if: steps.browseros-cache.outputs.cache-hit != 'true'
if: matrix.needs_browser == true && steps.browseros-cache.outputs.cache-hit != 'true'
run: |
mkdir -p .ci/bin
curl -fsSL "$BROWSEROS_APPIMAGE_URL" -o .ci/bin/BrowserOS.AppImage
chmod +x .ci/bin/BrowserOS.AppImage
- name: Prepare BrowserOS wrapper
if: matrix.needs_browser == true
run: |
mkdir -p .ci/bin
cat > .ci/bin/browseros <<'EOF'
@@ -96,16 +137,23 @@ jobs:
BROWSEROS_BINARY: ${{ github.workspace }}/packages/browseros-agent/.ci/bin/browseros
BROWSEROS_TEST_HEADLESS: "true"
BROWSEROS_TEST_EXTRA_ARGS: --no-sandbox --disable-dev-shm-usage
BROWSEROS_JUNIT_PATH: ${{ github.workspace }}/packages/browseros-agent/${{ matrix.junit_path }}
run: |
set +e
mkdir -p test-results
cd apps/server
bun run test:cleanup
bun --env-file=.env.development test "${{ matrix.test_path }}" --reporter=junit --reporter-outfile="../../${{ matrix.junit_path }}"
${{ matrix.command }}
exit_code=$?
cd ../..
if [ ! -f "${{ matrix.junit_path }}" ]; then
cat > "${{ matrix.junit_path }}" <<EOF
if [ "$exit_code" = "0" ]; then
cat > "${{ matrix.junit_path }}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="0" failures="0">
<testsuite name="${{ matrix.suite }}" tests="0" failures="0">
</testsuite>
</testsuites>
EOF
else
cat > "${{ matrix.junit_path }}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="1" failures="1">
<testsuite name="${{ matrix.suite }}" tests="1" failures="1">
@@ -115,6 +163,7 @@ jobs:
</testsuite>
</testsuites>
EOF
fi
fi
echo "exit_code=$exit_code" >> "$GITHUB_OUTPUT"
@@ -136,3 +185,124 @@ jobs:
echo "See the uploaded \`junit-${{ matrix.suite }}\` artifact for details." >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
comment:
name: PR test summary
needs: test
if: >-
always()
&& github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
pull-requests: write
actions: read
steps:
- name: Download JUnit artifacts
uses: actions/download-artifact@v4
continue-on-error: true
with:
path: junit
pattern: junit-*
- name: Build comment body
run: |
python3 <<'PY'
import glob, os, xml.etree.ElementTree as ET
run_url = f"{os.environ['GITHUB_SERVER_URL']}/{os.environ['GITHUB_REPOSITORY']}/actions/runs/{os.environ['GITHUB_RUN_ID']}"
marker = "<!-- browseros-agent-tests-summary -->"
suites = []
failed_cases = []
total_tests = total_failed = total_skipped = 0
for xml_path in sorted(glob.glob("junit/junit-*/*.xml")):
suite_name = os.path.basename(os.path.dirname(xml_path)).removeprefix("junit-")
try:
root = ET.parse(xml_path).getroot()
except ET.ParseError:
suites.append({"name": suite_name, "passed": 0, "failed": 1, "skipped": 0, "total": 1})
total_tests += 1
total_failed += 1
failed_cases.append((suite_name, "(could not parse junit XML)"))
continue
testsuites = root.findall("testsuite") if root.tag == "testsuites" else [root]
s_tests = s_fail = s_err = s_skip = 0
for ts in testsuites:
s_tests += int(ts.get("tests") or 0)
s_fail += int(ts.get("failures") or 0)
s_err += int(ts.get("errors") or 0)
s_skip += int(ts.get("skipped") or 0)
for tc in ts.iter("testcase"):
if tc.find("failure") is not None or tc.find("error") is not None:
cls = tc.get("classname") or ""
name = tc.get("name") or "(unnamed)"
label = f"{cls} > {name}" if cls else name
failed_cases.append((suite_name, label))
s_failed = s_fail + s_err
s_passed = max(s_tests - s_failed - s_skip, 0)
suites.append({"name": suite_name, "passed": s_passed, "failed": s_failed, "skipped": s_skip, "total": s_tests})
total_tests += s_tests
total_failed += s_failed
total_skipped += s_skip
total_passed = max(total_tests - total_failed - total_skipped, 0)
if total_tests == 0:
header = "## :warning: No test results were produced"
elif total_failed == 0:
header = f"## :white_check_mark: Tests passed — {total_passed}/{total_tests}"
else:
header = f"## :x: Tests failed — {total_failed}/{total_tests} failed"
lines = [marker, header, ""]
if suites:
lines.append("| Suite | Passed | Failed | Skipped |")
lines.append("|-------|--------|--------|---------|")
for s in suites:
icon = ":white_check_mark:" if s["failed"] == 0 and s["total"] > 0 else ":warning:" if s["total"] == 0 else ":x:"
lines.append(f"| {icon} `{s['name']}` | {s['passed']}/{s['total']} | {s['failed']} | {s['skipped']} |")
if failed_cases:
lines += ["", "<details open>", "<summary><b>Failed tests</b></summary>", ""]
for suite_name, label in failed_cases[:50]:
lines.append(f"- **{suite_name}** — `{label}`")
if len(failed_cases) > 50:
lines.append(f"- …and {len(failed_cases) - 50} more")
lines += ["", "</details>"]
lines += ["", f"[View workflow run]({run_url})"]
with open("comment.md", "w") as f:
f.write("\n".join(lines) + "\n")
PY
- name: Upsert sticky PR comment
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const body = fs.readFileSync('comment.md', 'utf8');
const marker = '<!-- browseros-agent-tests-summary -->';
const { owner, repo } = context.repo;
const issue_number = context.payload.pull_request.number;
const triggerSha = context.payload.pull_request.head.sha;
const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: issue_number });
if (pr.head.sha !== triggerSha) {
core.info(`PR head has moved (${pr.head.sha} vs ${triggerSha}) — skipping stale comment.`);
return;
}
const comments = await github.paginate(github.rest.issues.listComments, {
owner, repo, issue_number, per_page: 100,
});
const existing = comments.find(c => c.body && c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body });
} else {
await github.rest.issues.createComment({ owner, repo, issue_number, body });
}

2
.gitignore vendored
View File

@@ -1,4 +1,6 @@
**/.DS_Store
**.auctor/**
.auctor.json
.gcs_entries
**/dmg
**/env

View File

@@ -14,6 +14,7 @@ lerna-debug.log*
# Ignore all .env files except .env.example
**/.env.*
!**/.env.example
!**/.env.sample
!**/.env.production.example
@@ -179,6 +180,7 @@ packages/*/dist
browseros-server
browseros-server.exe
browseros-server-*
tools/dogfood/browseros-dogfood
tools/dev/browseros-dev
log.txt

View File

@@ -218,3 +218,9 @@ This uses the same element resolution as the server's MCP tools — no coordinat
The `<target>` argument can be:
- An **index** from the `targets` output (e.g., `3`)
- A **URL substring** (e.g., `sidepanel`, `newtab`, `chrome-extension://`)
## Release gating — bundled-VM runtime migration (2026-Q2)
Between the Lima server-prod-resources cutover (WS3) and the ContainerRuntime migration (WS6) landing, `resources/bin/third_party/` ships `limactl` instead of `podman`. The current OpenClaw runtime (`apps/server/src/api/services/openclaw/podman-runtime.ts`, `container-runtime.ts`) still invokes `podman`; it will fail to find the binary on builds cut from `dev`.
Do **not** cut a release branch off `dev` during this window. Track WS6 progress before any release cut. See `specs/bundled-vm-runtime-spec.md` + `specs/workstreams.md` for context.

View File

@@ -1,6 +1,6 @@
# BrowserOS Agent
The agent platform powering [BrowserOS](https://github.com/browseros-ai/BrowserOS) — contains the MCP server, agent UI, CLI, evaluation framework, and SDK.
The agent platform powering [BrowserOS](https://github.com/browseros-ai/BrowserOS) — contains the MCP server, agent UI, CLI, and evaluation framework.
## Monorepo Structure
@@ -12,7 +12,6 @@ apps/
eval/ # Evaluation framework for benchmarking agents
packages/
agent-sdk/ # Node.js SDK (@browseros-ai/agent-sdk)
cdp-protocol/ # Type-safe Chrome DevTools Protocol bindings
shared/ # Shared constants (ports, timeouts, limits)
```
@@ -23,7 +22,6 @@ packages/
| `apps/agent` | Agent UI — Chrome extension for the chat interface |
| `apps/cli` | Go CLI — control BrowserOS from the terminal or AI coding agents |
| `apps/eval` | Benchmark framework — WebVoyager, Mind2Web evaluation |
| `packages/agent-sdk` | Node.js SDK for browser automation with natural language |
| `packages/cdp-protocol` | Auto-generated CDP type bindings used by the server |
| `packages/shared` | Shared constants used across packages |
@@ -75,26 +73,20 @@ packages/
### Setup
Requires [process-compose](https://github.com/F1bonacc1/process-compose):
```bash
brew install process-compose
```
```bash
# Copy environment files for each package
cp apps/server/.env.example apps/server/.env.development
cp apps/agent/.env.example apps/agent/.env.development
cp apps/server/.env.production.example apps/server/.env.production
# Install deps, generate agent code, and sync the VM cache
bun run dev:setup
# Start the full dev environment
process-compose up
bun run dev:watch
```
The `process-compose up` command runs the following in order:
1. `bun install` — installs dependencies
2. `bun --cwd apps/agent codegen` — generates agent code
3. `bun --cwd apps/server start` and `bun --cwd apps/agent dev` — starts server and agent in parallel
`dev:watch` exits when the VM cache manifest is missing, but setup stays in `dev:setup`.
### Environment Variables

View File

@@ -0,0 +1,50 @@
import type { Provider } from './chatComponentTypes'
export interface ProviderOptionGroup {
key: 'llm' | 'acp'
label: string
options: Provider[]
}
export function groupProviderOptions(
providers: Provider[],
): ProviderOptionGroup[] {
const llm = providers.filter((provider) => provider.kind !== 'acp')
const acp = providers.filter((provider) => provider.kind === 'acp')
return [
...(llm.length
? [{ key: 'llm' as const, label: 'AI Providers', options: llm }]
: []),
...(acp.length
? [{ key: 'acp' as const, label: 'Agents', options: acp }]
: []),
]
}
export function getProviderSearchValue(
provider: Provider,
groupLabel: string,
): string {
return [
provider.id,
provider.name,
provider.type,
groupLabel,
provider.adapterName,
provider.modelLabel,
]
.filter(Boolean)
.join(' ')
}
export function getProviderSubtitle(provider: Provider): string | undefined {
if (provider.kind !== 'acp') return undefined
return [
provider.adapterName,
provider.modelLabel,
provider.modelControl === 'best-effort' ? 'best effort' : undefined,
]
.filter(Boolean)
.join(' · ')
}

View File

@@ -0,0 +1,72 @@
import { describe, expect, it } from 'bun:test'
import {
getProviderSearchValue,
getProviderSubtitle,
groupProviderOptions,
} from './ChatProviderSelector.helpers'
import type { Provider } from './chatComponentTypes'
const options: Provider[] = [
{ kind: 'llm', id: 'browseros', name: 'BrowserOS', type: 'browseros' },
{
kind: 'llm',
id: 'anthropic-sonnet',
name: 'Anthropic Sonnet',
type: 'anthropic',
},
{
kind: 'acp',
id: 'agent-claude-review',
name: 'Review Bot',
type: 'acp',
adapterName: 'Claude Code',
modelLabel: 'Haiku',
modelControl: 'best-effort',
},
{
kind: 'acp',
id: 'agent-codex-browser',
name: 'Browser Driver',
type: 'acp',
adapterName: 'Codex',
modelLabel: 'GPT-5.5',
modelControl: 'runtime-supported',
},
]
describe('groupProviderOptions', () => {
it('groups normal providers separately from created agents', () => {
expect(groupProviderOptions(options)).toEqual([
{
key: 'llm',
label: 'AI Providers',
options: [options[0], options[1]],
},
{
key: 'acp',
label: 'Agents',
options: [options[2], options[3]],
},
])
})
})
describe('getProviderSearchValue', () => {
it('matches created-agent group labels and item labels', () => {
expect(getProviderSearchValue(options[2], 'Agents')).toContain('Agents')
expect(getProviderSearchValue(options[2], 'Agents')).toContain('Review Bot')
expect(getProviderSearchValue(options[2], 'Agents')).toContain(
'Claude Code',
)
})
})
describe('getProviderSubtitle', () => {
it('describes created-agent runtime context without model-target copy', () => {
expect(getProviderSubtitle(options[2])).toBe(
'Claude Code · Haiku · best effort',
)
expect(getProviderSubtitle(options[3])).toBe('Codex · GPT-5.5')
expect(getProviderSubtitle(options[0])).toBeUndefined()
})
})

View File

@@ -1,4 +1,4 @@
import { Check, Plus } from 'lucide-react'
import { Bot, Check, Plus } from 'lucide-react'
import type { FC, PropsWithChildren } from 'react'
import { useState } from 'react'
import {
@@ -17,6 +17,11 @@ import {
import { BrowserOSIcon, ProviderIcon } from '@/lib/llm-providers/providerIcons'
import type { ProviderType } from '@/lib/llm-providers/types'
import { cn } from '@/lib/utils'
import {
getProviderSearchValue,
getProviderSubtitle,
groupProviderOptions,
} from './ChatProviderSelector.helpers'
import type { Provider } from './chatComponentTypes'
interface ChatProviderSelectorProps {
@@ -29,54 +34,58 @@ export const ChatProviderSelector: FC<
PropsWithChildren<ChatProviderSelectorProps>
> = ({ children, providers, selectedProvider, onSelectProvider }) => {
const [open, setOpen] = useState(false)
const groups = groupProviderOptions(providers)
return (
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>{children}</PopoverTrigger>
<PopoverContent side="bottom" align="start" className="w-48 p-0">
<PopoverContent side="bottom" align="start" className="w-64 p-0">
<Command>
<CommandInput placeholder="Search providers..." className="h-9" />
<CommandInput
placeholder="Search providers or agents..."
className="h-9"
/>
<CommandList>
<div className="my-2 px-2 font-semibold text-muted-foreground text-xs uppercase tracking-wide">
AI Provider
</div>
<CommandEmpty>No provider found</CommandEmpty>
<CommandGroup>
{providers.map((provider) => {
const isSelected = selectedProvider.id === provider.id
return (
<CommandItem
key={provider.id}
value={`${provider.id} ${provider.name}`}
onSelect={() => {
onSelectProvider(provider)
setOpen(false)
}}
className={cn(
'flex w-full items-center gap-3 rounded-md p-2 transition-colors',
isSelected && 'bg-[var(--accent-orange)]/10',
)}
>
<span className="text-muted-foreground">
{provider.type === 'browseros' ? (
<BrowserOSIcon size={18} />
) : (
<ProviderIcon
type={provider.type as ProviderType}
size={18}
/>
{groups.map((group) => (
<CommandGroup key={group.key} heading={group.label}>
{group.options.map((provider) => {
const isSelected = selectedProvider.id === provider.id
const subtitle = getProviderSubtitle(provider)
return (
<CommandItem
key={provider.id}
value={getProviderSearchValue(provider, group.label)}
onSelect={() => {
onSelectProvider(provider)
setOpen(false)
}}
className={cn(
'flex w-full items-center gap-3 rounded-md p-2 transition-colors',
isSelected && 'bg-[var(--accent-orange)]/10',
)}
</span>
<span className="flex-1 text-left text-sm">
{provider.name}
</span>
{isSelected && (
<Check className="h-3.5 w-3.5 text-[var(--accent-orange)]" />
)}
</CommandItem>
)
})}
</CommandGroup>
>
<span className="text-muted-foreground">
<ProviderOptionIcon provider={provider} />
</span>
<span className="min-w-0 flex-1 text-left">
<span className="block truncate text-sm">
{provider.name}
</span>
{subtitle && (
<span className="block truncate text-muted-foreground text-xs">
{subtitle}
</span>
)}
</span>
{isSelected && (
<Check className="h-3.5 w-3.5 text-[var(--accent-orange)]" />
)}
</CommandItem>
)
})}
</CommandGroup>
))}
<div className="border-border border-t p-1">
<button
type="button"
@@ -96,3 +105,9 @@ export const ChatProviderSelector: FC<
</Popover>
)
}
function ProviderOptionIcon({ provider }: { provider: Provider }) {
if (provider.kind === 'acp') return <Bot size={18} />
if (provider.type === 'browseros') return <BrowserOSIcon size={18} />
return <ProviderIcon type={provider.type as ProviderType} size={18} />
}

View File

@@ -1,7 +1,14 @@
import type { ProviderType } from '@/lib/llm-providers/types'
export type ChatProviderType = ProviderType | 'acp'
export interface Provider {
id: string
name: string
type: ProviderType
type: ChatProviderType
kind: 'llm' | 'acp'
agentId?: string
adapterName?: string
modelLabel?: string
modelControl?: 'runtime-supported' | 'best-effort'
}

View File

@@ -74,6 +74,18 @@ const primaryNavItems: NavItem[] = [
{ name: 'Settings', to: '/settings/ai', icon: Settings },
]
function isNavItemActive(item: NavItem, pathname: string): boolean {
if (item.to === '/settings/ai') {
return pathname.startsWith('/settings')
}
if (item.to === '/agents') {
return pathname === '/agents' || pathname.startsWith('/agents/')
}
return pathname === item.to
}
export const SidebarNavigation: FC<SidebarNavigationProps> = ({
expanded = true,
}) => {
@@ -90,10 +102,7 @@ export const SidebarNavigation: FC<SidebarNavigationProps> = ({
<nav className="space-y-1">
{filteredItems.map((item) => {
const Icon = item.icon
const isActive =
item.to === '/settings/ai'
? location.pathname.startsWith('/settings')
: location.pathname === item.to
const isActive = isNavItemActive(item, location.pathname)
const navItem = (
<NavLink

View File

@@ -113,7 +113,22 @@ export const App: FC = () => {
<Route path="connect-apps" element={<ConnectMCP />} />
<Route path="scheduled" element={<ScheduledTasksPage />} />
{alphaEnabled ? (
<Route path="agents" element={<AgentsPage />} />
<>
<Route path="agents" element={<AgentsPage />} />
<Route element={<AgentCommandLayout />}>
<Route
path="agents/:agentId"
element={
<AgentCommandConversation
variant="page"
backPath="/agents"
agentPathPrefix="/agents"
createAgentPath="/agents"
/>
}
/>
</Route>
</>
) : null}
{alphaEnabled ? (
<Route path="admin" element={<AdminDashboardPage />} />

View File

@@ -1,4 +1,4 @@
import { Bot } from 'lucide-react'
import { Bot, Loader2, Wrench } from 'lucide-react'
import type { FC } from 'react'
import type { AgentCardData } from '@/lib/agent-conversations/types'
import { cn } from '@/lib/utils'
@@ -32,6 +32,11 @@ function getStatusTone(status: AgentCardData['status']): string {
return 'bg-emerald-500'
}
function formatCost(usd: number): string {
if (usd < 0.005) return `$${usd.toFixed(4)}`
return `$${usd.toFixed(2)}`
}
export const AgentCardExpanded: FC<AgentCardProps> = ({
agent,
onClick,
@@ -81,9 +86,26 @@ export const AgentCardExpanded: FC<AgentCardProps> = ({
</p>
</div>
<div className="mt-4 flex items-center justify-between gap-3 text-muted-foreground text-xs">
<span>{formatTimestamp(agent.lastMessageTimestamp)}</span>
<span>Open conversation</span>
<div className="mt-4 space-y-1.5 text-muted-foreground text-xs">
<div className="flex items-center justify-between gap-3">
<span>{formatTimestamp(agent.lastMessageTimestamp)}</span>
{agent.costUsd ? (
<span className="tabular-nums opacity-70">
{formatCost(agent.costUsd)}
</span>
) : null}
</div>
{agent.status === 'working' && agent.currentTool ? (
<div className="flex items-center gap-1.5 text-[var(--accent-orange)]/70">
<Loader2 className="size-3 shrink-0 animate-spin" />
<span className="truncate">{agent.currentTool}</span>
</div>
) : agent.activitySummary ? (
<div className="flex items-center gap-1.5 text-muted-foreground/60">
<Wrench className="size-3 shrink-0" />
<span className="truncate">{agent.activitySummary}</span>
</div>
) : null}
</div>
</button>
)

View File

@@ -1,190 +1,308 @@
import { Bot, Home, RotateCcw } from 'lucide-react'
import { type FC, useEffect, useRef } from 'react'
import { ArrowLeft, Bot, Home } from 'lucide-react'
import { type FC, useEffect, useMemo, useRef } from 'react'
import { Navigate, useNavigate, useParams, useSearchParams } from 'react-router'
import { Button } from '@/components/ui/button'
import type { AgentEntry } from '@/entrypoints/app/agents/useOpenClaw'
import {
type AgentEntry,
getModelDisplayName,
} from '@/entrypoints/app/agents/useOpenClaw'
import { cn } from '@/lib/utils'
import { useAgentCommandData } from './agent-command-layout'
import { ClawChat } from './ClawChat'
import { ConversationInput } from './ConversationInput'
import { ConversationMessage } from './ConversationMessage'
import {
buildChatHistoryFromClawMessages,
filterTurnsPersistedInHistory,
flattenHistoryPages,
} from './claw-chat-types'
import { useAgentConversation } from './useAgentConversation'
import { useHarnessChatHistory } from './useHarnessChatHistory'
function StatusBadge({ status }: { status: string }) {
return (
<div className="inline-flex items-center gap-2 rounded-full border border-border/60 bg-card px-3 py-1 text-[11px] text-muted-foreground uppercase tracking-[0.18em]">
<span
className={cn(
'size-1.5 rounded-full',
status === 'Working on your request'
? 'bg-amber-500'
: status === 'Ready'
? 'bg-emerald-500'
: status === 'Offline'
? 'bg-muted-foreground/50'
: 'bg-[var(--accent-orange)]',
)}
/>
<span>{status}</span>
</div>
)
}
function AgentIdentity({
name,
meta,
className,
}: {
name: string
meta: string
className?: string
}) {
return (
<div className={cn('min-w-0', className)}>
<div className="truncate font-semibold text-[15px] leading-5">{name}</div>
<div className="truncate text-muted-foreground text-xs leading-5">
{meta}
</div>
</div>
)
}
function ConversationHeader({
agentName,
agentMeta,
status,
backLabel,
backTarget,
onGoHome,
onReset,
}: {
agentName: string
agentMeta: string
status: string
backLabel: string
backTarget: 'home' | 'page'
onGoHome: () => void
onReset: () => void
}) {
const BackIcon = backTarget === 'home' ? Home : ArrowLeft
return (
<div className="overflow-hidden rounded-[1.5rem] border border-border/60 bg-card/95 shadow-sm backdrop-blur">
<div className="flex items-center justify-between gap-3 px-5 py-4">
<div className="flex min-w-0 items-center gap-3">
<Button
variant="ghost"
size="icon"
onClick={onGoHome}
className="rounded-xl"
title="Back to home"
>
<Home className="size-4" />
</Button>
<div className="flex size-11 shrink-0 items-center justify-center rounded-2xl bg-muted text-muted-foreground">
<Bot className="size-5" />
</div>
<div className="min-w-0">
<div className="truncate font-semibold text-sm">{agentName}</div>
<div className="truncate text-muted-foreground text-sm">
{status}
</div>
</div>
</div>
<div className="flex h-14 items-center justify-between gap-4 border-border/50 border-b px-5">
<div className="flex min-w-0 items-center gap-3">
<Button
variant="ghost"
size="sm"
onClick={onReset}
className="rounded-xl text-muted-foreground"
size="icon"
onClick={onGoHome}
className="size-8 rounded-xl lg:hidden"
title={backLabel}
>
<RotateCcw className="mr-2 size-4" />
New conversation
<BackIcon className="size-4" />
</Button>
</div>
</div>
)
}
function EmptyConversationState({ agentName }: { agentName: string }) {
return (
<div className="flex min-h-full items-center justify-center py-10">
<div className="max-w-md rounded-[1.5rem] border border-border/60 bg-card/90 px-8 py-10 text-center shadow-sm backdrop-blur">
<div className="mx-auto flex size-14 items-center justify-center rounded-2xl bg-muted text-muted-foreground">
<Bot className="size-6" />
<div className="flex size-8 shrink-0 items-center justify-center rounded-xl bg-muted text-muted-foreground">
<Bot className="size-4" />
</div>
<AgentIdentity name={agentName} meta={agentMeta} />
</div>
<StatusBadge status={status} />
</div>
)
}
function AgentRailHeader({ onGoHome }: { onGoHome: () => void }) {
return (
<div className="hidden h-14 items-center border-border/50 border-r border-b bg-background/70 px-4 lg:flex">
<div className="flex min-w-0 items-center gap-3">
<Button
variant="ghost"
size="icon"
onClick={onGoHome}
className="size-8 rounded-xl"
title="Back to home"
>
<ArrowLeft className="size-4" />
</Button>
<div className="truncate font-semibold text-[15px] leading-5">
Agents
</div>
<h2 className="mt-4 font-semibold text-lg">{agentName}</h2>
<p className="mt-2 text-muted-foreground text-sm">
Send a message to start a focused conversation with this agent.
</p>
</div>
</div>
)
}
function getConversationStatusCopy(
status: string | undefined,
streaming: boolean,
): string {
if (streaming) return 'Working on your request'
if (status === 'running') return 'Ready for the next task'
if (status === 'starting') return 'Connecting to OpenClaw'
if (status === 'error') return 'OpenClaw needs attention'
if (status === 'stopped') return 'OpenClaw is offline'
return 'Open agent setup to continue'
function AgentRailList({
activeAgentId,
agents,
onSelectAgent,
}: {
activeAgentId: string
agents: AgentEntry[]
onSelectAgent: (entry: AgentEntry) => void
}) {
return (
<aside className="hidden min-h-0 flex-col border-border/50 border-r bg-background/70 lg:flex">
<div className="styled-scrollbar min-h-0 flex-1 space-y-2 overflow-y-auto px-3 py-3">
{agents.map((entry) => {
const active = entry.agentId === activeAgentId
const modelName = getAgentEntryMeta(entry)
return (
<button
key={entry.agentId}
type="button"
onClick={() => onSelectAgent(entry)}
className={cn(
'w-full rounded-2xl border px-3 py-3 text-left transition-all',
active
? 'border-[var(--accent-orange)]/30 bg-[var(--accent-orange)]/8 shadow-sm'
: 'border-transparent bg-transparent hover:border-border/60 hover:bg-card',
)}
>
<div className="flex items-center gap-3">
<div
className={cn(
'flex size-9 items-center justify-center rounded-xl',
active
? 'bg-[var(--accent-orange)]/12 text-[var(--accent-orange)]'
: 'bg-muted text-muted-foreground',
)}
>
<Bot className="size-4" />
</div>
<AgentIdentity name={entry.name} meta={modelName} />
</div>
</button>
)
})}
</div>
</aside>
)
}
export const AgentCommandConversation: FC = () => {
const { agentId } = useParams<{ agentId: string }>()
const [searchParams, setSearchParams] = useSearchParams()
function getAgentEntryMeta(agent: AgentEntry | undefined): string {
if (agent?.source === 'agent-harness') {
return getModelDisplayName(agent.model) ?? 'ACP agent'
}
return getModelDisplayName(agent?.model) ?? 'OpenClaw agent'
}
function AgentConversationController({
agentId,
initialMessage,
onInitialMessageConsumed,
agents,
agentPathPrefix,
createAgentPath,
}: {
agentId: string
initialMessage: string | null
onInitialMessageConsumed: () => void
agents: AgentEntry[]
agentPathPrefix: string
createAgentPath: string
}) {
const navigate = useNavigate()
const scrollRef = useRef<HTMLDivElement>(null)
const initialQuerySent = useRef(false)
const { status, agents } = useAgentCommandData()
const shouldRedirectHome = !agentId
const resolvedAgentId = agentId ?? ''
const agent = agents.find((entry) => entry.agentId === resolvedAgentId)
const agentName = agent?.name || resolvedAgentId || 'Agent'
const { turns, streaming, loading, send, resetConversation } =
useAgentConversation(resolvedAgentId, agentName)
const lastTurn = turns[turns.length - 1]
const lastTurnPartCount = lastTurn?.parts.length ?? 0
const initialMessageSentRef = useRef<string | null>(null)
const onInitialMessageConsumedRef = useRef(onInitialMessageConsumed)
const agent = agents.find((entry) => entry.agentId === agentId)
const agentName = agent?.name || agentId || 'Agent'
// Routing is now harness-only. Every OpenClaw agent has a harness
// record post the gateway → harness backfill, so the chat panel
// always talks to /agents/<id>/chat. The legacy ClawChat surface
// was deleted with the /claw/agents/:id/chat server route.
const harnessHistoryQuery = useHarnessChatHistory(agentId, Boolean(agent))
const historyMessages = useMemo(
() =>
flattenHistoryPages(
harnessHistoryQuery.data ? [harnessHistoryQuery.data] : [],
),
[harnessHistoryQuery.data],
)
const chatHistory = useMemo(
() => buildChatHistoryFromClawMessages(historyMessages),
[historyMessages],
)
const { turns, streaming, send } = useAgentConversation(agentId, {
runtime: 'agent-harness',
sessionKey: null,
history: chatHistory,
onComplete: () => {
void harnessHistoryQuery.refetch()
},
onSessionKeyChange: () => {},
})
const visibleTurns = useMemo(
() => filterTurnsPersistedInHistory(turns, historyMessages),
[historyMessages, turns],
)
onInitialMessageConsumedRef.current = onInitialMessageConsumed
const disabled = !agent
const historyReady =
harnessHistoryQuery.isFetched || harnessHistoryQuery.isError
const initialMessageKey = initialMessage
? `${agentId}:${initialMessage}`
: null
const error = harnessHistoryQuery.error ?? null
const sendRef = useRef(send)
sendRef.current = send
useEffect(() => {
if (shouldRedirectHome) return
const query = searchParams.get('q')
if (query && !initialQuerySent.current && !loading) {
initialQuerySent.current = true
setSearchParams({}, { replace: true })
void send(query)
const query = initialMessage?.trim()
if (!initialMessageKey) {
initialMessageSentRef.current = null
return
}
}, [loading, searchParams, send, setSearchParams, shouldRedirectHome])
useEffect(() => {
if (
shouldRedirectHome ||
(turns.length === 0 && lastTurnPartCount === 0 && !streaming)
!query ||
initialMessageSentRef.current === initialMessageKey ||
disabled ||
!historyReady
) {
return
}
scrollRef.current?.scrollTo({
top: scrollRef.current.scrollHeight,
behavior: 'smooth',
})
}, [lastTurnPartCount, shouldRedirectHome, streaming, turns.length])
if (shouldRedirectHome) {
return <Navigate to="/home" replace />
}
initialMessageSentRef.current = initialMessageKey
onInitialMessageConsumedRef.current()
void sendRef.current({ text: query })
}, [disabled, historyReady, initialMessage, initialMessageKey])
const handleSelectAgent = (entry: AgentEntry) => {
navigate(`/home/agents/${entry.agentId}`)
navigate(`${agentPathPrefix}/${entry.agentId}`)
}
const statusCopy = getConversationStatusCopy(status?.status, streaming)
return (
<div className="absolute inset-0 overflow-hidden">
<div className="fade-in slide-in-from-bottom-5 mx-auto flex h-full w-full max-w-3xl animate-in flex-col gap-3 px-4 pt-4 pb-2 duration-300">
<ConversationHeader
agentName={agentName}
status={statusCopy}
onGoHome={() => navigate('/home')}
onReset={resetConversation}
/>
<div className="flex min-h-0 flex-col overflow-hidden">
<ClawChat
agentName={agentName}
historyMessages={historyMessages}
turns={visibleTurns}
streaming={streaming}
isInitialLoading={harnessHistoryQuery.isLoading}
error={error}
hasNextPage={false}
isFetchingNextPage={false}
onFetchNextPage={() => {}}
onRetry={() => {
void harnessHistoryQuery.refetch()
}}
/>
<main
ref={scrollRef}
className={cn(
'styled-scrollbar min-h-0 flex-1 overflow-y-auto overflow-x-hidden rounded-[1.5rem] border border-border/50 bg-card/85 px-5 py-5 shadow-sm',
'[&_[data-streamdown="code-block"]]:!max-w-full [&_[data-streamdown="table-wrapper"]]:!max-w-full [&_[data-streamdown="code-block"]]:overflow-x-auto [&_[data-streamdown="table-wrapper"]]:overflow-x-auto',
)}
>
{loading ? (
<div className="flex h-full items-center justify-center text-muted-foreground text-sm">
Loading conversation...
</div>
) : turns.length === 0 ? (
<EmptyConversationState agentName={agentName} />
) : (
<div className="w-full space-y-4">
{turns.map((turn, index) => (
<ConversationMessage
key={turn.id}
turn={turn}
streaming={streaming && index === turns.length - 1}
/>
))}
</div>
)}
</main>
<div className="w-full flex-shrink-0">
<div className="border-border/50 border-t bg-background/88 px-4 py-3 backdrop-blur-md">
<div className="mx-auto max-w-3xl">
<ConversationInput
variant="conversation"
agents={agents}
selectedAgentId={resolvedAgentId}
selectedAgentId={agentId}
onSelectAgent={handleSelectAgent}
onSend={(text) => {
void send(text)
onSend={(input) => {
const attachments = input.attachments.map((a) => a.payload)
const attachmentPreviews = input.attachments.map((a) => ({
id: a.id,
kind: a.kind,
mediaType: a.mediaType,
name: a.name,
dataUrl: a.dataUrl,
}))
void send({ text: input.text, attachments, attachmentPreviews })
}}
onCreateAgent={() => navigate('/agents')}
onCreateAgent={() => navigate(createAgentPath)}
streaming={streaming}
disabled={status?.status !== 'running'}
status={status?.status}
disabled={disabled}
status="running"
attachmentsEnabled={true}
placeholder={`Message ${agentName}...`}
/>
</div>
@@ -192,3 +310,79 @@ export const AgentCommandConversation: FC = () => {
</div>
)
}
interface AgentCommandConversationProps {
variant?: 'command' | 'page'
backPath?: string
agentPathPrefix?: string
createAgentPath?: string
}
export const AgentCommandConversation: FC<AgentCommandConversationProps> = ({
variant = 'command',
backPath = '/home',
agentPathPrefix = '/home/agents',
createAgentPath = '/agents',
}) => {
const { agentId } = useParams<{ agentId: string }>()
const [searchParams, setSearchParams] = useSearchParams()
const navigate = useNavigate()
const { agents } = useAgentCommandData()
const shouldRedirectHome = !agentId
const resolvedAgentId = agentId ?? ''
const agent = agents.find((entry) => entry.agentId === resolvedAgentId)
const agentName = agent?.name || resolvedAgentId || 'Agent'
const agentMeta = getAgentEntryMeta(agent)
const initialMessage = searchParams.get('q')
const isPageVariant = variant === 'page'
const backLabel = isPageVariant ? 'Back to agents' : 'Back to home'
if (shouldRedirectHome) {
return <Navigate to="/home" replace />
}
const handleSelectAgent = (entry: AgentEntry) => {
navigate(`${agentPathPrefix}/${entry.agentId}`)
}
// Every visible agent runs through the harness now, so per-agent
// runtime status doesn't gate chat the way OpenClaw's legacy
// gateway lifecycle did. Show "Ready" once the agent record is
// resolved from the rail, "Setup" otherwise.
const statusCopy = agent ? 'Ready' : 'Setup'
return (
<div className="absolute inset-0 overflow-hidden bg-background md:pl-[theme(spacing.14)]">
<div className="mx-auto grid h-full w-full max-w-[1480px] lg:grid-cols-[288px_minmax(0,1fr)] lg:grid-rows-[3.5rem_minmax(0,1fr)]">
<AgentRailHeader onGoHome={() => navigate(backPath)} />
<ConversationHeader
agentName={agentName}
agentMeta={agentMeta}
status={statusCopy}
backLabel={backLabel}
backTarget={isPageVariant ? 'page' : 'home'}
onGoHome={() => navigate(backPath)}
/>
<AgentRailList
activeAgentId={resolvedAgentId}
agents={agents}
onSelectAgent={handleSelectAgent}
/>
<AgentConversationController
key={resolvedAgentId}
agentId={resolvedAgentId}
agents={agents}
initialMessage={initialMessage}
onInitialMessageConsumed={() =>
setSearchParams({}, { replace: true })
}
agentPathPrefix={agentPathPrefix}
createAgentPath={createAgentPath}
/>
</div>
</div>
)
}

View File

@@ -1,89 +1,87 @@
import { ArrowRight } from 'lucide-react'
import { Plus } from 'lucide-react'
import { type FC, useEffect, useState } from 'react'
import { useNavigate } from 'react-router'
import { Button } from '@/components/ui/button'
import { Card, CardContent } from '@/components/ui/card'
import { Separator } from '@/components/ui/separator'
import type { AgentEntry } from '@/entrypoints/app/agents/useOpenClaw'
import { ImportDataHint } from '@/entrypoints/newtab/index/ImportDataHint'
import { NewTabBranding } from '@/entrypoints/newtab/index/NewTabBranding'
import { NewTabTip } from '@/entrypoints/newtab/index/NewTabTip'
import { ScheduleResults } from '@/entrypoints/newtab/index/ScheduleResults'
import { SignInHint } from '@/entrypoints/newtab/index/SignInHint'
import { TopSites } from '@/entrypoints/newtab/index/TopSites'
import { useActiveHint } from '@/entrypoints/newtab/index/useActiveHint'
import type { AgentCardData } from '@/lib/agent-conversations/types'
import { AgentCardDock } from './AgentCardDock'
import { useAgentCommandData } from './agent-command-layout'
import { ConversationInput } from './ConversationInput'
import { useAgentCardData } from './useAgentCardData'
function AgentCommandSetupState({
onOpenAgents,
}: {
onOpenAgents: () => void
}) {
return (
<Card className="border-border/60 bg-card/85 shadow-sm">
<CardContent className="flex flex-col items-center gap-4 p-6 text-center">
<p className="max-w-xl text-muted-foreground text-sm">
Set up OpenClaw agents to turn your new tab into an agent command
center.
</p>
<Button onClick={onOpenAgents} className="gap-2">
Open Agent Setup
<ArrowRight className="size-4" />
</Button>
</CardContent>
</Card>
)
}
import { buildAgentCardData } from './useAgentCardData'
function EmptyAgentsState({ onOpenAgents }: { onOpenAgents: () => void }) {
return (
<Card className="border-border/60 bg-card/85 shadow-sm">
<CardContent className="flex flex-col items-center gap-4 p-6 text-center">
<p className="max-w-xl text-muted-foreground text-sm">
OpenClaw is running, but you do not have any agents yet.
</p>
<Button variant="outline" onClick={onOpenAgents}>
Create your first agent
<Card className="border-border/60 bg-card/90 shadow-sm">
<CardContent className="flex flex-col items-center gap-4 p-8 text-center">
<div className="flex size-12 items-center justify-center rounded-2xl bg-muted text-muted-foreground">
<Plus className="size-5" />
</div>
<div className="space-y-2">
<h2 className="font-semibold text-lg">No agents yet</h2>
<p className="max-w-md text-muted-foreground text-sm leading-6">
Create an agent to start using BrowserOS as an agent-first new tab.
</p>
</div>
<Button variant="outline" onClick={onOpenAgents} className="rounded-xl">
Create agent
</Button>
</CardContent>
</Card>
)
}
function OpenClawUnavailableState({
function RecentThreads({
activeAgentId,
agents,
onOpenAgents,
onSelectAgent,
}: {
activeAgentId?: string | null
agents: AgentCardData[]
onOpenAgents: () => void
onSelectAgent: (agentId: string) => void
}) {
if (agents.length === 0) return null
return (
<Card className="border-border/60 bg-card/85 shadow-sm">
<CardContent className="flex flex-col items-center gap-4 p-6 text-center">
<p className="max-w-xl text-muted-foreground text-sm">
OpenClaw is unavailable right now. Open the Agents page to restart the
gateway or review setup.
</p>
<Button onClick={onOpenAgents} className="gap-2">
Open Agent Setup
<ArrowRight className="size-4" />
<section className="space-y-4">
<div className="flex items-center justify-between gap-4">
<div>
<h2 className="font-semibold text-base">Recent agents</h2>
<p className="text-muted-foreground text-sm">
Continue from where you left off.
</p>
</div>
<Button
variant="outline"
onClick={onOpenAgents}
className="rounded-xl"
size="sm"
>
Manage agents
</Button>
</CardContent>
</Card>
</div>
<AgentCardDock
agents={agents}
activeAgentId={activeAgentId ?? undefined}
onSelectAgent={onSelectAgent}
onCreateAgent={onOpenAgents}
/>
</section>
)
}
export const AgentCommandHome: FC = () => {
const navigate = useNavigate()
const activeHint = useActiveHint()
const { status, agents } = useAgentCommandData()
const [mounted, setMounted] = useState(false)
const { agents, status } = useAgentCommandData()
const [selectedAgentId, setSelectedAgentId] = useState<string | null>(null)
const cardData = useAgentCardData(agents, status?.status)
useEffect(() => {
setMounted(true)
}, [])
const cardData = buildAgentCardData(agents, status?.status, undefined)
useEffect(() => {
if (agents.length === 0) {
@@ -101,78 +99,77 @@ export const AgentCommandHome: FC = () => {
}
}, [agents, selectedAgentId])
const handleSend = (text: string) => {
const handleSend = (input: { text: string }) => {
if (!selectedAgentId) return
navigate(`/home/agents/${selectedAgentId}?q=${encodeURIComponent(text)}`)
navigate(
`/home/agents/${selectedAgentId}?q=${encodeURIComponent(input.text)}`,
)
}
const handleSelectAgent = (agent: AgentEntry) => {
setSelectedAgentId(agent.agentId)
}
const openClawStatus = status?.status
const isSetup = openClawStatus != null && openClawStatus !== 'uninitialized'
const shouldShowUnavailableState =
openClawStatus != null &&
openClawStatus !== 'running' &&
openClawStatus !== 'uninitialized' &&
cardData.length === 0
const selectedAgent = agents.find(
(agent) => agent.agentId === selectedAgentId,
)
const selectedAgentReady = selectedAgent
? selectedAgent.source === 'agent-harness' || status?.status === 'running'
: false
const selectedAgentStatus =
selectedAgent?.source === 'agent-harness' ? 'running' : status?.status
const selectedCard =
cardData.find((agent) => agent.agentId === selectedAgentId) ?? cardData[0]
return (
<div className="pt-[max(25vh,16px)]">
<div className="relative w-full space-y-8 md:w-3xl">
<NewTabBranding />
<ConversationInput
variant="home"
agents={agents}
selectedAgentId={selectedAgentId}
onSelectAgent={handleSelectAgent}
onSend={handleSend}
onCreateAgent={() => navigate('/agents')}
streaming={false}
disabled={status?.status !== 'running'}
status={status?.status}
placeholder={
status?.status === 'running'
? undefined
: 'OpenClaw is not running...'
}
/>
{mounted ? <NewTabTip /> : null}
{isSetup ? (
shouldShowUnavailableState ? (
<OpenClawUnavailableState
onOpenAgents={() => navigate('/agents')}
/>
) : cardData.length > 0 ? (
<section className="space-y-3">
<div className="flex items-center justify-between">
<div>
<h2 className="font-semibold text-base">Agents</h2>
<p className="text-muted-foreground text-sm">
Pick up where your agents left off.
</p>
</div>
<div className="min-h-full px-4 py-6">
<div className="mx-auto flex w-full max-w-5xl flex-col gap-8">
{cardData.length > 0 ? (
<>
<div className="flex flex-col items-center gap-5 pt-[max(10vh,24px)] text-center">
<div className="space-y-3">
<h1 className="font-semibold text-[clamp(2rem,4vw,3.25rem)] leading-tight tracking-tight">
What should your agent work on next?
</h1>
<p className="mx-auto max-w-2xl text-muted-foreground text-sm leading-6">
Start with a task, continue a thread, or switch to another
agent without leaving the new tab.
</p>
</div>
<AgentCardDock
agents={cardData}
activeAgentId={selectedAgentId ?? undefined}
onSelectAgent={(agentId) => navigate(`/home/agents/${agentId}`)}
onCreateAgent={() => navigate('/agents')}
/>
</section>
) : (
<EmptyAgentsState onOpenAgents={() => navigate('/agents')} />
)
) : (
<AgentCommandSetupState onOpenAgents={() => navigate('/agents')} />
)}
{mounted ? <TopSites /> : null}
{mounted ? <ScheduleResults /> : null}
<div className="w-full max-w-3xl">
<ConversationInput
variant="home"
agents={agents}
selectedAgentId={selectedAgentId}
onSelectAgent={handleSelectAgent}
onSend={handleSend}
onCreateAgent={() => navigate('/agents')}
streaming={false}
disabled={!selectedAgentReady}
status={selectedAgentStatus}
attachmentsEnabled={false}
placeholder={
selectedAgentReady
? `Ask ${selectedCard?.name ?? 'your agent'} to handle a task...`
: 'Agent runtime is not running...'
}
/>
</div>
</div>
<Separator />
<RecentThreads
activeAgentId={selectedAgentId}
agents={cardData}
onOpenAgents={() => navigate('/agents')}
onSelectAgent={(agentId) => navigate(`/home/agents/${agentId}`)}
/>
</>
) : (
<EmptyAgentsState onOpenAgents={() => navigate('/agents')} />
)}
</div>
{activeHint === 'signin' ? <SignInHint /> : null}

View File

@@ -0,0 +1,172 @@
import { Bot, Loader2, RefreshCw } from 'lucide-react'
import { type FC, useEffect, useRef } from 'react'
import {
Conversation,
ConversationContent,
ConversationScrollButton,
} from '@/components/ai-elements/conversation'
import type { AgentConversationTurn } from '@/lib/agent-conversations/types'
import { cn } from '@/lib/utils'
import { ClawChatMessage } from './ClawChatMessage'
import { ConversationMessage } from './ConversationMessage'
import type { ClawChatMessage as ClawChatMessageModel } from './claw-chat-types'
interface ClawChatProps {
agentName: string
historyMessages: ClawChatMessageModel[]
turns: AgentConversationTurn[]
streaming: boolean
isInitialLoading: boolean
error: Error | null
hasNextPage: boolean
isFetchingNextPage: boolean
onFetchNextPage: () => void
onRetry: () => void
className?: string
}
function EmptyConversationState({ agentName }: { agentName: string }) {
return (
<div className="flex h-full items-center justify-center px-6 py-12">
<div className="max-w-md text-center">
<div className="mx-auto flex size-14 items-center justify-center rounded-3xl bg-muted text-muted-foreground">
<Bot className="size-6" />
</div>
<h2 className="mt-5 font-semibold text-xl">{agentName}</h2>
<p className="mt-2 text-muted-foreground text-sm leading-6">
Ask {agentName} to start a task.
</p>
</div>
</div>
)
}
function LoadingConversationState() {
return (
<div className="flex h-full items-center justify-center gap-2 text-muted-foreground text-sm">
<Loader2 className="size-4 animate-spin" />
Loading conversation...
</div>
)
}
function ConversationErrorState({
message,
onRetry,
}: {
message: string
onRetry: () => void
}) {
return (
<div className="flex h-full items-center justify-center px-6 py-12">
<div className="max-w-md rounded-2xl border border-border/60 bg-card px-5 py-4 text-center shadow-sm">
<p className="text-sm">{message}</p>
<button
type="button"
onClick={onRetry}
className="mt-3 inline-flex items-center gap-2 rounded-lg border border-border/60 px-3 py-1.5 font-medium text-muted-foreground text-xs transition-colors hover:bg-accent hover:text-foreground"
>
<RefreshCw className="size-3.5" />
Retry
</button>
</div>
</div>
)
}
export const ClawChat: FC<ClawChatProps> = ({
agentName,
historyMessages,
turns,
streaming,
isInitialLoading,
error,
hasNextPage,
isFetchingNextPage,
onFetchNextPage,
onRetry,
className,
}) => {
const topSentinelRef = useRef<HTMLDivElement>(null)
const onFetchNextPageRef = useRef(onFetchNextPage)
onFetchNextPageRef.current = onFetchNextPage
const hasMessages = historyMessages.length > 0 || turns.length > 0
useEffect(() => {
const sentinel = topSentinelRef.current
if (!sentinel) return
const observer = new IntersectionObserver(
(entries) => {
const [entry] = entries
if (!entry?.isIntersecting || !hasNextPage || isFetchingNextPage) {
return
}
onFetchNextPageRef.current()
},
{
root: null,
rootMargin: '160px 0px 0px 0px',
threshold: 0,
},
)
observer.observe(sentinel)
return () => observer.disconnect()
}, [hasNextPage, isFetchingNextPage])
return (
<div
className={cn('flex min-h-0 flex-1 flex-col overflow-hidden', className)}
>
<Conversation
className={cn(
'bg-background',
'[&_[data-streamdown="code-block"]]:!w-full [&_[data-streamdown="code-block"]]:!max-w-full [&_[data-streamdown="table-wrapper"]]:!w-full [&_[data-streamdown="table-wrapper"]]:!max-w-full [&_[data-streamdown="code-block"]]:overflow-x-auto [&_[data-streamdown="table-wrapper"]]:overflow-x-auto',
)}
>
<ConversationContent className="min-h-full px-5 py-5">
{isInitialLoading ? (
<LoadingConversationState />
) : error && !hasMessages ? (
<ConversationErrorState message={error.message} onRetry={onRetry} />
) : !hasMessages ? (
<EmptyConversationState agentName={agentName} />
) : (
<div className="mx-auto flex w-full max-w-3xl flex-col gap-3">
<div ref={topSentinelRef} aria-hidden="true" className="h-px" />
{isFetchingNextPage ? (
<div className="flex justify-center py-2 text-muted-foreground text-xs">
<Loader2 className="mr-2 size-3.5 animate-spin" />
Loading older messages...
</div>
) : null}
{!hasNextPage && historyMessages.length > 0 ? (
<div className="py-1 text-center text-muted-foreground text-xs">
Start of conversation
</div>
) : null}
{historyMessages.map((message) => (
<ClawChatMessage key={message.id} message={message} />
))}
{turns.map((turn, index) => (
<ConversationMessage
key={turn.id}
turn={turn}
streaming={streaming && index === turns.length - 1}
/>
))}
{error ? (
<div className="rounded-xl border border-border/60 bg-card px-4 py-3 text-muted-foreground text-sm">
{error.message}
</div>
) : null}
</div>
)}
</ConversationContent>
<ConversationScrollButton />
</Conversation>
</div>
)
}

View File

@@ -0,0 +1,248 @@
import { CheckCircle2, Copy, Loader2, Wrench, XCircle } from 'lucide-react'
import { type FC, useCallback, useMemo } from 'react'
import {
Message,
MessageAction,
MessageActions,
MessageAttachment,
MessageAttachments,
MessageContent,
MessageResponse,
MessageToolbar,
} from '@/components/ai-elements/message'
import {
Reasoning,
ReasoningContent,
ReasoningTrigger,
} from '@/components/ai-elements/reasoning'
import {
Task,
TaskContent,
TaskItem,
TaskTrigger,
} from '@/components/ai-elements/task'
import { cn } from '@/lib/utils'
import type {
ClawChatMessagePart,
ClawChatMessage as ClawChatMessageType,
} from './claw-chat-types'
function formatCost(usd: number): string {
if (usd < 0.005) return `$${usd.toFixed(4)}`
return `$${usd.toFixed(2)}`
}
type ToolCallPart = Extract<ClawChatMessagePart, { type: 'tool-call' }>
type AttachmentPart = Extract<ClawChatMessagePart, { type: 'attachment' }>
interface RenderEntry {
kind: 'text' | 'reasoning' | 'meta' | 'task' | 'attachments'
partIndex: number
part?: ClawChatMessagePart
tools?: ToolCallPart[]
attachments?: AttachmentPart[]
}
/**
* Build a render plan that groups all tool-call parts into a single Task
* collapsible and all attachment parts into a single attachment strip at
* their respective first-appearance positions. Other parts render in place.
*/
function buildRenderEntries(parts: ClawChatMessagePart[]): RenderEntry[] {
const entries: RenderEntry[] = []
const tools: ToolCallPart[] = []
const attachments: AttachmentPart[] = []
let taskInserted = false
let attachmentsInserted = false
parts.forEach((part, partIndex) => {
if (part.type === 'tool-call') {
tools.push(part)
if (!taskInserted) {
entries.push({ kind: 'task', partIndex, tools })
taskInserted = true
}
} else if (part.type === 'attachment') {
attachments.push(part)
if (!attachmentsInserted) {
entries.push({ kind: 'attachments', partIndex, attachments })
attachmentsInserted = true
}
} else if (part.type === 'text') {
entries.push({ kind: 'text', partIndex, part })
} else if (part.type === 'reasoning') {
entries.push({ kind: 'reasoning', partIndex, part })
} else if (part.type === 'meta') {
entries.push({ kind: 'meta', partIndex, part })
}
})
return entries
}
function ToolStatusIcon({ status }: { status: ToolCallPart['status'] }) {
if (status === 'running' || status === 'pending') {
return (
<Loader2 className="size-3.5 shrink-0 animate-spin text-muted-foreground" />
)
}
if (status === 'completed') {
return <CheckCircle2 className="size-3.5 shrink-0 text-green-500" />
}
return <XCircle className="size-3.5 shrink-0 text-destructive" />
}
interface ClawChatMessageProps {
message: ClawChatMessageType
}
export const ClawChatMessage: FC<ClawChatMessageProps> = ({ message }) => {
const messageText = message.parts
.filter((p) => p.type === 'text')
.map((p) => p.text)
.join('\n')
const handleCopy = useCallback(() => {
if (messageText) navigator.clipboard.writeText(messageText)
}, [messageText])
const entries = useMemo(
() => buildRenderEntries(message.parts),
[message.parts],
)
return (
<Message
from={message.role}
className="max-w-full group-[.is-user]:max-w-[80%]"
>
<MessageContent className="max-w-full overflow-hidden group-[.is-assistant]:w-full group-[.is-user]:max-w-full">
{entries.map((entry) => {
const key = `${message.id}-entry-${entry.partIndex}`
if (entry.kind === 'attachments' && entry.attachments) {
return (
<MessageAttachments key={key}>
{entry.attachments.map((attachment, idx) => (
<MessageAttachment
// biome-ignore lint/suspicious/noArrayIndexKey: attachment order is stable within a finalized message
key={`${attachment.kind}-${idx}`}
data={{
type: 'file',
url: attachment.dataUrl ?? '',
mediaType: attachment.mediaType,
filename: attachment.name,
}}
/>
))}
</MessageAttachments>
)
}
if (entry.kind === 'text' && entry.part?.type === 'text') {
return (
<MessageResponse
key={key}
// Historical messages are finalized — render immediately.
// Streamdown's default "streaming" mode uses an idle-callback
// debounce (300ms / 500ms idle) that paints empty content
// first, which made history flash blank tool collapsibles
// before text on every load.
mode="static"
parseIncompleteMarkdown={false}
className={cn(
'max-w-full overflow-hidden break-words',
'[&_[data-streamdown="code-block"]]:!w-full [&_[data-streamdown="code-block"]]:!max-w-full [&_[data-streamdown="code-block"]]:overflow-x-auto',
'[&_[data-streamdown="table-wrapper"]]:!w-full [&_[data-streamdown="table-wrapper"]]:!max-w-full [&_[data-streamdown="table-wrapper"]]:overflow-x-auto',
'[&_table]:w-max [&_table]:min-w-full',
)}
>
{entry.part.text}
</MessageResponse>
)
}
if (entry.kind === 'reasoning' && entry.part?.type === 'reasoning') {
return (
<Reasoning
key={key}
className="w-full"
defaultOpen={false}
duration={entry.part.duration}
>
<ReasoningTrigger />
<ReasoningContent>{entry.part.text}</ReasoningContent>
</Reasoning>
)
}
if (entry.kind === 'meta' && entry.part?.type === 'meta') {
return (
<div key={key} className="text-muted-foreground text-xs">
{entry.part.label}: {entry.part.value}
</div>
)
}
if (entry.kind === 'task' && entry.tools) {
const tools = entry.tools
const errorCount = tools.filter((t) => t.status === 'failed').length
const taskTitle = `Agent activity (${tools.length} ${tools.length === 1 ? 'action' : 'actions'}${errorCount > 0 ? `, ${errorCount} failed` : ''})`
return (
<Task key={key} defaultOpen={false}>
<TaskTrigger title={taskTitle} TriggerIcon={Wrench} />
<TaskContent>
{tools.map((tool, idx) => (
<TaskItem
// biome-ignore lint/suspicious/noArrayIndexKey: tool order is stable within a finalized historical message
key={`${tool.name}-${tool.status}-${idx}`}
className="flex items-center gap-2"
>
<ToolStatusIcon status={tool.status} />
<span className="text-foreground text-xs">
{tool.label}
</span>
{tool.subject ? (
<span className="ml-1.5 truncate text-muted-foreground/70 text-xs">
· {tool.subject}
</span>
) : null}
{tool.error ? (
<span className="ml-2 truncate text-destructive text-xs">
{tool.error}
</span>
) : null}
{tool.durationMs != null ? (
<span className="ml-auto text-muted-foreground/60 text-xs tabular-nums">
{(tool.durationMs / 1000).toFixed(1)}s
</span>
) : null}
</TaskItem>
))}
</TaskContent>
</Task>
)
}
return null
})}
{message.role === 'assistant' && messageText ? (
<MessageToolbar>
<MessageActions>
<MessageAction tooltip="Copy" onClick={handleCopy}>
<Copy className="size-3.5" />
</MessageAction>
</MessageActions>
{message.costUsd ? (
<span className="text-[11px] text-muted-foreground/50 tabular-nums">
{formatCost(message.costUsd)}
</span>
) : null}
</MessageToolbar>
) : null}
</MessageContent>
</Message>
)
}

View File

@@ -2,20 +2,33 @@ import {
ArrowRight,
Bot,
ChevronDown,
FileText,
Folder,
Layers,
Loader2,
Mic,
Paperclip,
Square,
X,
} from 'lucide-react'
import { type FC, type ReactNode, useEffect, useState } from 'react'
import {
type DragEvent,
type FC,
type ReactNode,
useEffect,
useLayoutEffect,
useRef,
useState,
} from 'react'
import { AppSelector } from '@/components/elements/AppSelector'
import { TabPickerPopover } from '@/components/elements/tab-picker-popover'
import { WorkspaceSelector } from '@/components/elements/workspace-selector'
import { Button } from '@/components/ui/button'
import { Textarea } from '@/components/ui/textarea'
import type { AgentEntry } from '@/entrypoints/app/agents/useOpenClaw'
import { McpServerIcon } from '@/entrypoints/app/connect-mcp/McpServerIcon'
import { useGetUserMCPIntegrations } from '@/entrypoints/app/connect-mcp/useGetUserMCPIntegrations'
import { type StagedAttachment, stageAttachments } from '@/lib/attachments'
import { Feature } from '@/lib/browseros/capabilities'
import { useCapabilities } from '@/lib/browseros/useCapabilities'
import { useMcpServers } from '@/lib/mcp/mcpServerStorage'
@@ -24,16 +37,22 @@ import { useVoiceInput } from '@/lib/voice/useVoiceInput'
import { useWorkspace } from '@/lib/workspace/use-workspace'
import { AgentSelector } from './AgentSelector'
export interface ConversationInputSendInput {
text: string
attachments: StagedAttachment[]
}
interface ConversationInputProps {
agents: AgentEntry[]
selectedAgentId: string | null
onSelectAgent: (agent: AgentEntry) => void
onSend: (text: string) => void
onSend: (input: ConversationInputSendInput) => void
onCreateAgent?: () => void
streaming: boolean
disabled?: boolean
status?: string
placeholder?: string
attachmentsEnabled?: boolean
variant?: 'home' | 'conversation'
}
@@ -123,6 +142,9 @@ function ContextControls({
onToggleTab,
showAgentSelector,
status,
onAttachClick,
attachDisabled,
attachmentsEnabled,
}: {
agents: AgentEntry[]
onCreateAgent?: () => void
@@ -132,6 +154,9 @@ function ContextControls({
onToggleTab: (tab: chrome.tabs.Tab) => void
showAgentSelector: boolean
status?: string
onAttachClick: () => void
attachDisabled: boolean
attachmentsEnabled: boolean
}) {
const { supports } = useCapabilities()
const { selectedFolder } = useWorkspace()
@@ -146,7 +171,7 @@ function ContextControls({
})
return (
<div className="flex items-center justify-between border-border/50 border-t px-5 py-3">
<div className="flex items-center justify-between border-border/40 border-t px-4 py-2.5">
<div className="flex items-center gap-1">
{showAgentSelector ? (
<AgentSelector
@@ -191,6 +216,20 @@ function ContextControls({
<span>Tabs</span>
</Button>
</TabPickerPopover>
<Button
type="button"
variant="ghost"
onClick={onAttachClick}
disabled={attachDisabled || !attachmentsEnabled}
title="Attach files"
className={cn(
'flex items-center gap-2 rounded-lg px-3 py-1.5 font-medium text-sm transition-all',
'bg-transparent text-muted-foreground hover:bg-accent hover:text-accent-foreground',
)}
>
<Paperclip className="h-4 w-4" />
<span>Attach</span>
</Button>
</div>
{supports(Feature.MANAGED_MCP_SUPPORT) ? (
@@ -234,7 +273,7 @@ function ContextControls({
function HomeShell({ children }: { children: ReactNode }) {
return (
<div className="overflow-hidden rounded-[1.5rem] border border-border/60 bg-card/95 shadow-sm backdrop-blur">
<div className="overflow-hidden rounded-[1.55rem] border border-border/60 bg-card/95 shadow-sm">
{children}
</div>
)
@@ -242,7 +281,7 @@ function HomeShell({ children }: { children: ReactNode }) {
function ConversationShell({ children }: { children: ReactNode }) {
return (
<div className="overflow-hidden rounded-[1.5rem] border border-border/60 bg-card/95 shadow-sm backdrop-blur">
<div className="overflow-hidden rounded-[1.35rem] border border-border/50 bg-background/95 shadow-[0_10px_30px_rgba(15,23,42,0.06)] backdrop-blur-md">
{children}
</div>
)
@@ -258,14 +297,63 @@ export const ConversationInput: FC<ConversationInputProps> = ({
disabled,
status,
placeholder,
attachmentsEnabled = true,
variant = 'conversation',
}) => {
const [input, setInput] = useState('')
const [selectedTabs, setSelectedTabs] = useState<chrome.tabs.Tab[]>([])
const [isExpandedDraft, setIsExpandedDraft] = useState(false)
const [attachments, setAttachments] = useState<StagedAttachment[]>([])
const [attachmentError, setAttachmentError] = useState<string | null>(null)
const [isStaging, setIsStaging] = useState(false)
const [isDragOver, setIsDragOver] = useState(false)
const fileInputRef = useRef<HTMLInputElement>(null)
const voice = useVoiceInput()
const textareaRef = useRef<HTMLTextAreaElement>(null)
const selectedAgent = agents.find(
(agent) => agent.agentId === selectedAgentId,
)
const isConversation = variant === 'conversation'
const stageFiles = async (files: File[]) => {
if (files.length === 0) return
if (!attachmentsEnabled) {
setAttachmentError('Attachments are not supported for this agent yet.')
return
}
setIsStaging(true)
setAttachmentError(null)
try {
const result = await stageAttachments(files, attachments.length)
if (result.staged.length > 0) {
setAttachments((prev) => [...prev, ...result.staged])
}
if (result.errors.length > 0) {
setAttachmentError(result.errors.map((e) => e.message).join(' \u2022 '))
}
} finally {
setIsStaging(false)
}
}
const removeAttachment = (id: string) => {
setAttachments((prev) => prev.filter((a) => a.id !== id))
setAttachmentError(null)
}
useLayoutEffect(() => {
const element = textareaRef.current
if (!element) return
const maxHeight = isConversation ? 176 : 100
const collapsedHeight = isConversation ? 56 : 72
element.style.height = '0px'
const nextHeight = Math.min(element.scrollHeight, maxHeight)
element.style.height = `${nextHeight}px`
element.style.overflowY =
element.scrollHeight > maxHeight ? 'auto' : 'hidden'
setIsExpandedDraft(nextHeight > collapsedHeight)
})
useEffect(() => {
if (voice.transcript && !voice.isTranscribing) {
@@ -274,6 +362,12 @@ export const ConversationInput: FC<ConversationInputProps> = ({
}
}, [voice.transcript, voice.isTranscribing, voice])
useEffect(() => {
if (attachmentsEnabled) return
setAttachments([])
setAttachmentError(null)
}, [attachmentsEnabled])
const toggleTab = (tab: chrome.tabs.Tab) => {
setSelectedTabs((prev) => {
const isSelected = prev.some((selected) => selected.id === tab.id)
@@ -284,11 +378,70 @@ export const ConversationInput: FC<ConversationInputProps> = ({
})
}
const hasContent = input.trim().length > 0 || attachments.length > 0
const handleSend = () => {
const text = input.trim()
if (!text || streaming || disabled) return
onSend(text)
if (disabled || isStaging || streaming) return
if (!text && attachments.length === 0) return
onSend({ text, attachments })
setInput('')
setAttachments([])
setAttachmentError(null)
}
const handlePaste = (event: React.ClipboardEvent<HTMLTextAreaElement>) => {
const items = event.clipboardData?.items
if (!items) return
const files: File[] = []
for (const item of items) {
if (item.kind === 'file') {
const file = item.getAsFile()
if (file) files.push(file)
}
}
if (files.length > 0) {
event.preventDefault()
void stageFiles(files)
}
}
const handleDrop = (event: DragEvent<HTMLDivElement>) => {
event.preventDefault()
setIsDragOver(false)
const files = Array.from(event.dataTransfer?.files ?? [])
if (files.length > 0) {
void stageFiles(files)
}
}
const handleDragOver = (event: DragEvent<HTMLDivElement>) => {
if (!event.dataTransfer?.types.includes('Files')) return
event.preventDefault()
setIsDragOver(true)
}
const handleDragLeave = (event: DragEvent<HTMLDivElement>) => {
if (event.currentTarget.contains(event.relatedTarget as Node | null)) {
return
}
setIsDragOver(false)
}
const openFilePicker = () => {
if (!attachmentsEnabled) {
setAttachmentError('Attachments are not supported for this agent yet.')
return
}
fileInputRef.current?.click()
}
const handleFileInputChange = (
event: React.ChangeEvent<HTMLInputElement>,
) => {
const files = Array.from(event.target.files ?? [])
event.target.value = ''
if (files.length > 0) void stageFiles(files)
}
const shell = variant === 'home' ? HomeShell : ConversationShell
@@ -296,73 +449,201 @@ export const ConversationInput: FC<ConversationInputProps> = ({
return (
<Shell>
<div className="flex items-center gap-3 px-5 py-4">
<BotInputIcon variant={variant} />
<section
// Drag/drop on a region isn't a click affordance — wrap the
// composer in a labeled <section> so the a11y rule is satisfied
// without misrepresenting the surface as interactive.
aria-label="Message composer"
className={cn('relative', isDragOver && 'ring-2 ring-primary/60')}
onDragOver={handleDragOver}
onDragLeave={handleDragLeave}
onDrop={handleDrop}
>
<input
type="text"
value={input}
onChange={(event) => setInput(event.currentTarget.value)}
onKeyDown={(event) => {
if (event.key === 'Enter') {
event.preventDefault()
handleSend()
ref={fileInputRef}
type="file"
multiple
accept="image/png,image/jpeg,image/webp,image/gif,text/*,application/json"
className="hidden"
onChange={handleFileInputChange}
/>
{attachments.length > 0 || attachmentError ? (
<AttachmentStrip
attachments={attachments}
onRemove={removeAttachment}
error={attachmentError}
/>
) : null}
<div
className={cn(
'flex gap-3',
variant === 'home' ? 'px-4 py-3' : 'px-4 py-3',
isExpandedDraft ? 'items-end' : 'items-center',
)}
>
<BotInputIcon variant={variant} />
<div className="flex-1">
<Textarea
ref={textareaRef}
value={input}
onChange={(event) => setInput(event.currentTarget.value)}
onKeyDown={(event) => {
if (event.key === 'Enter' && !event.shiftKey) {
event.preventDefault()
handleSend()
}
}}
onPaste={handlePaste}
rows={1}
placeholder={
voice.isTranscribing
? 'Transcribing...'
: (placeholder ??
`Message ${selectedAgent?.name ?? 'agent'}...`)
}
disabled={disabled || voice.isTranscribing}
className={cn(
'resize-none border-none bg-transparent px-0 text-[15px] shadow-none focus-visible:ring-0',
'[field-sizing:fixed]',
variant === 'home'
? 'min-h-[40px] py-2 leading-6'
: 'min-h-[40px] py-2 leading-6',
'placeholder:text-muted-foreground/80',
)}
/>
</div>
<VoiceButton
isRecording={voice.isRecording}
isTranscribing={voice.isTranscribing}
onStart={() => {
void voice.startRecording()
}}
onStop={() => {
void voice.stopRecording()
}}
/>
<InputActionButton
disabled={
!hasContent ||
isStaging ||
!!disabled ||
voice.isRecording ||
voice.isTranscribing ||
streaming
}
}}
placeholder={
voice.isTranscribing
? 'Transcribing...'
: (placeholder ?? `Message ${selectedAgent?.name ?? 'agent'}...`)
}
disabled={disabled || voice.isTranscribing}
className="flex-1 border-none bg-transparent text-base text-foreground outline-none placeholder:text-muted-foreground disabled:opacity-60"
onClick={handleSend}
// Spinner stays the user-facing "agent is busy" hint; with the
// queue active we still spin while a turn is in flight.
streaming={streaming}
/>
</div>
{voice.error ? (
<div className="px-5 pb-2 text-destructive text-xs">
{voice.error}
</div>
) : null}
<ContextControls
agents={agents}
onCreateAgent={onCreateAgent}
onSelectAgent={onSelectAgent}
selectedAgentId={selectedAgentId}
selectedTabs={selectedTabs}
onToggleTab={toggleTab}
showAgentSelector={variant === 'home'}
status={status}
onAttachClick={openFilePicker}
attachDisabled={attachments.length >= 10 || isStaging || !!disabled}
attachmentsEnabled={attachmentsEnabled}
/>
<VoiceButton
isRecording={voice.isRecording}
isTranscribing={voice.isTranscribing}
onStart={() => {
void voice.startRecording()
}}
onStop={() => {
void voice.stopRecording()
}}
/>
<InputActionButton
disabled={
!input.trim() ||
streaming ||
!!disabled ||
voice.isRecording ||
voice.isTranscribing
}
onClick={handleSend}
streaming={streaming}
/>
</div>
{voice.error ? (
<div className="px-5 pb-2 text-destructive text-xs">{voice.error}</div>
) : null}
<ContextControls
agents={agents}
onCreateAgent={onCreateAgent}
onSelectAgent={onSelectAgent}
selectedAgentId={selectedAgentId}
selectedTabs={selectedTabs}
onToggleTab={toggleTab}
showAgentSelector={variant === 'home'}
status={status}
/>
{isDragOver ? (
<div className="pointer-events-none absolute inset-0 flex items-center justify-center rounded-[inherit] bg-background/80 font-medium text-foreground text-sm backdrop-blur-sm">
Drop files to attach
</div>
) : null}
</section>
</Shell>
)
}
function AttachmentStrip({
attachments,
onRemove,
error,
}: {
attachments: StagedAttachment[]
onRemove: (id: string) => void
error: string | null
}) {
return (
<div className="border-border/40 border-b px-4 pt-3 pb-2">
{attachments.length > 0 ? (
<div className="flex flex-wrap gap-2">
{attachments.map((attachment) => (
<AttachmentChip
key={attachment.id}
attachment={attachment}
onRemove={() => onRemove(attachment.id)}
/>
))}
</div>
) : null}
{error ? (
<div className="mt-2 text-destructive text-xs">{error}</div>
) : null}
</div>
)
}
function AttachmentChip({
attachment,
onRemove,
}: {
attachment: StagedAttachment
onRemove: () => void
}) {
if (attachment.kind === 'image' && attachment.dataUrl) {
return (
<div className="group relative size-16 overflow-hidden rounded-md border border-border/60">
<img
src={attachment.dataUrl}
alt={attachment.name}
className="size-full object-cover"
/>
<button
type="button"
onClick={onRemove}
className="absolute top-1 right-1 inline-flex size-5 items-center justify-center rounded-full bg-background/80 text-muted-foreground opacity-0 transition-opacity hover:text-foreground group-hover:opacity-100"
aria-label={`Remove ${attachment.name}`}
>
<X className="size-3" />
</button>
</div>
)
}
return (
<div className="group flex max-w-[220px] items-center gap-2 rounded-md border border-border/60 bg-background/60 px-2 py-1.5">
<FileText className="size-4 shrink-0 text-muted-foreground" />
<span className="truncate text-xs">{attachment.name}</span>
<button
type="button"
onClick={onRemove}
className="ml-1 inline-flex size-4 items-center justify-center text-muted-foreground hover:text-foreground"
aria-label={`Remove ${attachment.name}`}
>
<X className="size-3" />
</button>
</div>
)
}
function BotInputIcon({ variant }: { variant: 'home' | 'conversation' }) {
return (
<div
className={cn(
'flex items-center justify-center text-[var(--accent-orange)]',
variant === 'home'
? 'h-10 w-10 rounded-xl bg-[var(--accent-orange)]/10'
: 'h-9 w-9 rounded-xl bg-[var(--accent-orange)]/12',
? 'h-8 w-8 rounded-lg bg-[var(--accent-orange)]/10'
: 'h-8 w-8 rounded-lg bg-[var(--accent-orange)]/10',
)}
>
<Bot className="h-4 w-4" />

View File

@@ -1,7 +1,9 @@
import { Bot, CheckCircle2, Loader2, XCircle } from 'lucide-react'
import type { FC } from 'react'
import { Bot, CheckCircle2, Loader2, Wrench, XCircle } from 'lucide-react'
import { type FC, useMemo } from 'react'
import {
Message,
MessageAttachment,
MessageAttachments,
MessageContent,
MessageResponse,
} from '@/components/ai-elements/message'
@@ -10,96 +12,191 @@ import {
ReasoningContent,
ReasoningTrigger,
} from '@/components/ai-elements/reasoning'
import type { AgentConversationTurn } from '@/lib/agent-conversations/types'
import {
Task,
TaskContent,
TaskItem,
TaskTrigger,
} from '@/components/ai-elements/task'
import type {
AgentConversationTurn,
ToolEntry,
} from '@/lib/agent-conversations/types'
interface ConversationMessageProps {
turn: AgentConversationTurn
streaming: boolean
}
interface RenderEntry {
kind: 'thinking' | 'text' | 'task'
partIndex: number
text?: string
done?: boolean
tools?: ToolEntry[]
}
/**
* Build the render plan for an assistant turn:
* - thinking and text parts render in place
* - all tool-batch parts collapse into a single Task entry at their first
* appearance position, with tools listed in arrival order
*/
function buildRenderEntries(turn: AgentConversationTurn): RenderEntry[] {
const entries: RenderEntry[] = []
const aggregatedTools: ToolEntry[] = []
let taskInserted = false
turn.parts.forEach((part, partIndex) => {
if (part.kind === 'thinking') {
entries.push({
kind: 'thinking',
partIndex,
text: part.text,
done: part.done,
})
} else if (part.kind === 'text') {
entries.push({ kind: 'text', partIndex, text: part.text })
} else if (part.kind === 'tool-batch') {
aggregatedTools.push(...part.tools)
if (!taskInserted) {
entries.push({
kind: 'task',
partIndex,
tools: aggregatedTools,
})
taskInserted = true
}
}
})
return entries
}
function ToolStatusIcon({ status }: { status: ToolEntry['status'] }) {
if (status === 'running') {
return (
<Loader2 className="size-3.5 shrink-0 animate-spin text-muted-foreground" />
)
}
if (status === 'completed') {
return <CheckCircle2 className="size-3.5 shrink-0 text-green-500" />
}
return <XCircle className="size-3.5 shrink-0 text-destructive" />
}
export const ConversationMessage: FC<ConversationMessageProps> = ({
turn,
streaming,
}) => (
<div className="space-y-3">
<Message from="user">
<MessageContent>
<pre className="whitespace-pre-wrap font-sans text-sm">
{turn.userText}
</pre>
</MessageContent>
</Message>
}) => {
const entries = useMemo(() => buildRenderEntries(turn), [turn])
{turn.parts.length > 0 && (
<Message from="assistant">
return (
<div className="space-y-3">
<Message from="user">
<MessageContent>
{turn.parts.map((part, i) => {
const key = `${turn.id}-part-${i}`
{turn.userAttachments && turn.userAttachments.length > 0 && (
<MessageAttachments>
{turn.userAttachments.map((attachment) => (
<MessageAttachment
key={attachment.id}
data={{
type: 'file',
url: attachment.dataUrl ?? '',
mediaType: attachment.mediaType,
filename: attachment.name,
}}
/>
))}
</MessageAttachments>
)}
{turn.userText && (
<pre className="whitespace-pre-wrap font-sans text-sm">
{turn.userText}
</pre>
)}
</MessageContent>
</Message>
switch (part.kind) {
case 'thinking':
{entries.length > 0 && (
<Message from="assistant">
<MessageContent>
{entries.map((entry) => {
const key = `${turn.id}-entry-${entry.partIndex}`
if (entry.kind === 'thinking') {
return (
<Reasoning
key={key}
className="w-full"
isStreaming={!part.done}
defaultOpen={!part.done}
isStreaming={!entry.done}
defaultOpen={!entry.done}
>
<ReasoningTrigger />
<ReasoningContent>{part.text}</ReasoningContent>
<ReasoningContent>{entry.text ?? ''}</ReasoningContent>
</Reasoning>
)
}
case 'tool-batch':
if (entry.kind === 'text') {
return (
<div key={key} className="w-full space-y-1">
{part.tools.map((tool) => (
<div
<MessageResponse key={key}>
{entry.text ?? ''}
</MessageResponse>
)
}
const tools = entry.tools ?? []
const allDone = tools.every((t) => t.status !== 'running')
const taskTitle = allDone
? `Agent activity (${tools.length} ${tools.length === 1 ? 'action' : 'actions'})`
: `Working… (${tools.length} ${tools.length === 1 ? 'action' : 'actions'})`
return (
<Task key={key} defaultOpen={!turn.done}>
<TaskTrigger title={taskTitle} TriggerIcon={Wrench} />
<TaskContent>
{tools.map((tool) => (
<TaskItem
key={tool.id}
className="flex items-center gap-2 rounded-md border px-3 py-2 text-sm"
className="flex items-center gap-2"
>
{tool.status === 'running' && (
<Loader2 className="size-3.5 animate-spin text-muted-foreground" />
)}
{tool.status === 'completed' && (
<CheckCircle2 className="size-3.5 text-green-500" />
)}
{tool.status === 'error' && (
<XCircle className="size-3.5 text-destructive" />
)}
<span className="font-mono text-xs">{tool.name}</span>
<ToolStatusIcon status={tool.status} />
<span className="text-foreground text-xs">
{tool.label}
</span>
{tool.subject ? (
<span className="ml-1.5 truncate text-muted-foreground/70 text-xs">
· {tool.subject}
</span>
) : null}
{tool.durationMs != null && (
<span className="ml-auto text-muted-foreground text-xs">
<span className="ml-auto text-muted-foreground/60 text-xs tabular-nums">
{(tool.durationMs / 1000).toFixed(1)}s
</span>
)}
</div>
</TaskItem>
))}
</div>
)
</TaskContent>
</Task>
)
})}
</MessageContent>
</Message>
)}
case 'text':
return <MessageResponse key={key}>{part.text}</MessageResponse>
default:
return null
}
})}
</MessageContent>
</Message>
)}
{!turn.done && turn.parts.length === 0 && streaming && (
<div className="flex gap-2">
<div className="flex size-7 shrink-0 items-center justify-center rounded-full bg-[var(--accent-orange)] text-white">
<Bot className="size-3.5" />
{!turn.done && turn.parts.length === 0 && streaming && (
<div className="flex gap-2">
<div className="flex size-7 shrink-0 items-center justify-center rounded-full bg-[var(--accent-orange)] text-white">
<Bot className="size-3.5" />
</div>
<div className="flex items-center gap-1 rounded-xl rounded-tl-none border border-border/50 bg-card px-3 py-2.5 shadow-sm">
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.3s]" />
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.15s]" />
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)]" />
</div>
</div>
<div className="flex items-center gap-1 rounded-xl rounded-tl-none border border-border/50 bg-card px-3 py-2.5 shadow-sm">
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.3s]" />
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.15s]" />
<span className="size-1.5 animate-bounce rounded-full bg-[var(--accent-orange)]" />
</div>
</div>
)}
</div>
)
)}
</div>
)
}

View File

@@ -1,8 +1,11 @@
import type { FC } from 'react'
import { Outlet, useOutletContext } from 'react-router'
import { useHarnessAgents } from '@/entrypoints/app/agents/useAgents'
import type {
AgentEntry,
OpenClawStatus,
} from '@/entrypoints/app/agents/useOpenClaw'
import {
type AgentEntry,
type OpenClawStatus,
useOpenClawAgents,
useOpenClawStatus,
} from '@/entrypoints/app/agents/useOpenClaw'
@@ -16,16 +19,32 @@ interface AgentCommandContextValue {
export const AgentCommandLayout: FC = () => {
const { status, loading: statusLoading } = useOpenClawStatus(5000)
const { agents, loading: agentsLoading } = useOpenClawAgents(
status?.status === 'running' && status.controlPlaneStatus === 'connected',
const openClawEnabled =
status?.status === 'running' && status.controlPlaneStatus === 'connected'
const { agents: openClawAgents, loading: openClawAgentsLoading } =
useOpenClawAgents(openClawEnabled)
const { agents: harnessAgents, loading: harnessAgentsLoading } =
useHarnessAgents()
const visibleOpenClawAgents = openClawEnabled ? openClawAgents : []
// Dual-created OpenClaw agents appear in both `/claw/agents` (gateway
// record) and `/agents` (harness record) under the same id. Prefer the
// harness entry so the chat panel can route through the harness path
// and the rail doesn't show duplicates.
const harnessAgentIds = new Set(harnessAgents.map((entry) => entry.agentId))
const dedupedOpenClawAgents = visibleOpenClawAgents.filter(
(entry) => !harnessAgentIds.has(entry.agentId),
)
const agents = [...dedupedOpenClawAgents, ...harnessAgents]
return (
<Outlet
context={
{
agents,
agentsLoading,
agentsLoading:
harnessAgentsLoading ||
statusLoading ||
(openClawEnabled && openClawAgentsLoading),
status,
statusLoading,
} satisfies AgentCommandContextValue

View File

@@ -0,0 +1,12 @@
import { describe, expect, it } from 'bun:test'
import { mapAgentHarnessToolStatus } from './agent-stream-events'
describe('mapAgentHarnessToolStatus', () => {
it('normalizes ACP tool statuses for the chat renderer', () => {
expect(mapAgentHarnessToolStatus('running')).toBe('running')
expect(mapAgentHarnessToolStatus('completed')).toBe('completed')
expect(mapAgentHarnessToolStatus('failed')).toBe('error')
expect(mapAgentHarnessToolStatus('incomplete')).toBe('running')
expect(mapAgentHarnessToolStatus(undefined)).toBe('running')
})
})

View File

@@ -0,0 +1,19 @@
import type { ToolEntry } from '@/lib/agent-conversations/types'
export function mapAgentHarnessToolStatus(
status: string | undefined,
): ToolEntry['status'] {
if (!status) return 'running'
const normalized = status.toLowerCase()
if (['error', 'failed', 'failure', 'denied'].includes(normalized)) {
return 'error'
}
if (
['complete', 'completed', 'done', 'success', 'succeeded'].includes(
normalized,
)
) {
return 'completed'
}
return 'running'
}

View File

@@ -0,0 +1,183 @@
import { describe, expect, it } from 'bun:test'
import type { AgentConversationTurn } from '@/lib/agent-conversations/types'
import {
type AgentHistoryPageResponse,
type BrowserOSChatHistoryItem,
buildChatHistoryFromClawMessages,
filterTurnsPersistedInHistory,
flattenHistoryPages,
mapHistoryItemToClawMessage,
} from './claw-chat-types'
function historyItem(
overrides: Partial<BrowserOSChatHistoryItem>,
): BrowserOSChatHistoryItem {
return {
id: 'session-1:0',
role: 'user',
text: 'Hello',
timestamp: 1000,
messageSeq: 0,
sessionKey: 'session-1',
source: 'user-chat',
...overrides,
}
}
function page(items: BrowserOSChatHistoryItem[]): AgentHistoryPageResponse {
return {
agentId: 'main',
sessionKey: 'session-1',
session: null,
items,
page: {
hasMore: false,
limit: 50,
},
}
}
describe('claw-chat-types', () => {
it('maps backend history items into text-first ClawChat messages', () => {
const message = mapHistoryItemToClawMessage(
historyItem({
id: 'session-1:1',
role: 'assistant',
text: 'Hi there',
messageSeq: 1,
}),
)
expect(message).toEqual({
id: 'session-1:1',
role: 'assistant',
sessionKey: 'session-1',
timestamp: 1000,
source: 'user-chat',
messageSeq: 1,
status: 'historical',
parts: [{ type: 'text', text: 'Hi there' }],
})
})
it('flattens paginated history into oldest-to-newest render order', () => {
const messages = flattenHistoryPages([
page([
historyItem({
id: 'session-1:2',
role: 'user',
text: 'newer',
timestamp: 3000,
messageSeq: 2,
}),
]),
page([
historyItem({
id: 'session-1:0',
role: 'user',
text: 'older',
timestamp: 1000,
messageSeq: 0,
}),
historyItem({
id: 'session-1:1',
role: 'assistant',
text: 'middle',
timestamp: 2000,
messageSeq: 1,
}),
]),
])
expect(messages.map((message) => message.id)).toEqual([
'session-1:0',
'session-1:1',
'session-1:2',
])
})
it('builds OpenClaw chat history from text message parts only', () => {
const history = buildChatHistoryFromClawMessages([
{
id: 'user-1',
role: 'user',
sessionKey: 'session-1',
parts: [{ type: 'text', text: ' User request ' }],
},
{
id: 'assistant-1',
role: 'assistant',
sessionKey: 'session-1',
parts: [
{ type: 'reasoning', text: 'private reasoning' },
{ type: 'text', text: 'Assistant answer' },
],
},
])
expect(history).toEqual([
{ role: 'user', content: 'User request' },
{ role: 'assistant', content: 'Assistant answer' },
])
})
it('hides completed live turns once harness history contains the same turn', () => {
const turn: AgentConversationTurn = {
id: 'live-turn',
userText: 'hello',
parts: [{ kind: 'text', text: 'hi there' }],
done: true,
timestamp: 1_000,
}
const visible = filterTurnsPersistedInHistory(
[turn],
[
{
id: 'history-user',
role: 'user',
sessionKey: 'main',
timestamp: 1_050,
status: 'historical',
parts: [{ type: 'text', text: 'hello' }],
},
{
id: 'history-assistant',
role: 'assistant',
sessionKey: 'main',
timestamp: 1_100,
status: 'historical',
parts: [{ type: 'text', text: 'hi there' }],
},
],
)
expect(visible).toEqual([])
})
it('keeps completed live turns until matching assistant history arrives', () => {
const turn: AgentConversationTurn = {
id: 'live-turn',
userText: 'hello',
parts: [{ kind: 'text', text: 'hi there' }],
done: true,
timestamp: 1_000,
}
const visible = filterTurnsPersistedInHistory(
[turn],
[
{
id: 'history-user',
role: 'user',
sessionKey: 'main',
timestamp: 1_050,
status: 'historical',
parts: [{ type: 'text', text: 'hello' }],
},
],
)
expect(visible).toEqual([turn])
})
})

View File

@@ -0,0 +1,287 @@
import type { OpenClawChatHistoryMessage } from '@/entrypoints/app/agents/useOpenClaw'
import type { AgentConversationTurn } from '@/lib/agent-conversations/types'
export type ClawChatRole = 'user' | 'assistant'
export type ClawChatSource = 'user-chat' | 'cron' | 'hook' | 'channel' | 'other'
export interface BrowserOSOpenClawSession {
key: string
updatedAt: number
sessionId: string
agentId: string
kind: string
source: ClawChatSource
status?: string
totalTokens?: number
model?: string
modelProvider?: string
}
export interface BrowserOSChatHistoryToolCall {
toolCallId?: string
toolName: string
label: string
subject?: string
status: 'pending' | 'running' | 'completed' | 'failed'
input?: unknown
output?: unknown
error?: string
durationMs?: number
}
export interface BrowserOSChatHistoryReasoning {
text: string
durationMs?: number
}
export interface BrowserOSChatHistoryAttachment {
kind: 'image' | 'file'
mediaType: string
// Images carry a `data:` URL so we can render directly without any
// additional fetch; files (text/PDF) currently round-trip via inline
// text in the message body and do not populate this field in v1.
dataUrl?: string
name?: string
}
export interface BrowserOSChatHistoryItem {
id: string
role: ClawChatRole
text: string
timestamp?: number
messageSeq: number
sessionKey: string
source: ClawChatSource
costUsd?: number
tokensIn?: number
tokensOut?: number
toolCalls?: BrowserOSChatHistoryToolCall[]
reasoning?: BrowserOSChatHistoryReasoning
attachments?: BrowserOSChatHistoryAttachment[]
}
export interface AgentHistoryPageResponse {
agentId: string
sessionKey: string | null
session: BrowserOSOpenClawSession | null
items: BrowserOSChatHistoryItem[]
page: {
cursor?: string
hasMore: boolean
limit: number
}
}
export type ClawChatMessageStatus =
| 'historical'
| 'sending'
| 'streaming'
| 'error'
export type ClawChatMessagePart =
| { type: 'text'; text: string }
| { type: 'reasoning'; text: string; duration?: number }
| {
type: 'tool-call'
name: string
label: string
subject?: string
status: 'pending' | 'running' | 'completed' | 'failed'
input?: unknown
output?: unknown
error?: string
durationMs?: number
}
| {
type: 'attachment'
kind: 'image' | 'file'
mediaType: string
dataUrl?: string
name?: string
}
| { type: 'meta'; label: string; value: string }
export interface ClawChatMessage {
id: string
role: ClawChatRole
sessionKey: string
timestamp?: number
source?: ClawChatSource
messageSeq?: number
status?: ClawChatMessageStatus
parts: ClawChatMessagePart[]
costUsd?: number
tokensIn?: number
tokensOut?: number
}
export function mapHistoryItemToClawMessage(
item: BrowserOSChatHistoryItem,
): ClawChatMessage {
const parts: ClawChatMessagePart[] = []
// Attachments first — they belong above the text in user messages and
// never appear on assistant messages today (assistant images come back
// through tool results, which render via the Task collapsible).
if (item.attachments && item.attachments.length > 0) {
for (const attachment of item.attachments) {
parts.push({
type: 'attachment',
kind: attachment.kind,
mediaType: attachment.mediaType,
dataUrl: attachment.dataUrl,
name: attachment.name,
})
}
}
// Reasoning, then tool calls, then text — the chronological order the
// agent produced them (think → act → answer).
if (item.reasoning && item.reasoning.text.trim().length > 0) {
// 0ms means thinking and the final answer were emitted in the same JSONL
// line (no tool calls between them) — there's no real elapsed wall-clock,
// so fall through to the "Thinking" trigger instead of "Thought for 0
// seconds" / streaming shimmer. Real multi-line turns floor at 1s.
const durationMs = item.reasoning.durationMs ?? 0
const duration =
durationMs > 0 ? Math.max(1, Math.round(durationMs / 1000)) : undefined
parts.push({
type: 'reasoning',
text: item.reasoning.text,
duration,
})
}
if (item.toolCalls && item.toolCalls.length > 0) {
for (const tc of item.toolCalls) {
parts.push({
type: 'tool-call',
name: tc.toolName,
label: tc.label,
subject: tc.subject,
status: tc.status,
input: tc.input,
output: tc.output,
error: tc.error,
durationMs: tc.durationMs,
})
}
}
// Only emit a text part when there's actual content. User messages with
// only attachments and no caption shouldn't render an empty bubble.
if (item.text.trim().length > 0) {
parts.push({ type: 'text', text: item.text })
}
return {
id: item.id,
role: item.role,
sessionKey: item.sessionKey,
timestamp: item.timestamp,
source: item.source,
messageSeq: item.messageSeq,
status: 'historical',
parts,
costUsd: item.costUsd,
tokensIn: item.tokensIn,
tokensOut: item.tokensOut,
}
}
export function flattenHistoryPages(
pages: AgentHistoryPageResponse[],
): ClawChatMessage[] {
return pages
.flatMap((page) => page.items)
.sort((a, b) => {
if (a.timestamp != null && b.timestamp != null) {
return a.timestamp - b.timestamp
}
return a.messageSeq - b.messageSeq
})
.map(mapHistoryItemToClawMessage)
}
export function buildChatHistoryFromClawMessages(
messages: ClawChatMessage[],
): OpenClawChatHistoryMessage[] {
return messages
.map((message) => {
const content = message.parts
.filter((part): part is { type: 'text'; text: string } => {
return part.type === 'text' && part.text.trim().length > 0
})
.map((part) => part.text.trim())
.join('\n\n')
return content ? { role: message.role, content } : null
})
.filter((message): message is OpenClawChatHistoryMessage =>
Boolean(message),
)
}
const TURN_HISTORY_MATCH_WINDOW_MS = 5_000
export function filterTurnsPersistedInHistory(
turns: AgentConversationTurn[],
historyMessages: ClawChatMessage[],
): AgentConversationTurn[] {
return turns.filter(
(turn) => !isTurnPersistedInHistory(turn, historyMessages),
)
}
function isTurnPersistedInHistory(
turn: AgentConversationTurn,
historyMessages: ClawChatMessage[],
): boolean {
if (!turn.done) return false
const assistantText = getTurnAssistantText(turn)
if (!assistantText) return false
const minTimestamp = turn.timestamp - TURN_HISTORY_MATCH_WINDOW_MS
const userText = turn.userText.trim()
const userPersisted =
!userText ||
historyMessages.some(
(message) =>
message.role === 'user' &&
isHistoryMessageAfter(message, minTimestamp) &&
getClawMessageText(message) === userText,
)
const assistantPersisted = historyMessages.some(
(message) =>
message.role === 'assistant' &&
isHistoryMessageAfter(message, minTimestamp) &&
getClawMessageText(message) === assistantText,
)
return userPersisted && assistantPersisted
}
function isHistoryMessageAfter(
message: ClawChatMessage,
minTimestamp: number,
): boolean {
return message.timestamp == null || message.timestamp >= minTimestamp
}
function getTurnAssistantText(turn: AgentConversationTurn): string {
return turn.parts
.filter((part) => part.kind === 'text')
.map((part) => part.text)
.join('')
.trim()
}
function getClawMessageText(message: ClawChatMessage): string {
return message.parts
.filter((part) => part.type === 'text')
.map((part) => part.text)
.join('')
.trim()
}

View File

@@ -0,0 +1,71 @@
import { buildToolLabel } from '../../../lib/tool-labels'
import type { HarnessAgentHistoryPage } from '../agents/agent-harness-types'
import type {
AgentHistoryPageResponse,
BrowserOSChatHistoryItem,
BrowserOSChatHistoryToolCall,
} from './claw-chat-types'
export function mapHarnessHistoryPage(
page: HarnessAgentHistoryPage,
): AgentHistoryPageResponse {
const items: BrowserOSChatHistoryItem[] = page.items.map((item, index) => {
const toolCalls = item.toolCalls?.map(
(tool): BrowserOSChatHistoryToolCall => {
const input = asRecord(tool.input)
const { label, subject } = buildToolLabel(tool.toolName, input)
return {
toolName: tool.toolName,
label,
status: tool.status,
...(tool.toolCallId ? { toolCallId: tool.toolCallId } : {}),
...(subject ? { subject } : {}),
...(tool.input !== undefined ? { input: tool.input } : {}),
...(tool.output !== undefined ? { output: tool.output } : {}),
...(tool.error ? { error: tool.error } : {}),
...(tool.durationMs != null ? { durationMs: tool.durationMs } : {}),
}
},
)
return {
id: item.id,
role: item.role,
text: item.text,
timestamp: item.createdAt,
messageSeq: index + 1,
sessionKey: 'main',
source: 'user-chat',
...(item.reasoning ? { reasoning: item.reasoning } : {}),
...(toolCalls && toolCalls.length > 0 ? { toolCalls } : {}),
}
})
const updatedAt =
page.items.length > 0
? Math.max(...page.items.map((item) => item.createdAt))
: Date.now()
return {
agentId: page.agentId,
sessionKey: 'main',
session: {
key: 'main',
updatedAt,
sessionId: 'main',
agentId: page.agentId,
kind: 'agent-harness',
source: 'user-chat',
},
items,
page: {
hasMore: false,
limit: items.length,
},
}
}
function asRecord(value: unknown): Record<string, unknown> | undefined {
return value && typeof value === 'object' && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined
}

View File

@@ -1,69 +1,53 @@
import { useEffect, useState } from 'react'
import {
type AgentEntry,
getModelDisplayName,
type OpenClawStatus,
} from '@/entrypoints/app/agents/useOpenClaw'
import { getLatestConversation } from '@/lib/agent-conversations/storage'
import type { AgentCardData } from '@/lib/agent-conversations/types'
import type { AgentOverview } from './useAgentDashboard'
function getAgentStatusTone(
status: OpenClawStatus['status'] | undefined,
function resolveAgentStatus(
gatewayStatus: OpenClawStatus['status'] | undefined,
liveStatus: AgentOverview['status'] | undefined,
): AgentCardData['status'] {
if (status === 'error') return 'error'
if (status === 'starting') return 'working'
// Gateway-level errors take precedence
if (gatewayStatus === 'error') return 'error'
if (gatewayStatus === 'starting') return 'working'
// Per-agent live status from the WS observer
if (liveStatus === 'working') return 'working'
if (liveStatus === 'error') return 'error'
return 'idle'
}
async function getAgentCardData(
agent: AgentEntry,
status: OpenClawStatus['status'] | undefined,
): Promise<AgentCardData> {
const conversation = await getLatestConversation(agent.agentId)
const lastTurn = conversation?.turns[conversation.turns.length - 1]
const lastTextPart = lastTurn?.parts.findLast((part) => part.kind === 'text')
return {
agentId: agent.agentId,
name: agent.name,
model: getModelDisplayName(agent.model),
status: getAgentStatusTone(status),
lastMessage:
lastTextPart?.kind === 'text'
? lastTextPart.text.slice(0, 120)
: undefined,
lastMessageTimestamp: lastTurn?.timestamp,
}
}
export function useAgentCardData(
/**
* Build agent card display data by merging the raw agent entries from
* the gateway with enriched overview data from the dashboard API.
*
* Pure function — no hooks, no IndexedDB, no async.
*/
export function buildAgentCardData(
agents: AgentEntry[],
status: OpenClawStatus['status'] | undefined,
) {
const [cardData, setCardData] = useState<AgentCardData[]>([])
dashboard: AgentOverview[] | undefined,
): AgentCardData[] {
return agents.map((agent) => {
const overview = dashboard?.find((d) => d.agentId === agent.agentId)
useEffect(() => {
let active = true
const loadCardData = async () => {
const nextCardData = await Promise.all(
agents.map((agent) => getAgentCardData(agent, status)),
)
if (active) {
setCardData(nextCardData)
}
return {
agentId: agent.agentId,
name: agent.name,
model: getModelDisplayName(agent.model),
status:
agent.source === 'agent-harness'
? 'idle'
: resolveAgentStatus(status, overview?.status),
lastMessage: overview?.latestMessage?.slice(0, 200) ?? undefined,
lastMessageTimestamp: overview?.latestMessageAt ?? undefined,
activitySummary: overview?.activitySummary ?? undefined,
currentTool: overview?.currentTool ?? undefined,
costUsd: overview?.totalCostUsd ?? undefined,
}
if (agents.length > 0) {
void loadCardData()
} else {
setCardData([])
}
return () => {
active = false
}
}, [agents, status])
return cardData
})
}

View File

@@ -1,51 +1,77 @@
import { useEffect, useRef, useState } from 'react'
import {
chatWithAgent,
type OpenClawStreamEvent,
} from '@/entrypoints/app/agents/useOpenClaw'
import {
getLatestConversation,
saveConversation,
} from '@/lib/agent-conversations/storage'
type AgentHarnessStreamEvent,
attachToHarnessTurn,
cancelHarnessTurn,
chatWithHarnessAgent,
fetchActiveHarnessTurn,
} from '@/entrypoints/app/agents/useAgents'
import type { OpenClawChatHistoryMessage } from '@/entrypoints/app/agents/useOpenClaw'
import type {
AgentConversation,
AgentConversationTurn,
AssistantPart,
ToolEntry,
UserAttachmentPreview,
} from '@/lib/agent-conversations/types'
import type { ServerAttachmentPayload } from '@/lib/attachments'
import { consumeSSEStream } from '@/lib/sse'
import { buildToolLabel } from '@/lib/tool-labels'
import { mapAgentHarnessToolStatus } from './agent-stream-events'
export function useAgentConversation(agentId: string, agentName: string) {
export interface SendInput {
text: string
attachments?: ServerAttachmentPayload[]
// Optional preview metadata used to render the optimistic user turn.
// Built by the composer at staging time; the server only sees the
// payload array.
attachmentPreviews?: UserAttachmentPreview[]
}
interface UseAgentConversationOptions {
// The hook always speaks to the harness chat path now; the OpenClaw
// legacy /claw/agents/:id/chat surface was removed in Step 12. The
// option remains for forward-compatibility.
runtime?: 'agent-harness'
sessionKey?: string | null
history?: OpenClawChatHistoryMessage[]
onComplete?: () => void
onSessionKeyChange?: (sessionKey: string) => void
}
export function useAgentConversation(
agentId: string,
options: UseAgentConversationOptions = {},
) {
const [turns, setTurns] = useState<AgentConversationTurn[]>([])
const [streaming, setStreaming] = useState(false)
const [loading, setLoading] = useState(true)
const sessionKeyRef = useRef('')
const sessionKeyRef = useRef(options.sessionKey ?? '')
const historyRef = useRef<OpenClawChatHistoryMessage[]>(options.history ?? [])
const textAccRef = useRef('')
const thinkAccRef = useRef('')
const streamAbortRef = useRef<AbortController | null>(null)
const onCompleteRef = useRef(options.onComplete)
const onSessionKeyChangeRef = useRef(options.onSessionKeyChange)
// Per-turn resume bookkeeping. `turnId` is captured from the response
// header; `lastSeq` advances with every SSE event so a reconnect can
// resume via Last-Event-ID.
const turnIdRef = useRef<string | null>(null)
const lastSeqRef = useRef<number | null>(null)
useEffect(() => {
let active = true
getLatestConversation(agentId)
.then((conv) => {
if (!active) return
if (conv) {
setTurns(conv.turns)
sessionKeyRef.current = conv.sessionKey
} else {
sessionKeyRef.current = crypto.randomUUID()
}
setLoading(false)
})
.catch(() => {
if (active) {
sessionKeyRef.current = crypto.randomUUID()
setLoading(false)
}
})
return () => {
active = false
}
}, [agentId])
sessionKeyRef.current = options.sessionKey ?? ''
}, [options.sessionKey])
useEffect(() => {
historyRef.current = options.history ?? []
}, [options.history])
useEffect(() => {
onCompleteRef.current = options.onComplete
}, [options.onComplete])
useEffect(() => {
onSessionKeyChangeRef.current = options.onSessionKeyChange
}, [options.onSessionKeyChange])
useEffect(() => {
return () => {
@@ -53,17 +79,11 @@ export function useAgentConversation(agentId: string, agentName: string) {
}
}, [])
const persistTurns = (updatedTurns: AgentConversationTurn[]) => {
const conv: AgentConversation = {
agentId,
agentName,
sessionKey: sessionKeyRef.current,
turns: updatedTurns,
createdAt: updatedTurns[0]?.timestamp ?? Date.now(),
updatedAt: Date.now(),
}
saveConversation(conv).catch(() => {})
}
// Indirection for the resume effect below: lets it call the latest
// event handler without re-subscribing on every render.
const processEventRef = useRef<(event: AgentHarnessStreamEvent) => void>(
() => {},
)
const updateCurrentTurnParts = (
updater: (parts: AssistantPart[]) => AssistantPart[],
@@ -75,122 +95,210 @@ export function useAgentConversation(agentId: string, agentName: string) {
})
}
const processStreamEvent = (event: OpenClawStreamEvent) => {
switch (event.type) {
case 'text-delta': {
const delta = (event.data.text as string) ?? ''
textAccRef.current += delta
const text = textAccRef.current
updateCurrentTurnParts((parts) => {
const last = parts[parts.length - 1]
if (last?.kind === 'text') {
return [...parts.slice(0, -1), { ...last, text }]
}
return [...parts, { kind: 'text', text }]
})
break
const appendTextDelta = (delta: string) => {
textAccRef.current += delta
const text = textAccRef.current
updateCurrentTurnParts((parts) => {
const last = parts[parts.length - 1]
if (last?.kind === 'text') {
return [...parts.slice(0, -1), { ...last, text }]
}
case 'thinking': {
const delta = (event.data.text as string) ?? ''
thinkAccRef.current += delta
const text = thinkAccRef.current
updateCurrentTurnParts((parts) => {
const idx = parts.findIndex((p) => p.kind === 'thinking' && !p.done)
if (idx >= 0) {
return [
...parts.slice(0, idx),
{ ...parts[idx], text, done: false },
...parts.slice(idx + 1),
]
}
return [...parts, { kind: 'thinking', text, done: false }]
})
break
}
case 'tool-start': {
const tool = {
id: (event.data.toolCallId as string) ?? crypto.randomUUID(),
name: (event.data.toolName as string) ?? 'unknown',
status: 'running' as const,
}
updateCurrentTurnParts((parts) => {
const last = parts[parts.length - 1]
if (last?.kind === 'tool-batch') {
return [
...parts.slice(0, -1),
{ ...last, tools: [...last.tools, tool] },
]
}
return [...parts, { kind: 'tool-batch', tools: [tool] }]
})
break
}
case 'tool-end': {
const toolId = event.data.toolCallId as string
const toolStatus: 'completed' | 'error' =
(event.data.status as string) === 'error' ? 'error' : 'completed'
const durationMs = event.data.durationMs as number | undefined
updateCurrentTurnParts((parts) => {
for (let i = parts.length - 1; i >= 0; i--) {
const part = parts[i]
if (
part.kind === 'tool-batch' &&
part.tools.some((t) => t.id === toolId)
) {
const updatedTools = part.tools.map((t) =>
t.id === toolId ? { ...t, status: toolStatus, durationMs } : t,
)
return [
...parts.slice(0, i),
{ ...part, tools: updatedTools },
...parts.slice(i + 1),
]
}
}
return parts
})
break
}
case 'done': {
updateCurrentTurnParts((parts) =>
parts.map((part) =>
part.kind === 'thinking' ? { ...part, done: true } : part,
),
)
setTurns((prev) => {
const last = prev[prev.length - 1]
if (!last) return prev
const updated = [...prev.slice(0, -1), { ...last, done: true }]
persistTurns(updated)
return updated
})
break
}
case 'error': {
const msg =
(event.data.message as string) ??
(event.data.error as string) ??
'Unknown error'
updateCurrentTurnParts((parts) => [
...parts,
{ kind: 'text', text: `Error: ${msg}` },
])
break
}
}
return [...parts, { kind: 'text', text }]
})
}
const send = async (text: string) => {
if (!text.trim() || streaming) return
const appendThinkingDelta = (delta: string) => {
thinkAccRef.current += delta
const text = thinkAccRef.current
updateCurrentTurnParts((parts) => {
const idx = parts.findIndex((p) => p.kind === 'thinking' && !p.done)
if (idx >= 0) {
return [
...parts.slice(0, idx),
{ ...parts[idx], text, done: false },
...parts.slice(idx + 1),
]
}
return [...parts, { kind: 'thinking', text, done: false }]
})
}
const appendErrorText = (message: string) => {
updateCurrentTurnParts((parts) => [
...parts,
{ kind: 'text', text: `Error: ${message}` },
])
}
const markCurrentTurnDone = () => {
updateCurrentTurnParts((parts) =>
parts.map((part) =>
part.kind === 'thinking' ? { ...part, done: true } : part,
),
)
setTurns((prev) => {
const last = prev[prev.length - 1]
if (!last) return prev
return [...prev.slice(0, -1), { ...last, done: true }]
})
}
const upsertAgentHarnessTool = (event: AgentHarnessStreamEvent) => {
if (event.type !== 'tool_call') return
const rawName = event.title || event.rawType || 'tool call'
const { label, subject } = buildToolLabel(
rawName,
event.text ? { description: event.text } : undefined,
)
const tool: ToolEntry = {
id: event.id ?? crypto.randomUUID(),
name: rawName,
label,
subject,
status: mapAgentHarnessToolStatus(event.status),
}
updateCurrentTurnParts((parts) => {
for (let i = parts.length - 1; i >= 0; i--) {
const part = parts[i]
if (
part.kind === 'tool-batch' &&
part.tools.some((existing) => existing.id === tool.id)
) {
const tools = part.tools.map((existing) =>
existing.id === tool.id ? { ...existing, ...tool } : existing,
)
return [
...parts.slice(0, i),
{ ...part, tools },
...parts.slice(i + 1),
]
}
}
const last = parts[parts.length - 1]
if (last?.kind === 'tool-batch') {
return [
...parts.slice(0, -1),
{ ...last, tools: [...last.tools, tool] },
]
}
return [...parts, { kind: 'tool-batch', tools: [tool] }]
})
}
const processAgentHarnessStreamEvent = (event: AgentHarnessStreamEvent) => {
switch (event.type) {
case 'text_delta':
if (event.stream === 'thought') {
appendThinkingDelta(event.text)
} else {
appendTextDelta(event.text)
}
break
case 'tool_call':
upsertAgentHarnessTool(event)
break
case 'done':
markCurrentTurnDone()
break
case 'error':
appendErrorText(event.message)
break
case 'status':
break
}
}
processEventRef.current = processAgentHarnessStreamEvent
// On mount (and whenever the agent changes), check whether the
// server has an in-flight turn for this agent and reattach to it.
// This is what makes the chat resilient across tab close/reopen,
// refresh, and navigation: the runtime call kept running on the
// server while we were away. Effect only depends on `agentId` —
// the event handler is read off a ref so this doesn't re-subscribe
// every render.
useEffect(() => {
let cancelled = false
const abortController = new AbortController()
const attemptResume = async () => {
try {
const active = await fetchActiveHarnessTurn(agentId)
if (cancelled || !active || active.status !== 'running') return
if (streamAbortRef.current) return // a fresh send already in flight
// Stage a placeholder turn so the streamed events have a row
// to render into. We don't have the user message text on
// resume; the assistant turn is what we're catching up on.
setTurns((prev) => [
...prev,
{
id: crypto.randomUUID(),
userText: '',
parts: [],
done: false,
timestamp: active.startedAt,
},
])
textAccRef.current = ''
thinkAccRef.current = ''
turnIdRef.current = active.turnId
lastSeqRef.current = null
streamAbortRef.current = abortController
setStreaming(true)
const response = await attachToHarnessTurn(agentId, {
turnId: active.turnId,
signal: abortController.signal,
})
if (!response.ok) return
await consumeSSEStream<AgentHarnessStreamEvent>(
response,
(event, meta) => {
if (typeof meta.seq === 'number') lastSeqRef.current = meta.seq
processEventRef.current(event)
},
abortController.signal,
)
} catch {
// Resume is best-effort; transient errors fall back to the
// user starting a new turn manually.
} finally {
if (!cancelled) {
if (streamAbortRef.current === abortController) {
streamAbortRef.current = null
}
turnIdRef.current = null
lastSeqRef.current = null
setStreaming(false)
}
}
}
void attemptResume()
return () => {
cancelled = true
abortController.abort()
}
}, [agentId])
const send = async (input: string | SendInput) => {
const normalized: SendInput =
typeof input === 'string' ? { text: input } : input
const trimmed = normalized.text.trim()
const attachments = normalized.attachments ?? []
if (streaming) return
if (!trimmed && attachments.length === 0) return
const turn: AgentConversationTurn = {
id: crypto.randomUUID(),
userText: text.trim(),
userText: trimmed,
userAttachments:
normalized.attachmentPreviews &&
normalized.attachmentPreviews.length > 0
? normalized.attachmentPreviews
: undefined,
parts: [],
done: false,
timestamp: Date.now(),
@@ -203,12 +311,37 @@ export function useAgentConversation(agentId: string, agentName: string) {
streamAbortRef.current = abortController
try {
const response = await chatWithAgent(
let response = await chatWithHarnessAgent(
agentId,
text.trim(),
sessionKeyRef.current,
trimmed,
abortController.signal,
attachments,
)
// 409 means the server already has an active turn for this
// agent (e.g. a previous tab kicked one off and we're a fresh
// mount that missed the resume window). Attach to it instead of
// double-sending.
if (response.status === 409) {
const body = (await response.json()) as { turnId?: string }
if (body.turnId) {
response = await attachToHarnessTurn(agentId, {
turnId: body.turnId,
signal: abortController.signal,
})
}
}
const responseSessionKey =
response.headers.get('X-Session-Key') ??
response.headers.get('X-Session-Id')
if (responseSessionKey) {
sessionKeyRef.current = responseSessionKey
onSessionKeyChangeRef.current?.(responseSessionKey)
}
const responseTurnId = response.headers.get('X-Turn-Id')
if (responseTurnId) {
turnIdRef.current = responseTurnId
lastSeqRef.current = null
}
if (!response.ok) {
const err = await response.text()
updateCurrentTurnParts((parts) => [
@@ -217,9 +350,12 @@ export function useAgentConversation(agentId: string, agentName: string) {
])
return
}
await consumeSSEStream(
await consumeSSEStream<AgentHarnessStreamEvent>(
response,
processStreamEvent,
(event, meta) => {
if (typeof meta.seq === 'number') lastSeqRef.current = meta.seq
processAgentHarnessStreamEvent(event)
},
abortController.signal,
)
} catch (err) {
@@ -233,24 +369,45 @@ export function useAgentConversation(agentId: string, agentName: string) {
if (streamAbortRef.current === abortController) {
streamAbortRef.current = null
}
turnIdRef.current = null
lastSeqRef.current = null
onCompleteRef.current?.()
setStreaming(false)
}
}
const resetConversation = () => {
/**
* Stop button. The fetch abort only detaches *this* SSE subscriber
* now — the underlying turn would otherwise keep running on the
* server. So we explicitly cancel via the new endpoint, then unwind
* the local stream.
*/
const stop = async () => {
const turnId = turnIdRef.current ?? undefined
streamAbortRef.current?.abort()
streamAbortRef.current = null
try {
await cancelHarnessTurn(agentId, {
turnId,
reason: 'user pressed stop',
})
} catch {
// Best-effort — UI already aborted.
}
}
const resetConversation = () => {
void stop()
setTurns([])
setStreaming(false)
sessionKeyRef.current = crypto.randomUUID()
}
return {
turns,
streaming,
loading,
sessionKey: sessionKeyRef.current,
send,
stop,
resetConversation,
}
}

View File

@@ -0,0 +1,95 @@
import { useQuery, useQueryClient } from '@tanstack/react-query'
import { useEffect } from 'react'
import { useAgentServerUrl } from '@/lib/browseros/useBrowserOSProviders'
export interface AgentOverview {
agentId: string
status: 'working' | 'idle' | 'error' | 'unknown'
latestMessage: string | null
latestMessageAt: number | null
activitySummary: string | null
currentTool: string | null
totalCostUsd: number
sessionCount: number
}
export interface DashboardResponse {
agents: AgentOverview[]
summary: {
totalAgents: number
totalCostUsd: number
}
}
interface StatusEvent {
agentId: string
status: AgentOverview['status']
currentTool: string | null
error: string | null
timestamp: number
}
const DASHBOARD_QUERY_KEY = ['claw', 'dashboard']
export function useAgentDashboard(enabled: boolean) {
const { baseUrl, isLoading: urlLoading } = useAgentServerUrl()
const queryClient = useQueryClient()
const ready = enabled && Boolean(baseUrl) && !urlLoading
// Initial data load + periodic refresh as fallback
const query = useQuery<DashboardResponse>({
queryKey: [...DASHBOARD_QUERY_KEY, baseUrl],
queryFn: async () => {
const url = new URL('/claw/dashboard', baseUrl as string)
const response = await fetch(url.toString())
if (!response.ok) throw new Error('Failed to fetch dashboard')
return response.json()
},
enabled: ready,
})
// SSE subscription for real-time status patches
useEffect(() => {
if (!ready || !baseUrl) return
const streamUrl = new URL('/claw/dashboard/stream', baseUrl)
const eventSource = new EventSource(streamUrl.toString())
eventSource.addEventListener('snapshot', (event) => {
try {
const dashboard = JSON.parse(event.data) as DashboardResponse
queryClient.setQueryData([...DASHBOARD_QUERY_KEY, baseUrl], dashboard)
} catch {}
})
eventSource.addEventListener('status', (event) => {
try {
const status = JSON.parse(event.data) as StatusEvent
queryClient.setQueryData<DashboardResponse>(
[...DASHBOARD_QUERY_KEY, baseUrl],
(prev) => {
if (!prev) return prev
return {
...prev,
agents: prev.agents.map((agent) =>
agent.agentId === status.agentId
? {
...agent,
status: status.status,
currentTool: status.currentTool,
}
: agent,
),
}
},
)
} catch {}
})
return () => {
eventSource.close()
}
}, [ready, baseUrl, queryClient])
return query
}

View File

@@ -0,0 +1,55 @@
import { describe, expect, it } from 'bun:test'
import { mapHarnessHistoryPage } from './harness-history-mapper'
describe('mapHarnessHistoryPage', () => {
it('maps rich harness history into chat history items', () => {
const page = mapHarnessHistoryPage({
agentId: 'agent-1',
sessionId: 'main',
items: [
{
id: 'agent:agent-1:main:1',
agentId: 'agent-1',
sessionId: 'main',
role: 'assistant',
text: 'Done.',
createdAt: 1000,
reasoning: { text: 'checking state' },
toolCalls: [
{
toolCallId: 'tool-1',
toolName: 'read_file',
status: 'completed',
input: { path: 'src/index.ts' },
output: 'file contents',
},
],
},
],
})
expect(page.items).toEqual([
{
id: 'agent:agent-1:main:1',
role: 'assistant',
text: 'Done.',
timestamp: 1000,
messageSeq: 1,
sessionKey: 'main',
source: 'user-chat',
reasoning: { text: 'checking state' },
toolCalls: [
{
toolCallId: 'tool-1',
toolName: 'read_file',
label: 'Read file',
subject: 'index.ts',
status: 'completed',
input: { path: 'src/index.ts' },
output: 'file contents',
},
],
},
])
})
})

View File

@@ -0,0 +1,29 @@
import { useQuery } from '@tanstack/react-query'
import { fetchHarnessAgentHistory } from '@/entrypoints/app/agents/useAgents'
import { useAgentServerUrl } from '@/lib/browseros/useBrowserOSProviders'
import type { AgentHistoryPageResponse } from './claw-chat-types'
import { mapHarnessHistoryPage } from './harness-history-mapper'
const HISTORY_QUERY_KEY = 'harness-agent-history'
export function useHarnessChatHistory(agentId: string, enabled = true) {
const {
baseUrl,
isLoading: urlLoading,
error: urlError,
} = useAgentServerUrl()
const query = useQuery<AgentHistoryPageResponse, Error>({
queryKey: [HISTORY_QUERY_KEY, baseUrl, agentId, 'main'],
queryFn: async () => {
return mapHarnessHistoryPage(await fetchHarnessAgentHistory(agentId))
},
enabled: Boolean(baseUrl) && !urlLoading && enabled && Boolean(agentId),
})
return {
...query,
error: query.error ?? urlError,
isLoading: query.isLoading || urlLoading,
}
}

View File

@@ -0,0 +1,42 @@
import { Bot, Cpu, Sparkles } from 'lucide-react'
import type { FC } from 'react'
import type { HarnessAgentAdapter } from './agent-harness-types'
/**
* Single icon component for any adapter the agent rail can render.
* Falls back to a generic bot when the adapter is unknown so future
* adapters land without a code change at the call site.
*/
interface AdapterIconProps {
adapter: HarnessAgentAdapter | 'unknown'
className?: string
}
export const AdapterIcon: FC<AdapterIconProps> = ({ adapter, className }) => {
switch (adapter) {
case 'claude':
// Claude Code — text-based agent, sparkles to evoke the "AI assistant" feel.
return <Sparkles className={className} aria-label="Claude Code" />
case 'codex':
// Codex — code-leaning, CPU mark.
return <Cpu className={className} aria-label="Codex" />
case 'openclaw':
// OpenClaw — bot/automation framing.
return <Bot className={className} aria-label="OpenClaw" />
default:
return <Bot className={className} aria-label="Agent" />
}
}
export function adapterLabel(adapter: HarnessAgentAdapter | 'unknown'): string {
switch (adapter) {
case 'claude':
return 'Claude Code'
case 'codex':
return 'Codex'
case 'openclaw':
return 'OpenClaw'
default:
return 'Agent'
}
}

View File

@@ -1,393 +0,0 @@
import {
ArrowLeft,
Bot,
CheckCircle2,
Loader2,
Send,
XCircle,
} from 'lucide-react'
import { type FC, useEffect, useRef, useState } from 'react'
import {
Message,
MessageContent,
MessageResponse,
} from '@/components/ai-elements/message'
import {
Reasoning,
ReasoningContent,
ReasoningTrigger,
} from '@/components/ai-elements/reasoning'
import { Button } from '@/components/ui/button'
import { Textarea } from '@/components/ui/textarea'
import { consumeSSEStream } from '@/lib/sse'
import { chatWithAgent, type OpenClawStreamEvent } from './useOpenClaw'
interface ToolEntry {
id: string
name: string
status: 'running' | 'completed' | 'error'
durationMs?: number
}
type AssistantPart =
| { kind: 'thinking'; text: string; done: boolean }
| { kind: 'tool-batch'; tools: ToolEntry[] }
| { kind: 'text'; text: string }
interface ChatTurn {
id: string
userText: string
parts: AssistantPart[]
done: boolean
}
interface AgentChatProps {
agentId: string
agentName: string
onBack: () => void
}
export const AgentChat: FC<AgentChatProps> = ({
agentId,
agentName,
onBack,
}) => {
const [turns, setTurns] = useState<ChatTurn[]>([])
const [input, setInput] = useState('')
const [streaming, setStreaming] = useState(false)
const scrollRef = useRef<HTMLDivElement>(null)
const sessionKeyRef = useRef(crypto.randomUUID())
const streamAbortRef = useRef<AbortController | null>(null)
const textAccRef = useRef('')
const thinkAccRef = useRef('')
const scrollToBottom = () => {
scrollRef.current?.scrollTo(0, scrollRef.current.scrollHeight)
}
// biome-ignore lint/correctness/useExhaustiveDependencies: scroll on every turns change
useEffect(() => {
scrollToBottom()
}, [turns])
useEffect(() => {
return () => {
streamAbortRef.current?.abort()
}
}, [])
const updateCurrentTurnParts = (
updater: (parts: AssistantPart[]) => AssistantPart[],
) => {
setTurns((prev) => {
const last = prev[prev.length - 1]
if (!last) return prev
return [...prev.slice(0, -1), { ...last, parts: updater(last.parts) }]
})
}
const processStreamEvent = (event: OpenClawStreamEvent) => {
switch (event.type) {
case 'text-delta': {
const delta = (event.data.text as string) ?? ''
textAccRef.current += delta
const text = textAccRef.current
updateCurrentTurnParts((parts) => {
const last = parts[parts.length - 1]
if (last?.kind === 'text') {
return [...parts.slice(0, -1), { ...last, text }]
}
return [...parts, { kind: 'text', text }]
})
break
}
case 'thinking': {
const delta = (event.data.text as string) ?? ''
thinkAccRef.current += delta
const text = thinkAccRef.current
updateCurrentTurnParts((parts) => {
const idx = parts.findIndex((p) => p.kind === 'thinking' && !p.done)
if (idx >= 0) {
return [
...parts.slice(0, idx),
{ ...parts[idx], text, done: false },
...parts.slice(idx + 1),
]
}
return [...parts, { kind: 'thinking', text, done: false }]
})
break
}
case 'tool-start': {
const tool: ToolEntry = {
id: (event.data.toolCallId as string) ?? crypto.randomUUID(),
name: (event.data.toolName as string) ?? 'unknown',
status: 'running',
}
updateCurrentTurnParts((parts) => {
const last = parts[parts.length - 1]
if (last?.kind === 'tool-batch') {
return [
...parts.slice(0, -1),
{ ...last, tools: [...last.tools, tool] },
]
}
return [...parts, { kind: 'tool-batch', tools: [tool] }]
})
break
}
case 'tool-end': {
const toolId = event.data.toolCallId as string
const status =
(event.data.status as string) === 'error' ? 'error' : 'completed'
const durationMs = event.data.durationMs as number | undefined
updateCurrentTurnParts((parts) => {
for (let i = parts.length - 1; i >= 0; i--) {
const part = parts[i]
if (
part.kind === 'tool-batch' &&
part.tools.some((t) => t.id === toolId)
) {
const updatedTools = part.tools.map((t) =>
t.id === toolId
? {
...t,
status: status as ToolEntry['status'],
durationMs,
}
: t,
)
return [
...parts.slice(0, i),
{ ...part, tools: updatedTools },
...parts.slice(i + 1),
]
}
}
return parts
})
break
}
case 'done': {
updateCurrentTurnParts((parts) =>
parts.map((part) =>
part.kind === 'thinking' ? { ...part, done: true } : part,
),
)
setTurns((prev) => {
const last = prev[prev.length - 1]
if (!last) return prev
return [...prev.slice(0, -1), { ...last, done: true }]
})
break
}
case 'error': {
const msg =
(event.data.message as string) ??
(event.data.error as string) ??
'Unknown error'
updateCurrentTurnParts((parts) => [
...parts,
{ kind: 'text', text: `Error: ${msg}` },
])
break
}
}
}
const handleSend = async () => {
const text = input.trim()
if (!text || streaming) return
const turn: ChatTurn = {
id: crypto.randomUUID(),
userText: text,
parts: [],
done: false,
}
setTurns((prev) => [...prev, turn])
setInput('')
setStreaming(true)
textAccRef.current = ''
thinkAccRef.current = ''
const abortController = new AbortController()
streamAbortRef.current = abortController
try {
const response = await chatWithAgent(
agentId,
text,
sessionKeyRef.current,
abortController.signal,
)
if (!response.ok) {
const err = await response.text()
updateCurrentTurnParts((parts) => [
...parts,
{ kind: 'text', text: `Error: ${err}` },
])
return
}
await consumeSSEStream(
response,
processStreamEvent,
abortController.signal,
)
} catch (err) {
if (abortController.signal.aborted) return
const msg = err instanceof Error ? err.message : String(err)
updateCurrentTurnParts((parts) => [
...parts,
{ kind: 'text', text: `Error: ${msg}` },
])
} finally {
if (streamAbortRef.current === abortController) {
streamAbortRef.current = null
}
setStreaming(false)
}
}
return (
<div className="flex h-[calc(100vh-4rem)] flex-col">
<div className="flex items-center gap-2 border-b px-4 py-3">
<Button variant="ghost" size="icon" onClick={onBack}>
<ArrowLeft className="size-4" />
</Button>
<h2 className="font-semibold text-lg">{agentName}</h2>
</div>
<div ref={scrollRef} className="flex-1 space-y-4 overflow-y-auto p-4">
{turns.map((turn) => (
<div key={turn.id} className="space-y-3">
{/* User message */}
<Message from="user">
<MessageContent>
<pre className="whitespace-pre-wrap font-sans text-sm">
{turn.userText}
</pre>
</MessageContent>
</Message>
{/* Assistant response — all parts grouped */}
{turn.parts.length > 0 && (
<Message from="assistant">
<MessageContent>
{turn.parts.map((part, i) => {
const key = `${turn.id}-part-${i}`
switch (part.kind) {
case 'thinking':
return (
<Reasoning
key={key}
className="w-full"
isStreaming={!part.done}
defaultOpen={!part.done}
>
<ReasoningTrigger />
<ReasoningContent>{part.text}</ReasoningContent>
</Reasoning>
)
case 'tool-batch':
return (
<div key={key} className="w-full space-y-1">
{part.tools.map((tool) => (
<div
key={tool.id}
className="flex items-center gap-2 rounded-md border px-3 py-2 text-sm"
>
{tool.status === 'running' && (
<Loader2 className="size-3.5 animate-spin text-muted-foreground" />
)}
{tool.status === 'completed' && (
<CheckCircle2 className="size-3.5 text-green-500" />
)}
{tool.status === 'error' && (
<XCircle className="size-3.5 text-destructive" />
)}
<span className="font-mono text-xs">
{tool.name}
</span>
{tool.durationMs != null && (
<span className="ml-auto text-muted-foreground text-xs">
{(tool.durationMs / 1000).toFixed(1)}s
</span>
)}
</div>
))}
</div>
)
case 'text':
return (
<MessageResponse key={key}>
{part.text}
</MessageResponse>
)
default:
return null
}
})}
</MessageContent>
</Message>
)}
{/* Streaming indicator when waiting for first part */}
{!turn.done && turn.parts.length === 0 && streaming && (
<div className="flex gap-2">
<div className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full bg-[var(--accent-orange)] text-white">
<Bot className="h-3.5 w-3.5" />
</div>
<div className="flex items-center gap-1 rounded-xl rounded-tl-none border border-border/50 bg-card px-3 py-2.5 shadow-sm">
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.3s]" />
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-[var(--accent-orange)] [animation-delay:-0.15s]" />
<span className="h-1.5 w-1.5 animate-bounce rounded-full bg-[var(--accent-orange)]" />
</div>
</div>
)}
</div>
))}
</div>
<div className="border-t p-4">
<div className="flex gap-2">
<Textarea
value={input}
onChange={(e) => setInput(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
handleSend()
}
}}
placeholder="Send a message..."
className="min-h-[44px] resize-none"
rows={1}
/>
<Button
onClick={handleSend}
disabled={!input.trim() || streaming}
size="icon"
>
{streaming ? (
<Loader2 className="size-4 animate-spin" />
) : (
<Send className="size-4" />
)}
</Button>
</div>
</div>
</div>
)
}

View File

@@ -0,0 +1,108 @@
import { Loader2 } from 'lucide-react'
import { type FC, useMemo } from 'react'
import { AgentRowCard } from './AgentRowCard'
import { AgentsEmptyState } from './AgentsEmptyState'
import type { HarnessAgent, HarnessAgentAdapter } from './agent-harness-types'
import type { AgentListItem } from './agents-page-types'
import type { AgentLiveness } from './LivenessDot'
interface AgentListProps {
agents: AgentListItem[]
/**
* Optional per-agent activity metadata. Keyed by `agentId`. Missing
* entries fall back to status='unknown' / lastUsedAt=null and the
* row renders an "unknown" dot. The server will populate this once
* the activity tracker ships; the page works without it.
*/
activity?: Record<
string,
{ status: AgentLiveness; lastUsedAt: number | null }
>
/**
* Lookup table from harness agent id → adapter + reasoning effort,
* sourced from `useHarnessAgents`. Lets the row card render the
* correct adapter icon and chips for harness agents (legacy
* /claw/agents entries fall back to inferring from `runtimeLabel`).
*/
harnessAgentLookup?: Map<string, HarnessAgent>
loading: boolean
deletingAgentKey: string | null
onCreateAgent: () => void
onDeleteAgent: (agent: AgentListItem) => void
}
export const AgentList: FC<AgentListProps> = ({
agents,
activity,
harnessAgentLookup,
loading,
deletingAgentKey,
onCreateAgent,
onDeleteAgent,
}) => {
// Sort by recency: most recently used first; never-used agents drop
// to the bottom in id-stable order so the list doesn't reshuffle on
// every refresh. The pinned exception is the gateway's `main` agent
// when it's never been touched — keep it at the top so a fresh
// install has an obvious starting point.
const ordered = useMemo(() => {
const withScore = agents.map((agent) => {
const lastUsedAt = activity?.[agent.agentId]?.lastUsedAt ?? null
return { agent, lastUsedAt }
})
return withScore
.sort((a, b) => {
const aPinned = a.agent.agentId === 'main' && a.lastUsedAt === null
const bPinned = b.agent.agentId === 'main' && b.lastUsedAt === null
if (aPinned && !bPinned) return -1
if (!aPinned && bPinned) return 1
const aValue = a.lastUsedAt ?? -Infinity
const bValue = b.lastUsedAt ?? -Infinity
if (aValue !== bValue) return bValue - aValue
return a.agent.agentId.localeCompare(b.agent.agentId)
})
.map((entry) => entry.agent)
}, [activity, agents])
if (loading && agents.length === 0) {
return (
<div className="flex h-36 items-center justify-center rounded-xl border border-border border-dashed bg-card/50">
<Loader2 className="size-5 animate-spin text-muted-foreground" />
</div>
)
}
if (agents.length === 0) {
return <AgentsEmptyState onCreateAgent={onCreateAgent} />
}
return (
<div className="grid gap-3">
{ordered.map((agent) => {
const harness = harnessAgentLookup?.get(agent.agentId)
const adapter: HarnessAgentAdapter | undefined =
harness?.adapter ?? inferAdapterFromLabel(agent.runtimeLabel)
return (
<AgentRowCard
key={agent.key}
agent={agent}
status={activity?.[agent.agentId]?.status}
lastUsedAt={activity?.[agent.agentId]?.lastUsedAt}
adapter={adapter}
reasoningEffort={harness?.reasoningEffort ?? null}
onDelete={onDeleteAgent}
deleting={deletingAgentKey === agent.key}
/>
)
})}
</div>
)
}
function inferAdapterFromLabel(label: string): HarnessAgentAdapter | undefined {
const lower = label?.toLowerCase()
if (lower === 'claude code') return 'claude'
if (lower === 'codex') return 'codex'
if (lower === 'openclaw') return 'openclaw'
return undefined
}

View File

@@ -0,0 +1,270 @@
import {
Copy,
Loader2,
MessageSquare,
MoreHorizontal,
Pencil,
RotateCcw,
Trash2,
} from 'lucide-react'
import type { FC } from 'react'
import { useNavigate } from 'react-router'
import { toast } from 'sonner'
import { Badge } from '@/components/ui/badge'
import { Button } from '@/components/ui/button'
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from '@/components/ui/dropdown-menu'
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from '@/components/ui/tooltip'
import { cn } from '@/lib/utils'
import { AdapterIcon, adapterLabel } from './AdapterIcon'
import {
canDelete as canDeleteAgent,
canRename as canRenameAgent,
displayName,
formatRelativeTime,
workspaceLabel,
} from './agent-display.helpers'
import type { HarnessAgentAdapter } from './agent-harness-types'
import type { AgentListItem } from './agents-page-types'
import { type AgentLiveness, LivenessDot } from './LivenessDot'
interface AgentRowCardProps {
agent: AgentListItem
/**
* Per-agent extras the listing surface provides on top of the
* minimal `AgentListItem` shape. `lastUsedAt` survives server
* restart (sourced from acpx session record); `status` is in-memory
* server-side.
*/
status?: AgentLiveness
lastUsedAt?: number | null
/** Adapter the agent belongs to. Drives icon + label. */
adapter?: HarnessAgentAdapter
/** Reasoning effort chip (claude/codex/openclaw catalog). */
reasoningEffort?: string | null
/** Modeled directly off the inbound delete handler so the parent owns the dialog. */
onDelete: (agent: AgentListItem) => void
/** Whether THIS agent is mid-delete; renders a spinner in place of the trash icon. */
deleting?: boolean
}
export const AgentRowCard: FC<AgentRowCardProps> = ({
agent,
status = 'unknown',
lastUsedAt,
adapter,
reasoningEffort,
onDelete,
deleting,
}) => {
const navigate = useNavigate()
const adapterId = adapter ?? inferAdapterFromListItem(agent)
const workspace = workspaceLabel(agent)
const lastUsedLabel = formatRelativeTime(lastUsedAt ?? null)
const allowDelete = canDeleteAgent(agent)
const allowRename = canRenameAgent(agent)
const handleChat = () => navigate(`/agents/${agent.agentId}`)
const handleCopyId = async () => {
try {
await navigator.clipboard.writeText(agent.agentId)
toast.success('Agent id copied')
} catch {
toast.error('Could not copy agent id')
}
}
return (
<div
className={cn(
'group rounded-xl border border-border bg-card p-4 shadow-sm transition-all',
'hover:border-[var(--accent-orange)]/50 hover:shadow-sm',
)}
>
<div className="flex items-start gap-4">
{/* Adapter tile + liveness dot in the corner. */}
<div className="relative shrink-0">
<div className="flex h-12 w-12 items-center justify-center rounded-xl bg-muted text-muted-foreground">
<AdapterIcon adapter={adapterId} className="h-6 w-6" />
</div>
<LivenessDot
status={status}
detail={livenessDetail(status, lastUsedAt)}
className="absolute -right-0.5 -bottom-0.5"
/>
</div>
<div className="min-w-0 flex-1">
<div className="mb-1 flex items-center gap-2">
<span className="truncate font-semibold">{displayName(agent)}</span>
{status === 'working' && (
<Badge
variant="secondary"
className="bg-amber-50 text-amber-900 hover:bg-amber-50"
>
Working
</Badge>
)}
{status === 'asleep' && (
<Badge variant="outline" className="text-muted-foreground">
Asleep
</Badge>
)}
{status === 'error' && (
<Badge variant="destructive">Attention</Badge>
)}
</div>
<div className="mb-2 flex flex-wrap items-center gap-1.5 text-xs">
<Badge variant="secondary" className="font-normal">
{adapterLabel(adapterId)}
</Badge>
{agent.modelLabel && agent.modelLabel !== 'default' && (
<Badge variant="outline" className="font-normal">
{agent.modelLabel}
</Badge>
)}
{reasoningEffort && reasoningEffort !== 'medium' && (
<Badge variant="outline" className="font-normal">
{reasoningEffort}
</Badge>
)}
</div>
<div className="flex flex-wrap items-center gap-2 text-muted-foreground text-xs">
<span>Last used {lastUsedLabel}</span>
{workspace && (
<>
<span aria-hidden></span>
<span className="truncate font-mono" title={workspace}>
{workspace}
</span>
</>
)}
</div>
</div>
<div className="flex shrink-0 items-center gap-2">
<Button variant="outline" size="sm" onClick={handleChat}>
<MessageSquare className="mr-1.5 h-3 w-3" />
Chat
</Button>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button
variant="ghost"
size="icon"
aria-label={`More actions for ${displayName(agent)}`}
className="h-8 w-8"
>
<MoreHorizontal className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-44">
<DropdownMenuItem onSelect={() => void handleCopyId()}>
<Copy className="mr-2 h-3.5 w-3.5" />
Copy id
</DropdownMenuItem>
<RenameMenuItem disabled={!allowRename} />
<ResetHistoryMenuItem />
<DropdownMenuSeparator />
<DropdownMenuItem
onSelect={() => onDelete(agent)}
disabled={!allowDelete || deleting}
className="text-destructive focus:text-destructive"
>
{deleting ? (
<Loader2 className="mr-2 h-3.5 w-3.5 animate-spin" />
) : (
<Trash2 className="mr-2 h-3.5 w-3.5" />
)}
Delete
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
</div>
</div>
)
}
const RenameMenuItem: FC<{ disabled: boolean }> = ({ disabled }) => {
const item = (
<DropdownMenuItem disabled className="text-muted-foreground">
<Pencil className="mr-2 h-3.5 w-3.5" />
Rename
</DropdownMenuItem>
)
if (!disabled) return item
// Disabled but with a hint so users know it's coming, not broken.
return (
<TooltipProvider delayDuration={300}>
<Tooltip>
<TooltipTrigger asChild>
<span className="block w-full">{item}</span>
</TooltipTrigger>
<TooltipContent side="left" className="text-xs">
Rename coming soon
</TooltipContent>
</Tooltip>
</TooltipProvider>
)
}
const ResetHistoryMenuItem: FC = () => {
const item = (
<DropdownMenuItem disabled className="text-muted-foreground">
<RotateCcw className="mr-2 h-3.5 w-3.5" />
Reset history
</DropdownMenuItem>
)
return (
<TooltipProvider delayDuration={300}>
<Tooltip>
<TooltipTrigger asChild>
<span className="block w-full">{item}</span>
</TooltipTrigger>
<TooltipContent side="left" className="text-xs">
Reset history coming soon
</TooltipContent>
</Tooltip>
</TooltipProvider>
)
}
function inferAdapterFromListItem(
agent: AgentListItem,
): HarnessAgentAdapter | 'unknown' {
const label = agent.runtimeLabel?.toLowerCase()
if (label?.includes('claude')) return 'claude'
if (label?.includes('codex')) return 'codex'
if (label?.includes('openclaw')) return 'openclaw'
return 'unknown'
}
function livenessDetail(
status: AgentLiveness,
lastUsedAt: number | null | undefined,
): string | undefined {
if (lastUsedAt == null) return undefined
const diffMin = Math.floor((Date.now() - lastUsedAt) / 60_000)
if (status === 'idle') return `Idle for ${Math.max(0, diffMin)} min`
if (status === 'asleep') {
if (diffMin < 60) return `Asleep — quiet for ${diffMin} min`
const hr = Math.floor(diffMin / 60)
return `Asleep — quiet for ${hr} hr`
}
if (status === 'working') return 'Working on a turn'
if (status === 'error') return 'Attention — last turn failed'
return undefined
}

View File

@@ -5,14 +5,16 @@ import {
import { FitAddon } from '@xterm/addon-fit'
import { WebLinksAddon } from '@xterm/addon-web-links'
import { Terminal } from '@xterm/xterm'
import { ArrowLeft } from 'lucide-react'
import { type FC, useEffect, useRef } from 'react'
import { ArrowLeft, Check, Copy } from 'lucide-react'
import { type FC, useEffect, useRef, useState } from 'react'
import '@xterm/xterm/css/xterm.css'
import { Button } from '@/components/ui/button'
import { getAgentServerUrl } from '@/lib/browseros/helpers'
interface AgentTerminalProps {
onBack: () => void
initialCommand?: string
onSessionExit?: () => void
}
type TerminalServerMessage =
@@ -36,26 +38,22 @@ function resolveCssColor(variableName: string): string {
return color
}
function withAlpha(color: string, alpha: number): string {
const channels = color.match(/[\d.]+/g)
if (!channels || channels.length < 3) return color
const [red, green, blue] = channels
return `rgb(${red} ${green} ${blue} / ${alpha})`
}
function createTerminalTheme() {
const isDark = document.documentElement.classList.contains('dark')
const background = resolveCssColor('--background')
const foreground = resolveCssColor('--foreground')
const muted = resolveCssColor('--muted-foreground')
const accent = resolveCssColor('--accent-orange')
return {
background,
foreground,
cursor: foreground,
cursorAccent: background,
selectionBackground: withAlpha(accent, isDark ? 0.3 : 0.2),
// Solid terminal-standard selection colors. Deriving from a CSS var
// with alpha composed against the background produced near-white
// rectangles on light mode, making selection invisible.
selectionBackground: isDark ? '#3a4463' : '#b4d4f4',
selectionInactiveBackground: isDark ? '#2b3348' : '#d9e5f3',
selectionForeground: foreground,
black: isDark ? '#16131a' : '#1f1b22',
red: isDark ? '#ef8c7c' : '#c25544',
@@ -118,8 +116,38 @@ function parseTerminalMessage(data: unknown): TerminalServerMessage | null {
return null
}
export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
export const AgentTerminal: FC<AgentTerminalProps> = ({
onBack,
initialCommand,
onSessionExit,
}) => {
const containerRef = useRef<HTMLDivElement>(null)
const terminalRef = useRef<Terminal | null>(null)
// Refs keep the mount-once effect from tearing down the PTY when the
// parent re-renders with new inline callbacks.
const initialCommandRef = useRef(initialCommand)
const onSessionExitRef = useRef(onSessionExit)
initialCommandRef.current = initialCommand
onSessionExitRef.current = onSessionExit
const [copied, setCopied] = useState(false)
// Copy the current xterm selection to the browser clipboard. No-op
// if nothing is selected — users who want the whole buffer can
// Cmd+A first. Uses the browser clipboard, not the container's, so
// it works even when the running TUI has mouse tracking enabled
// (Opt+drag forces a selection regardless, see terminal config).
const handleCopy = async (): Promise<void> => {
const text = terminalRef.current?.getSelection()
if (!text) return
try {
await navigator.clipboard.writeText(text)
setCopied(true)
window.setTimeout(() => setCopied(false), 1500)
} catch {
// clipboard permission denied or unavailable — swallow, user will retry
}
}
useEffect(() => {
if (!containerRef.current) return
@@ -132,6 +160,34 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
lineHeight: 1.25,
scrollback: 8000,
theme: createTerminalTheme(),
// Opt+click+drag forces a native text selection even when the
// running TUI has mouse-tracking enabled (xterm would otherwise
// forward every click to the app and selection wouldn't work).
macOptionClickForcesSelection: true,
})
terminalRef.current = terminal
// Cmd+A → select all, Cmd+C → copy selection via the browser
// clipboard. Return false so xterm doesn't also forward the keys
// to the running program.
terminal.attachCustomKeyEventHandler((event) => {
if (event.type !== 'keydown') return true
const isMac = navigator.platform.toUpperCase().includes('MAC')
const mod = isMac ? event.metaKey : event.ctrlKey
if (!mod) return true
const key = event.key.toLowerCase()
if (key === 'a') {
terminal.selectAll()
return false
}
if (key === 'c') {
const sel = terminal.getSelection()
if (sel) {
void navigator.clipboard.writeText(sel)
return false
}
}
return true
})
const fitAddon = new FitAddon()
@@ -139,6 +195,12 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
terminal.loadAddon(new WebLinksAddon())
terminal.open(containerRef.current)
// React 18 StrictMode double-invokes effects in dev. Everything
// async inside this effect is scoped to an AbortController; the
// cleanup aborts it and any pending awaits bail out, so we never
// leak a second live WebSocket or duplicate xterm listeners.
const ac = new AbortController()
const cleanups: Array<() => void> = []
let ws: WebSocket | null = null
let sawExit = false
@@ -159,17 +221,28 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
sendMessage({ type: 'resize', cols, rows })
}
const connect = async () => {
const connect = async (): Promise<void> => {
const baseUrl = await getAgentServerUrl()
if (ac.signal.aborted) return
const wsUrl = new URL('/terminal/ws', baseUrl)
wsUrl.protocol = wsUrl.protocol === 'https:' ? 'wss:' : 'ws:'
ws = new WebSocket(wsUrl)
// If the effect was cleaned up between the await above and now,
// close the socket we just opened and bail.
if (ac.signal.aborted) {
ws.close()
ws = null
return
}
cleanups.push(() => ws?.close())
ws.onopen = () => {
fitAddon.fit()
terminal.focus()
sendResize()
const cmd = initialCommandRef.current
if (cmd) sendMessage({ type: 'input', data: `${cmd}\n` })
}
ws.onmessage = (event) => {
@@ -185,6 +258,7 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
terminal.write(
`\r\n\x1b[90m[session ended with exit ${message.exitCode}]\x1b[0m\r\n`,
)
onSessionExitRef.current?.()
}
}
@@ -200,49 +274,41 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
const inputDisposable = terminal.onData((data) => {
sendMessage({ type: 'input', data })
})
const resizeDisposable = terminal.onResize(({ cols, rows }) => {
sendResize(cols, rows)
})
return () => {
inputDisposable.dispose()
resizeDisposable.dispose()
}
cleanups.push(() => inputDisposable.dispose())
cleanups.push(() => resizeDisposable.dispose())
}
let disposeSocketBindings: (() => void) | undefined
void connect().then((disposeBindings) => {
disposeSocketBindings = disposeBindings
})
void connect()
const resizeObserver = new ResizeObserver(() => {
fitAddon.fit()
sendResize()
})
resizeObserver.observe(containerRef.current)
cleanups.push(() => resizeObserver.disconnect())
const themeObserver = new MutationObserver(() => {
applyTheme()
})
const themeObserver = new MutationObserver(() => applyTheme())
themeObserver.observe(document.documentElement, {
attributes: true,
attributeFilter: ['class'],
})
cleanups.push(() => themeObserver.disconnect())
return () => {
resizeObserver.disconnect()
themeObserver.disconnect()
disposeSocketBindings?.()
ws?.close()
ac.abort()
for (const dispose of cleanups) dispose()
terminal.dispose()
terminalRef.current = null
}
}, [])
return (
<div className="flex h-[calc(100dvh-10rem)] min-h-[32rem] w-full flex-col py-2 sm:min-h-[42rem] sm:py-4">
<div className="flex min-h-0 flex-1 flex-col overflow-hidden rounded-xl border border-border bg-card shadow-sm">
<div className="flex items-center gap-3 border-border border-b px-4 py-3 sm:px-6">
<div className="flex items-center justify-between gap-3 border-border border-b px-4 py-3 sm:px-6">
<div className="flex min-w-0 items-center gap-3">
<Button variant="ghost" size="icon" onClick={onBack}>
<ArrowLeft className="size-4" />
@@ -256,6 +322,14 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
</div>
</div>
</div>
<Button variant="outline" size="sm" onClick={handleCopy}>
{copied ? (
<Check className="mr-1 size-3.5" />
) : (
<Copy className="mr-1 size-3.5" />
)}
{copied ? 'Copied' : 'Copy'}
</Button>
</div>
<div className="min-h-0 flex-1 p-4 sm:p-6">
@@ -269,7 +343,7 @@ export const AgentTerminal: FC<AgentTerminalProps> = ({ onBack }) => {
</div>
</div>
<div className="min-h-0 flex-1 px-4 py-4 sm:px-5 sm:py-5">
<div className="min-h-0 flex-1 cursor-text px-4 py-4 sm:px-5 sm:py-5">
<div ref={containerRef} className="h-full w-full" />
</div>
</div>

View File

@@ -0,0 +1,32 @@
import { Bot, Plus } from 'lucide-react'
import type { FC } from 'react'
import { Button } from '@/components/ui/button'
interface AgentsEmptyStateProps {
onCreateAgent: () => void
}
export const AgentsEmptyState: FC<AgentsEmptyStateProps> = ({
onCreateAgent,
}) => {
return (
<div className="rounded-xl border border-border border-dashed bg-card/50 p-12 text-center">
<div className="mx-auto mb-4 flex h-12 w-12 items-center justify-center rounded-xl bg-[var(--accent-orange)]/10">
<Bot className="h-6 w-6 text-[var(--accent-orange)]" />
</div>
<h3 className="mb-1 font-semibold">No agents yet</h3>
<p className="mx-auto mb-4 max-w-sm text-muted-foreground text-sm">
Spin up an OpenClaw, Claude Code, or Codex agent to chat with, schedule,
or run in the background.
</p>
<Button
onClick={onCreateAgent}
variant="outline"
className="border-[var(--accent-orange)] bg-[var(--accent-orange)]/10 text-[var(--accent-orange)] hover:bg-[var(--accent-orange)]/20 hover:text-[var(--accent-orange)]"
>
<Plus className="mr-1.5 h-4 w-4" />
Create your first agent
</Button>
</div>
)
}

View File

@@ -0,0 +1,41 @@
import { Bot, Plus } from 'lucide-react'
import type { FC } from 'react'
import { Button } from '@/components/ui/button'
interface AgentsHeaderProps {
onCreateAgent: () => void
}
/**
* Mirrors the visual shape of `SoulHeader` and `ScheduledTasksHeader`
* so the page reads as part of the same family. Loose lifecycle
* controls that used to sit next to the title moved into
* `GatewayStatusBar` — they're OpenClaw-specific and don't apply to
* Claude/Codex agents.
*/
export const AgentsHeader: FC<AgentsHeaderProps> = ({ onCreateAgent }) => {
return (
<div className="rounded-xl border border-border bg-card p-6 shadow-sm transition-all hover:shadow-md">
<div className="flex items-start gap-4">
<div className="flex h-12 w-12 shrink-0 items-center justify-center rounded-xl bg-[var(--accent-orange)]/10">
<Bot className="h-6 w-6 text-[var(--accent-orange)]" />
</div>
<div className="flex-1">
<h2 className="mb-1 font-semibold text-xl">Agents</h2>
<p className="text-muted-foreground text-sm">
OpenClaw, Claude Code, and Codex agents chat, schedule, and run
them in the background.
</p>
</div>
<Button
onClick={onCreateAgent}
className="border-[var(--accent-orange)] bg-[var(--accent-orange)]/10 text-[var(--accent-orange)] hover:bg-[var(--accent-orange)]/20 hover:text-[var(--accent-orange)]"
variant="outline"
>
<Plus className="mr-1.5 h-4 w-4" />
New Agent
</Button>
</div>
</div>
)
}

View File

@@ -0,0 +1,206 @@
import { Loader2, RotateCcw, Terminal } from 'lucide-react'
import type { FC, ReactNode } from 'react'
import { Badge } from '@/components/ui/badge'
import { Button } from '@/components/ui/button'
import { Separator } from '@/components/ui/separator'
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from '@/components/ui/tooltip'
import { cn } from '@/lib/utils'
import type { OpenClawStatus } from './useOpenClaw'
interface GatewayStatusBarProps {
status: OpenClawStatus | null
/** Disabled while a gateway lifecycle mutation is mid-flight. */
actionInProgress: boolean
onOpenTerminal: () => void
onRestart: () => void
}
/**
* Compact one-line status bar for the OpenClaw gateway. Renders the
* lifecycle pills (Running / Control plane connected) plus a Terminal
* escape hatch and a Restart Gateway action. Lives between the page
* header and the agent list when at least one OpenClaw agent is in
* the merged list; collapses to nothing for Claude/Codex-only setups.
*
* Status is sourced from `GET /agents`'s `gateway` field — the agents
* page no longer polls `/claw/status` directly. One endpoint, one
* 5s interval, no duplicate state.
*/
export const GatewayStatusBar: FC<GatewayStatusBarProps> = ({
status,
actionInProgress,
onOpenTerminal,
onRestart,
}) => {
if (!status) return null
const runningPill = pillForRuntimeStatus(status.status)
const controlPlanePill = pillForControlPlane(status.controlPlaneStatus)
return (
<div className="rounded-xl border border-border bg-card px-4 py-3 shadow-sm">
<div className="flex items-center gap-3 text-sm">
<span className="font-medium text-muted-foreground">
OpenClaw gateway
</span>
<Badge
variant={runningPill.variant}
className={cn('gap-1.5', runningPill.className)}
>
<span
className={cn(
'inline-block h-1.5 w-1.5 rounded-full',
runningPill.dot,
)}
/>
{runningPill.label}
</Badge>
<Badge
variant={controlPlanePill.variant}
className={cn('gap-1.5', controlPlanePill.className)}
>
<span
className={cn(
'inline-block h-1.5 w-1.5 rounded-full',
controlPlanePill.dot,
)}
/>
{controlPlanePill.label}
</Badge>
<Separator orientation="vertical" className="h-4" />
<WithTooltip label="Open a shell into the OpenClaw gateway container for raw CLI access (config edits, session inspection).">
<Button variant="ghost" size="sm" onClick={onOpenTerminal}>
<Terminal className="mr-1.5 h-3.5 w-3.5" />
Terminal
</Button>
</WithTooltip>
<WithTooltip label="Restart the OpenClaw gateway. Useful when the gateway is stuck or after editing provider config.">
<Button
variant="ghost"
size="sm"
onClick={onRestart}
disabled={actionInProgress}
className="ml-auto"
>
{actionInProgress ? (
<Loader2 className="mr-1.5 h-3.5 w-3.5 animate-spin" />
) : (
<RotateCcw className="mr-1.5 h-3.5 w-3.5" />
)}
Restart Gateway
</Button>
</WithTooltip>
</div>
</div>
)
}
const WithTooltip: FC<{ label: string; children: ReactNode }> = ({
label,
children,
}) => (
<TooltipProvider delayDuration={250}>
<Tooltip>
<TooltipTrigger asChild>{children}</TooltipTrigger>
<TooltipContent side="bottom" className="max-w-xs text-xs">
{label}
</TooltipContent>
</Tooltip>
</TooltipProvider>
)
type PillKind = {
variant: 'default' | 'secondary' | 'outline' | 'destructive'
label: string
dot: string
className?: string
}
function pillForRuntimeStatus(status: OpenClawStatus['status']): PillKind {
switch (status) {
case 'running':
return {
variant: 'secondary',
label: 'Running',
dot: 'bg-emerald-500',
className: 'bg-emerald-50 text-emerald-900 hover:bg-emerald-50',
}
case 'starting':
return {
variant: 'secondary',
label: 'Starting',
dot: 'bg-amber-500 animate-pulse',
className: 'bg-amber-50 text-amber-900 hover:bg-amber-50',
}
case 'stopped':
return {
variant: 'outline',
label: 'Stopped',
dot: 'bg-muted-foreground/40',
}
case 'error':
return {
variant: 'destructive',
label: 'Error',
dot: 'bg-destructive-foreground',
}
default:
return {
variant: 'outline',
label: 'Unknown',
dot: 'bg-muted-foreground/40',
}
}
}
function pillForControlPlane(
status: OpenClawStatus['controlPlaneStatus'],
): PillKind {
switch (status) {
case 'connected':
return {
variant: 'secondary',
label: 'Control plane connected',
dot: 'bg-emerald-500',
className: 'bg-emerald-50 text-emerald-900 hover:bg-emerald-50',
}
case 'connecting':
return {
variant: 'secondary',
label: 'Connecting',
dot: 'bg-amber-500 animate-pulse',
className: 'bg-amber-50 text-amber-900 hover:bg-amber-50',
}
case 'reconnecting':
return {
variant: 'secondary',
label: 'Reconnecting',
dot: 'bg-amber-500 animate-pulse',
className: 'bg-amber-50 text-amber-900 hover:bg-amber-50',
}
case 'recovering':
return {
variant: 'secondary',
label: 'Recovering',
dot: 'bg-amber-500 animate-pulse',
className: 'bg-amber-50 text-amber-900 hover:bg-amber-50',
}
case 'failed':
return {
variant: 'destructive',
label: 'Needs attention',
dot: 'bg-destructive-foreground',
}
default:
return {
variant: 'outline',
label: 'Disconnected',
dot: 'bg-muted-foreground/40',
}
}
}

View File

@@ -0,0 +1,83 @@
import type { FC } from 'react'
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from '@/components/ui/tooltip'
import { cn } from '@/lib/utils'
export type AgentLiveness = 'working' | 'idle' | 'asleep' | 'error' | 'unknown'
interface LivenessDotProps {
status: AgentLiveness
/**
* Optional human-friendly secondary line, e.g. "Idle for 4 min" or
* "Asleep — no activity for 22 min". When absent the tooltip just
* reads the status label.
*/
detail?: string
className?: string
}
const VARIANT: Record<
AgentLiveness,
{ dot: string; ring: string; label: string }
> = {
working: {
// Animated amber pulse + soft halo so the eye catches an active
// agent in a long list without the dot screaming for attention.
dot: 'bg-amber-500 animate-pulse',
ring: 'ring-2 ring-amber-200',
label: 'Working on a turn',
},
idle: {
dot: 'bg-emerald-500',
ring: 'ring-2 ring-emerald-100',
label: 'Idle',
},
asleep: {
dot: 'bg-muted-foreground/40',
ring: 'ring-2 ring-muted',
label: 'Asleep',
},
error: {
dot: 'bg-destructive',
ring: 'ring-2 ring-destructive/30',
label: 'Attention',
},
unknown: {
dot: 'bg-muted-foreground/30',
ring: 'ring-2 ring-muted',
label: 'Status unknown',
},
}
export const LivenessDot: FC<LivenessDotProps> = ({
status,
detail,
className,
}) => {
const variant = VARIANT[status]
return (
<TooltipProvider delayDuration={150}>
<Tooltip>
<TooltipTrigger asChild>
<span
role="img"
aria-label={detail ?? variant.label}
className={cn(
'inline-block h-3 w-3 rounded-full',
variant.dot,
variant.ring,
className,
)}
/>
</TooltipTrigger>
<TooltipContent side="right" className="text-xs">
{detail ?? variant.label}
</TooltipContent>
</Tooltip>
</TooltipProvider>
)
}

View File

@@ -0,0 +1,260 @@
import { AlertCircle, Loader2 } from 'lucide-react'
import type { FC } from 'react'
import { Alert, AlertDescription, AlertTitle } from '@/components/ui/alert'
import { Button } from '@/components/ui/button'
import {
Dialog,
DialogContent,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select'
import type {
HarnessAdapterDescriptor,
HarnessAgentAdapter,
} from './agent-harness-types'
import type { CreateAgentRuntime, ProviderOption } from './agents-page-types'
import { ProviderSelector } from './OpenClawControls'
import {
type OpenClawCliProvider,
type OpenClawCliProviderAuthStatus,
OpenClawCliProviderStatusPanel,
} from './openclaw-cli-providers'
interface NewAgentDialogProps {
adapters: HarnessAdapterDescriptor[]
canManageOpenClaw: boolean
createError: string | null
createRuntime: CreateAgentRuntime
creating: boolean
defaultProviderId: string
harnessAdapterId: HarnessAgentAdapter
harnessModelId: string
harnessReasoningEffort: string
name: string
open: boolean
providers: ProviderOption[]
selectedCliProvider: OpenClawCliProvider | undefined
selectedProviderId: string
cliAuthError: Error | null
cliAuthLoading: boolean
cliAuthStatus: OpenClawCliProviderAuthStatus | undefined
onConnectCliProvider: () => void
onCreate: () => void
onOpenChange: (open: boolean) => void
onRuntimeChange: (runtime: CreateAgentRuntime) => void
onHarnessAdapterChange: (adapter: HarnessAgentAdapter) => void
onHarnessModelChange: (modelId: string) => void
onHarnessReasoningChange: (reasoningEffort: string) => void
onNameChange: (name: string) => void
onProviderChange: (providerId: string) => void
}
export const NewAgentDialog: FC<NewAgentDialogProps> = ({
adapters,
canManageOpenClaw,
createError,
createRuntime,
creating,
defaultProviderId,
harnessAdapterId,
harnessModelId,
harnessReasoningEffort,
name,
open,
providers,
selectedCliProvider,
selectedProviderId,
cliAuthError,
cliAuthLoading,
cliAuthStatus,
onConnectCliProvider,
onCreate,
onOpenChange,
onRuntimeChange,
onHarnessAdapterChange,
onHarnessModelChange,
onHarnessReasoningChange,
onNameChange,
onProviderChange,
}) => {
const selectedHarnessAdapter =
adapters.find((adapter) => adapter.id === harnessAdapterId) ?? adapters[0]
const isHarnessRuntime = createRuntime !== 'openclaw'
const openClawBlocked = createRuntime === 'openclaw' && !canManageOpenClaw
const cliBlocked =
createRuntime === 'openclaw' &&
!!selectedCliProvider &&
!cliAuthStatus?.loggedIn
const canCreate =
Boolean(name.trim()) &&
!creating &&
!openClawBlocked &&
!cliBlocked &&
(createRuntime === 'openclaw'
? providers.length > 0
: Boolean(selectedHarnessAdapter))
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent>
<DialogHeader>
<DialogTitle>New Agent</DialogTitle>
</DialogHeader>
<div className="grid gap-4 py-2">
{createError ? (
<Alert variant="destructive">
<AlertCircle className="size-4" />
<AlertTitle>Create failed</AlertTitle>
<AlertDescription>{createError}</AlertDescription>
</Alert>
) : null}
<div className="grid gap-2">
<Label htmlFor="agent-name">Name</Label>
<Input
id="agent-name"
value={name}
onChange={(event) => onNameChange(event.target.value)}
placeholder={
createRuntime === 'openclaw' ? 'research-agent' : 'Review bot'
}
onKeyDown={(event) => {
if (event.key === 'Enter' && canCreate) onCreate()
}}
/>
</div>
<div className="grid gap-2">
<Label htmlFor="agent-runtime">Adapter</Label>
<Select
value={createRuntime}
onValueChange={(value) => {
if (
value === 'openclaw' ||
value === 'claude' ||
value === 'codex'
) {
onRuntimeChange(value)
if (value !== 'openclaw') onHarnessAdapterChange(value)
}
}}
>
<SelectTrigger id="agent-runtime">
<SelectValue />
</SelectTrigger>
<SelectContent>
{adapters.map((adapter) => (
<SelectItem key={adapter.id} value={adapter.id}>
{adapter.name}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
{createRuntime === 'openclaw' ? (
<>
{openClawBlocked ? (
<Alert>
<AlertCircle className="size-4" />
<AlertTitle>OpenClaw is not ready</AlertTitle>
<AlertDescription>
Start or set up the OpenClaw gateway before creating an
OpenClaw agent.
</AlertDescription>
</Alert>
) : null}
<ProviderSelector
providers={providers}
defaultProviderId={defaultProviderId}
selectedId={selectedProviderId}
onSelect={onProviderChange}
hideApiKeyHint={!!selectedCliProvider}
/>
{selectedCliProvider ? (
<OpenClawCliProviderStatusPanel
provider={selectedCliProvider}
status={cliAuthStatus}
loading={cliAuthLoading}
fetchError={cliAuthError}
onConnect={onConnectCliProvider}
/>
) : null}
</>
) : null}
{isHarnessRuntime ? (
<>
<div className="grid gap-2">
<Label htmlFor="harness-model">Model</Label>
<Select
value={harnessModelId}
onValueChange={onHarnessModelChange}
>
<SelectTrigger id="harness-model">
<SelectValue />
</SelectTrigger>
<SelectContent>
{(selectedHarnessAdapter?.models ?? []).map((model) => (
<SelectItem key={model.id} value={model.id}>
{model.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
<div className="grid gap-2">
<Label htmlFor="harness-effort">Reasoning</Label>
<Select
value={harnessReasoningEffort}
onValueChange={onHarnessReasoningChange}
>
<SelectTrigger id="harness-effort">
<SelectValue />
</SelectTrigger>
<SelectContent>
{(selectedHarnessAdapter?.reasoningEfforts ?? []).map(
(effort) => (
<SelectItem key={effort.id} value={effort.id}>
{effort.label}
</SelectItem>
),
)}
</SelectContent>
</Select>
</div>
</>
) : null}
</div>
<DialogFooter>
<Button
variant="outline"
onClick={() => onOpenChange(false)}
disabled={creating}
>
Cancel
</Button>
<Button disabled={!canCreate} onClick={onCreate}>
{creating ? <Loader2 className="mr-2 size-4 animate-spin" /> : null}
Create
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
)
}

View File

@@ -0,0 +1,387 @@
import {
AlertCircle,
Cpu,
Loader2,
Plus,
RefreshCw,
ShieldAlert,
Square,
TerminalSquare,
WifiOff,
Wrench,
} from 'lucide-react'
import type { FC } from 'react'
import { Alert, AlertDescription, AlertTitle } from '@/components/ui/alert'
import { Badge } from '@/components/ui/badge'
import { Button } from '@/components/ui/button'
import { Card, CardContent } from '@/components/ui/card'
import { Label } from '@/components/ui/label'
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select'
import type { ProviderOption } from './agents-page-types'
import {
CONTROL_PLANE_COPY,
FALLBACK_CONTROL_PLANE_COPY,
} from './agents-page-types'
import type { getControlPlaneCopy } from './agents-page-utils'
import type { OpenClawStatus } from './useOpenClaw'
const StatusBadge: FC<{ status: OpenClawStatus['status'] }> = ({ status }) => {
const variants: Record<
OpenClawStatus['status'],
{
variant: 'default' | 'secondary' | 'outline' | 'destructive'
label: string
}
> = {
running: { variant: 'default', label: 'Running' },
starting: { variant: 'secondary', label: 'Starting...' },
stopped: { variant: 'outline', label: 'Stopped' },
error: { variant: 'destructive', label: 'Error' },
uninitialized: { variant: 'outline', label: 'Not Set Up' },
}
const current = variants[status] ?? {
variant: 'outline' as const,
label: 'Unknown',
}
return <Badge variant={current.variant}>{current.label}</Badge>
}
const ControlPlaneBadge: FC<{
status: OpenClawStatus['controlPlaneStatus']
}> = ({ status }) => {
const current = CONTROL_PLANE_COPY[status] ?? FALLBACK_CONTROL_PLANE_COPY
return <Badge variant={current.badgeVariant}>{current.badgeLabel}</Badge>
}
interface ProviderSelectorProps {
providers: ProviderOption[]
defaultProviderId: string
selectedId: string
onSelect: (id: string) => void
hideApiKeyHint?: boolean
}
export const ProviderSelector: FC<ProviderSelectorProps> = ({
providers,
defaultProviderId,
selectedId,
onSelect,
hideApiKeyHint,
}) => {
if (providers.length === 0) {
return (
<div className="space-y-2">
<p className="font-medium text-sm">LLM Provider</p>
<p className="text-muted-foreground text-sm">
No compatible LLM providers configured.{' '}
<a href="#/settings/ai" className="underline">
Add one in AI settings
</a>{' '}
first.
</p>
</div>
)
}
return (
<div className="space-y-2">
<Label htmlFor="provider-select">LLM Provider</Label>
<Select value={selectedId} onValueChange={onSelect}>
<SelectTrigger id="provider-select">
<SelectValue placeholder="Select a provider" />
</SelectTrigger>
<SelectContent>
{providers.map((provider) => (
<SelectItem key={provider.id} value={provider.id}>
{provider.name} - {provider.modelId}
{provider.id === defaultProviderId ? ' (default)' : ''}
</SelectItem>
))}
</SelectContent>
</Select>
{!hideApiKeyHint && (
<p className="text-muted-foreground text-xs">
Uses your existing API key from BrowserOS settings. The key is passed
to the container and never leaves your machine.
</p>
)}
</div>
)
}
interface AgentsPageHeaderProps {
actionInProgress: boolean
controlPlaneBusy: boolean
reconnecting: boolean
status: OpenClawStatus | null
onCreateAgent: () => void
onOpenTerminal: () => void
onReconnect: () => void
onRefresh: () => void
onRestart: () => void
onStop: () => void
}
export const AgentsPageHeader: FC<AgentsPageHeaderProps> = ({
actionInProgress,
controlPlaneBusy,
reconnecting,
status,
onCreateAgent,
onOpenTerminal,
onReconnect,
onRefresh,
onRestart,
onStop,
}) => (
<div className="flex flex-wrap items-center justify-between gap-3">
<div>
<h1 className="font-semibold text-2xl tracking-normal">Agents</h1>
<p className="text-muted-foreground text-sm">
OpenClaw, Claude Code, and Codex agents
</p>
</div>
<div className="flex flex-wrap items-center gap-2">
{status ? (
<>
<StatusBadge status={status.status} />
{status.status !== 'uninitialized' && (
<ControlPlaneBadge status={status.controlPlaneStatus} />
)}
</>
) : null}
{status?.status === 'running' &&
status.controlPlaneStatus !== 'connected' ? (
<Button
variant="outline"
onClick={onReconnect}
disabled={actionInProgress || controlPlaneBusy}
>
{reconnecting ? (
<Loader2 className="mr-2 size-4 animate-spin" />
) : (
<RefreshCw className="mr-2 size-4" />
)}
Retry Connection
</Button>
) : null}
{status?.status === 'running' ? (
<>
<Button
variant="ghost"
size="icon"
onClick={onRestart}
disabled={actionInProgress}
title="Restart gateway"
>
<RefreshCw className="size-4" />
</Button>
<Button
variant="ghost"
size="icon"
onClick={onStop}
disabled={actionInProgress}
title="Stop gateway"
>
<Square className="size-4" />
</Button>
<Button variant="outline" onClick={onOpenTerminal}>
<TerminalSquare className="mr-2 size-4" />
Terminal
</Button>
</>
) : null}
<Button variant="ghost" size="icon" onClick={onRefresh} title="Refresh">
<RefreshCw className="size-4" />
</Button>
<Button onClick={onCreateAgent}>
<Plus className="mr-2 size-4" />
New Agent
</Button>
</div>
</div>
)
export function LifecycleAlert({ message }: { message: string }) {
return (
<Alert>
<Loader2 className="size-4 animate-spin" />
<AlertTitle>{message}</AlertTitle>
</Alert>
)
}
export function InlineErrorAlert({
message,
onDismiss,
}: {
message: string
onDismiss: () => void
}) {
return (
<Alert variant="destructive">
<AlertCircle className="size-4" />
<AlertTitle>Agent action failed</AlertTitle>
<AlertDescription>
<p>{message}</p>
<div className="mt-2">
<Button variant="outline" size="sm" onClick={onDismiss}>
Dismiss
</Button>
</div>
</AlertDescription>
</Alert>
)
}
interface ControlPlaneAlertProps {
actionInProgress: boolean
controlPlaneBusy: boolean
controlPlaneCopy: ReturnType<typeof getControlPlaneCopy>
reconnecting: boolean
recoveryDetail: string | null
status: OpenClawStatus
onReconnect: () => void
onRestart: () => void
}
export const ControlPlaneAlert: FC<ControlPlaneAlertProps> = ({
actionInProgress,
controlPlaneBusy,
controlPlaneCopy,
reconnecting,
recoveryDetail,
status,
onReconnect,
onRestart,
}) => (
<Alert
variant={status.controlPlaneStatus === 'failed' ? 'destructive' : 'default'}
>
{status.controlPlaneStatus === 'failed' ? (
<ShieldAlert className="size-4" />
) : status.controlPlaneStatus === 'recovering' ? (
<Wrench className="size-4" />
) : (
<WifiOff className="size-4" />
)}
<AlertTitle>{controlPlaneCopy.title}</AlertTitle>
<AlertDescription>
<p>{controlPlaneCopy.description}</p>
{recoveryDetail ? <p>{recoveryDetail}</p> : null}
<div className="mt-2 flex flex-wrap gap-2">
<Button
variant="outline"
size="sm"
onClick={onReconnect}
disabled={actionInProgress || controlPlaneBusy}
>
{reconnecting ? (
<Loader2 className="mr-2 size-4 animate-spin" />
) : (
<RefreshCw className="mr-2 size-4" />
)}
Retry Connection
</Button>
<Button
variant="outline"
size="sm"
onClick={onRestart}
disabled={actionInProgress}
>
Restart Gateway
</Button>
</div>
</AlertDescription>
</Alert>
)
interface GatewayStateCardsProps {
actionInProgress: boolean
status: OpenClawStatus | null
onOpenSetup: () => void
onRestart: () => void
onStart: () => void
}
export const GatewayStateCards: FC<GatewayStateCardsProps> = ({
actionInProgress,
status,
onOpenSetup,
onRestart,
onStart,
}) => (
<>
{status?.status === 'uninitialized' ? (
<Card>
<CardContent className="flex flex-col items-center gap-4 py-12">
<Cpu className="size-12 text-muted-foreground" />
<div className="text-center">
<h3 className="font-semibold text-lg">Set Up OpenClaw</h3>
<p className="text-muted-foreground text-sm">
{status.podmanAvailable
? 'Create a local BrowserOS VM to run autonomous agents with full tool access.'
: 'BrowserOS VM runtime is unavailable on this system.'}
</p>
</div>
{status.podmanAvailable ? (
<Button onClick={onOpenSetup}>Set Up Now</Button>
) : null}
</CardContent>
</Card>
) : null}
{status?.status === 'stopped' ? (
<Card>
<CardContent className="flex flex-col items-center gap-4 py-12">
<Cpu className="size-12 text-muted-foreground" />
<div className="text-center">
<h3 className="font-semibold text-lg">Gateway Stopped</h3>
<p className="text-muted-foreground text-sm">
The OpenClaw gateway is not running.
</p>
</div>
<Button onClick={onStart} disabled={actionInProgress}>
Start Gateway
</Button>
</CardContent>
</Card>
) : null}
{status?.status === 'error' ? (
<Card className="border-destructive">
<CardContent className="flex flex-col items-center gap-4 py-12">
<AlertCircle className="size-12 text-destructive" />
<div className="text-center">
<h3 className="font-semibold text-lg">Gateway Error</h3>
<p className="text-muted-foreground text-sm">
{status.error ?? status.lastGatewayError}
</p>
</div>
<div className="flex gap-2">
<Button onClick={onStart} disabled={actionInProgress}>
Start Gateway
</Button>
<Button
variant="outline"
onClick={onRestart}
disabled={actionInProgress}
>
Restart Gateway
</Button>
</div>
</CardContent>
</Card>
) : null}
</>
)

View File

@@ -0,0 +1,76 @@
import { Loader2 } from 'lucide-react'
import type { FC } from 'react'
import { Button } from '@/components/ui/button'
import {
Dialog,
DialogContent,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog'
import type { ProviderOption } from './agents-page-types'
import { ProviderSelector } from './OpenClawControls'
import type { OpenClawCliProvider } from './openclaw-cli-providers'
interface SetupOpenClawDialogProps {
defaultProviderId: string
open: boolean
providers: ProviderOption[]
selectedProviderId: string
selectedCliProvider: OpenClawCliProvider | undefined
settingUp: boolean
onOpenChange: (open: boolean) => void
onProviderChange: (providerId: string) => void
onSetup: () => void
}
export const SetupOpenClawDialog: FC<SetupOpenClawDialogProps> = ({
defaultProviderId,
open,
providers,
selectedProviderId,
selectedCliProvider,
settingUp,
onOpenChange,
onProviderChange,
onSetup,
}) => (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent>
<DialogHeader>
<DialogTitle>Set Up OpenClaw</DialogTitle>
</DialogHeader>
<div className="space-y-4 py-2">
<ProviderSelector
providers={providers}
defaultProviderId={defaultProviderId}
selectedId={selectedProviderId}
onSelect={onProviderChange}
hideApiKeyHint={!!selectedCliProvider}
/>
{selectedCliProvider ? (
<p className="rounded-md border border-border bg-muted/30 px-3 py-2 text-muted-foreground text-xs">
{selectedCliProvider.description}. Clicking{' '}
<span className="font-medium">Set Up &amp; Start</span> starts the
gateway and opens a terminal to sign in.
</p>
) : null}
<Button
onClick={onSetup}
disabled={settingUp || providers.length === 0}
className="w-full"
>
{settingUp ? (
<>
<Loader2 className="mr-2 size-4 animate-spin" />
Setting up...
</>
) : (
'Set Up & Start'
)}
</Button>
</div>
</DialogContent>
</Dialog>
)

View File

@@ -0,0 +1,4 @@
export function buildAgentApiUrl(baseUrl: string, path: string): string {
const normalizedPath = path === '/' ? '' : path
return `${baseUrl}/agents${normalizedPath}`
}

View File

@@ -0,0 +1,84 @@
import type { AgentListItem } from './agents-page-types'
/**
* Display rules for the redesigned agent rows. Pure helpers — no React,
* no API calls — so they're trivial to unit-test and the row card stays
* focused on layout.
*/
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
const OC_UUID_PATTERN =
/^oc-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
/**
* The agent rail used to render whatever the gateway returned for `name`.
* Post-migration that's frequently the agent's UUID — readable to nobody.
* Prefer the explicit `name` when it differs meaningfully from the id;
* otherwise fall back to a short prefix users can recognize on second
* glance.
*/
export function displayName(agent: AgentListItem): string {
const name = agent.name?.trim()
const id = agent.agentId
if (!name || name === id) {
if (OC_UUID_PATTERN.test(id)) return id.slice(0, 11) // "oc-XXXXXXXX"
if (UUID_PATTERN.test(id)) return id.slice(0, 8)
return id
}
return name
}
export function canDelete(agent: AgentListItem): boolean {
// The gateway's protected `main` agent must not be deletable. The
// server enforces this too, but disabling the menu item avoids users
// hitting an opaque 400.
if (agent.agentId === 'main') return false
return agent.canDelete
}
/**
* Rename will be wired to a future `PATCH /agents/:id` endpoint. The
* legacy `/claw/agents` create flow named the agent on the gateway via
* the `name` field but the field isn't editable post-create today.
*/
export function canRename(_agent: AgentListItem): boolean {
return false
}
/**
* The detail line carries the agent's workspace path. The `detail`
* field on AgentListItem already holds it for OpenClaw entries
* (`/home/node/.openclaw/workspace-...`); for harness agents it's the
* synthetic `<adapter>:main` marker that's not informative — hide it.
*/
export function workspaceLabel(agent: AgentListItem): string | null {
if (!agent.detail) return null
if (/^(claude|codex|openclaw):main$/.test(agent.detail)) return null
return agent.detail
}
const ONE_MINUTE = 60_000
const ONE_HOUR = 60 * ONE_MINUTE
const ONE_DAY = 24 * ONE_HOUR
/**
* Lightweight relative-time formatter. We don't want to drag in
* `dayjs/relativeTime` just for a few labels.
*/
export function formatRelativeTime(epochMs: number | null): string {
if (epochMs === null || !Number.isFinite(epochMs)) return 'never'
const diff = Math.max(0, Date.now() - epochMs)
if (diff < ONE_MINUTE) return 'just now'
if (diff < ONE_HOUR) {
const m = Math.floor(diff / ONE_MINUTE)
return `${m} min ago`
}
if (diff < ONE_DAY) {
const h = Math.floor(diff / ONE_HOUR)
return h === 1 ? '1 hr ago' : `${h} hr ago`
}
const d = Math.floor(diff / ONE_DAY)
return d === 1 ? '1 day ago' : `${d} days ago`
}

View File

@@ -0,0 +1,118 @@
import type { AgentEntry } from './useOpenClaw'
export type HarnessAgentAdapter = 'claude' | 'codex' | 'openclaw'
export type AgentHarnessStreamEvent =
| {
type: 'text_delta'
text: string
stream: 'output' | 'thought'
rawType?: string
}
| {
type: 'tool_call'
text: string
title: string
id?: string
status?: string
rawType?: string
}
| {
type: 'status'
text: string
rawType?: string
}
| {
type: 'done'
text?: string
stopReason?: string
}
| {
type: 'error'
message: string
code?: string
}
export type HarnessAgentLiveness = 'working' | 'idle' | 'asleep' | 'error'
export interface HarnessAgent {
id: string
name: string
adapter: HarnessAgentAdapter
modelId?: string
reasoningEffort?: string
permissionMode: 'approve-all'
sessionKey: string
createdAt: number
updatedAt: number
/**
* Server-derived liveness state. When the listing endpoint hasn't
* been enriched yet (older deployments) this is undefined and the UI
* falls back to `unknown`.
*/
status?: HarnessAgentLiveness
/**
* Wall-clock ms of the last persisted turn. `null` for never-used
* agents. Drives the recency sort and the "Last used X min ago" copy.
*/
lastUsedAt?: number | null
}
export interface HarnessAdapterDescriptor {
id: HarnessAgentAdapter
name: string
defaultModelId: string
defaultReasoningEffort: string
modelControl: 'runtime-supported' | 'best-effort'
models: Array<{ id: string; label: string; recommended?: boolean }>
reasoningEfforts: Array<{ id: string; label: string; recommended?: boolean }>
}
export interface CreateHarnessAgentInput {
name: string
adapter: HarnessAgentAdapter
modelId?: string
reasoningEffort?: string
}
export interface HarnessHistoryReasoning {
text: string
durationMs?: number
}
export interface HarnessHistoryToolCall {
toolCallId?: string
toolName: string
status: 'pending' | 'running' | 'completed' | 'failed'
input?: unknown
output?: unknown
error?: string
durationMs?: number
}
export interface HarnessHistoryEntry {
id: string
agentId: string
sessionId: 'main'
role: 'user' | 'assistant'
text: string
createdAt: number
reasoning?: HarnessHistoryReasoning
toolCalls?: HarnessHistoryToolCall[]
}
export interface HarnessAgentHistoryPage {
agentId: string
sessionId: 'main'
items: HarnessHistoryEntry[]
}
export function mapHarnessAgentToEntry(agent: HarnessAgent): AgentEntry {
return {
agentId: agent.id,
name: agent.name,
workspace: `${agent.adapter}:main`,
model: agent.modelId,
source: 'agent-harness',
}
}

View File

@@ -0,0 +1,172 @@
import type { NavigateFunction } from 'react-router'
import {
AGENT_CREATED_EVENT,
AGENT_DELETED_EVENT,
} from '@/lib/constants/analyticsEvents'
import { track } from '@/lib/metrics/track'
import type { HarnessAgent, HarnessAgentAdapter } from './agent-harness-types'
import type {
AgentListItem,
CreateAgentRuntime,
ProviderOption,
} from './agents-page-types'
import { findOpenClawCliProviderById } from './openclaw-cli-providers'
import type {
AgentEntry,
OpenClawAgentMutationInput,
OpenClawSetupInput,
} from './useOpenClaw'
export interface AgentPageActionInput {
createProviderId: string
createRuntime: CreateAgentRuntime
harnessModelId: string
harnessReasoningEffort: string
navigate: NavigateFunction
newName: string
selectableOpenClawProviders: ProviderOption[]
setupProviderId: string
createHarnessAgent: (input: {
name: string
adapter: HarnessAgentAdapter
modelId?: string
reasoningEffort?: string
}) => Promise<HarnessAgent>
createOpenClawAgent: (
input: OpenClawAgentMutationInput,
) => Promise<{ agent: AgentEntry }>
deleteHarnessAgent: (agentId: string) => Promise<unknown>
deleteOpenClawAgent: (agentId: string) => Promise<unknown>
setCliAuthModalOpen: (open: boolean) => void
setCreateError: (error: string | null) => void
setCreateOpen: (open: boolean) => void
setDeletingAgentKey: (key: string | null) => void
setNewName: (name: string) => void
setPageError: (error: string | null) => void
setSetupOpen: (open: boolean) => void
setupOpenClaw: (input: OpenClawSetupInput) => Promise<unknown>
}
export function createAgentPageActions(input: AgentPageActionInput) {
const runWithPageErrorHandling = async (fn: () => Promise<unknown>) => {
input.setPageError(null)
try {
await fn()
} catch (err) {
input.setPageError(err instanceof Error ? err.message : String(err))
}
}
const handleSetup = async () => {
const option = input.selectableOpenClawProviders.find(
(item) => item.id === input.setupProviderId,
)
const isCli = !!option && !!findOpenClawCliProviderById(option.type)
const llmOption = !isCli && option ? option : undefined
await runWithPageErrorHandling(async () => {
await input.setupOpenClaw({
providerType: option?.type,
providerName: isCli ? undefined : option?.name,
baseUrl: llmOption?.baseUrl,
apiKey: llmOption?.apiKey,
modelId: option?.modelId,
})
input.setSetupOpen(false)
if (isCli) input.setCliAuthModalOpen(true)
})
}
const handleOpenClawCreate = async () => {
if (!input.newName.trim()) return
const option = input.selectableOpenClawProviders.find(
(item) => item.id === input.createProviderId,
)
const normalizedName = input.newName
.trim()
.toLowerCase()
.replace(/\s+/g, '-')
const isCli = !!option && !!findOpenClawCliProviderById(option.type)
const llmOption = !isCli && option ? option : undefined
input.setCreateError(null)
try {
const result = await input.createOpenClawAgent({
name: normalizedName,
providerType: option?.type,
providerName: isCli ? undefined : option?.name,
baseUrl: llmOption?.baseUrl,
apiKey: llmOption?.apiKey,
modelId: option?.modelId,
})
input.setCreateOpen(false)
input.setNewName('')
track(AGENT_CREATED_EVENT, {
runtime: 'openclaw',
provider_type: option?.type,
})
input.navigate(`/agents/${result.agent.agentId}`)
} catch (err) {
input.setCreateError(err instanceof Error ? err.message : String(err))
}
}
const handleHarnessCreate = async () => {
if (!input.newName.trim()) return
input.setCreateError(null)
try {
const agent = await input.createHarnessAgent({
name: input.newName.trim(),
adapter: input.createRuntime as HarnessAgentAdapter,
modelId: input.harnessModelId || undefined,
reasoningEffort: input.harnessReasoningEffort || undefined,
})
input.setCreateOpen(false)
input.setNewName('')
track(AGENT_CREATED_EVENT, {
runtime: input.createRuntime,
model_id: input.harnessModelId || undefined,
reasoning_effort: input.harnessReasoningEffort || undefined,
})
input.navigate(`/agents/${agent.id}`)
} catch (err) {
input.setCreateError(err instanceof Error ? err.message : String(err))
}
}
const handleCreate = () => {
const createByRuntime: Record<CreateAgentRuntime, () => Promise<void>> = {
openclaw: handleOpenClawCreate,
claude: handleHarnessCreate,
codex: handleHarnessCreate,
}
void createByRuntime[input.createRuntime]()
}
const handleDelete = async (agent: AgentListItem) => {
input.setDeletingAgentKey(agent.key)
await runWithPageErrorHandling(async () => {
const deleteBySource: Record<
AgentListItem['source'],
(agentId: string) => Promise<unknown>
> = {
openclaw: (agentId) => input.deleteOpenClawAgent(agentId),
'agent-harness': (agentId) => input.deleteHarnessAgent(agentId),
}
await deleteBySource[agent.source](agent.agentId)
track(AGENT_DELETED_EVENT, {
runtime: agent.source,
agent_id: agent.agentId,
})
})
input.setDeletingAgentKey(null)
}
return {
handleCreate,
handleDelete,
handleSetup,
runWithPageErrorHandling,
}
}

View File

@@ -0,0 +1,173 @@
import { type Dispatch, type SetStateAction, useEffect, useMemo } from 'react'
import type { LlmProviderConfig } from '@/lib/llm-providers/types'
import type {
HarnessAdapterDescriptor,
HarnessAgentAdapter,
} from './agent-harness-types'
import type { CreateAgentRuntime } from './agents-page-types'
import { toProviderOptions } from './agents-page-utils'
import {
buildOpenClawCliProviderOptions,
findOpenClawCliProviderById,
useOpenClawCliProviderAuthStatus,
} from './openclaw-cli-providers'
export function useDefaultAgentName(
createOpen: boolean,
setNewName: Dispatch<SetStateAction<string>>,
): void {
useEffect(() => {
if (!createOpen) return
setNewName((current) => current || 'agent')
}, [createOpen, setNewName])
}
export function useHarnessAgentDefaults(input: {
adapters: HarnessAdapterDescriptor[]
createOpen: boolean
harnessAdapterId: HarnessAgentAdapter
setHarnessAdapterId: Dispatch<SetStateAction<HarnessAgentAdapter>>
setHarnessModelId: Dispatch<SetStateAction<string>>
setHarnessReasoningEffort: Dispatch<SetStateAction<string>>
}): void {
const {
adapters,
createOpen,
harnessAdapterId,
setHarnessAdapterId,
setHarnessModelId,
setHarnessReasoningEffort,
} = input
useEffect(() => {
if (!createOpen) return
const adapter =
adapters.find((entry) => entry.id === harnessAdapterId) ?? adapters[0]
if (!adapter) return
setHarnessAdapterId(adapter.id)
setHarnessModelId((current) => current || adapter.defaultModelId)
setHarnessReasoningEffort(
(current) => current || adapter.defaultReasoningEffort,
)
}, [
adapters,
createOpen,
harnessAdapterId,
setHarnessAdapterId,
setHarnessModelId,
setHarnessReasoningEffort,
])
}
export function useOpenClawProviderSelection(input: {
providers: LlmProviderConfig[]
defaultProviderId: string
createOpen: boolean
createRuntime: CreateAgentRuntime
createProviderId: string
setCreateProviderId: Dispatch<SetStateAction<string>>
setupOpen: boolean
setupProviderId: string
setSetupProviderId: Dispatch<SetStateAction<string>>
cliAuthModalOpen: boolean
setCliAuthModalOpen: Dispatch<SetStateAction<boolean>>
}) {
const {
providers,
defaultProviderId,
createOpen,
createRuntime,
createProviderId,
setCreateProviderId,
setupOpen,
setupProviderId,
setSetupProviderId,
cliAuthModalOpen,
setCliAuthModalOpen,
} = input
const cliProviderOptions = useMemo(
() => buildOpenClawCliProviderOptions(),
[],
)
const selectableOpenClawProviders = useMemo(
() => toProviderOptions(providers, cliProviderOptions),
[providers, cliProviderOptions],
)
useEffect(() => {
if (selectableOpenClawProviders.length === 0) return
const fallbackId =
selectableOpenClawProviders.find(
(provider) => provider.id === defaultProviderId,
)?.id ?? selectableOpenClawProviders[0].id
if (createOpen && !createProviderId) {
setCreateProviderId(fallbackId)
}
}, [
createOpen,
createProviderId,
defaultProviderId,
selectableOpenClawProviders,
setCreateProviderId,
])
useEffect(() => {
if (selectableOpenClawProviders.length === 0) return
const fallbackId =
selectableOpenClawProviders.find(
(provider) => provider.id === defaultProviderId,
)?.id ?? selectableOpenClawProviders[0].id
if (setupOpen && !setupProviderId) {
setSetupProviderId(fallbackId)
}
}, [
defaultProviderId,
selectableOpenClawProviders,
setSetupProviderId,
setupOpen,
setupProviderId,
])
const selectedCreateOption = selectableOpenClawProviders.find(
(provider) => provider.id === createProviderId,
)
const selectedCliProvider = selectedCreateOption
? findOpenClawCliProviderById(selectedCreateOption.type)
: undefined
const selectedSetupOption = selectableOpenClawProviders.find(
(provider) => provider.id === setupProviderId,
)
const selectedSetupCliProvider = selectedSetupOption
? findOpenClawCliProviderById(selectedSetupOption.type)
: undefined
const activeCliProvider =
(setupOpen && selectedSetupCliProvider) ||
(createOpen && createRuntime === 'openclaw' && selectedCliProvider) ||
undefined
const {
data: cliAuthStatus,
isLoading: cliAuthLoading,
error: cliAuthError,
} = useOpenClawCliProviderAuthStatus(
activeCliProvider?.id ?? '',
!!activeCliProvider,
)
useEffect(() => {
if (cliAuthModalOpen && cliAuthStatus?.loggedIn) {
setCliAuthModalOpen(false)
}
}, [cliAuthModalOpen, cliAuthStatus?.loggedIn, setCliAuthModalOpen])
return {
selectableOpenClawProviders,
selectedCliProvider,
selectedSetupCliProvider,
authTerminalProvider: selectedSetupCliProvider ?? selectedCliProvider,
cliAuthStatus,
cliAuthLoading,
cliAuthError,
}
}

View File

@@ -0,0 +1,119 @@
import type { HarnessAgentAdapter } from './agent-harness-types'
import type { GatewayLifecycleAction, OpenClawStatus } from './useOpenClaw'
export type CreateAgentRuntime = 'openclaw' | HarnessAgentAdapter
export interface ProviderOption {
id: string
type: string
name: string
modelId: string
baseUrl?: string
apiKey?: string
}
export interface AgentListItem {
key: string
agentId: string
name: string
source: 'openclaw' | 'agent-harness'
runtimeLabel: string
modelLabel: string
detail: string
canChat: boolean
canDelete: boolean
}
export interface GatewayUiState {
canManageAgents: boolean
controlPlaneDegraded: boolean
controlPlaneBusy: boolean
}
export const DEFAULT_HARNESS_ADAPTER: HarnessAgentAdapter = 'claude'
export const DEFAULT_CREATE_RUNTIME: CreateAgentRuntime = 'openclaw'
export const LIFECYCLE_BANNER_COPY: Record<GatewayLifecycleAction, string> = {
setup: 'Setting up OpenClaw...',
start: 'Starting gateway...',
stop: 'Stopping gateway...',
restart: 'Restarting gateway...',
reconnect: 'Restoring gateway connection...',
}
export const CONTROL_PLANE_COPY: Record<
OpenClawStatus['controlPlaneStatus'],
{
badgeVariant: 'default' | 'secondary' | 'outline' | 'destructive'
badgeLabel: string
title: string
description: string
}
> = {
connected: {
badgeVariant: 'default',
badgeLabel: 'Control Plane Ready',
title: 'Gateway Connected',
description: 'OpenClaw can create, manage, and chat with agents normally.',
},
connecting: {
badgeVariant: 'secondary',
badgeLabel: 'Connecting',
title: 'Connecting to Gateway',
description:
'BrowserOS is establishing the OpenClaw control channel for agent operations.',
},
reconnecting: {
badgeVariant: 'secondary',
badgeLabel: 'Reconnecting',
title: 'Reconnecting Control Plane',
description:
'The gateway process is up, but BrowserOS is restoring the control channel.',
},
recovering: {
badgeVariant: 'secondary',
badgeLabel: 'Recovering',
title: 'Recovering Gateway Connection',
description:
'BrowserOS detected a control-plane fault and is trying a safe recovery path.',
},
disconnected: {
badgeVariant: 'outline',
badgeLabel: 'Disconnected',
title: 'Gateway Disconnected',
description: 'The gateway process is not available to BrowserOS right now.',
},
failed: {
badgeVariant: 'destructive',
badgeLabel: 'Needs Attention',
title: 'Gateway Recovery Failed',
description:
'BrowserOS could not restore the OpenClaw control channel automatically.',
},
}
export const FALLBACK_CONTROL_PLANE_COPY = {
badgeVariant: 'outline' as const,
badgeLabel: 'Unknown',
title: 'Gateway State Unknown',
description:
'BrowserOS received a gateway status it does not recognize yet. Refreshing or reconnecting should restore a known state.',
}
export const RECOVERY_REASON_COPY: Record<
NonNullable<OpenClawStatus['lastRecoveryReason']>,
string
> = {
transient_disconnect:
'The control channel dropped briefly and BrowserOS is retrying it.',
signature_expired:
'The gateway rejected the signed device handshake because its clock drifted.',
pairing_required:
'The gateway asked BrowserOS to approve its local device identity again.',
token_mismatch:
'BrowserOS had to reload the gateway token before reconnecting.',
container_not_ready:
'The OpenClaw gateway process is not ready yet, so control-plane recovery cannot start.',
unknown:
'BrowserOS hit an unexpected gateway error and could not classify it cleanly.',
}

View File

@@ -0,0 +1,167 @@
import type { LlmProviderConfig } from '@/lib/llm-providers/types'
import type { HarnessAgent, HarnessAgentAdapter } from './agent-harness-types'
import {
type AgentListItem,
CONTROL_PLANE_COPY,
FALLBACK_CONTROL_PLANE_COPY,
type GatewayUiState,
LIFECYCLE_BANNER_COPY,
type ProviderOption,
RECOVERY_REASON_COPY,
} from './agents-page-types'
import { getOpenClawSupportedProviders } from './openclaw-supported-providers'
import {
type AgentEntry,
type GatewayLifecycleAction,
getModelDisplayName,
type OpenClawStatus,
} from './useOpenClaw'
export function getControlPlaneCopy(
status: OpenClawStatus['controlPlaneStatus'],
) {
return CONTROL_PLANE_COPY[status] ?? FALLBACK_CONTROL_PLANE_COPY
}
export function getRecoveryDetail(status: OpenClawStatus): string | null {
if (!status.lastRecoveryReason && !status.lastGatewayError) return null
const detail = status.lastRecoveryReason
? RECOVERY_REASON_COPY[status.lastRecoveryReason]
: null
if (status.lastGatewayError && detail) {
return `${detail} Latest gateway error: ${status.lastGatewayError}`
}
return status.lastGatewayError ?? detail
}
export function formatHarnessAdapter(adapter: HarnessAgentAdapter): string {
return adapter === 'claude' ? 'Claude Code' : 'Codex'
}
export function toProviderOptions(
providers: LlmProviderConfig[],
cliProviders: ProviderOption[],
): ProviderOption[] {
return [...getOpenClawSupportedProviders(providers), ...cliProviders]
}
export function toOpenClawListItem(
agent: AgentEntry,
canManageAgents: boolean,
): AgentListItem {
return {
key: `openclaw:${agent.agentId}`,
agentId: agent.agentId,
name: agent.name,
source: 'openclaw',
runtimeLabel: 'OpenClaw',
modelLabel: getModelDisplayName(agent.model) ?? 'default',
detail: agent.workspace,
canChat: canManageAgents,
canDelete: canManageAgents && agent.agentId !== 'main',
}
}
export function toHarnessListItem(agent: HarnessAgent): AgentListItem {
return {
key: `agent-harness:${agent.id}`,
agentId: agent.id,
name: agent.name,
source: 'agent-harness',
runtimeLabel: formatHarnessAdapter(agent.adapter),
modelLabel: agent.modelId ?? 'default',
detail: `${agent.adapter}:main`,
canChat: true,
canDelete: true,
}
}
export function getGatewayUiState(
status: OpenClawStatus | null,
): GatewayUiState {
if (!status) {
return {
canManageAgents: false,
controlPlaneDegraded: false,
controlPlaneBusy: false,
}
}
const controlPlaneBusy =
status.controlPlaneStatus === 'connecting' ||
status.controlPlaneStatus === 'reconnecting' ||
status.controlPlaneStatus === 'recovering'
return {
canManageAgents:
status.status === 'running' && status.controlPlaneStatus === 'connected',
controlPlaneBusy,
controlPlaneDegraded:
status.status === 'running' && status.controlPlaneStatus !== 'connected',
}
}
export function getLifecycleBanner(
action: GatewayLifecycleAction | null,
): string | null {
return action ? LIFECYCLE_BANNER_COPY[action] : null
}
export function canManageOpenClawAgents(
state: GatewayUiState,
lifecyclePending: boolean,
): boolean {
return state.canManageAgents && !lifecyclePending
}
export function shouldShowControlPlaneDegraded(
state: GatewayUiState,
lifecyclePending: boolean,
): boolean {
return state.controlPlaneDegraded && !lifecyclePending
}
export function getControlPlaneCopyForStatus(status: OpenClawStatus | null) {
return status
? getControlPlaneCopy(status.controlPlaneStatus)
: FALLBACK_CONTROL_PLANE_COPY
}
export function getVisibleOpenClawAgents(
enabled: boolean,
agents: AgentEntry[],
): AgentEntry[] {
return enabled ? agents : []
}
export function getAgentsLoading(input: {
adaptersLoading: boolean
harnessAgentsLoading: boolean
openClawAgentsLoading: boolean
}): boolean {
return (
input.adaptersLoading ||
input.harnessAgentsLoading ||
input.openClawAgentsLoading
)
}
export function getInlineError(input: {
lifecyclePending: boolean
pageError: string | null
openClawAgentsError: Error | null
adaptersError: Error | null
harnessAgentsError: Error | null
}): string | null {
if (input.lifecyclePending) return null
return (
input.pageError ??
input.openClawAgentsError?.message ??
input.adaptersError?.message ??
input.harnessAgentsError?.message ??
null
)
}

View File

@@ -0,0 +1,185 @@
import { useQuery } from '@tanstack/react-query'
import { CheckCircle2, Loader2, Terminal, TriangleAlert } from 'lucide-react'
import type { FC } from 'react'
import { Button } from '@/components/ui/button'
import { useAgentServerUrl } from '@/lib/browseros/useBrowserOSProviders'
export interface OpenClawCliProvider {
id: string
displayName: string
description: string
models: readonly string[]
authLoginCommand: string
}
export interface OpenClawCliProviderAuthStatus {
installed: boolean
loggedIn: boolean
accountLabel?: string
subscriptionLabel?: string
error?: string
}
export interface OpenClawCliProviderOption {
id: string
type: string
name: string
modelId: string
}
const CLAUDE_CLI_PROVIDER: OpenClawCliProvider = {
id: 'claude-cli',
displayName: 'Anthropic Claude CLI',
description: 'Uses your Claude.ai subscription via the Claude Code CLI',
models: ['claude-sonnet-4-6', 'claude-opus-4-6', 'claude-haiku-4-5'],
authLoginCommand: 'claude /login',
}
export const OPENCLAW_CLI_PROVIDERS: readonly OpenClawCliProvider[] = [
CLAUDE_CLI_PROVIDER,
]
export function findOpenClawCliProviderById(
id: string,
): OpenClawCliProvider | undefined {
return OPENCLAW_CLI_PROVIDERS.find((provider) => provider.id === id)
}
export function buildOpenClawCliProviderOptions(): OpenClawCliProviderOption[] {
return OPENCLAW_CLI_PROVIDERS.flatMap((provider) =>
provider.models.map((modelId) => ({
id: `${provider.id}/${modelId}`,
type: provider.id,
name: provider.displayName,
modelId,
})),
)
}
async function fetchCliProviderAuthStatus(
baseUrl: string,
providerId: string,
): Promise<OpenClawCliProviderAuthStatus> {
const res = await fetch(`${baseUrl}/claw/providers/${providerId}/auth-status`)
if (!res.ok) {
let message = `Auth status request failed (${res.status})`
try {
const body = (await res.json()) as { error?: string }
if (body.error) message = body.error
} catch {}
throw new Error(message)
}
return res.json() as Promise<OpenClawCliProviderAuthStatus>
}
export function useOpenClawCliProviderAuthStatus(
providerId: string,
enabled: boolean,
) {
const { baseUrl, isLoading: urlLoading } = useAgentServerUrl()
return useQuery<OpenClawCliProviderAuthStatus, Error>({
queryKey: ['openclaw-cli-auth', baseUrl, providerId],
queryFn: () => fetchCliProviderAuthStatus(baseUrl as string, providerId),
enabled: !!baseUrl && !urlLoading && enabled,
refetchInterval: enabled ? 2000 : false,
})
}
interface OpenClawCliProviderStatusPanelProps {
provider: OpenClawCliProvider
status: OpenClawCliProviderAuthStatus | undefined
loading: boolean
fetchError: Error | null
onConnect: () => void
}
export const OpenClawCliProviderStatusPanel: FC<
OpenClawCliProviderStatusPanelProps
> = ({ provider, status, loading, fetchError, onConnect }) => {
// Initial fetch (no data yet).
if (loading && !status) {
return (
<div className="flex items-center gap-2 rounded-md border border-border bg-muted/30 px-3 py-2 text-sm">
<Loader2 className="size-4 animate-spin text-muted-foreground" />
<span className="text-muted-foreground">
Checking {provider.displayName} status
</span>
</div>
)
}
if (fetchError) {
return (
<div className="flex items-start gap-2 rounded-md border border-destructive/30 bg-destructive/5 px-3 py-2 text-sm">
<TriangleAlert className="mt-0.5 size-4 text-destructive" />
<div>
<div className="font-medium text-destructive">
Could not read {provider.displayName} status
</div>
<div className="text-muted-foreground text-xs">
{fetchError.message}
</div>
</div>
</div>
)
}
if (!status) return null
// Install failed or binary missing.
if (!status.installed) {
return (
<div className="flex items-start gap-2 rounded-md border border-amber-500/40 bg-amber-500/5 px-3 py-2 text-sm">
<TriangleAlert className="mt-0.5 size-4 text-amber-600" />
<div>
<div className="font-medium">
{provider.displayName} not installed
</div>
<div className="text-muted-foreground text-xs">
The gateway will try to install it on the next restart. If this
persists, check your network and the gateway logs.
</div>
</div>
</div>
)
}
// Happy path.
if (status.loggedIn) {
const identityBits = [
status.accountLabel,
status.subscriptionLabel ? `(${status.subscriptionLabel})` : null,
].filter(Boolean)
const identity = identityBits.length > 0 ? identityBits.join(' ') : 'Ready'
return (
<div className="flex items-center gap-2 rounded-md border border-emerald-500/40 bg-emerald-500/5 px-3 py-2 text-sm">
<CheckCircle2 className="size-4 text-emerald-600" />
<div className="min-w-0 flex-1">
<div className="font-medium">Connected to {provider.displayName}</div>
<div className="truncate text-muted-foreground text-xs">
{identity}
</div>
</div>
</div>
)
}
// Installed but not logged in.
return (
<div className="flex flex-col gap-2 rounded-md border border-border bg-muted/30 px-3 py-3 text-sm">
<div>
<div className="font-medium">{provider.displayName} not set up</div>
<div className="text-muted-foreground text-xs">
{provider.description}
</div>
{status.error && (
<div className="mt-1 text-destructive text-xs">{status.error}</div>
)}
</div>
<Button size="sm" variant="outline" onClick={onConnect} className="w-fit">
<Terminal className="mr-1 size-4" />
Connect {provider.displayName}
</Button>
</div>
)
}

View File

@@ -0,0 +1,24 @@
import type { LlmProviderConfig, ProviderType } from '@/lib/llm-providers/types'
const OPENCLAW_SUPPORTED_PROVIDER_TYPES: ProviderType[] = [
'openrouter',
'openai',
'openai-compatible',
'anthropic',
'moonshot',
]
export function isOpenClawSupportedProviderType(
providerType: ProviderType,
): boolean {
return OPENCLAW_SUPPORTED_PROVIDER_TYPES.includes(providerType)
}
export function getOpenClawSupportedProviders(
providers: LlmProviderConfig[],
): LlmProviderConfig[] {
return providers.filter(
(provider) =>
!!provider.apiKey && isOpenClawSupportedProviderType(provider.type),
)
}

View File

@@ -0,0 +1,38 @@
import { describe, expect, it } from 'bun:test'
import { buildAgentApiUrl } from './agent-api-url'
import { mapHarnessAgentToEntry } from './agent-harness-types'
describe('mapHarnessAgentToEntry', () => {
it('maps created harness agents into chat-compatible entries', () => {
expect(
mapHarnessAgentToEntry({
id: 'agent-1',
name: 'Review bot',
adapter: 'codex',
modelId: 'gpt-5.5',
reasoningEffort: 'medium',
permissionMode: 'approve-all',
sessionKey: 'agent:agent-1:main',
createdAt: 1000,
updatedAt: 1000,
}),
).toEqual({
agentId: 'agent-1',
name: 'Review bot',
workspace: 'codex:main',
model: 'gpt-5.5',
source: 'agent-harness',
})
})
})
describe('buildAgentApiUrl', () => {
it('does not add a trailing slash for the harness root route', () => {
expect(buildAgentApiUrl('http://127.0.0.1:9105', '/')).toBe(
'http://127.0.0.1:9105/agents',
)
expect(buildAgentApiUrl('http://127.0.0.1:9105', '/adapters')).toBe(
'http://127.0.0.1:9105/agents/adapters',
)
})
})

View File

@@ -0,0 +1,262 @@
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import { getAgentServerUrl } from '@/lib/browseros/helpers'
import { useAgentServerUrl } from '@/lib/browseros/useBrowserOSProviders'
import { buildAgentApiUrl } from './agent-api-url'
import {
type AgentHarnessStreamEvent,
type CreateHarnessAgentInput,
type HarnessAdapterDescriptor,
type HarnessAgent,
type HarnessAgentHistoryPage,
mapHarnessAgentToEntry,
} from './agent-harness-types'
import type { OpenClawStatus } from './useOpenClaw'
/**
* Combined response shape of `GET /agents`. The page polls this once
* and consumes both fields, replacing the dedicated `/claw/status`
* poll the previous design carried.
*/
interface HarnessAgentsResponse {
agents: HarnessAgent[]
gateway: OpenClawStatus | null
}
export type { AgentHarnessStreamEvent }
const AGENT_QUERY_KEYS = {
adapters: 'agent-harness-adapters',
agents: 'agent-harness-agents',
} as const
async function agentsFetch<T>(
baseUrl: string,
path: string,
init?: RequestInit,
): Promise<T> {
const res = await fetch(buildAgentApiUrl(baseUrl, path), init)
if (!res.ok) {
let message = `Request failed with status ${res.status}`
try {
const body = (await res.json()) as { error?: string }
if (body.error) message = body.error
} catch {}
throw new Error(message)
}
return res.json() as Promise<T>
}
export function useAgentAdapters(enabled = true) {
const {
baseUrl,
isLoading: urlLoading,
error: urlError,
} = useAgentServerUrl()
const query = useQuery<HarnessAdapterDescriptor[], Error>({
queryKey: [AGENT_QUERY_KEYS.adapters, baseUrl],
queryFn: async () => {
const data = await agentsFetch<{ adapters: HarnessAdapterDescriptor[] }>(
baseUrl as string,
'/adapters',
)
return data.adapters ?? []
},
enabled: Boolean(baseUrl) && !urlLoading && enabled,
})
return {
adapters: query.data ?? [],
loading: query.isLoading || urlLoading,
error: query.error ?? urlError,
refetch: query.refetch,
}
}
export function useHarnessAgents(enabled = true) {
const {
baseUrl,
isLoading: urlLoading,
error: urlError,
} = useAgentServerUrl()
const query = useQuery<HarnessAgentsResponse, Error>({
queryKey: [AGENT_QUERY_KEYS.agents, baseUrl],
queryFn: async () => {
const data = await agentsFetch<HarnessAgentsResponse>(
baseUrl as string,
'/',
)
return {
agents: data.agents ?? [],
gateway: data.gateway ?? null,
}
},
enabled: Boolean(baseUrl) && !urlLoading && enabled,
// Poll every 5s so the per-agent liveness state (working / idle /
// asleep / error) and last-used timestamps stay fresh without a
// websocket. `refetchIntervalInBackground: false` lets a hidden
// tab go quiet — react-query's default, made explicit.
refetchInterval: 5_000,
refetchIntervalInBackground: false,
})
return {
agents: (query.data?.agents ?? []).map(mapHarnessAgentToEntry),
harnessAgents: query.data?.agents ?? [],
gateway: query.data?.gateway ?? null,
loading: query.isLoading || urlLoading,
error: query.error ?? urlError,
refetch: query.refetch,
}
}
export function useCreateHarnessAgent() {
const { baseUrl, isLoading: urlLoading } = useAgentServerUrl()
const queryClient = useQueryClient()
return useMutation({
mutationFn: async (input: CreateHarnessAgentInput) => {
if (!baseUrl || urlLoading) {
throw new Error('BrowserOS agent server URL is not ready')
}
const data = await agentsFetch<{ agent: HarnessAgent }>(baseUrl, '/', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(input),
})
return data.agent
},
onSuccess: async () => {
await queryClient.invalidateQueries({
queryKey: [AGENT_QUERY_KEYS.agents],
})
},
})
}
export function useDeleteHarnessAgent() {
const { baseUrl, isLoading: urlLoading } = useAgentServerUrl()
const queryClient = useQueryClient()
return useMutation({
mutationFn: async (agentId: string) => {
if (!baseUrl || urlLoading) {
throw new Error('BrowserOS agent server URL is not ready')
}
return agentsFetch<{ success: boolean }>(
baseUrl,
`/${encodeURIComponent(agentId)}`,
{ method: 'DELETE' },
)
},
onSuccess: async () => {
await queryClient.invalidateQueries({
queryKey: [AGENT_QUERY_KEYS.agents],
})
},
})
}
export async function chatWithHarnessAgent(
agentId: string,
message: string,
signal?: AbortSignal,
attachments?: ReadonlyArray<unknown>,
): Promise<Response> {
const baseUrl = await getAgentServerUrl()
return fetch(`${baseUrl}/agents/${encodeURIComponent(agentId)}/chat`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
message,
...(attachments && attachments.length > 0 ? { attachments } : {}),
}),
signal,
})
}
/**
* Subscribe to an existing turn (the server's `ActiveTurnRegistry`
* decoupled the turn lifecycle from POST /chat). `lastSeq` lets the
* client resume after a disconnect — the server replays buffered
* frames with seq > lastSeq, then tails new ones.
*/
export async function attachToHarnessTurn(
agentId: string,
options: { turnId?: string; lastSeq?: number; signal?: AbortSignal } = {},
): Promise<Response> {
const baseUrl = await getAgentServerUrl()
const url = new URL(
`${baseUrl}/agents/${encodeURIComponent(agentId)}/chat/stream`,
)
if (options.turnId) url.searchParams.set('turnId', options.turnId)
const headers: Record<string, string> = {}
if (typeof options.lastSeq === 'number') {
headers['Last-Event-ID'] = String(options.lastSeq)
}
return fetch(url.toString(), { signal: options.signal, headers })
}
export interface HarnessActiveTurnInfo {
turnId: string
agentId: string
sessionId: 'main'
status: 'running' | 'done' | 'error' | 'cancelled'
lastSeq: number
startedAt: number
endedAt?: number
}
/**
* Discover an in-flight turn for an agent. Used on chat mount so the
* UI reattaches instead of starting a new turn after a tab/refresh.
*/
export async function fetchActiveHarnessTurn(
agentId: string,
): Promise<HarnessActiveTurnInfo | null> {
const baseUrl = await getAgentServerUrl()
const response = await fetch(
`${baseUrl}/agents/${encodeURIComponent(agentId)}/chat/active`,
)
if (!response.ok) return null
const body = (await response.json()) as {
active: HarnessActiveTurnInfo | null
}
return body.active
}
/**
* Stop button. Hits the explicit cancel endpoint instead of just
* aborting the fetch (which now only detaches *this* subscriber from
* the buffer; the underlying turn would otherwise keep running).
*/
export async function cancelHarnessTurn(
agentId: string,
options: { turnId?: string; reason?: string } = {},
): Promise<{ cancelled: boolean }> {
const baseUrl = await getAgentServerUrl()
const response = await fetch(
`${baseUrl}/agents/${encodeURIComponent(agentId)}/chat/cancel`,
{
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
...(options.turnId ? { turnId: options.turnId } : {}),
...(options.reason ? { reason: options.reason } : {}),
}),
},
)
if (!response.ok) return { cancelled: false }
return (await response.json()) as { cancelled: boolean }
}
export async function fetchHarnessAgentHistory(
agentId: string,
): Promise<HarnessAgentHistoryPage> {
const baseUrl = await getAgentServerUrl()
return agentsFetch<HarnessAgentHistoryPage>(
baseUrl,
`/${encodeURIComponent(agentId)}/sessions/main/history`,
)
}

View File

@@ -1,9 +1,4 @@
import type {
BrowserOSAgentRoleId,
BrowserOSCustomRoleInput,
} from '@browseros/shared/types/role-aware-agents'
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import { getAgentServerUrl } from '@/lib/browseros/helpers'
import { useAgentServerUrl } from '@/lib/browseros/useBrowserOSProviders'
export interface AgentEntry {
@@ -11,27 +6,7 @@ export interface AgentEntry {
name: string
workspace: string
model?: unknown
role?: {
roleSource: 'builtin' | 'custom'
roleId?: BrowserOSAgentRoleId
roleName: string
shortDescription: string
}
}
export interface RoleTemplateSummary {
id: BrowserOSAgentRoleId
name: string
shortDescription: string
longDescription: string
recommendedApps: string[]
defaultAgentName: string
boundaries: Array<{
key: string
label: string
description: string
defaultMode: 'allow' | 'ask' | 'block'
}>
source?: 'openclaw' | 'agent-harness'
}
export interface OpenClawStatus {
@@ -61,13 +36,12 @@ export interface OpenClawStatus {
export interface OpenClawAgentMutationInput {
name: string
roleId?: BrowserOSAgentRoleId
customRole?: BrowserOSCustomRoleInput
providerType?: string
providerName?: string
baseUrl?: string
apiKey?: string
modelId?: string
supportsImages?: boolean
}
export interface OpenClawSetupInput {
@@ -76,6 +50,10 @@ export interface OpenClawSetupInput {
baseUrl?: string
apiKey?: string
modelId?: string
// Mirrors LlmProviderConfig.supportsImages — pass-through so the gateway
// can declare the model's input modalities correctly when persisting the
// custom-provider config.
supportsImages?: boolean
}
export function getModelDisplayName(model: unknown): string | undefined {
@@ -86,9 +64,15 @@ export function getModelDisplayName(model: unknown): string | undefined {
export const OPENCLAW_QUERY_KEYS = {
status: 'openclaw-status',
agents: 'openclaw-agents',
roles: 'openclaw-roles',
} as const
export type GatewayLifecycleAction =
| 'setup'
| 'start'
| 'stop'
| 'restart'
| 'reconnect'
async function clawFetch<T>(
baseUrl: string,
path: string,
@@ -114,17 +98,10 @@ async function fetchOpenClawStatus(baseUrl: string): Promise<OpenClawStatus> {
async function fetchOpenClawAgents(baseUrl: string): Promise<AgentEntry[]> {
const data = await clawFetch<{ agents: AgentEntry[] }>(baseUrl, '/agents')
return data.agents ?? []
}
async function fetchOpenClawRoles(
baseUrl: string,
): Promise<RoleTemplateSummary[]> {
const data = await clawFetch<{ roles: RoleTemplateSummary[] }>(
baseUrl,
'/roles',
)
return data.roles ?? []
return (data.agents ?? []).map((agent) => ({
...agent,
source: 'openclaw',
}))
}
async function invalidateOpenClawQueries(
@@ -179,28 +156,6 @@ export function useOpenClawAgents(enabled = true) {
}
}
export function useOpenClawRoles() {
const {
baseUrl,
isLoading: urlLoading,
error: urlError,
} = useAgentServerUrl()
const query = useQuery<RoleTemplateSummary[], Error>({
queryKey: [OPENCLAW_QUERY_KEYS.roles, baseUrl],
queryFn: () => fetchOpenClawRoles(baseUrl as string),
enabled: !!baseUrl && !urlLoading,
staleTime: 60_000,
})
return {
roles: query.data ?? [],
loading: query.isLoading || urlLoading,
error: query.error ?? urlError,
refetch: query.refetch,
}
}
export function useOpenClawMutations() {
const { baseUrl, isLoading: urlLoading } = useAgentServerUrl()
const queryClient = useQueryClient()
@@ -278,6 +233,13 @@ export function useOpenClawMutations() {
onSuccess,
})
let pendingGatewayAction: GatewayLifecycleAction | null = null
if (setupMutation.isPending) pendingGatewayAction = 'setup'
else if (restartMutation.isPending) pendingGatewayAction = 'restart'
else if (stopMutation.isPending) pendingGatewayAction = 'stop'
else if (startMutation.isPending) pendingGatewayAction = 'start'
else if (reconnectMutation.isPending) pendingGatewayAction = 'reconnect'
return {
setupOpenClaw: setupMutation.mutateAsync,
createAgent: createMutation.mutateAsync,
@@ -298,6 +260,7 @@ export function useOpenClawMutations() {
creating: createMutation.isPending,
deleting: deleteMutation.isPending,
reconnecting: reconnectMutation.isPending,
pendingGatewayAction,
}
}
@@ -314,17 +277,44 @@ export interface OpenClawStreamEvent {
data: Record<string, unknown>
}
export async function chatWithAgent(
agentId: string,
message: string,
sessionKey?: string,
signal?: AbortSignal,
): Promise<Response> {
const baseUrl = await getAgentServerUrl()
return fetch(`${baseUrl}/claw/agents/${agentId}/chat`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ message, sessionKey }),
signal,
})
export interface OpenClawChatHistoryMessage {
role: 'user' | 'assistant'
content: string
}
interface ChatHistoryTurnLike {
userText: string
parts: Array<{ kind: string; text?: string }>
}
export function buildChatHistoryFromTurns(
turns: ChatHistoryTurnLike[],
): OpenClawChatHistoryMessage[] {
const messages: OpenClawChatHistoryMessage[] = []
for (const turn of turns) {
const userText = turn.userText.trim()
if (userText) {
messages.push({ role: 'user', content: userText })
}
const assistantText = turn.parts
.filter(
(
part,
): part is {
kind: 'text'
text: string
} => part.kind === 'text' && typeof part.text === 'string',
)
.map((part) => part.text.trim())
.filter(Boolean)
.join('\n\n')
if (assistantText) {
messages.push({ role: 'assistant', content: assistantText })
}
}
return messages
}

View File

@@ -164,9 +164,17 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
const resolvedProvider: Provider | null = (() => {
const id = selectedProviderId ?? defaultProviderId
const found = providers.find((p) => p.id === id)
if (found) return { id: found.id, name: found.name, type: found.type }
if (found) {
return {
kind: 'llm' as const,
id: found.id,
name: found.name,
type: found.type,
}
}
if (providers[0])
return {
kind: 'llm' as const,
id: providers[0].id,
name: providers[0].name,
type: providers[0].type,
@@ -175,6 +183,7 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
})()
const providerOptions: Provider[] = providers.map((p) => ({
kind: 'llm',
id: p.id,
name: p.name,
type: p.type,

View File

@@ -18,8 +18,8 @@ describe('route-utils', () => {
expect(shouldUseChatSession('/home/chat')).toBe(true)
})
it('keeps the focus grid on home while hiding it on dedicated full-screen routes', () => {
expect(shouldHideFocusGrid('/home')).toBe(false)
it('hides the focus grid on full-screen routes', () => {
expect(shouldHideFocusGrid('/home')).toBe(true)
expect(shouldHideFocusGrid('/home/agents/main')).toBe(true)
expect(shouldHideFocusGrid('/home/chat')).toBe(true)
expect(shouldHideFocusGrid('/home/skills')).toBe(true)

View File

@@ -1,4 +1,5 @@
const HIDE_FOCUS_GRID_PATHS = new Set([
'/home',
'/home/soul',
'/home/memory',
'/home/skills',

View File

@@ -1,4 +1,4 @@
import { Github, History, Plus, SettingsIcon } from 'lucide-react'
import { Bot, Github, History, Plus, SettingsIcon } from 'lucide-react'
import type { FC } from 'react'
import { Link, useLocation, useNavigate } from 'react-router'
import { ChatProviderSelector } from '@/components/chat/ChatProviderSelector'
@@ -64,7 +64,9 @@ export const ChatHeader: FC<ChatHeaderProps> = ({
className="group relative inline-flex cursor-pointer items-center gap-2 rounded-lg p-2 text-muted-foreground transition-colors hover:bg-muted/50 hover:text-foreground data-[state=open]:bg-accent"
title="Change AI Provider"
>
{selectedProvider.type === 'browseros' ? (
{selectedProvider.kind === 'acp' ? (
<Bot className="h-[18px] w-[18px]" />
) : selectedProvider.type === 'browseros' ? (
<BrowserOSIcon size={18} />
) : (
<ProviderIcon

View File

@@ -0,0 +1,258 @@
import { describe, expect, it } from 'bun:test'
import type {
HarnessAdapterDescriptor,
HarnessAgent,
} from '@/entrypoints/app/agents/agent-harness-types'
import type { LlmProviderConfig } from '@/lib/llm-providers/types'
import {
buildSidepanelChatTargets,
persistSidepanelChatTargetSelection,
resolveSidepanelChatTarget,
type SidepanelChatTargetSelection,
toLlmProviderConfig,
} from './sidepanel-chat-targets'
const timestamp = 1000
const providers: LlmProviderConfig[] = [
{
id: 'browseros',
type: 'browseros',
name: 'BrowserOS',
baseUrl: 'https://api.browseros.com/v1',
modelId: 'browseros-auto',
supportsImages: true,
contextWindow: 200000,
temperature: 0.2,
createdAt: timestamp,
updatedAt: timestamp,
},
{
id: 'anthropic-sonnet',
type: 'anthropic',
name: 'Anthropic Sonnet',
modelId: 'claude-sonnet-4-6',
apiKey: 'sk-ant',
supportsImages: true,
contextWindow: 200000,
temperature: 0.2,
createdAt: timestamp,
updatedAt: timestamp,
},
]
const adapters: HarnessAdapterDescriptor[] = [
{
id: 'claude',
name: 'Claude Code',
defaultModelId: 'haiku',
defaultReasoningEffort: 'medium',
modelControl: 'best-effort',
models: [
{ id: 'sonnet', label: 'Sonnet' },
{ id: 'haiku', label: 'Haiku', recommended: true },
],
reasoningEfforts: [
{ id: 'medium', label: 'Medium', recommended: true },
{ id: 'high', label: 'High' },
],
},
{
id: 'codex',
name: 'Codex',
defaultModelId: 'gpt-5.5',
defaultReasoningEffort: 'medium',
modelControl: 'runtime-supported',
models: [{ id: 'gpt-5.5', label: 'GPT-5.5', recommended: true }],
reasoningEfforts: [{ id: 'medium', label: 'Medium', recommended: true }],
},
{
id: 'openclaw',
name: 'OpenClaw',
defaultModelId: 'default',
defaultReasoningEffort: 'medium',
modelControl: 'best-effort',
models: [],
reasoningEfforts: [
{ id: 'medium', label: 'Medium', recommended: true },
{ id: 'high', label: 'High' },
],
},
]
const agents: HarnessAgent[] = [
{
id: 'agent-codex',
name: 'Review Bot',
adapter: 'codex',
modelId: 'gpt-5.5',
reasoningEffort: 'medium',
permissionMode: 'approve-all',
sessionKey: 'agent:agent-codex:main',
createdAt: timestamp,
updatedAt: timestamp,
},
{
id: 'agent-openclaw',
name: 'Research Claw',
adapter: 'openclaw',
modelId: 'default',
reasoningEffort: 'high',
permissionMode: 'approve-all',
sessionKey: 'agent:agent-openclaw:main',
createdAt: timestamp,
updatedAt: timestamp,
},
]
describe('buildSidepanelChatTargets', () => {
it('returns LLM targets plus one ACP target per persisted harness agent', () => {
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
expect(targets.map((target) => target.id)).toEqual([
'browseros',
'anthropic-sonnet',
'agent-codex',
'agent-openclaw',
])
})
it('does not emit catalog-only ACP targets without persisted agents', () => {
const targets = buildSidepanelChatTargets({
providers,
adapters,
agents: [],
})
expect(targets.map((target) => target.id)).toEqual([
'browseros',
'anthropic-sonnet',
])
})
it('uses the created OpenClaw agent name instead of a generic adapter target', () => {
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
const openclaw = targets.find((target) => target.id === 'agent-openclaw')
expect(openclaw).toMatchObject({
kind: 'acp',
id: 'agent-openclaw',
agentId: 'agent-openclaw',
adapter: 'openclaw',
adapterName: 'OpenClaw',
modelId: 'default',
modelLabel: 'default',
name: 'Research Claw',
modelControl: 'best-effort',
reasoningEffort: 'high',
})
})
it('preserves adapter metadata for created agent targets', () => {
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
const codex = targets.find((target) => target.id === 'agent-codex')
expect(codex).toMatchObject({
kind: 'acp',
agentId: 'agent-codex',
adapter: 'codex',
adapterName: 'Codex',
modelId: 'gpt-5.5',
modelLabel: 'GPT-5.5',
modelControl: 'runtime-supported',
recommended: true,
reasoningEffort: 'medium',
reasoningEffortLabel: 'Medium',
})
})
it('still returns LLM targets when agents and adapters are unavailable', () => {
expect(
buildSidepanelChatTargets({ providers, adapters: [], agents: [] }),
).toEqual([
{
kind: 'llm',
id: 'browseros',
name: 'BrowserOS',
type: 'browseros',
provider: providers[0],
},
{
kind: 'llm',
id: 'anthropic-sonnet',
name: 'Anthropic Sonnet',
type: 'anthropic',
provider: providers[1],
},
])
})
})
describe('resolveSidepanelChatTarget', () => {
it('resolves selected LLM targets back to their provider config', () => {
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
const resolved = resolveSidepanelChatTarget({
targets,
defaultProviderId: 'browseros',
selection: { kind: 'llm', id: 'anthropic-sonnet' },
})
expect(resolved?.kind).toBe('llm')
expect(toLlmProviderConfig(resolved)?.modelId).toBe('claude-sonnet-4-6')
})
it('falls back to the current default LLM provider when a persisted ACP target is stale', () => {
const targets = buildSidepanelChatTargets({
providers,
adapters,
agents: [],
})
expect(
resolveSidepanelChatTarget({
targets,
defaultProviderId: 'anthropic-sonnet',
selection: { kind: 'acp', id: 'agent-codex' },
}),
).toMatchObject({
kind: 'llm',
id: 'anthropic-sonnet',
})
})
it('falls back when an old catalog-style ACP target id is persisted', () => {
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
expect(
resolveSidepanelChatTarget({
targets,
defaultProviderId: 'anthropic-sonnet',
selection: { kind: 'acp', id: 'acp:codex:gpt-5.5:medium' },
}),
).toMatchObject({
kind: 'llm',
id: 'anthropic-sonnet',
})
})
})
describe('persistSidepanelChatTargetSelection', () => {
it('stores only target identity and does not mutate LLM provider arrays', async () => {
let savedSelection: SidepanelChatTargetSelection | null = null
const originalProviders = providers.map((provider) => ({ ...provider }))
const targets = buildSidepanelChatTargets({ providers, adapters, agents })
const target = targets.find((candidate) => candidate.id === 'agent-codex')
await persistSidepanelChatTargetSelection(target, {
setValue: async (value) => {
savedSelection = value
},
})
expect(savedSelection as SidepanelChatTargetSelection | null).toEqual({
kind: 'acp',
id: 'agent-codex',
})
expect(providers).toEqual(originalProviders)
})
})

View File

@@ -0,0 +1,178 @@
import type {
HarnessAdapterDescriptor,
HarnessAgent,
HarnessAgentAdapter,
} from '@/entrypoints/app/agents/agent-harness-types'
import type { LlmProviderConfig, ProviderType } from '@/lib/llm-providers/types'
export type SidepanelTargetKind = 'llm' | 'acp'
export type SidepanelChatTarget =
| {
kind: 'llm'
id: string
name: string
type: ProviderType
provider: LlmProviderConfig
}
| {
kind: 'acp'
id: string
name: string
type: 'acp'
agentId: string
adapter: HarnessAgentAdapter
adapterName: string
modelId: string
modelLabel: string
modelControl: HarnessAdapterDescriptor['modelControl']
recommended?: boolean
reasoningEffort: string
reasoningEffortLabel?: string
}
export type SidepanelChatTargetSelection = Pick<
SidepanelChatTarget,
'kind' | 'id'
>
interface BuildSidepanelChatTargetsInput {
providers: LlmProviderConfig[]
adapters: HarnessAdapterDescriptor[]
agents?: HarnessAgent[]
}
interface ResolveSidepanelChatTargetInput {
targets: SidepanelChatTarget[]
defaultProviderId: string
selection?: SidepanelChatTargetSelection | null
}
interface SidepanelChatTargetSelectionWriter {
setValue(value: SidepanelChatTargetSelection | null): Promise<void>
}
interface SidepanelChatTargetSelectionReader {
getValue(): Promise<SidepanelChatTargetSelection | null>
}
type SidepanelChatTargetSelectionStore = SidepanelChatTargetSelectionReader &
SidepanelChatTargetSelectionWriter
let sidepanelChatTargetSelectionStorage:
| SidepanelChatTargetSelectionStore
| undefined
export function buildSidepanelChatTargets({
providers,
adapters,
agents = [],
}: BuildSidepanelChatTargetsInput): SidepanelChatTarget[] {
return [
...providers.map(toLlmTarget),
...agents.map((agent) => toAcpTargetForAgent(agent, adapters)),
]
}
function toAcpTargetForAgent(
agent: HarnessAgent,
adapters: HarnessAdapterDescriptor[],
): SidepanelChatTarget {
const adapter = adapters.find((entry) => entry.id === agent.adapter)
const modelId = agent.modelId ?? adapter?.defaultModelId ?? 'default'
const reasoningEffort =
agent.reasoningEffort ?? adapter?.defaultReasoningEffort ?? 'medium'
const model = adapter?.models.find((entry) => entry.id === modelId)
const reasoning = adapter?.reasoningEfforts.find(
(effort) => effort.id === reasoningEffort,
)
return {
kind: 'acp',
id: agent.id,
name: agent.name,
type: 'acp',
agentId: agent.id,
adapter: agent.adapter,
adapterName: adapter?.name ?? formatAdapterName(agent.adapter),
modelId,
modelLabel: model?.label ?? modelId,
modelControl: adapter?.modelControl ?? 'best-effort',
recommended: model?.recommended,
reasoningEffort,
reasoningEffortLabel: reasoning?.label,
}
}
function formatAdapterName(adapter: HarnessAgentAdapter): string {
if (adapter === 'claude') return 'Claude Code'
if (adapter === 'codex') return 'Codex'
if (adapter === 'openclaw') return 'OpenClaw'
return adapter
}
export function resolveSidepanelChatTarget({
targets,
defaultProviderId,
selection,
}: ResolveSidepanelChatTargetInput): SidepanelChatTarget | undefined {
if (selection) {
const selected = targets.find(
(target) => target.kind === selection.kind && target.id === selection.id,
)
if (selected) return selected
}
return (
targets.find(
(target) => target.kind === 'llm' && target.id === defaultProviderId,
) ?? targets.find((target) => target.kind === 'llm')
)
}
export function toLlmProviderConfig(
target: SidepanelChatTarget | undefined,
): LlmProviderConfig | undefined {
return target?.kind === 'llm' ? target.provider : undefined
}
export async function persistSidepanelChatTargetSelection(
target: SidepanelChatTarget | undefined,
store?: SidepanelChatTargetSelectionWriter,
): Promise<void> {
const targetStore = store ?? (await getSidepanelChatTargetSelectionStorage())
await targetStore.setValue(
target ? { kind: target.kind, id: target.id } : null,
)
}
export async function loadSidepanelChatTargetSelection(
store?: SidepanelChatTargetSelectionReader,
): Promise<SidepanelChatTargetSelection | null> {
const targetStore = store ?? (await getSidepanelChatTargetSelectionStorage())
return targetStore.getValue()
}
function toLlmTarget(provider: LlmProviderConfig): SidepanelChatTarget {
return {
kind: 'llm',
id: provider.id,
name: provider.name,
type: provider.type,
provider,
}
}
async function getSidepanelChatTargetSelectionStorage(): Promise<SidepanelChatTargetSelectionStore> {
if (sidepanelChatTargetSelectionStorage) {
return sidepanelChatTargetSelectionStorage
}
const { storage } = await import('@wxt-dev/storage')
sidepanelChatTargetSelectionStorage =
storage.defineItem<SidepanelChatTargetSelection | null>(
'local:sidepanel-chat-target-selection',
{ fallback: null },
)
return sidepanelChatTargetSelectionStorage
}

View File

@@ -1,9 +1,21 @@
import { useEffect, useRef } from 'react'
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import useDeepCompareEffect from 'use-deep-compare-effect'
import {
useAgentAdapters,
useHarnessAgents,
} from '@/entrypoints/app/agents/useAgents'
import type { LlmProviderConfig } from '@/lib/llm-providers/types'
import { useLlmProviders } from '@/lib/llm-providers/useLlmProviders'
import { type McpServer, useMcpServers } from '@/lib/mcp/mcpServerStorage'
import { usePersonalization } from '@/lib/personalization/personalizationStorage'
import {
buildSidepanelChatTargets,
loadSidepanelChatTargetSelection,
persistSidepanelChatTargetSelection,
resolveSidepanelChatTarget,
type SidepanelChatTarget,
type SidepanelChatTargetSelection,
} from './sidepanel-chat-targets'
const constructMcpServers = (servers: McpServer[]) => {
return servers
@@ -23,14 +35,53 @@ const constructCustomServers = (servers: McpServer[]) => {
export const useChatRefs = () => {
const { servers: mcpServers } = useMcpServers()
const {
providers: llmProviders,
selectedProvider: selectedLlmProvider,
setDefaultProvider,
isLoading: isLoadingProviders,
} = useLlmProviders()
const { adapters, loading: isLoadingAdapters } = useAgentAdapters()
const { harnessAgents, loading: isLoadingAgents } = useHarnessAgents()
const { personalization } = usePersonalization()
const [targetSelection, setTargetSelection] =
useState<SidepanelChatTargetSelection | null>(null)
useEffect(() => {
let cancelled = false
loadSidepanelChatTargetSelection().then((selection) => {
if (!cancelled) setTargetSelection(selection)
})
return () => {
cancelled = true
}
}, [])
const chatTargets = useMemo(
() =>
buildSidepanelChatTargets({
providers: llmProviders,
adapters,
agents: harnessAgents,
}),
[llmProviders, adapters, harnessAgents],
)
const selectedChatTarget = useMemo(
() =>
resolveSidepanelChatTarget({
targets: chatTargets,
defaultProviderId: selectedLlmProvider?.id ?? llmProviders[0]?.id ?? '',
selection: targetSelection,
}),
[chatTargets, llmProviders, selectedLlmProvider, targetSelection],
)
const selectedLlmProviderRef = useRef<LlmProviderConfig | null>(
selectedLlmProvider,
)
const selectedChatTargetRef = useRef<SidepanelChatTarget | undefined>(
selectedChatTarget,
)
const enabledMcpServersRef = useRef(constructMcpServers(mcpServers))
const enabledCustomServersRef = useRef(constructCustomServers(mcpServers))
const personalizationRef = useRef(personalization)
@@ -41,16 +92,36 @@ export const useChatRefs = () => {
enabledCustomServersRef.current = constructCustomServers(mcpServers)
}, [selectedLlmProvider, mcpServers])
useEffect(() => {
selectedChatTargetRef.current = selectedChatTarget
}, [selectedChatTarget])
useEffect(() => {
personalizationRef.current = personalization
}, [personalization])
const selectChatTarget = useCallback(
async (target: SidepanelChatTarget | undefined) => {
selectedChatTargetRef.current = target
setTargetSelection(target ? { kind: target.kind, id: target.id } : null)
await persistSidepanelChatTargetSelection(target)
},
[],
)
return {
selectedLlmProviderRef,
selectedChatTargetRef,
enabledMcpServersRef,
enabledCustomServersRef,
personalizationRef,
llmProviders,
setDefaultProvider,
chatTargets,
selectedChatTarget,
selectChatTarget,
selectedLlmProvider,
isLoadingProviders,
isLoadingProviders:
isLoadingProviders || isLoadingAdapters || isLoadingAgents,
}
}

View File

@@ -0,0 +1,153 @@
import { describe, expect, it } from 'bun:test'
import type { LlmProviderConfig } from '@/lib/llm-providers/types'
import type { ChatMode } from './chatTypes'
import type { SidepanelChatTarget } from './sidepanel-chat-targets'
import { buildSidepanelPreparedSendMessagesRequest } from './useChatSessionRequest'
const conversationId = '00000000-0000-4000-8000-000000000001'
describe('buildSidepanelPreparedSendMessagesRequest', () => {
it('keeps LLM targets on the existing /chat request body', () => {
const request = buildSidepanelPreparedSendMessagesRequest({
agentServerUrl: 'http://127.0.0.1:5151',
target: llmTarget,
fallbackProvider,
message: 'Summarize this page',
...commonRequestInput(),
})
expect(request.api).toBe('http://127.0.0.1:5151/chat')
expect(request.body).toMatchObject({
message: 'Summarize this page',
conversationId,
provider: 'browseros',
providerType: 'browseros',
providerName: 'BrowserOS',
model: 'gpt-5',
mode: 'agent',
browserContext: {
activeTab: { id: 10, url: 'https://example.com', title: 'Example' },
enabledMcpServers: ['slack'],
},
userSystemPrompt: 'Be concise',
userWorkingDir: '/tmp/work',
previousConversation: [{ role: 'assistant', content: 'Prior answer' }],
selectedText: 'selected text',
selectedTextSource: {
url: 'https://example.com',
title: 'Example',
},
})
})
it('sends created-agent targets to the agent-id sidepanel route', () => {
const request = buildSidepanelPreparedSendMessagesRequest({
agentServerUrl: 'http://127.0.0.1:5151',
target: acpTarget,
fallbackProvider,
message: 'Inspect the current tab',
approvalResponses: [
{ approvalId: 'approval-1', approved: true, reason: 'ok' },
],
...commonRequestInput(),
})
expect(request.api).toBe(
'http://127.0.0.1:5151/agents/agent-codex/sidepanel/chat',
)
expect(request.body).toEqual({
conversationId,
message: 'Inspect the current tab',
browserContext: {
activeTab: { id: 10, url: 'https://example.com', title: 'Example' },
enabledMcpServers: ['slack'],
},
userSystemPrompt: 'Be concise',
userWorkingDir: '/tmp/work',
selectedText: 'selected text',
selectedTextSource: {
url: 'https://example.com',
title: 'Example',
},
})
})
it('keeps tool approval retry payloads scoped to LLM chat', () => {
const request = buildSidepanelPreparedSendMessagesRequest({
agentServerUrl: 'http://127.0.0.1:5151',
target: llmTarget,
fallbackProvider,
approvalResponses: [
{ approvalId: 'approval-1', approved: false, reason: 'no' },
],
...commonRequestInput(),
})
expect(request.api).toBe('http://127.0.0.1:5151/chat')
expect(request.body).toMatchObject({
message: '',
toolApprovalResponses: [
{ approvalId: 'approval-1', approved: false, reason: 'no' },
],
})
})
})
function commonRequestInput() {
return {
conversationId,
mode: 'agent' as ChatMode,
browserContext: {
activeTab: { id: 10, url: 'https://example.com', title: 'Example' },
enabledMcpServers: ['slack'],
},
userSystemPrompt: 'Be concise',
userWorkingDir: '/tmp/work',
previousConversation: [
{ role: 'assistant' as const, content: 'Prior answer' },
],
declinedApps: ['gmail'],
aclRules: [{ id: 'rule-1', sitePattern: '*://*/*', enabled: true }],
selectedText: 'selected text',
selectedTextSource: {
url: 'https://example.com',
title: 'Example',
},
toolApprovalConfig: { categories: { navigation: true } },
}
}
const fallbackProvider: LlmProviderConfig = {
id: 'browseros',
type: 'browseros',
name: 'BrowserOS',
modelId: 'gpt-5',
supportsImages: true,
contextWindow: 128000,
temperature: 0.7,
createdAt: 1000,
updatedAt: 1000,
}
const llmTarget: SidepanelChatTarget = {
kind: 'llm',
id: fallbackProvider.id,
name: fallbackProvider.name,
type: fallbackProvider.type,
provider: fallbackProvider,
}
const acpTarget: SidepanelChatTarget = {
kind: 'acp',
id: 'agent-codex',
name: 'Review bot',
type: 'acp',
agentId: 'agent-codex',
adapter: 'codex',
adapterName: 'Codex',
modelId: 'gpt-5.5',
modelLabel: 'GPT-5.5',
modelControl: 'best-effort',
reasoningEffort: 'medium',
reasoningEffortLabel: 'Medium',
}

View File

@@ -26,15 +26,14 @@ import { useInvalidateCredits } from '@/lib/credits/useCredits'
import { declinedAppsStorage } from '@/lib/declined-apps/storage'
import { useGraphqlQuery } from '@/lib/graphql/useGraphqlQuery'
import { createDefaultBrowserOSProvider } from '@/lib/llm-providers/storage'
import { useLlmProviders } from '@/lib/llm-providers/useLlmProviders'
import {
type ApprovalResponseData,
buildChatRequestBody,
type ChatRequestBrowserContext,
import type {
ApprovalResponseData,
ChatRequestBrowserContext,
} from '@/lib/messaging/server/buildChatRequestBody'
import { track } from '@/lib/metrics/track'
import { searchActionsStorage } from '@/lib/search-actions/searchActionsStorage'
import { selectedTextStorage } from '@/lib/selected-text/selectedTextStorage'
import { sentry } from '@/lib/sentry/sentry'
import { stopAgentStorage } from '@/lib/stop-agent/stop-agent-storage'
import {
type ApprovalResponse,
@@ -52,7 +51,12 @@ import {
import { selectedWorkspaceStorage } from '@/lib/workspace/workspace-storage'
import type { ChatMode } from './chatTypes'
import { GetConversationWithMessagesDocument } from './graphql/chatSessionDocument'
import { toLlmProviderConfig } from './sidepanel-chat-targets'
import { useChatRefs } from './useChatRefs'
import {
buildSidepanelPreparedSendMessagesRequest,
toProviderOption,
} from './useChatSessionRequest'
import { useExecutionHistoryTracker } from './useExecutionHistoryTracker'
import { useNotifyActiveTab } from './useNotifyActiveTab'
import { useRemoteConversationSave } from './useRemoteConversationSave'
@@ -186,16 +190,19 @@ const buildRequestBrowserContext = ({
export const useChatSession = (options?: ChatSessionOptions) => {
const {
selectedLlmProviderRef,
selectedChatTargetRef,
enabledMcpServersRef,
enabledCustomServersRef,
personalizationRef,
setDefaultProvider,
chatTargets,
selectedChatTarget,
selectChatTarget,
selectedLlmProvider,
isLoadingProviders,
} = useChatRefs()
const invalidateCredits = useInvalidateCredits()
const { providers: llmProviders, setDefaultProvider } = useLlmProviders()
const {
baseUrl: agentServerUrl,
isLoading: isLoadingAgentUrl,
@@ -218,11 +225,7 @@ export const useChatSession = (options?: ChatSessionOptions) => {
agentUrlRef.current = agentServerUrl
}, [agentServerUrl])
const providers: Provider[] = llmProviders.map((p) => ({
id: p.id,
name: p.name,
type: p.type,
}))
const providers: Provider[] = chatTargets.map(toProviderOption)
const [mode, setMode] = useState<ChatMode>('agent')
const [textToAction, setTextToAction] = useState<Map<string, ChatAction>>(
@@ -324,15 +327,8 @@ export const useChatSession = (options?: ChatSessionOptions) => {
textToActionRef.current = textToAction
}, [mode, textToAction])
const selectedProvider = selectedLlmProvider
? {
id: selectedLlmProvider.id,
name: selectedLlmProvider.name,
type:
selectedLlmProvider.id === 'browseros'
? ('browseros' as const)
: selectedLlmProvider.type,
}
const selectedProvider = selectedChatTarget
? toProviderOption(selectedChatTarget)
: providers[0]
const {
@@ -346,7 +342,8 @@ export const useChatSession = (options?: ChatSessionOptions) => {
} = useChat({
transport: new DefaultChatTransport({
prepareSendMessagesRequest: async ({ messages }) => {
const provider =
const target = selectedChatTargetRef.current
const fallbackProvider =
selectedLlmProviderRef.current ?? createDefaultBrowserOSProvider()
const activeTabsList = await chrome.tabs.query({
active: true,
@@ -395,51 +392,46 @@ export const useChatSession = (options?: ChatSessionOptions) => {
personalizationRef.current,
)
const approvalResponses = extractApprovalResponses(messages)
const commonRequest = {
conversationId: conversationIdRef.current,
mode: currentMode,
browserContext: requestBrowserContext,
userSystemPrompt,
userWorkingDir: workingDirRef.current,
previousConversation,
declinedApps,
aclRules: enabledAclRules,
toolApprovalConfig: approvalConfig,
}
const approvalResponses =
target?.kind === 'acp' ? null : extractApprovalResponses(messages)
if (approvalResponses) {
return {
api: `${agentUrlRef.current}/chat`,
body: buildChatRequestBody({
conversationId: conversationIdRef.current,
provider,
mode: currentMode,
browserContext: requestBrowserContext,
userSystemPrompt,
userWorkingDir: workingDirRef.current,
previousConversation,
declinedApps,
aclRules: enabledAclRules,
toolApprovalConfig: approvalConfig,
toolApprovalResponses: approvalResponses,
}),
}
return buildSidepanelPreparedSendMessagesRequest({
agentServerUrl: agentUrlRef.current ?? undefined,
target,
fallbackProvider,
...commonRequest,
approvalResponses,
})
}
const message = getLastMessageText(messages)
const result = {
api: `${agentUrlRef.current}/chat`,
body: buildChatRequestBody({
message,
conversationId: conversationIdRef.current,
provider,
mode: currentMode,
browserContext: requestBrowserContext,
userSystemPrompt,
userWorkingDir: workingDirRef.current,
previousConversation,
declinedApps,
aclRules: enabledAclRules,
selectedText: activeTabSelection?.text,
selectedTextSource: activeTabSelection
? {
url: activeTabSelection.url,
title: activeTabSelection.title,
}
: undefined,
toolApprovalConfig: approvalConfig,
}),
}
const result = buildSidepanelPreparedSendMessagesRequest({
agentServerUrl: agentUrlRef.current ?? undefined,
target,
fallbackProvider,
message,
...commonRequest,
selectedText: activeTabSelection?.text,
selectedTextSource: activeTabSelection
? {
url: activeTabSelection.url,
title: activeTabSelection.title,
}
: undefined,
})
// Track which tab's selection was sent so we can clear it on success
pendingSelectionTabKeyRef.current =
@@ -451,7 +443,7 @@ export const useChatSession = (options?: ChatSessionOptions) => {
sendAutomaticallyWhen: () => {
if (approvalJustRespondedRef.current) {
approvalJustRespondedRef.current = false
return true
return selectedChatTargetRef.current?.kind !== 'acp'
}
return false
},
@@ -686,10 +678,22 @@ export const useChatSession = (options?: ChatSessionOptions) => {
}, [dispatchMessage, isIntegrationsSynced])
const sendMessage = (params: { text: string; action?: ChatAction }) => {
const target = selectedChatTargetRef.current
const llmTargetProvider = toLlmProviderConfig(target)
const agentTarget = target?.kind === 'acp' ? target : undefined
track(MESSAGE_SENT_EVENT, {
mode,
provider_type: selectedLlmProvider?.type,
model: selectedLlmProvider?.modelId,
provider_id:
agentTarget?.agentId ??
llmTargetProvider?.id ??
selectedLlmProvider?.id,
provider_type: agentTarget ? 'acp' : llmTargetProvider?.type,
agent_id: agentTarget?.agentId,
adapter: agentTarget?.adapter,
model:
agentTarget?.modelId ??
llmTargetProvider?.modelId ??
selectedLlmProvider?.modelId,
})
if (!isIntegrationsSyncedRef.current) {
@@ -741,14 +745,54 @@ export const useChatSession = (options?: ChatSessionOptions) => {
addToolApprovalResponse(params)
}
const resetConversationState = () => {
stop()
void finishExecutionTask({ isAbort: true })
setConversationId(crypto.randomUUID())
setMessages([])
setTextToAction(new Map())
setLiked({})
setDisliked({})
setRestoredConversationId(null)
resetRemoteConversation()
}
const handleSelectProvider = (provider: Provider) => {
const fullProvider = llmProviders.find((p) => p.id === provider.id)
const target = chatTargets.find(
(candidate) =>
candidate.id === provider.id && candidate.kind === provider.kind,
)
if (!target) return
const previousTarget = selectedChatTargetRef.current
track(PROVIDER_SELECTED_EVENT, {
provider_id: provider.id,
provider_type: provider.type,
model_id: fullProvider?.modelId,
provider_id: target.id,
provider_type: target.kind === 'acp' ? 'acp' : target.type,
model_id:
target.kind === 'acp' ? target.modelId : target.provider.modelId,
agent_id: target.kind === 'acp' ? target.agentId : undefined,
adapter: target.kind === 'acp' ? target.adapter : undefined,
})
setDefaultProvider(provider.id)
void selectChatTarget(target).catch((error) => {
sentry.captureException(error, {
extra: {
message: 'Failed to persist sidepanel chat target selection',
targetId: target.id,
targetKind: target.kind,
},
})
})
if (target.kind === 'llm') setDefaultProvider(target.provider.id)
if (
previousTarget &&
(previousTarget.kind !== target.kind ||
previousTarget.id !== target.id) &&
messagesRef.current.length > 0
) {
resetConversationState()
}
}
const getActionForMessage = (message: UIMessage) => {
@@ -762,15 +806,7 @@ export const useChatSession = (options?: ChatSessionOptions) => {
const resetConversation = () => {
track(CONVERSATION_RESET_EVENT, { message_count: messages.length })
stop()
void finishExecutionTask({ isAbort: true })
setConversationId(crypto.randomUUID())
setMessages([])
setTextToAction(new Map())
setLiked({})
setDisliked({})
setRestoredConversationId(null)
resetRemoteConversation()
resetConversationState()
}
const isRestoringConversation =

View File

@@ -0,0 +1,74 @@
import type { Provider } from '../../../components/chat/chatComponentTypes'
import type { LlmProviderConfig } from '../../../lib/llm-providers/types'
import {
type ApprovalResponseData,
buildChatRequestBody,
} from '../../../lib/messaging/server/buildChatRequestBody'
import {
type SidepanelChatTarget,
toLlmProviderConfig,
} from './sidepanel-chat-targets'
type LlmChatRequestBodyInput = Parameters<typeof buildChatRequestBody>[0]
type CommonSidepanelRequestInput = Omit<
LlmChatRequestBodyInput,
'provider' | 'message' | 'toolApprovalResponses' | 'isScheduledTask'
>
interface BuildSidepanelPreparedSendMessagesRequestInput
extends CommonSidepanelRequestInput {
agentServerUrl: string | undefined
target: SidepanelChatTarget | undefined
fallbackProvider: LlmProviderConfig
message?: string
approvalResponses?: ApprovalResponseData[] | null
}
export function buildSidepanelPreparedSendMessagesRequest({
agentServerUrl,
target,
fallbackProvider,
message,
approvalResponses,
...common
}: BuildSidepanelPreparedSendMessagesRequestInput) {
if (target?.kind === 'acp') {
return {
api: `${agentServerUrl}/agents/${encodeURIComponent(target.agentId)}/sidepanel/chat`,
body: {
conversationId: common.conversationId,
message: message ?? '',
browserContext: common.browserContext,
userSystemPrompt: common.userSystemPrompt,
userWorkingDir: common.userWorkingDir,
selectedText: common.selectedText,
selectedTextSource: common.selectedTextSource,
},
}
}
const provider = toLlmProviderConfig(target) ?? fallbackProvider
return {
api: `${agentServerUrl}/chat`,
body: buildChatRequestBody({
...common,
provider,
message,
toolApprovalResponses: approvalResponses ?? undefined,
}),
}
}
export function toProviderOption(target: SidepanelChatTarget): Provider {
return {
id: target.id,
name: target.name,
type: target.type,
kind: target.kind,
agentId: target.kind === 'acp' ? target.agentId : undefined,
adapterName: target.kind === 'acp' ? target.adapterName : undefined,
modelLabel: target.kind === 'acp' ? target.modelLabel : undefined,
modelControl: target.kind === 'acp' ? target.modelControl : undefined,
}
}

View File

@@ -12,6 +12,8 @@ export interface AssistantThinkingPart {
export interface ToolEntry {
id: string
name: string
label: string
subject?: string
status: 'running' | 'completed' | 'error'
durationMs?: number
}
@@ -26,9 +28,24 @@ export type AssistantPart =
| AssistantThinkingPart
| AssistantToolBatchPart
/**
* Attachments rendered alongside the user's text on the optimistic turn
* — populated when the composer staged any images/files. The dataUrl is
* the same one the server received; we keep it in memory only for the
* lifetime of the live turn (history reload re-fetches via the JSONL).
*/
export interface UserAttachmentPreview {
id: string
kind: 'image' | 'file'
mediaType: string
name: string
dataUrl?: string
}
export interface AgentConversationTurn {
id: string
userText: string
userAttachments?: UserAttachmentPreview[]
parts: AssistantPart[]
done: boolean
timestamp: number
@@ -50,4 +67,7 @@ export interface AgentCardData {
status: 'idle' | 'working' | 'error'
lastMessage?: string
lastMessageTimestamp?: number
activitySummary?: string
currentTool?: string
costUsd?: number
}

View File

@@ -0,0 +1,369 @@
/**
* Composer attachment helpers — validation, image compression, and the
* client-side payload shape sent to /agents/:id/chat.
*
* Image attachments travel as `data:` URLs (base64) so the gateway, which
* runs on 127.0.0.1 over Lima virtiofs, can ingest them as standard
* OpenAI-style content blocks. Non-image text-shaped files are read into
* memory and travel as their extracted text body — the server inlines
* them as a fenced `<attachment>` block on the user message.
*/
export const MAX_ATTACHMENTS_PER_MESSAGE = 10
export const MAX_IMAGE_BYTES = 5 * 1024 * 1024 // 5 MB after compression
export const MAX_FILE_TEXT_BYTES = 1 * 1024 * 1024 // 1 MB extracted text
export const IMAGE_LONG_EDGE_CAP = 2048
export const ALLOWED_IMAGE_MEDIA_TYPES = [
'image/png',
'image/jpeg',
'image/jpg',
'image/webp',
'image/gif',
] as const
export const ALLOWED_FILE_MEDIA_TYPE_PREFIXES = [
'text/',
'application/json',
] as const
export type ServerImageAttachment = {
kind: 'image'
mediaType: string
dataUrl: string
name?: string
}
export type ServerFileAttachment = {
kind: 'file'
mediaType: string
name: string
text: string
}
export type ServerAttachmentPayload =
| ServerImageAttachment
| ServerFileAttachment
/** UI-side representation: what the composer needs to render a chip. */
export interface StagedAttachment {
id: string
kind: 'image' | 'file'
mediaType: string
name: string
// Set for images so the chip thumbnail can render directly. For files
// we don't need a preview yet, but the field exists for v2 PDF previews.
dataUrl?: string
// Pre-computed payload for the server. Built once at staging time so
// re-renders don't re-encode large blobs.
payload: ServerAttachmentPayload
}
export type AttachmentValidationError =
| { code: 'too_many'; message: string }
| { code: 'unsupported_type'; message: string; mediaType: string }
| { code: 'too_large'; message: string }
| { code: 'read_failed'; message: string }
export type StageAttachmentResult =
| { ok: true; attachment: StagedAttachment }
| { ok: false; error: AttachmentValidationError }
function isImageMediaType(mediaType: string): boolean {
return (ALLOWED_IMAGE_MEDIA_TYPES as readonly string[]).includes(mediaType)
}
function isAllowedFileMediaType(mediaType: string): boolean {
return ALLOWED_FILE_MEDIA_TYPE_PREFIXES.some((prefix) =>
mediaType.startsWith(prefix),
)
}
/** Build a unique id without depending on `crypto.randomUUID` outside DOM. */
function makeId(): string {
if (typeof crypto !== 'undefined' && crypto.randomUUID) {
return crypto.randomUUID()
}
return `att-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 10)}`
}
/**
* Read a `File` and produce the staged-attachment shape — validate type,
* compress if it's a large image, and pre-build the server payload.
*/
export async function stageAttachment(
file: File,
): Promise<StageAttachmentResult> {
const mediaType = file.type || 'application/octet-stream'
if (isImageMediaType(mediaType)) {
try {
const compressed = await compressImageIfNeeded(file)
const dataUrl = await readAsDataUrl(compressed)
// Rough byte ceiling — `data:image/png;base64,...` doubles size with
// base64. Reject early so we never POST something the route will 400.
if (dataUrl.length > MAX_IMAGE_BYTES * 2) {
return {
ok: false,
error: {
code: 'too_large',
message: `Image "${file.name}" is too large (max ${humanBytes(
MAX_IMAGE_BYTES,
)}).`,
},
}
}
return {
ok: true,
attachment: {
id: makeId(),
kind: 'image',
mediaType,
name: file.name || 'image',
dataUrl,
payload: {
kind: 'image',
mediaType,
dataUrl,
name: file.name || undefined,
},
},
}
} catch (err) {
return {
ok: false,
error: {
code: 'read_failed',
message:
err instanceof Error
? err.message
: `Failed to read image "${file.name}".`,
},
}
}
}
if (isAllowedFileMediaType(mediaType)) {
let text: string
try {
text = await file.text()
} catch (err) {
return {
ok: false,
error: {
code: 'read_failed',
message:
err instanceof Error
? err.message
: `Failed to read file "${file.name}".`,
},
}
}
if (text.length > MAX_FILE_TEXT_BYTES) {
return {
ok: false,
error: {
code: 'too_large',
message: `File "${file.name}" is too large (max ${humanBytes(
MAX_FILE_TEXT_BYTES,
)}).`,
},
}
}
return {
ok: true,
attachment: {
id: makeId(),
kind: 'file',
mediaType,
name: file.name || 'attachment',
payload: {
kind: 'file',
mediaType,
name: file.name || 'attachment',
text,
},
},
}
}
return {
ok: false,
error: {
code: 'unsupported_type',
message: `Unsupported attachment type: ${mediaType || 'unknown'}`,
mediaType,
},
}
}
/**
* Stage multiple files at once, enforcing the per-message cap. The result
* partitions successful stages and any errors so the caller can show
* granular toasts.
*/
export async function stageAttachments(
files: File[],
alreadyStaged: number,
): Promise<{
staged: StagedAttachment[]
errors: AttachmentValidationError[]
}> {
const remainingSlots = Math.max(
0,
MAX_ATTACHMENTS_PER_MESSAGE - alreadyStaged,
)
const staged: StagedAttachment[] = []
const errors: AttachmentValidationError[] = []
if (remainingSlots === 0 && files.length > 0) {
errors.push({
code: 'too_many',
message: `At most ${MAX_ATTACHMENTS_PER_MESSAGE} attachments per message.`,
})
return { staged, errors }
}
const overflow = files.length - remainingSlots
if (overflow > 0) {
errors.push({
code: 'too_many',
message: `Only the first ${remainingSlots} of ${files.length} files were attached (max ${MAX_ATTACHMENTS_PER_MESSAGE}).`,
})
}
for (const file of files.slice(0, remainingSlots)) {
const result = await stageAttachment(file)
if (result.ok) {
staged.push(result.attachment)
} else {
errors.push(result.error)
}
}
return { staged, errors }
}
/**
* Resize images that are oversized to a sane long-edge cap. JPEG/WebP
* source files are re-encoded to JPEG; PNGs/GIFs that are already small
* are passed through untouched.
*/
export async function compressImageIfNeeded(file: File): Promise<Blob> {
// Cheap path: small files don't need any transform.
if (file.size <= 1.5 * 1024 * 1024) return file
const bitmap = await blobToImageBitmap(file)
const { width, height } = bitmap
const longEdge = Math.max(width, height)
if (longEdge <= IMAGE_LONG_EDGE_CAP && file.size <= MAX_IMAGE_BYTES) {
bitmap.close?.()
return file
}
const scale = Math.min(1, IMAGE_LONG_EDGE_CAP / longEdge)
const targetWidth = Math.max(1, Math.round(width * scale))
const targetHeight = Math.max(1, Math.round(height * scale))
const canvas =
typeof OffscreenCanvas !== 'undefined'
? new OffscreenCanvas(targetWidth, targetHeight)
: Object.assign(document.createElement('canvas'), {
width: targetWidth,
height: targetHeight,
})
const ctx = canvas.getContext('2d') as
| CanvasRenderingContext2D
| OffscreenCanvasRenderingContext2D
| null
if (!ctx) {
bitmap.close?.()
return file
}
ctx.drawImage(bitmap, 0, 0, targetWidth, targetHeight)
bitmap.close?.()
const outputType = 'image/jpeg'
if (canvas instanceof HTMLCanvasElement) {
return await new Promise<Blob>((resolve, reject) => {
canvas.toBlob(
(blob) => {
if (blob) resolve(blob)
else reject(new Error('Image compression failed.'))
},
outputType,
0.85,
)
})
}
return await (canvas as OffscreenCanvas).convertToBlob({
type: outputType,
quality: 0.85,
})
}
async function blobToImageBitmap(blob: Blob): Promise<ImageBitmap> {
if (typeof createImageBitmap === 'function') {
return createImageBitmap(blob)
}
// Fallback: load via an Image element and use the canvas decode path.
const url = URL.createObjectURL(blob)
try {
const img = await new Promise<HTMLImageElement>((resolve, reject) => {
const el = new Image()
el.onload = () => resolve(el)
el.onerror = () =>
reject(new Error('Failed to decode image for compression.'))
el.src = url
})
const canvas = document.createElement('canvas')
canvas.width = img.naturalWidth
canvas.height = img.naturalHeight
const ctx = canvas.getContext('2d')
if (!ctx) throw new Error('Canvas 2D context unavailable.')
ctx.drawImage(img, 0, 0)
const blobOut = await new Promise<Blob | null>((resolve) =>
canvas.toBlob(resolve, 'image/png'),
)
if (!blobOut) throw new Error('Canvas toBlob returned null.')
return await createImageBitmap(blobOut)
} finally {
URL.revokeObjectURL(url)
}
}
async function readAsDataUrl(blob: Blob): Promise<string> {
if ('arrayBuffer' in blob && typeof FileReader === 'undefined') {
const buffer = await blob.arrayBuffer()
const base64 = arrayBufferToBase64(buffer)
const type = blob.type || 'application/octet-stream'
return `data:${type};base64,${base64}`
}
return await new Promise<string>((resolve, reject) => {
const reader = new FileReader()
reader.onload = () => resolve(reader.result as string)
reader.onerror = () =>
reject(reader.error ?? new Error('FileReader failed to read blob.'))
reader.readAsDataURL(blob)
})
}
function arrayBufferToBase64(buffer: ArrayBuffer): string {
const bytes = new Uint8Array(buffer)
let binary = ''
const chunkSize = 0x8000
for (let i = 0; i < bytes.byteLength; i += chunkSize) {
binary += String.fromCharCode.apply(
null,
Array.from(bytes.subarray(i, Math.min(i + chunkSize, bytes.byteLength))),
)
}
return btoa(binary)
}
function humanBytes(bytes: number): string {
if (bytes >= 1024 * 1024) return `${(bytes / 1024 / 1024).toFixed(0)} MB`
if (bytes >= 1024) return `${(bytes / 1024).toFixed(0)} KB`
return `${bytes} B`
}

View File

@@ -75,6 +75,12 @@ export const MCP_EXTERNAL_ACCESS_DISABLED_EVENT =
/** @public */
export const MCP_SERVER_RESTARTED_EVENT = 'settings.mcp_server.restarted'
/** @public */
export const AGENT_CREATED_EVENT = 'agents.agent.created'
/** @public */
export const AGENT_DELETED_EVENT = 'agents.agent.deleted'
/** @public */
export const NEW_SCHEDULED_TASK_CREATED_EVENT =
'settings.scheduled_task.created'

View File

@@ -2,29 +2,75 @@ function isAbortError(error: unknown): boolean {
return error instanceof DOMException && error.name === 'AbortError'
}
export interface ParsedSSEEvent<T> {
data: T
/** Numeric `id:` line on the same SSE event, if any. */
seq?: number
}
export function parseSSELines<T>(buffer: string): {
events: T[]
events: ParsedSSEEvent<T>[]
remainder: string
} {
// SSE events are separated by blank lines. Buffer lines until we hit
// a blank, then assemble each event. Lines we recognise: `id: <n>`
// and `data: <payload>`. Everything else is ignored.
const events: ParsedSSEEvent<T>[] = []
const lines = buffer.split('\n')
const remainder = lines.pop() ?? ''
const events: T[] = []
for (const line of lines) {
if (!line.startsWith('data: ')) continue
const payload = line.slice(6)
if (payload === '[DONE]') continue
try {
events.push(JSON.parse(payload) as T)
} catch {}
// Find the last blank-line boundary; everything after it is the
// remainder (next event partially received).
let lastBoundary = -1
for (let i = lines.length - 1; i >= 0; i--) {
if (lines[i] === '') {
lastBoundary = i
break
}
}
const completeLines = lastBoundary >= 0 ? lines.slice(0, lastBoundary) : []
const remainder =
lastBoundary >= 0 ? lines.slice(lastBoundary + 1).join('\n') : buffer
let currentSeq: number | undefined
let currentData: string | null = null
const flush = () => {
if (currentData != null && currentData !== '[DONE]') {
try {
events.push({
data: JSON.parse(currentData) as T,
seq: currentSeq,
})
} catch {
// ignore
}
}
currentSeq = undefined
currentData = null
}
for (const line of completeLines) {
if (line === '') {
flush()
continue
}
if (line.startsWith('id: ')) {
const n = Number.parseInt(line.slice(4).trim(), 10)
if (Number.isFinite(n)) currentSeq = n
continue
}
if (line.startsWith('data: ')) {
currentData = line.slice(6)
}
}
// Catch a complete trailing event with no terminating blank line —
// shouldn't happen in well-formed SSE, but be tolerant.
flush()
return { events, remainder }
}
export async function consumeSSEStream<T>(
response: Response,
onEvent: (event: T) => void,
onEvent: (event: T, meta: { seq?: number }) => void,
signal?: AbortSignal,
): Promise<void> {
const reader = response.body?.getReader()
@@ -49,7 +95,7 @@ export async function consumeSSEStream<T>(
buffer = remainder
for (const event of events) {
onEvent(event)
onEvent(event.data, { seq: event.seq })
}
}
} catch (error) {
@@ -64,7 +110,7 @@ export async function consumeSSEStream<T>(
if (buffer) {
const { events } = parseSSELines<T>(buffer)
for (const event of events) {
onEvent(event)
onEvent(event.data, { seq: event.seq })
}
}
}

View File

@@ -0,0 +1,325 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*
* Maps raw tool names + arguments to human-readable activity labels for
* the chat UI activity view. The MCP ToolRegistry is the source of truth
* for tool *existence*; this file is the editorial layer that turns
* snake_case identifiers into agent-speak verbs.
*/
const VERB_OVERRIDES: Record<string, string> = {
// Navigation
navigate_page: 'Navigated to',
new_page: 'Opened tab',
new_hidden_page: 'Opened tab',
show_page: 'Showed tab',
close_page: 'Closed tab',
list_pages: 'Listed open tabs',
get_active_page: 'Got active tab',
move_page: 'Moved tab',
group_tabs: 'Grouped tabs',
// Page reading
take_snapshot: 'Captured page snapshot',
take_enhanced_snapshot: 'Captured detailed snapshot',
get_page_content: 'Read page content',
get_page_links: 'Extracted page links',
get_dom: 'Read page DOM',
search_dom: 'Searched page DOM',
take_screenshot: 'Took screenshot',
// Input
click: 'Clicked',
click_at: 'Clicked at coordinates',
hover: 'Hovered',
hover_at: 'Hovered at coordinates',
type_at: 'Typed at coordinates',
drag_at: 'Dragged',
focus: 'Focused element',
fill: 'Filled field',
clear: 'Cleared field',
check: 'Checked box',
uncheck: 'Unchecked box',
press_key: 'Pressed key',
upload_file: 'Uploaded file',
// Console / scripts
evaluate_script: 'Ran script',
get_console_logs: 'Read console logs',
// History / bookmarks
search_history: 'Searched history',
get_recent_history: 'Read recent history',
delete_history_url: 'Deleted history entry',
delete_history_range: 'Deleted history range',
get_bookmarks: 'Listed bookmarks',
create_bookmark: 'Created bookmark',
remove_bookmark: 'Removed bookmark',
update_bookmark: 'Updated bookmark',
move_bookmark: 'Moved bookmark',
search_bookmarks: 'Searched bookmarks',
// Filesystem (sandboxed)
read_file: 'Read file',
write_file: 'Wrote file',
find_files: 'Searched files',
// Memory
read_soul: 'Read soul memory',
read_core: 'Read core memory',
write_memory: 'Wrote memory',
search_memory: 'Searched memory',
update_soul: 'Updated soul memory',
update_core: 'Updated core memory',
// Web
web_search: 'Searched the web',
web_fetch: 'Fetched URL',
// Klavis / external apps (Strata)
connector_mcp_servers: 'Listed connected apps',
discover_server_categories_or_actions: 'Browsed available actions',
get_category_actions: 'Listed actions',
get_action_details: 'Looked up action',
execute_action: 'Ran external action',
search_documentation: 'Searched docs',
handle_auth_failure: 'Handled auth issue',
// Suggestions
suggest_schedule: 'Suggested schedule',
suggest_app_connection: 'Suggested app connect',
// BrowserOS info
browseros_info: 'Read BrowserOS info',
// Windows
list_windows: 'Listed windows',
focus_window: 'Focused window',
close_window: 'Closed window',
create_window: 'Created window',
}
// ──────────────────────────────────────────────────────────────────────
// Helpers
// ──────────────────────────────────────────────────────────────────────
function asString(value: unknown): string | undefined {
return typeof value === 'string' && value.length > 0 ? value : undefined
}
function stringField(
input: Record<string, unknown>,
...keys: string[]
): string | undefined {
for (const k of keys) {
const v = asString(input[k])
if (v) return v
}
return undefined
}
function truncate(text: string | undefined, max: number): string | undefined {
if (!text) return undefined
return text.length > max ? `${text.slice(0, max - 1)}` : text
}
function quote(value: string | undefined): string | undefined {
if (!value) return undefined
return `"${truncate(value, 60)}"`
}
function basename(path: string | undefined): string | undefined {
if (!path) return undefined
const parts = path.split(/[/\\]/).filter(Boolean)
return parts[parts.length - 1] ?? path
}
function formatUrl(value: unknown): string | undefined {
const url = asString(value)
if (!url) return undefined
try {
const parsed = new URL(url)
const host = parsed.host
const path = parsed.pathname === '/' ? '' : parsed.pathname
const display = path && path.length > 0 ? `${host}${path}` : host
return truncate(display, 60)
} catch {
return truncate(url, 60)
}
}
function coords(x: unknown, y: unknown): string | undefined {
if (typeof x === 'number' && typeof y === 'number') {
return `${Math.round(x)}, ${Math.round(y)}`
}
return undefined
}
// ──────────────────────────────────────────────────────────────────────
// Subject extractors
// ──────────────────────────────────────────────────────────────────────
type SubjectExtractor = (input: Record<string, unknown>) => string | undefined
const SUBJECT_EXTRACTORS: Record<string, SubjectExtractor> = {
// URL-bearing tools
new_page: (i) => formatUrl(i.url),
new_hidden_page: (i) => formatUrl(i.url),
navigate_page: (i) => {
const action = asString(i.action)
if (action === 'back') return 'back'
if (action === 'forward') return 'forward'
if (action === 'reload') return 'reload'
return formatUrl(i.url)
},
web_fetch: (i) => formatUrl(i.url),
// Search queries
web_search: (i) => quote(stringField(i, 'query', 'q')),
search_history: (i) => quote(stringField(i, 'query', 'text')),
search_bookmarks: (i) => quote(stringField(i, 'query', 'text')),
search_memory: (i) => quote(stringField(i, 'query', 'q')),
search_dom: (i) => quote(stringField(i, 'query', 'selector')),
search_documentation: (i) => quote(stringField(i, 'query', 'q')),
find_files: (i) => quote(stringField(i, 'pattern', 'query')),
// Element interactions
click: (i) => stringField(i, 'element'),
hover: (i) => stringField(i, 'element'),
focus: (i) => stringField(i, 'element'),
clear: (i) => stringField(i, 'element'),
check: (i) => stringField(i, 'element'),
uncheck: (i) => stringField(i, 'element'),
fill: (i) => {
const target = stringField(i, 'element')
const text = stringField(i, 'text')
if (target && text) return `${target}: ${truncate(text, 40)}`
return target ?? truncate(text, 40)
},
press_key: (i) => stringField(i, 'key'),
// Coordinate-based input
click_at: (i) => coords(i.x, i.y),
hover_at: (i) => coords(i.x, i.y),
type_at: (i) => {
const at = coords(i.x, i.y)
const text = stringField(i, 'text')
if (at && text) return `${at}: ${truncate(text, 40)}`
return at ?? truncate(text, 40)
},
drag_at: (i) => {
const from = coords(i.fromX, i.fromY)
const to = coords(i.toX, i.toY)
if (from && to) return `${from}${to}`
return from ?? to
},
// Tab management
show_page: (i) => {
const page = i.page
return typeof page === 'number' ? `tab ${page}` : asString(page)
},
close_page: (i) => {
const page = i.page
return typeof page === 'number' ? `tab ${page}` : asString(page)
},
move_page: (i) => {
const page = i.page
return typeof page === 'number' ? `tab ${page}` : asString(page)
},
// Page reads (take_snapshot, take_enhanced_snapshot, get_page_content,
// get_page_links, get_dom, take_screenshot) intentionally omit a
// subject — the only argument is a numeric page ID that's internal
// to the agent and meaningless to the user ("tab 4" tells them nothing).
// The verb alone communicates what happened.
// External actions via Strata
execute_action: (i) => {
const server = stringField(i, 'server_name')
const action = stringField(i, 'action_name')
if (server && action) return `${server} · ${action}`
return action ?? server
},
get_category_actions: (i) => stringField(i, 'category_name', 'server_name'),
get_action_details: (i) => stringField(i, 'action_name'),
discover_server_categories_or_actions: (i) =>
stringField(i, 'server_name', 'category_name'),
connector_mcp_servers: (i) => stringField(i, 'server_name'),
// Filesystem
read_file: (i) => basename(stringField(i, 'path')),
write_file: (i) => basename(stringField(i, 'path')),
// Memory writes — show first chars of content
write_memory: (i) => truncate(stringField(i, 'content', 'text'), 40),
update_soul: (i) => truncate(stringField(i, 'content'), 40),
update_core: (i) => truncate(stringField(i, 'content'), 40),
// Bookmarks
create_bookmark: (i) => stringField(i, 'title') ?? formatUrl(i.url),
remove_bookmark: (i) => stringField(i, 'id', 'title'),
update_bookmark: (i) => stringField(i, 'id', 'title'),
move_bookmark: (i) => stringField(i, 'id', 'title'),
// History
delete_history_url: (i) => formatUrl(i.url),
}
// ──────────────────────────────────────────────────────────────────────
// Public API
// ──────────────────────────────────────────────────────────────────────
export interface ToolLabelResult {
label: string
subject?: string
}
/**
* Strip MCP namespace prefixes (e.g. "browseros__", "mcp_") to find the
* canonical tool name used in the override maps.
*/
function canonicalName(rawName: string): string {
return rawName.replace(/^browseros__/, '').replace(/^mcp_/, '')
}
/**
* Convert a snake_case tool name into Sentence-case English as a fallback
* when no curated override exists.
*/
function humanizeToolName(rawName: string): string {
const stripped = canonicalName(rawName)
const words = stripped.split(/[_-]/).filter((w) => w.length > 0)
if (words.length === 0) return rawName
const first = words[0]!
return [
first.charAt(0).toUpperCase() + first.slice(1),
...words.slice(1),
].join(' ')
}
/**
* Build a human-readable label and subject string for a tool call,
* suitable for rendering in the chat activity view.
*/
export function buildToolLabel(
rawName: string,
input?: Record<string, unknown>,
): ToolLabelResult {
const canonical = canonicalName(rawName)
const label =
VERB_OVERRIDES[canonical] ??
VERB_OVERRIDES[rawName] ??
humanizeToolName(rawName)
const extractor = Object.hasOwn(SUBJECT_EXTRACTORS, canonical)
? SUBJECT_EXTRACTORS[canonical]
: Object.hasOwn(SUBJECT_EXTRACTORS, rawName)
? SUBJECT_EXTRACTORS[rawName]
: undefined
const subject = extractor && input ? extractor(input) : undefined
return { label, subject }
}

View File

@@ -8,6 +8,7 @@ const chromiumArgs = [
'--show-component-extension-options',
'--disable-browseros-server',
'--disable-browseros-extensions',
'--browseros-dock-icon=dev',
]
if (env.BROWSEROS_CDP_PORT) {

View File

@@ -7,8 +7,9 @@ import { PRODUCT_WEB_HOST } from './lib/constants/productWebHost'
// biome-ignore lint/style/noProcessEnv: build config file needs env access
const env = process.env
// biome-ignore lint/style/noNonNullAssertion: required env var
const apiUrl = new URL(env.VITE_PUBLIC_BROWSEROS_API!)
const apiUrl = new URL(
env.VITE_PUBLIC_BROWSEROS_API?.trim() || 'https://api.browseros.com',
)
const apiPattern = apiUrl.port
? `${apiUrl.hostname}:${apiUrl.port}`
: apiUrl.hostname

View File

@@ -0,0 +1,51 @@
# Copy to .env.development for local eval runs.
# Provider keys used by existing config files.
OPENROUTER_API_KEY=
FIREWORKS_API_KEY=
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
GOOGLE_GENERATIVE_AI_API_KEY=
# Claude Agent SDK token used by performance_grader.
CLAUDE_CODE_OAUTH_TOKEN=
# Suite-mode model selection.
EVAL_VARIANT=local
EVAL_AGENT_PROVIDER=openai-compatible
EVAL_AGENT_MODEL=
EVAL_AGENT_API_KEY=
EVAL_AGENT_BASE_URL=
EVAL_AGENT_SUPPORTS_IMAGES=true
# Optional suite-mode executor override for orchestrator suites.
EVAL_EXECUTOR_MODEL=
EVAL_EXECUTOR_API_KEY=
EVAL_EXECUTOR_BASE_URL=
# Clado visual action executor.
CLADO_ACTION_MODEL=
CLADO_ACTION_API_KEY=
CLADO_ACTION_BASE_URL=
# Backward-compatible alias used by older local scripts.
CLADO_ACTION_URL=
# BrowserOS runner.
BROWSEROS_BINARY=/Applications/BrowserOS.app/Contents/MacOS/BrowserOS
BROWSEROS_SERVER_URL=http://127.0.0.1:9110
BROWSEROS_SERVER_LOG_DIR=/tmp/browseros-server-logs
BROWSEROS_CONFIG_URL=
# Captcha solver extension.
NOPECHA_API_KEY=
# WebArena-Infinity.
WEBARENA_INFINITY_DIR=
INFINITY_APP_URL=
# R2 publishing and weekly report.
EVAL_R2_ACCOUNT_ID=
EVAL_R2_ACCESS_KEY_ID=
EVAL_R2_SECRET_ACCESS_KEY=
EVAL_R2_BUCKET=browseros-eval
EVAL_R2_CDN_BASE_URL=https://eval.browseros.com

View File

@@ -1,875 +0,0 @@
# Eval System - Production Grade Design Doc
## Current State Analysis
### What's Working Well
1. **Zod validation** - Already exists in `config-validator.ts`, reuses `LLMConfigSchema` from `@browseros/shared`
2. **Grader registry pattern** - `createGrader()` factory works well, easy to add new graders
3. **AgentEvaluator interface** - Clean interface: `execute() → AgentResult`
4. **Discriminated unions** - Messages, agent types use proper TypeScript patterns
5. **Capture utilities** - `ScreenshotCapture`, `MessageLogger`, `TrajectorySaver` are modular
### Key Problems
**1. No Agent Registry/Factory**
Agent creation is hardcoded if-else in `task-executor.ts`:
```typescript
// Current approach - not scalable
if (this.config.agent.type === 'single') {
const evaluator = new SingleAgentEvaluator(...)
} else if (this.config.agent.type === 'orchestrator-executor') {
const evaluator = new OrchestratorExecutorEvaluator(...)
}
// Adding new agent = modify this file
```
**2. Heavy Server Dependency**
Imports from `@browseros/server`:
- `GeminiAgent` - Core agent (necessary)
- `ToolExecutionHooks` - Hook interface
- `ResolvedAgentConfig` - Agent config type
- `AgentExecutionError` - Error type
- `VercelAIContentGenerator` - Provider adapter
- Gateway client functions
**3. Scattered Types**
- `src/types.ts` - Main types
- `agents/types.ts` - Agent interface
- `agents/orchestrator-executor/types.ts` - Orchestrator types
- `runner/types.ts` - Runner types
- `graders/types.ts` - Grader types
**4. Duplicated Capture Logic**
Both agent evaluators duplicate:
- Initialize ScreenshotCapture
- Initialize MessageLogger
- Set up tool hooks
- Handle timeouts
- Collect errors/warnings
**5. No Unified Utils**
Hooks, screenshot capture, message logging code is copy-pasted per agent type.
---
## Design Goals
1. **Easy to add new agents** - Register new agent type, implement interface, done
2. **Shared capture infrastructure** - All agents use same screenshot/logging utils
3. **Type-safe with Zod** - Config validation at entry point
4. **Minimal server coupling** - Only import what's necessary
5. **Clear folder structure** - Types where they belong
6. **Production patterns** - Factory, registry, composition
---
## Proposed Architecture
### Folder Structure
```
eval/src/
├── index.ts # Entry point, CLI
├── types/
│ ├── index.ts # Re-exports all types
│ ├── config.ts # EvalConfig, AgentConfig (Zod schemas + types)
│ ├── task.ts # Task, TaskMetadata
│ ├── message.ts # Message discriminated union
│ ├── result.ts # AgentResult, GraderResult
│ └── errors.ts # ErrorSource, TaskError, EvalWarning
├── agents/
│ ├── index.ts # Re-exports + auto-registration
│ ├── registry.ts # Agent registry + factory
│ ├── types.ts # AgentEvaluator interface, AgentContext
│ ├── single/
│ │ └── index.ts # SingleAgentEvaluator
│ └── orchestrator-executor/
│ ├── index.ts # OrchestratorExecutorEvaluator
│ ├── types.ts # Orchestrator-specific types only
│ ├── orchestrator.ts
│ ├── orchestrator-agent.ts
│ ├── orchestrator-tools.ts
│ ├── executor.ts
│ └── executor-store.ts
├── capture/
│ ├── index.ts # Re-exports
│ ├── types.ts # CaptureContext interface
│ ├── context.ts # CaptureContext class (bundles all capture)
│ ├── hooks.ts # createCaptureHooks() utility
│ ├── screenshot.ts # ScreenshotCapture
│ ├── message-logger.ts # MessageLogger
│ ├── trajectory-saver.ts # TrajectorySaver
│ └── window-manager.ts # WindowManager
├── graders/
│ ├── index.ts # Re-exports
│ ├── registry.ts # Grader registry (existing pattern)
│ ├── types.ts # Grader interface
│ ├── benchmark/
│ │ ├── webvoyager.ts
│ │ └── mind2web.ts
│ └── fara/
│ ├── alignment.ts
│ ├── rubric.ts
│ ├── multimodal.ts
│ └── combined.ts
├── runner/
│ ├── index.ts # runEval() main entry
│ ├── types.ts # RunEvalOptions, TaskResult, BatchSummary
│ ├── task-loader.ts
│ ├── task-executor.ts
│ └── parallel-executor.ts
└── utils/
├── env.ts # resolveEnvValue() helper
└── validation.ts # Config validation logic
```
---
## Key Components
### 1. Type System (`types/`)
**`types/config.ts`** - Zod schemas + inferred types:
```typescript
import { LLMConfigSchema, LLMProviderSchema } from '@browseros/shared/schemas/llm'
import { z } from 'zod'
// Single agent config
export const SingleAgentConfigSchema = LLMConfigSchema.extend({
type: z.literal('single'),
})
export type SingleAgentConfig = z.infer<typeof SingleAgentConfigSchema>
// Orchestrator-executor config
export const OrchestratorExecutorConfigSchema = z.object({
type: z.literal('orchestrator-executor'),
orchestrator: LLMConfigSchema.extend({
maxTurns: z.number().int().min(1).optional(),
}),
executor: LLMConfigSchema.extend({
maxStepsPerDelegation: z.number().int().min(1).optional(),
}),
})
export type OrchestratorExecutorConfig = z.infer<typeof OrchestratorExecutorConfigSchema>
// Discriminated union
export const AgentConfigSchema = z.discriminatedUnion('type', [
SingleAgentConfigSchema,
OrchestratorExecutorConfigSchema,
])
export type AgentConfig = z.infer<typeof AgentConfigSchema>
// Full eval config
export const EvalConfigSchema = z.object({
agent: AgentConfigSchema,
dataset: z.string().min(1),
output_dir: z.string().optional(),
num_workers: z.number().int().min(1).max(20).default(1),
browseros: z.object({
server_url: z.string().url(),
}),
grader_model: z.string().optional(),
grader_api_key_env: z.string().optional(),
grader_base_url: z.string().url().optional(),
timeout_ms: z.number().int().min(30000).max(3600000).optional(),
})
export type EvalConfig = z.infer<typeof EvalConfigSchema>
```
**`types/message.ts`** - Message types:
```typescript
import { z } from 'zod'
const BaseMessageSchema = z.object({
timestamp: z.string().datetime(),
})
export const UserMessageSchema = BaseMessageSchema.extend({
type: z.literal('user'),
content: z.string(),
})
export const AssistantMessageSchema = BaseMessageSchema.extend({
type: z.literal('assistant'),
content: z.string(),
})
export const ToolCallMessageSchema = BaseMessageSchema.extend({
type: z.literal('tool_call'),
tool: z.string(),
toolCallId: z.string(),
params: z.record(z.unknown()),
})
export const ToolResultMessageSchema = BaseMessageSchema.extend({
type: z.literal('tool_result'),
toolCallId: z.string(),
result: z.unknown(),
isError: z.boolean(),
screenshot: z.number().optional(),
})
export const ErrorMessageSchema = BaseMessageSchema.extend({
type: z.literal('error'),
content: z.string(),
errorCode: z.string().optional(),
})
// Orchestrator-specific messages
export const DelegationMessageSchema = BaseMessageSchema.extend({
type: z.literal('delegation'),
instruction: z.string(),
executorId: z.string(),
maxSteps: z.number().optional(),
})
export const DelegationResultMessageSchema = BaseMessageSchema.extend({
type: z.literal('delegation_result'),
executorId: z.string(),
summary: z.string(),
status: z.enum(['done', 'blocked', 'max_steps']),
stepsUsed: z.number(),
currentUrl: z.string().optional(),
})
export const MessageSchema = z.discriminatedUnion('type', [
UserMessageSchema,
AssistantMessageSchema,
ToolCallMessageSchema,
ToolResultMessageSchema,
ErrorMessageSchema,
DelegationMessageSchema,
DelegationResultMessageSchema,
])
export type Message = z.infer<typeof MessageSchema>
export type UserMessage = z.infer<typeof UserMessageSchema>
export type AssistantMessage = z.infer<typeof AssistantMessageSchema>
export type ToolCallMessage = z.infer<typeof ToolCallMessageSchema>
export type ToolResultMessage = z.infer<typeof ToolResultMessageSchema>
export type ErrorMessage = z.infer<typeof ErrorMessageSchema>
export type DelegationMessage = z.infer<typeof DelegationMessageSchema>
export type DelegationResultMessage = z.infer<typeof DelegationResultMessageSchema>
// Type guards
export const isToolCallMessage = (m: Message): m is ToolCallMessage => m.type === 'tool_call'
export const isDelegationMessage = (m: Message): m is DelegationMessage => m.type === 'delegation'
// ... etc
```
---
### 2. Agent Registry (`agents/registry.ts`)
```typescript
import type { AgentContext, AgentEvaluator } from './types'
type AgentFactory = (context: AgentContext) => AgentEvaluator
const registry = new Map<string, AgentFactory>()
/**
* Register an agent type
*/
export function registerAgent(type: string, factory: AgentFactory): void {
if (registry.has(type)) {
throw new Error(`Agent type "${type}" already registered`)
}
registry.set(type, factory)
}
/**
* Create agent evaluator from context
*/
export function createAgent(context: AgentContext): AgentEvaluator {
const factory = registry.get(context.config.agent.type)
if (!factory) {
const available = Array.from(registry.keys()).join(', ')
throw new Error(
`Unknown agent type: "${context.config.agent.type}". Available: ${available}`
)
}
return factory(context)
}
/**
* Get all registered agent types
*/
export function getRegisteredAgentTypes(): string[] {
return Array.from(registry.keys())
}
```
**`agents/index.ts`** - Auto-registration:
```typescript
import { registerAgent } from './registry'
import { SingleAgentEvaluator } from './single'
import { OrchestratorExecutorEvaluator } from './orchestrator-executor'
// Auto-register built-in agents
registerAgent('single', (ctx) => new SingleAgentEvaluator(ctx))
registerAgent('orchestrator-executor', (ctx) => new OrchestratorExecutorEvaluator(ctx))
// Re-exports
export { createAgent, registerAgent, getRegisteredAgentTypes } from './registry'
export type { AgentContext, AgentEvaluator, AgentResult } from './types'
```
---
### 3. Agent Context (`agents/types.ts`)
```typescript
import type { CaptureContext } from '../capture/types'
import type { EvalConfig, Task, TaskMetadata, Message } from '../types'
/**
* All dependencies an agent needs - passed to factory
*/
export interface AgentContext {
// Config
config: EvalConfig
task: Task
// Browser window
windowId: number
tabId: number
// Output
outputDir: string // Root output dir
taskOutputDir: string // Task-specific: outputDir/query_id/
// Capture infrastructure (pre-initialized)
capture: CaptureContext
}
/**
* Result returned by agent execution
*/
export interface AgentResult {
metadata: TaskMetadata
messages: Message[]
finalAnswer: string | null
}
/**
* Interface all agent evaluators must implement
*/
export interface AgentEvaluator {
/**
* Execute the agent on the task
*/
execute(): Promise<AgentResult>
}
```
---
### 4. Capture Context (`capture/context.ts`)
Bundle all capture utilities:
```typescript
import { randomUUID } from 'node:crypto'
import type { ToolExecutionHooks, ToolExecutionResult } from '@browseros/server/agent'
import type { Message, TaskError, EvalWarning, ErrorSource } from '../types'
import { MessageLogger } from './message-logger'
import { ScreenshotCapture } from './screenshot'
import { TrajectorySaver } from './trajectory-saver'
export interface CaptureContextConfig {
serverUrl: string
outputDir: string
taskId: string
tabId: number
windowId: number
}
/**
* Unified capture context - bundles screenshot, message logging, errors/warnings
*/
export class CaptureContext {
readonly screenshot: ScreenshotCapture
readonly messageLogger: MessageLogger
readonly trajectorySaver: TrajectorySaver
private errors: TaskError[] = []
private warnings: EvalWarning[] = []
private currentToolCallId: string | null = null
private readonly tabId: number
private readonly windowId: number
constructor(private config: CaptureContextConfig) {
this.tabId = config.tabId
this.windowId = config.windowId
this.trajectorySaver = new TrajectorySaver(config.outputDir, config.taskId)
}
/**
* Initialize - must be called before use
*/
async init(): Promise<string> {
const taskOutputDir = await this.trajectorySaver.init()
this.screenshot = new ScreenshotCapture(this.config.serverUrl, taskOutputDir)
await this.screenshot.init()
this.messageLogger = new MessageLogger(taskOutputDir)
return taskOutputDir
}
/**
* Create tool execution hooks for GeminiAgent
*/
createToolHooks(): ToolExecutionHooks {
return {
onBeforeToolCall: async (toolName: string, args: unknown) => {
try {
this.currentToolCallId = randomUUID()
await this.messageLogger.logToolCall(
toolName,
this.currentToolCallId,
args as Record<string, unknown>
)
} catch (err) {
this.addWarning('message_logging', `Failed to log tool call ${toolName}: ${err}`)
}
},
onAfterToolCall: async (toolName: string, result: ToolExecutionResult) => {
let screenshotNum = 0
// Capture screenshot
try {
screenshotNum = await this.screenshot.capture(this.tabId, this.windowId)
} catch (err) {
this.addWarning('screenshot', `Screenshot after ${toolName} failed: ${err}`)
screenshotNum = this.screenshot.getCount()
}
// Log tool errors
if (result.isError) {
this.addWarning('mcp_tool', `Tool ${toolName} error: ${result.errorMessage}`)
}
// Log result
if (this.currentToolCallId) {
try {
await this.messageLogger.logToolResult(
this.currentToolCallId,
result.isError ? { error: result.errorMessage } : result.parts,
result.isError,
screenshotNum
)
} catch (err) {
this.addWarning('message_logging', `Failed to log tool result: ${err}`)
}
}
this.currentToolCallId = null
},
}
}
// Error/warning collection
addError(source: ErrorSource, message: string, details?: Record<string, unknown>): void {
this.errors.push({ source, message, timestamp: new Date().toISOString(), details })
}
addWarning(source: ErrorSource, message: string): void {
this.warnings.push({ source, message, timestamp: new Date().toISOString() })
console.warn(`[${source}] ${message}`)
}
getErrors(): TaskError[] { return [...this.errors] }
getWarnings(): EvalWarning[] { return [...this.warnings] }
getMessages(): Message[] { return this.messageLogger.getMessages() }
getScreenshotCount(): number { return this.screenshot.getCount() }
getLastAssistantMessage(): string | null { return this.messageLogger.getLastAssistantMessage() }
// Delegation logging (for orchestrator-executor)
async logDelegation(instruction: string, executorId: string, maxSteps?: number): Promise<void> {
await this.messageLogger.logDelegation(instruction, executorId, maxSteps)
}
async logDelegationResult(
executorId: string,
summary: string,
status: 'done' | 'blocked' | 'max_steps',
stepsUsed: number,
currentUrl?: string
): Promise<void> {
await this.messageLogger.logDelegationResult(executorId, summary, status, stepsUsed, currentUrl)
}
}
```
---
### 5. Single Agent Evaluator (`agents/single/index.ts`)
Clean implementation using context:
```typescript
import { randomUUID } from 'node:crypto'
import { GeminiAgent } from '@browseros/server/agent'
import { AgentExecutionError } from '@browseros/server/agent/errors'
import type { ResolvedAgentConfig } from '@browseros/server/agent/types'
import { MCPServerConfig } from '@google/gemini-cli-core'
import type { AgentContext, AgentEvaluator, AgentResult } from '../types'
import type { SingleAgentConfig, TaskMetadata } from '../../types'
import { resolveEnvValue } from '../../utils/env'
const DEFAULT_TIMEOUT_MS = 15 * 60 * 1000
export class SingleAgentEvaluator implements AgentEvaluator {
constructor(private ctx: AgentContext) {}
async execute(): Promise<AgentResult> {
const startTime = Date.now()
const { config, task, capture } = this.ctx
const agentConfig = config.agent as SingleAgentConfig
const timeoutMs = config.timeout_ms ?? DEFAULT_TIMEOUT_MS
// Log initial user message
await capture.messageLogger.logUser(task.query)
// Set up timeout
const abortController = new AbortController()
const timeoutHandle = setTimeout(() => abortController.abort(), timeoutMs)
// Create agent
const resolvedConfig: ResolvedAgentConfig = {
conversationId: randomUUID(),
provider: agentConfig.provider,
model: agentConfig.model ?? 'gemini-2.0-flash',
apiKey: resolveEnvValue(agentConfig.apiKey),
baseUrl: agentConfig.baseUrl,
sessionExecutionDir: '/tmp/browseros-eval',
evalMode: true,
}
const mcpServers = {
'browseros-mcp': new MCPServerConfig(
undefined, undefined, undefined, undefined, undefined,
`${config.browseros.server_url}/mcp`,
{ Accept: 'application/json, text/event-stream', 'X-BrowserOS-Source': 'eval' },
undefined, undefined, true
),
}
const agent = await GeminiAgent.create(resolvedConfig, mcpServers)
// Set capture hooks
agent.setToolHooks(capture.createToolHooks())
// Create mock stream to capture assistant messages
let lastAssistantMessage = ''
const mockStream = {
write: async (data: string) => {
if (data.includes('"type":"text-delta"')) {
const match = data.match(/"delta":"((?:[^"\\]|\\.)*)"/)
if (match) lastAssistantMessage += JSON.parse(`"${match[1]}"`)
} else if (data.includes('"type":"finish"')) {
if (lastAssistantMessage) {
await capture.messageLogger.logAssistant(lastAssistantMessage)
lastAssistantMessage = ''
}
}
},
}
// Execute
let terminationReason: TaskMetadata['termination_reason'] = 'completed'
try {
await agent.execute(
task.query,
mockStream as Parameters<typeof agent.execute>[1],
abortController.signal,
{ windowId: this.ctx.windowId, activeTab: { id: this.ctx.tabId, url: task.start_url } }
)
} catch (err) {
const error = err instanceof Error ? err : new Error(String(err))
if (abortController.signal.aborted) {
terminationReason = 'timeout'
capture.addError('agent_execution', `Task timed out after ${timeoutMs / 1000}s`)
} else {
terminationReason = 'error'
const msg = err instanceof AgentExecutionError && err.originalError
? `${error.message}: ${err.originalError.message}`
: error.message
capture.addError('agent_execution', msg, { stack: error.stack })
}
await capture.messageLogger.logError(error.message)
} finally {
clearTimeout(timeoutHandle)
}
// Build metadata
const metadata: TaskMetadata = {
query_id: task.query_id,
dataset: task.dataset,
query: task.query,
started_at: new Date(startTime).toISOString(),
completed_at: new Date().toISOString(),
total_duration_ms: Date.now() - startTime,
total_steps: capture.getScreenshotCount(),
termination_reason: terminationReason,
final_answer: capture.getLastAssistantMessage(),
errors: capture.getErrors(),
warnings: capture.getWarnings(),
agent_config: { type: 'single', model: resolvedConfig.model },
grader_results: {},
}
await capture.trajectorySaver.saveMetadata(metadata)
return {
metadata,
messages: capture.getMessages(),
finalAnswer: metadata.final_answer,
}
}
}
```
---
### 6. Task Executor (`runner/task-executor.ts`)
Uses agent registry:
```typescript
import { createAgent } from '../agents'
import type { AgentContext } from '../agents/types'
import { CaptureContext } from '../capture/context'
import type { EvalConfig, Task } from '../types'
import type { WindowManager } from '../capture/window-manager'
export class TaskExecutor {
constructor(
private config: EvalConfig,
private outputDir: string,
private windowManager: WindowManager,
private graderOptions: GraderOptions | null,
) {}
async execute(task: Task): Promise<TaskResult> {
const startTime = Date.now()
let window: { windowId: number; tabId: number } | null = null
try {
// Create window
window = await this.windowManager.createWindow(task.query_id, task.start_url)
// Initialize capture context
const capture = new CaptureContext({
serverUrl: this.config.browseros.server_url,
outputDir: this.outputDir,
taskId: task.query_id,
tabId: window.tabId,
windowId: window.windowId,
})
const taskOutputDir = await capture.init()
// Build agent context
const context: AgentContext = {
config: this.config,
task,
windowId: window.windowId,
tabId: window.tabId,
outputDir: this.outputDir,
taskOutputDir,
capture,
}
// Create and execute agent (via registry)
const agent = createAgent(context)
const agentResult = await agent.execute()
// Run graders
const graderResults = await this.runGraders(task, agentResult)
return {
status: agentResult.metadata.termination_reason === 'timeout' ? 'timeout' : 'completed',
task,
agentResult,
graderResults,
durationMs: Date.now() - startTime,
}
} catch (error) {
return {
status: 'failed',
task,
error: error instanceof Error ? error : new Error(String(error)),
errorSource: 'unknown',
durationMs: Date.now() - startTime,
}
} finally {
if (window) {
await this.windowManager.closeWindow(task.query_id)
}
}
}
}
```
---
## Server Dependencies
### What We MUST Import from Server
These are necessary - `GeminiAgent` IS the agent:
```typescript
// Core agent
import { GeminiAgent, type ToolExecutionHooks, type ToolExecutionResult } from '@browseros/server/agent'
import { AgentExecutionError } from '@browseros/server/agent/errors'
import type { ResolvedAgentConfig } from '@browseros/server/agent/types'
// Provider adapter (for orchestrator-agent)
import { VercelAIContentGenerator } from '@browseros/server/agent/provider-adapter'
// Gateway client (for browseros provider only)
import { fetchBrowserOSConfig, getLLMConfigFromProvider } from '@browseros/server/lib/clients/gateway'
```
### What Could Move to Shared (Future)
If we want to decouple more:
```typescript
// These types could be in @browseros/shared
export interface ToolExecutionHooks { ... }
export interface ToolExecutionResult { ... }
export interface ResolvedAgentConfig { ... }
```
But for now, importing from server is fine - eval is tightly coupled to server anyway.
---
## Import Guidelines
```typescript
// Shared package - schemas, constants
import { LLMConfigSchema, LLMProviderSchema, LLM_PROVIDERS } from '@browseros/shared/schemas/llm'
import { TIMEOUTS } from '@browseros/shared/constants/timeouts'
import { AGENT_LIMITS } from '@browseros/shared/constants/limits'
import type { BrowserContext } from '@browseros/shared/schemas/browser-context'
// Server - only agent-related imports
import { GeminiAgent, type ToolExecutionHooks } from '@browseros/server/agent'
import type { ResolvedAgentConfig } from '@browseros/server/agent/types'
// Internal eval types - from types/ folder
import type { EvalConfig, Task, Message, AgentResult } from '../types'
import type { AgentContext, AgentEvaluator } from '../agents/types'
```
---
## Adding a New Agent Type
1. Create folder: `agents/my-new-agent/`
2. Implement `AgentEvaluator` interface:
```typescript
// agents/my-new-agent/index.ts
import type { AgentContext, AgentEvaluator, AgentResult } from '../types'
export class MyNewAgentEvaluator implements AgentEvaluator {
constructor(private ctx: AgentContext) {}
async execute(): Promise<AgentResult> {
const { config, task, capture } = this.ctx
// Use capture.createToolHooks() for screenshot/logging
// Use capture.messageLogger for messages
// Use capture.addError/addWarning for errors
// Return AgentResult
}
}
```
3. Register in `agents/index.ts`:
```typescript
import { MyNewAgentEvaluator } from './my-new-agent'
registerAgent('my-new-agent', (ctx) => new MyNewAgentEvaluator(ctx))
```
4. Add config schema in `types/config.ts`:
```typescript
export const MyNewAgentConfigSchema = z.object({
type: z.literal('my-new-agent'),
// ... specific fields
})
export const AgentConfigSchema = z.discriminatedUnion('type', [
SingleAgentConfigSchema,
OrchestratorExecutorConfigSchema,
MyNewAgentConfigSchema, // Add here
])
```
Done - no changes to runner code needed.
---
## Implementation Order
1. **Phase 1: Types** (~1 hour)
- Create `types/` folder with proper structure
- Move/consolidate all types
- Add Zod schemas for messages
2. **Phase 2: Capture Context** (~1 hour)
- Create `CaptureContext` class
- Add delegation message methods
- Create `createToolHooks()` utility
3. **Phase 3: Agent Registry** (~30 min)
- Create `registry.ts`
- Create `AgentContext` interface
- Update exports
4. **Phase 4: Refactor Single Agent** (~1 hour)
- Use `AgentContext`
- Use `CaptureContext`
- Clean up code
5. **Phase 5: Refactor Orchestrator-Executor** (~2 hours)
- Use `AgentContext`
- Integrate `CaptureContext`
- Wire up hooks properly
6. **Phase 6: Update Runner** (~30 min)
- Use `createAgent()` instead of if-else
- Initialize `CaptureContext` in executor
7. **Phase 7: Testing** (~1 hour)
- Run single-agent eval
- Run orchestrator-executor eval
- Verify screenshots/messages captured
---
## Summary
| Before | After |
|--------|-------|
| If-else agent creation | Registry + factory pattern |
| Duplicated capture code | Shared `CaptureContext` |
| Scattered types | Organized `types/` folder |
| Copy-paste hooks | `createToolHooks()` utility |
| Tight coupling | Clear interfaces |
| Hard to add agents | Register + implement |

View File

@@ -1,431 +0,0 @@
# Implementation Phases - Parallel Execution Plan
## Dependency Graph
```
Phase 1: Types (4 parallel subagents)
├──────────────────┬──────────────────┐
▼ ▼ │
Phase 2: Capture Phase 3: Agent │
(2 parallel) Registry │
│ (1 subagent) │
│ │ │
└────────┬─────────┘ │
▼ │
Phase 4: Agent Refactors │
(2 parallel - after 2+3) │
│ │
▼ │
Phase 5: Runner Update │
(1 subagent - after 4) │
│ │
▼ │
Phase 6: Cleanup & Test ◄─────────────────┘
(1 subagent)
```
---
## Phase 1: Types (4 Parallel Subagents)
No dependencies - can all run simultaneously.
### Subagent 1A: Config Types
```
Create /apps/eval/src/types/config.ts
Requirements:
1. Import LLMConfigSchema, LLMProviderSchema from @browseros/shared/schemas/llm
2. Import z from zod
Create Zod schemas:
- SingleAgentConfigSchema = LLMConfigSchema.extend({ type: z.literal('single') })
- OrchestratorExecutorConfigSchema with orchestrator + executor nested configs
- AgentConfigSchema = z.discriminatedUnion('type', [...])
- EvalConfigSchema with all fields (agent, dataset, output_dir, num_workers, browseros, grader_*, timeout_ms)
Export both schemas and inferred types (z.infer<>)
Reference: Current implementation in /apps/eval/src/utils/config-validator.ts (lines 1-42)
```
### Subagent 1B: Message Types
```
Create /apps/eval/src/types/message.ts
Requirements:
1. Use Zod for all schemas
2. Create BaseMessageSchema with timestamp field
Create schemas for:
- UserMessageSchema (type: 'user', content)
- AssistantMessageSchema (type: 'assistant', content)
- ToolCallMessageSchema (type: 'tool_call', tool, toolCallId, params)
- ToolResultMessageSchema (type: 'tool_result', toolCallId, result, isError, screenshot?)
- ErrorMessageSchema (type: 'error', content, errorCode?)
- DelegationMessageSchema (type: 'delegation', instruction, executorId, maxSteps?)
- DelegationResultMessageSchema (type: 'delegation_result', executorId, summary, status, stepsUsed, currentUrl?)
Create MessageSchema = z.discriminatedUnion('type', [...all schemas])
Export schemas, types, and type guards (isToolCallMessage, isDelegationMessage, etc.)
Reference: Current types in /apps/eval/src/types.ts (lines 62-127)
```
### Subagent 1C: Task & Result Types
```
Create /apps/eval/src/types/task.ts
Requirements:
1. Use Zod schemas with inferred types
Create:
- TaskMetadataSchema (original_task_id, website?, category?, additional?)
- TaskSchema (query_id, dataset, query, graders[], start_url?, setup_script?, metadata)
Export schemas and types.
---
Create /apps/eval/src/types/result.ts
Create:
- GraderResultSchema (score, pass, reasoning, details?)
- TaskMetadataSchema (query_id, dataset, query, started_at, completed_at, total_duration_ms, total_steps, termination_reason, final_answer, errors, warnings, agent_config, grader_results)
- AgentResultSchema (metadata, messages, finalAnswer)
Export schemas and types.
Reference: Current types in /apps/eval/src/types.ts (lines 6-20, 156-182)
```
### Subagent 1D: Error Types + Index
```
Create /apps/eval/src/types/errors.ts
Create:
- ErrorSourceSchema = z.enum(['window_creation', 'agent_execution', 'mcp_tool', 'screenshot', 'grader', 'message_logging', 'cleanup', 'unknown'])
- TaskErrorSchema (source, message, timestamp, details?)
- EvalWarningSchema (source, message, timestamp)
Export schemas and types.
---
Create /apps/eval/src/types/index.ts
Re-export everything from:
- ./config
- ./message
- ./task
- ./result
- ./errors
This becomes the single import point: import { EvalConfig, Message, Task } from '../types'
Reference: Current types in /apps/eval/src/types.ts (lines 129-154)
```
---
## Phase 2: Capture Infrastructure (2 Parallel Subagents)
**Depends on:** Phase 1 (types)
### Subagent 2A: CaptureContext Class
```
Create /apps/eval/src/capture/types.ts
Define interface:
- CaptureContextConfig { serverUrl, outputDir, taskId, tabId, windowId }
---
Create /apps/eval/src/capture/context.ts
Requirements:
1. Import ToolExecutionHooks, ToolExecutionResult from @browseros/server/agent
2. Import types from ../types
3. Import existing ScreenshotCapture, MessageLogger, TrajectorySaver
Implement CaptureContext class:
- Constructor takes CaptureContextConfig
- async init() - initializes screenshot, messageLogger, trajectorySaver, returns taskOutputDir
- createToolHooks(): ToolExecutionHooks - returns hooks for GeminiAgent
- addError(source, message, details?)
- addWarning(source, message)
- getErrors(), getWarnings(), getMessages(), getScreenshotCount(), getLastAssistantMessage()
- logDelegation(instruction, executorId, maxSteps?)
- logDelegationResult(executorId, summary, status, stepsUsed, currentUrl?)
Reference implementation details in DESIGN_DOC.md section "4. Capture Context"
Update /apps/eval/src/capture/index.ts to export CaptureContext
```
### Subagent 2B: MessageLogger Extensions
```
Update /apps/eval/src/capture/message-logger.ts
Add two new methods:
1. logDelegation(instruction: string, executorId: string, maxSteps?: number): Promise<void>
- Creates DelegationMessage with type: 'delegation'
- Appends to messages
2. logDelegationResult(executorId: string, summary: string, status: 'done' | 'blocked' | 'max_steps', stepsUsed: number, currentUrl?: string): Promise<void>
- Creates DelegationResultMessage with type: 'delegation_result'
- Appends to messages
Import DelegationMessage, DelegationResultMessage from ../types
Reference: Current MessageLogger in /apps/eval/src/capture/message-logger.ts
```
---
## Phase 3: Agent Registry (1 Subagent)
**Depends on:** Phase 1 (types)
**Can run parallel with:** Phase 2
### Subagent 3A: Agent Registry + Types
```
Create /apps/eval/src/agents/types.ts
Define:
- AgentContext interface:
{
config: EvalConfig
task: Task
windowId: number
tabId: number
outputDir: string
taskOutputDir: string
capture: CaptureContext
}
- AgentResult interface (re-export from ../types or define here)
- AgentEvaluator interface { execute(): Promise<AgentResult> }
---
Create /apps/eval/src/agents/registry.ts
Implement:
- type AgentFactory = (context: AgentContext) => AgentEvaluator
- const registry = new Map<string, AgentFactory>()
- registerAgent(type: string, factory: AgentFactory): void
- createAgent(context: AgentContext): AgentEvaluator
- getRegisteredAgentTypes(): string[]
Reference: DESIGN_DOC.md section "2. Agent Registry"
---
Update /apps/eval/src/agents/index.ts
- Import registerAgent from ./registry
- Import SingleAgentEvaluator (will be updated later)
- Import OrchestratorExecutorEvaluator (will be updated later)
- Call registerAgent for both
- Re-export createAgent, registerAgent, getRegisteredAgentTypes
- Re-export types
Note: Registration calls will fail initially until agents are refactored.
That's OK - add TODO comments for now.
```
---
## Phase 4: Agent Refactors (2 Parallel Subagents)
**Depends on:** Phase 2 + Phase 3
### Subagent 4A: Single Agent Refactor
```
Refactor /apps/eval/src/agents/single-agent.ts
Changes:
1. Change constructor to accept AgentContext instead of individual params:
constructor(private ctx: AgentContext) {}
2. Use ctx.capture instead of creating ScreenshotCapture/MessageLogger:
- Remove local ScreenshotCapture initialization
- Remove local MessageLogger initialization
- Remove local hooks setup
- Use ctx.capture.createToolHooks() for GeminiAgent hooks
- Use ctx.capture.messageLogger.logUser/logAssistant
- Use ctx.capture.addError/addWarning
- Use ctx.capture.getMessages(), getScreenshotCount(), etc.
3. Build metadata using capture methods
4. Remove TrajectorySaver init (done in CaptureContext)
5. Keep the core agent execution logic (GeminiAgent.create, agent.execute)
Reference:
- Current implementation: /apps/eval/src/agents/single-agent.ts
- Target implementation: DESIGN_DOC.md section "5. Single Agent Evaluator"
```
### Subagent 4B: Orchestrator-Executor Refactor
```
Refactor /apps/eval/src/agents/orchestrator-executor/index.ts
Changes:
1. Change OrchestratorExecutorEvaluator constructor to accept AgentContext:
constructor(private ctx: AgentContext) {}
2. Initialize capture from context (already done in runner)
3. Add hook integration:
- Create executor hooks that use ctx.capture.createToolHooks()
- Wire hooks through Orchestrator → ExecutorStore → Executor
- Call ctx.capture.logDelegation() when orchestrator delegates
- Call ctx.capture.logDelegationResult() when executor returns
4. Update return to include messages:
return {
metadata,
messages: ctx.capture.getMessages(), // Now populated!
finalAnswer,
}
Also update supporting files if needed:
- orchestrator.ts - add setExecutorHooks() method
- executor.ts - accept external hooks via setObservationHooks()
- executor-store.ts - pass hooks to new executors
Reference:
- Current: /apps/eval/src/agents/orchestrator-executor/index.ts
- Target: DESIGN_DOC.md and previous IMPLEMENTATION_PLAN.md
```
---
## Phase 5: Runner Update (1 Subagent)
**Depends on:** Phase 4
### Subagent 5A: Task Executor Update
```
Update /apps/eval/src/runner/task-executor.ts
Changes:
1. Import createAgent from ../agents instead of individual evaluators
2. Import CaptureContext from ../capture
3. In execute() method:
- Create CaptureContext and call init()
- Build AgentContext with all required fields
- Use createAgent(context) instead of if-else switch
- Remove the if (config.agent.type === 'single') / else if blocks
4. Remove direct imports of SingleAgentEvaluator, OrchestratorExecutorEvaluator
Before:
```typescript
if (this.config.agent.type === 'single') {
const evaluator = new SingleAgentEvaluator(this.config, task, window.windowId, ...)
} else if (this.config.agent.type === 'orchestrator-executor') {
const evaluator = new OrchestratorExecutorEvaluator(this.config, task, ...)
}
```
After:
```typescript
const capture = new CaptureContext({ serverUrl, outputDir, taskId, tabId, windowId })
const taskOutputDir = await capture.init()
const context: AgentContext = {
config: this.config,
task,
windowId: window.windowId,
tabId: window.tabId,
outputDir: this.outputDir,
taskOutputDir,
capture,
}
const agent = createAgent(context)
const agentResult = await agent.execute()
```
Reference:
- Current: /apps/eval/src/runner/task-executor.ts (lines 143-186)
- Target: DESIGN_DOC.md section "6. Task Executor"
```
---
## Phase 6: Cleanup & Test (1 Subagent)
**Depends on:** Phase 5
### Subagent 6A: Cleanup Old Files + Verify
```
Tasks:
1. Delete old /apps/eval/src/types.ts (replaced by types/ folder)
2. Update all imports across the codebase:
- Change: import { EvalConfig, Task, Message } from '../types'
- Keep same (types/index.ts re-exports everything)
3. Update /apps/eval/src/utils/config-validator.ts:
- Import schemas from ../types/config instead of defining locally
- Remove duplicate schema definitions
4. Verify no TypeScript errors:
- Run: cd apps/eval && bun run typecheck
5. Test single-agent eval:
- Run: cd apps/eval && bun run eval -c configs/webvoyager-test.json
- Verify screenshots captured
- Verify messages.jsonl populated
6. Test orchestrator-executor eval:
- Run: cd apps/eval && bun run eval -c configs/orchestrator-executor-test.json
- Verify screenshots captured
- Verify messages.jsonl has delegation messages
- Verify graders pass (no "no_screenshots" error)
Report any issues found.
```
---
## Execution Summary
| Phase | Subagents | Can Parallelize? | Dependencies |
|-------|-----------|------------------|--------------|
| 1 | 4 (1A, 1B, 1C, 1D) | Yes - all parallel | None |
| 2 | 2 (2A, 2B) | Yes - both parallel | Phase 1 |
| 3 | 1 (3A) | Yes - parallel with Phase 2 | Phase 1 |
| 4 | 2 (4A, 4B) | Yes - both parallel | Phase 2 + 3 |
| 5 | 1 (5A) | No | Phase 4 |
| 6 | 1 (6A) | No | Phase 5 |
**Total: 11 subagent tasks**
**Parallel execution timeline:**
```
Time →
─────────────────────────────────────────────────────────────────
Phase 1: [1A] [1B] [1C] [1D] (4 parallel)
─────────────────
Phase 2: [2A] [2B] (2 parallel)
Phase 3: [3A] (parallel with Phase 2)
───────────
Phase 4: [4A] [4B] (2 parallel)
──────────
Phase 5: [5A]
────
Phase 6: [6A]
────
```
**Maximum parallelism: 4 subagents** (Phase 1)

View File

@@ -1,888 +0,0 @@
# Eval System - Production Grade Implementation Plan
## Overview
This plan outlines the changes needed to make the eval system production-grade with uniform agent observation across all agent patterns (single-agent, orchestrator-executor, future patterns).
**Goal:** All agent evaluators produce consistent `AgentResult` with screenshots, message traces, and verifiable action sequences.
---
## Phase 1: Type System Extensions
### 1.1 Add New Message Types
**File:** `src/types.ts`
Add delegation-specific message types for orchestrator pattern:
```typescript
// After ErrorMessage definition (~line 99)
export interface DelegationMessage extends BaseMessage {
type: 'delegation'
instruction: string
executorId: string
maxSteps?: number
}
export interface DelegationResultMessage extends BaseMessage {
type: 'delegation_result'
executorId: string
summary: string
status: 'done' | 'blocked' | 'max_steps'
stepsUsed: number
currentUrl?: string
}
// Update Message union (~line 101)
export type Message =
| UserMessage
| AssistantMessage
| ToolCallMessage
| ToolResultMessage
| ErrorMessage
| DelegationMessage // NEW
| DelegationResultMessage // NEW
// Add type guards
export function isDelegationMessage(msg: Message): msg is DelegationMessage {
return msg.type === 'delegation'
}
export function isDelegationResultMessage(msg: Message): msg is DelegationResultMessage {
return msg.type === 'delegation_result'
}
```
### 1.2 Add Orchestrator Hook Types
**File:** `src/agents/orchestrator-executor/types.ts`
```typescript
// Add after existing types
export interface OrchestratorHooks {
onDelegation?: (instruction: string, executorId: string, maxSteps?: number) => Promise<void>
onDelegationResult?: (result: ExecutorResult) => Promise<void>
onTurnStart?: (turn: number) => Promise<void>
onTurnComplete?: (turn: number) => Promise<void>
onComplete?: (answer: string) => Promise<void>
onFailed?: (reason: string) => Promise<void>
}
export interface ExecutorObservationHooks {
onBeforeToolCall?: (toolName: string, args: unknown) => Promise<string> // returns toolCallId
onAfterToolCall?: (toolName: string, toolCallId: string, result: unknown, isError: boolean) => Promise<void>
}
```
---
## Phase 2: Unified Capture Infrastructure
### 2.1 Create EvalCapture Class
**File:** `src/capture/eval-capture.ts` (NEW)
```typescript
/**
* EvalCapture - Unified capture infrastructure for all agent evaluators
*
* Combines screenshot capture, message logging, and provides hooks for
* both single-agent and orchestrator-executor patterns.
*/
import { randomUUID } from 'node:crypto'
import type {
AssistantMessage,
DelegationMessage,
DelegationResultMessage,
ErrorMessage,
Message,
ToolCallMessage,
ToolResultMessage,
UserMessage,
} from '../types'
import { MessageLogger } from './message-logger'
import { ScreenshotCapture } from './screenshot'
export interface EvalCaptureConfig {
serverUrl: string
outputDir: string
tabId: number
windowId: number
}
export class EvalCapture {
private screenshotCapture: ScreenshotCapture
private messageLogger: MessageLogger
private tabId: number
private windowId: number
private currentToolCallId: string | null = null
constructor(config: EvalCaptureConfig) {
this.screenshotCapture = new ScreenshotCapture(config.serverUrl, config.outputDir)
this.messageLogger = new MessageLogger(config.outputDir)
this.tabId = config.tabId
this.windowId = config.windowId
}
async init(): Promise<void> {
await this.screenshotCapture.init()
}
// ============================================================================
// Screenshot Capture
// ============================================================================
async captureScreenshot(): Promise<number> {
return this.screenshotCapture.capture(this.tabId, this.windowId)
}
getScreenshotCount(): number {
return this.screenshotCapture.getCount()
}
// ============================================================================
// Message Logging - Basic Types
// ============================================================================
async logUser(content: string): Promise<void> {
await this.messageLogger.logUser(content)
}
async logAssistant(content: string): Promise<void> {
await this.messageLogger.logAssistant(content)
}
async logError(content: string, errorCode?: string): Promise<void> {
await this.messageLogger.logError(content, errorCode)
}
// ============================================================================
// Tool Call Logging (for single-agent and executor)
// ============================================================================
async logToolCall(tool: string, params: Record<string, unknown>): Promise<string> {
const toolCallId = randomUUID()
this.currentToolCallId = toolCallId
await this.messageLogger.logToolCall(tool, toolCallId, params)
return toolCallId
}
async logToolResult(
toolCallId: string,
result: unknown,
isError: boolean,
screenshot?: number,
): Promise<void> {
await this.messageLogger.logToolResult(toolCallId, result, isError, screenshot)
this.currentToolCallId = null
}
getCurrentToolCallId(): string | null {
return this.currentToolCallId
}
// ============================================================================
// Delegation Logging (for orchestrator-executor)
// ============================================================================
async logDelegation(
instruction: string,
executorId: string,
maxSteps?: number,
): Promise<void> {
const message: DelegationMessage = {
type: 'delegation',
timestamp: new Date().toISOString(),
instruction,
executorId,
...(maxSteps !== undefined && { maxSteps }),
}
// Extend MessageLogger to handle this, or append directly
await this.appendMessage(message)
}
async logDelegationResult(
executorId: string,
summary: string,
status: 'done' | 'blocked' | 'max_steps',
stepsUsed: number,
currentUrl?: string,
): Promise<void> {
const message: DelegationResultMessage = {
type: 'delegation_result',
timestamp: new Date().toISOString(),
executorId,
summary,
status,
stepsUsed,
...(currentUrl && { currentUrl }),
}
await this.appendMessage(message)
}
// ============================================================================
// Helpers
// ============================================================================
private async appendMessage(message: Message): Promise<void> {
// Access internal messages array and file
// This requires either extending MessageLogger or using a shared approach
const messages = this.messageLogger.getMessages()
messages.push(message)
// Write to file - MessageLogger needs extension for this
}
getMessages(): Message[] {
return this.messageLogger.getMessages()
}
getLastAssistantMessage(): string | null {
return this.messageLogger.getLastAssistantMessage()
}
}
```
### 2.2 Extend MessageLogger for New Types
**File:** `src/capture/message-logger.ts`
Add methods for delegation messages:
```typescript
// Add after logError method
async logDelegation(
instruction: string,
executorId: string,
maxSteps?: number,
): Promise<void> {
const message: DelegationMessage = {
type: 'delegation',
timestamp: new Date().toISOString(),
instruction,
executorId,
...(maxSteps !== undefined && { maxSteps }),
}
await this.append(message)
}
async logDelegationResult(
executorId: string,
summary: string,
status: 'done' | 'blocked' | 'max_steps',
stepsUsed: number,
currentUrl?: string,
): Promise<void> {
const message: DelegationResultMessage = {
type: 'delegation_result',
timestamp: new Date().toISOString(),
executorId,
summary,
status,
stepsUsed,
...(currentUrl && { currentUrl }),
}
await this.append(message)
}
```
---
## Phase 3: Executor Hook Integration
### 3.1 Modify Executor to Accept External Hooks
**File:** `src/agents/orchestrator-executor/executor.ts`
```typescript
// Add import
import type { ExecutorObservationHooks } from './types'
export class Executor {
private agent: GeminiAgent | null = null
private stepsUsed = 0
private currentUrl = ''
private config: ExecutorConfig
private serverUrl: string
private windowId: number
private tabId: number
private observationHooks?: ExecutorObservationHooks // NEW
// ... existing constructor ...
/**
* Set external observation hooks for capture integration
*/
setObservationHooks(hooks: ExecutorObservationHooks): void {
this.observationHooks = hooks
}
async execute(
instruction: string,
maxSteps?: number,
signal?: AbortSignal,
): Promise<Omit<ExecutorResult, 'executorId'>> {
// ... existing setup ...
// Track steps via hooks - MODIFIED to include external observation
let stepsThisRun = 0
const hooks: ToolExecutionHooks = {
onBeforeToolCall: async (toolName: string, args: unknown) => {
// Call external hook if set (for logging)
if (this.observationHooks?.onBeforeToolCall) {
await this.observationHooks.onBeforeToolCall(toolName, args)
}
},
onAfterToolCall: async (toolName: string, result: ToolExecutionResult) => {
stepsThisRun++
this.stepsUsed++
// Call external hook if set (for screenshot capture and logging)
if (this.observationHooks?.onAfterToolCall) {
const toolCallId = 'current' // Will be tracked by EvalCapture
await this.observationHooks.onAfterToolCall(
toolName,
toolCallId,
result.parts,
result.isError,
)
}
},
}
this.agent.setToolHooks(hooks)
// ... rest of execute method ...
}
}
```
### 3.2 Pass Hooks Through ExecutorStore
**File:** `src/agents/orchestrator-executor/executor-store.ts`
```typescript
import type { ExecutorObservationHooks } from './types'
export class ExecutorStore {
private executors = new Map<string, Executor>()
private observationHooks?: ExecutorObservationHooks // NEW
/**
* Set observation hooks that will be applied to all executors
*/
setObservationHooks(hooks: ExecutorObservationHooks): void {
this.observationHooks = hooks
// Apply to existing executors
for (const executor of this.executors.values()) {
executor.setObservationHooks(hooks)
}
}
getOrCreate(
id: string,
config: ExecutorConfig,
serverUrl: string,
windowId: number,
tabId: number,
): Executor {
if (!this.executors.has(id)) {
const executor = new Executor(config, serverUrl, windowId, tabId)
// Apply observation hooks to new executor
if (this.observationHooks) {
executor.setObservationHooks(this.observationHooks)
}
this.executors.set(id, executor)
}
return this.executors.get(id)!
}
// ... rest unchanged ...
}
```
---
## Phase 4: Orchestrator Hook Integration
### 4.1 Add Hooks to OrchestratorAgent
**File:** `src/agents/orchestrator-executor/orchestrator-agent.ts`
```typescript
import type { ExecutorObservationHooks, OrchestratorHooks } from './types'
export class OrchestratorAgent {
private orchestratorHooks?: OrchestratorHooks // NEW
private constructor(
private client: GeminiClient,
private geminiConfig: GeminiConfig,
private state: OrchestratorState,
private executorStore: ExecutorStore,
private maxTurns: number,
) {}
/**
* Set orchestrator-level hooks for delegation tracking
*/
setHooks(hooks: OrchestratorHooks): void {
this.orchestratorHooks = hooks
}
/**
* Set executor observation hooks (passed through to ExecutorStore)
*/
setExecutorObservationHooks(hooks: ExecutorObservationHooks): void {
this.executorStore.setObservationHooks(hooks)
}
/**
* Get hooks for tool context (used by orchestrator-tools.ts)
*/
getOrchestratorHooks(): OrchestratorHooks | undefined {
return this.orchestratorHooks
}
async run(taskQuery: string): Promise<OrchestratorAgentResult> {
let currentParts: Part[] = [{ text: taskQuery }]
let turns = 0
while (
!this.state.isComplete &&
!this.state.isFailed &&
turns < this.maxTurns
) {
turns++
// Fire turn start hook
await this.orchestratorHooks?.onTurnStart?.(turns)
// ... existing turn logic ...
// Fire turn complete hook
await this.orchestratorHooks?.onTurnComplete?.(turns)
}
// Fire completion hooks
if (this.state.isComplete && this.state.finalAnswer) {
await this.orchestratorHooks?.onComplete?.(this.state.finalAnswer)
} else if (this.state.isFailed && this.state.failureReason) {
await this.orchestratorHooks?.onFailed?.(this.state.failureReason)
}
return {
success: this.state.isComplete,
answer: this.state.finalAnswer,
reason: this.state.failureReason,
delegationCount: this.state.delegationCount,
totalExecutorSteps: this.state.totalExecutorSteps,
turns,
}
}
// ... rest unchanged ...
}
```
### 4.2 Fire Hooks in Orchestrator Tools
**File:** `src/agents/orchestrator-executor/orchestrator-tools.ts`
Modify the delegate tool handler to fire hooks:
```typescript
// In createOrchestratorTools function, modify the delegate tool handler
// Inside the delegate tool's handler:
handler: async (args) => {
const { instruction, executorId, maxSteps } = args as DelegateParams
// Fire delegation hook BEFORE execution
const hooks = context.getOrchestratorHooks?.()
const actualExecutorId = executorId ?? randomUUID()
await hooks?.onDelegation?.(instruction, actualExecutorId, maxSteps)
// Get or create executor
const executor = context.executorStore.getOrCreate(
actualExecutorId,
context.executorConfig,
context.serverUrl,
context.windowId,
context.tabId,
)
// Execute
const result = await executor.execute(instruction, maxSteps)
// Update state
context.state.delegationCount++
context.state.totalExecutorSteps += result.stepsUsed
// Fire delegation result hook AFTER execution
await hooks?.onDelegationResult?.({
...result,
executorId: actualExecutorId,
})
// Return result to orchestrator
return {
executorId: actualExecutorId,
...result,
}
}
```
---
## Phase 5: Update OrchestratorExecutorEvaluator
### 5.1 Full Integration
**File:** `src/agents/orchestrator-executor/index.ts`
```typescript
import { ScreenshotCapture } from '../../capture/screenshot'
import { MessageLogger } from '../../capture/message-logger'
import { TrajectorySaver } from '../../capture/trajectory-saver'
import type { ExecutorObservationHooks, OrchestratorHooks } from './types'
export class OrchestratorExecutorEvaluator implements AgentEvaluator {
constructor(
private config: EvalConfig,
private task: Task,
private windowId: number,
private tabId: number,
private outputDir: string,
) {}
async execute(): Promise<AgentResult> {
const startTime = Date.now()
const timeoutMs = this.config.timeout_ms ?? DEFAULT_TIMEOUT_MS
const errors: TaskError[] = []
const warnings: EvalWarning[] = []
const addError = (source: TaskError['source'], message: string, details?: Record<string, unknown>) => {
errors.push({ source, message, timestamp: new Date().toISOString(), details })
}
const addWarning = (source: EvalWarning['source'], message: string) => {
warnings.push({ source, message, timestamp: new Date().toISOString() })
console.warn(`[${source}] ${message}`)
}
// Initialize trajectory saver
const saver = new TrajectorySaver(this.outputDir, this.task.query_id)
const taskOutputDir = await saver.init()
// NEW: Initialize capture infrastructure (same as single-agent)
const screenshotCapture = new ScreenshotCapture(
this.config.browseros.server_url,
taskOutputDir,
)
await screenshotCapture.init()
const messageLogger = new MessageLogger(taskOutputDir)
// Log initial user message
await messageLogger.logUser(this.task.query)
// Validate config type
if (this.config.agent.type !== 'orchestrator-executor') {
throw new Error('OrchestratorExecutorEvaluator requires orchestrator-executor config')
}
const agentConfig = this.config.agent as OrchestratorExecutorConfig
const { orchestrator: orchestratorConfig, executor: executorConfig } =
resolveAgentConfig(agentConfig)
// Create orchestrator
const orchestrator = new Orchestrator(
orchestratorConfig,
executorConfig,
this.config.browseros.server_url,
this.windowId,
this.tabId,
)
// NEW: Set up executor observation hooks (for tool call/result capture)
let currentToolCallId: string | null = null
const executorHooks: ExecutorObservationHooks = {
onBeforeToolCall: async (toolName: string, args: unknown) => {
try {
currentToolCallId = randomUUID()
await messageLogger.logToolCall(toolName, currentToolCallId, args as Record<string, unknown>)
} catch (err) {
addWarning('message_logging', `Failed to log tool call ${toolName}: ${err instanceof Error ? err.message : String(err)}`)
}
return currentToolCallId
},
onAfterToolCall: async (toolName: string, _toolCallId: string, result: unknown, isError: boolean) => {
let screenshotNum = 0
// Capture screenshot after tool execution
try {
screenshotNum = await screenshotCapture.capture(this.tabId, this.windowId)
} catch (err) {
addWarning('screenshot', `Screenshot after ${toolName} failed: ${err instanceof Error ? err.message : String(err)}`)
screenshotNum = screenshotCapture.getCount()
}
// Log tool errors
if (isError) {
addWarning('mcp_tool', `Tool ${toolName} returned error`)
}
if (!currentToolCallId) {
addWarning('message_logging', 'Tool result without matching tool call')
return
}
try {
await messageLogger.logToolResult(currentToolCallId, result, isError, screenshotNum)
} catch (err) {
addWarning('message_logging', `Failed to log tool result: ${err instanceof Error ? err.message : String(err)}`)
}
currentToolCallId = null
},
}
// NEW: Set up orchestrator hooks (for delegation tracking)
const orchestratorHooks: OrchestratorHooks = {
onDelegation: async (instruction: string, executorId: string, maxSteps?: number) => {
try {
await messageLogger.logDelegation(instruction, executorId, maxSteps)
} catch (err) {
addWarning('message_logging', `Failed to log delegation: ${err instanceof Error ? err.message : String(err)}`)
}
},
onDelegationResult: async (result) => {
try {
await messageLogger.logDelegationResult(
result.executorId,
result.summary,
result.status,
result.stepsUsed,
result.currentUrl,
)
} catch (err) {
addWarning('message_logging', `Failed to log delegation result: ${err instanceof Error ? err.message : String(err)}`)
}
},
}
// Apply hooks to orchestrator
orchestrator.setHooks(orchestratorHooks)
orchestrator.setExecutorObservationHooks(executorHooks)
// Set up timeout
const abortController = new AbortController()
const timeoutHandle = setTimeout(() => {
abortController.abort()
}, timeoutMs)
let terminationReason: 'completed' | 'max_steps' | 'error' | 'timeout' = 'completed'
let finalAnswer: string | null = null
let orchestratorResult: Awaited<ReturnType<typeof orchestrator.run>> | null = null
try {
const runPromise = orchestrator.run(this.task.query)
orchestratorResult = await Promise.race([
runPromise,
new Promise<never>((_, reject) => {
abortController.signal.addEventListener('abort', () => {
reject(new Error('Timeout'))
})
}),
])
if (orchestratorResult.success) {
finalAnswer = orchestratorResult.answer
terminationReason = 'completed'
// Log final assistant message
if (finalAnswer) {
await messageLogger.logAssistant(finalAnswer)
}
} else {
terminationReason = 'error'
addError('agent_execution', orchestratorResult.reason ?? 'Unknown failure')
await messageLogger.logError(orchestratorResult.reason ?? 'Unknown failure')
}
} catch (err) {
const error = err instanceof Error ? err : new Error(String(err))
if (error.message === 'Timeout' || abortController.signal.aborted) {
terminationReason = 'timeout'
addError('agent_execution', `Task timed out after ${timeoutMs / 1000}s`)
} else {
terminationReason = 'error'
addError('agent_execution', error.message, { stack: error.stack })
}
await messageLogger.logError(error.message)
} finally {
clearTimeout(timeoutHandle)
orchestrator.getExecutorStore().clear()
}
const endTime = Date.now()
// Create metadata
const metadata: TaskMetadata = {
query_id: this.task.query_id,
dataset: this.task.dataset,
query: this.task.query,
started_at: new Date(startTime).toISOString(),
completed_at: new Date(endTime).toISOString(),
total_duration_ms: endTime - startTime,
total_steps: screenshotCapture.getCount(), // Now accurate
termination_reason: terminationReason,
final_answer: finalAnswer,
errors,
warnings,
agent_config: {
type: 'orchestrator-executor',
model: `${orchestratorConfig.model} / ${executorConfig.model}`,
},
grader_results: {},
}
await saver.saveMetadata(metadata)
return {
metadata,
messages: messageLogger.getMessages(), // NOW POPULATED
finalAnswer,
}
}
}
```
---
## Phase 6: Orchestrator Class Updates
### 6.1 Add Hook Passthrough Methods
**File:** `src/agents/orchestrator-executor/orchestrator.ts`
```typescript
import type { ExecutorObservationHooks, OrchestratorHooks } from './types'
export class Orchestrator {
private agent: OrchestratorAgent | null = null
private executorStore: ExecutorStore
private pendingOrchestratorHooks?: OrchestratorHooks
private pendingExecutorHooks?: ExecutorObservationHooks
constructor(
private orchestratorConfig: OrchestratorConfig,
private executorConfig: ExecutorConfig,
private serverUrl: string,
private windowId: number,
private tabId: number,
) {
this.executorStore = new ExecutorStore()
}
/**
* Set orchestrator-level hooks (must be called before run())
*/
setHooks(hooks: OrchestratorHooks): void {
this.pendingOrchestratorHooks = hooks
if (this.agent) {
this.agent.setHooks(hooks)
}
}
/**
* Set executor observation hooks (must be called before run())
*/
setExecutorObservationHooks(hooks: ExecutorObservationHooks): void {
this.pendingExecutorHooks = hooks
this.executorStore.setObservationHooks(hooks)
if (this.agent) {
this.agent.setExecutorObservationHooks(hooks)
}
}
async run(taskQuery: string): Promise<OrchestratorAgentResult> {
this.agent = await OrchestratorAgent.create(
this.orchestratorConfig,
this.executorConfig,
this.serverUrl,
this.windowId,
this.tabId,
)
// Apply pending hooks
if (this.pendingOrchestratorHooks) {
this.agent.setHooks(this.pendingOrchestratorHooks)
}
if (this.pendingExecutorHooks) {
this.agent.setExecutorObservationHooks(this.pendingExecutorHooks)
}
const result = await this.agent.run(taskQuery)
this.executorStore = this.agent.getExecutorStore()
return result
}
getExecutorStore(): ExecutorStore {
return this.agent?.getExecutorStore() ?? this.executorStore
}
}
```
---
## Implementation Order
1. **Phase 1** - Type extensions (types.ts) - 30 min
2. **Phase 2** - MessageLogger extensions - 30 min
3. **Phase 3** - Executor hook integration - 1 hour
4. **Phase 4** - OrchestratorAgent hooks - 1 hour
5. **Phase 5** - OrchestratorExecutorEvaluator update - 1.5 hours
6. **Phase 6** - Orchestrator passthrough - 30 min
7. **Testing** - End-to-end verification - 1 hour
**Total estimated time:** ~6 hours
---
## Testing Checklist
- [ ] Single-agent eval still works (regression test)
- [ ] Orchestrator-executor produces screenshots in output folder
- [ ] Orchestrator-executor produces messages.jsonl with:
- [ ] user message
- [ ] delegation messages
- [ ] tool_call messages (from executor)
- [ ] tool_result messages with screenshot numbers
- [ ] delegation_result messages
- [ ] assistant message (final answer)
- [ ] Graders pass with orchestrator-executor (no "no_screenshots" error)
- [ ] metadata.json has accurate `total_steps` count
- [ ] Error/warning capture works for both patterns
---
## Future Considerations
1. **New Agent Patterns:** Any new agent type just needs to:
- Accept hooks in constructor or via setter
- Fire hooks at appropriate points
- Use shared capture infrastructure
2. **Grader Updates:** May need to update graders to understand delegation messages
3. **Parallel Executors:** If orchestrator delegates to multiple executors in parallel, need to handle concurrent screenshot capture
4. **Memory/Performance:** Screenshot capture creates MCP connection per capture - consider connection pooling for high-volume evals

View File

@@ -2,155 +2,158 @@
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](../../../../LICENSE)
Evaluation framework for benchmarking BrowserOS browser automation agents. Runs tasks from standard datasets ([WebVoyager](https://arxiv.org/abs/2401.13919), [Mind2Web](https://arxiv.org/abs/2306.06070)), captures trajectories with screenshots, and grades results automatically.
Evaluation framework for BrowserOS browser automation agents. Runs tasks from standard datasets ([WebVoyager](https://arxiv.org/abs/2401.13919), [Mind2Web](https://arxiv.org/abs/2306.06070), AGI SDK / REAL Bench, WebArena-Infinity, WebBench), captures trajectories with screenshots, and grades results automatically.
## Prerequisites
- **BrowserOS binary** installed at `/Applications/BrowserOS.app` (macOS)
- **BrowserOS binary** at `/Applications/BrowserOS.app` (macOS) or `BROWSEROS_BINARY` pointing at it
- **Bun** runtime
- **API keys** for your chosen LLM provider and grader model
- **API keys** for your LLM provider (and `CLAUDE_CODE_OAUTH_TOKEN` if you use `performance_grader`)
## Quick Start
### 1. Set up environment
```bash
cd apps/eval
```
Edit `.env.development` and add your API keys:
```bash
# Pick ONE provider for the orchestrator (whichever you have access to)
OPENAI_API_KEY=sk-xxxxx
ANTHROPIC_API_KEY=sk-ant-xxxxx
FIREWORKS_API_KEY=fw_xxxxx
GOOGLE_API_KEY=AIza-xxxxx
# For grading results (OpenRouter recommended — gives access to many models)
OPENROUTER_API_KEY=sk-or-v1-xxxxx
```
### 2. Launch the dashboard
```bash
cp .env.example .env.development
# Edit .env.development with your keys, then:
bun run eval
```
Opens the **Eval Dashboard** at `http://localhost:9900` in config mode.
Opens the eval dashboard at `http://localhost:9900` in config mode. From there: load a preset, edit settings, click **Run**.
### 3. Configure and run
From the dashboard:
1. **Load a preset** — select from the dropdown or click **Load File** to import a config JSON
2. **Edit settings** — change agent type, provider, model, API keys, dataset, workers, timeouts
3. **Save Config** — export your configuration for reuse
4. **Click Run** — starts the evaluation with live progress
### Alternative: Run from CLI
### CLI mode
```bash
bun run eval -c configs/orchestrator-executor-clado-test.json
bun run eval -c configs/legacy/browseros-agent-weekly.json
bun run eval suite --config configs/legacy/browseros-agent-weekly.json --publish r2
```
Runs immediately. Dashboard still available at `http://localhost:9900` for live progress.
## Agent Types
The `suite` command is the workflow-compatible full loop: execute tasks, run graders, write artifacts, and optionally publish to R2. The old `-c` form remains supported during migration.
### Orchestrator-Executor with Clado
```bash
bun run eval run --config configs/legacy/browseros-agent-weekly.json
bun run eval suite --suite configs/suites/agisdk-daily-10.json --variant kimi-fireworks --publish r2
bun run eval grade --run results/browseros-agent-weekly/2026-04-29-1430
bun run eval publish --run results/browseros-agent-weekly/2026-04-29-1430 --target r2
```
The recommended architecture for visual model evals. Two tiers:
Config files live in two groups:
- **Orchestrator** — An LLM that plans and issues high-level instructions
- **Executor** — The **Clado Action** visual model that takes screenshots and predicts click/type/scroll coordinates
```txt
configs/legacy/ # Complete EvalConfig files used by older workflows and the dashboard
configs/suites/ # Suite definitions; model/provider comes from CLI flags or env
```
The orchestrator works with **any LLM provider**. Pick whichever you have access to:
Suite mode takes model settings from CLI flags first, then env:
#### OpenAI orchestrator
```bash
EVAL_VARIANT=kimi-fireworks \
EVAL_AGENT_PROVIDER=openai-compatible \
EVAL_AGENT_MODEL=accounts/fireworks/models/kimi-k2p5 \
EVAL_AGENT_API_KEY=$FIREWORKS_API_KEY \
EVAL_AGENT_BASE_URL=https://api.fireworks.ai/inference/v1 \
bun run eval suite --suite configs/suites/agisdk-daily-10.json --publish r2
```
### Suites and variants
A **suite** is what we run: the task dataset, graders, worker count, timeout, and browser settings. For example, `agisdk-daily-10` means "run these 10 AGI SDK tasks and grade them with `agisdk_state_diff`."
A **variant** is the model setup we are testing on that suite. `EVAL_VARIANT` is just the human-readable name for that setup. The actual model connection still comes from `EVAL_AGENT_PROVIDER`, `EVAL_AGENT_MODEL`, `EVAL_AGENT_API_KEY`, and `EVAL_AGENT_BASE_URL`.
This lets us run the same suite against multiple model setups without copying the benchmark config:
```txt
agisdk-daily-10 + kimi-fireworks
agisdk-daily-10 + claude-sonnet
agisdk-daily-10 + clado-action-000159
```
For `orchestrator-executor` suites, there can also be an executor model/backend. The `EVAL_AGENT_*` vars describe the main agent or orchestrator. The optional `EVAL_EXECUTOR_*` or `CLADO_ACTION_*` vars describe the delegated executor.
## Agent types
| Type | Description |
|------|-------------|
| `single` | Single LLM agent driven by the BrowserOS tool loop (CDP) |
| `orchestrator-executor` | High-level orchestrator + per-step executor (LLM or Clado visual model) |
### Single agent
```json
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "moonshotai/kimi-k2.5",
"apiKey": "OPENROUTER_API_KEY",
"baseUrl": "https://openrouter.ai/api/v1",
"supportsImages": true
}
}
```
### Orchestrator-Executor
The orchestrator works with any LLM provider. The executor can be another LLM, or the **Clado action** visual model that takes screenshots and predicts click/type/scroll coordinates.
```json
{
"agent": {
"type": "orchestrator-executor",
"orchestrator": {
"provider": "openai",
"model": "gpt-4o",
"apiKey": "OPENAI_API_KEY"
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1"
},
"executor": {
"provider": "clado-action",
"model": "qwen3-vl-30b-a3b-instruct",
"model": "Qwen3.5-35B-A3B-action-000159-merged",
"apiKey": "",
"baseUrl": "https://clado-ai--clado-browseros-action-actionmodel-generate.modal.run"
"baseUrl": "https://clado-ai--clado-browseros-action-000159-merged-actionmod-f4a6ef.modal.run"
}
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/oe-clado-openai",
"num_workers": 3,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"headless": true
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}
}
```
#### Anthropic orchestrator
## Graders
| Name | Description |
|------|-------------|
| `performance_grader` | Multi-axis grader running on Claude Agent SDK (uses its own credentials via `CLAUDE_CODE_OAUTH_TOKEN`) |
| `agisdk_state_diff` | AGI SDK / REAL Bench environment state-diff grader (deterministic) |
| `infinity_state` | WebArena-Infinity verifier-script grader (deterministic) |
Set `graders` in your config to override the per-task `graders` field from the dataset:
```json
"orchestrator": {
"provider": "anthropic",
"model": "claude-sonnet-4-20250514",
"apiKey": "ANTHROPIC_API_KEY"
}
"graders": ["performance_grader"]
```
#### Google orchestrator
```json
"orchestrator": {
"provider": "google",
"model": "gemini-2.0-flash",
"apiKey": "GOOGLE_API_KEY"
}
```
#### Fireworks orchestrator (OpenAI-compatible)
```json
"orchestrator": {
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1"
}
```
The executor config stays the same across all orchestrator providers — it always uses the Clado action model.
### Other Agent Types
| Type | Description | Example config |
|------|-------------|----------------|
| `single` | Single LLM agent via Gemini CLI + MCP | `webvoyager-test.json` |
| `tool-loop` | AI SDK tool loop, connects via CDP | `tool-loop-test.json` |
| `gemini-computer-use` | Google native computer use API | `gemini-computer-use.json` |
| `yutori-navigator` | Yutori N1 visual model | `yutori-navigator.json` |
## Configuration Reference
## Configuration reference
### API keys
The `apiKey` field supports two formats:
- **Env var name**: `"OPENAI_API_KEY"` — resolved from `.env.development` at runtime
- **Direct value**: `"sk-xxxxx"` — used as-is (not recommended, prefer env vars)
- **Direct value**: `"sk-xxxxx"` — used as-is (not recommended)
### Environment variables
| Variable | Used for |
|----------|----------|
| `EVAL_AGENT_PROVIDER`, `EVAL_AGENT_MODEL`, `EVAL_AGENT_API_KEY`, `EVAL_AGENT_BASE_URL`, `EVAL_AGENT_SUPPORTS_IMAGES` | Suite variant model selection |
| `FIREWORKS_API_KEY`, `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, provider-specific keys | Config-file or provider-backed model calls |
| `EVAL_EXECUTOR_MODEL`, `EVAL_EXECUTOR_API_KEY`, `EVAL_EXECUTOR_BASE_URL` | Suite-mode orchestrator executor override |
| `CLADO_ACTION_MODEL`, `CLADO_ACTION_API_KEY`, `CLADO_ACTION_BASE_URL` | Clado executor defaults |
| `BROWSEROS_BINARY` | BrowserOS binary path in CI/local smoke runs |
| `BROWSEROS_SERVER_URL` | Optional grader MCP URL override |
| `WEBARENA_INFINITY_DIR` | Local WebArena-Infinity checkout for Infinity tasks |
| `NOPECHA_API_KEY` | CAPTCHA solver extension |
| `EVAL_R2_ACCOUNT_ID`, `EVAL_R2_ACCESS_KEY_ID`, `EVAL_R2_SECRET_ACCESS_KEY`, `EVAL_R2_BUCKET`, `EVAL_R2_CDN_BASE_URL` | R2 upload and viewer URL |
### Supported providers
@@ -160,12 +163,26 @@ The `apiKey` field supports two formats:
| Anthropic | `anthropic` | No |
| Google | `google` | No |
| Azure OpenAI | `azure` | Yes |
| AWS Bedrock | `bedrock` | No (uses `region`, `accessKeyId`, `secretAccessKey`) |
| AWS Bedrock | `bedrock` | No |
| OpenRouter | `openrouter` | No |
| Fireworks, Together, etc. | `openai-compatible` | Yes |
| Ollama | `ollama` | No |
| Clado Action (executor only) | `clado-action` | Yes |
### R2 publishing
`suite --config ... --publish r2` and `publish --target r2` upload the run artifacts plus `viewer.html` to the viewer-compatible R2 layout:
```bash
export EVAL_R2_ACCOUNT_ID=...
export EVAL_R2_ACCESS_KEY_ID=...
export EVAL_R2_SECRET_ACCESS_KEY=...
export EVAL_R2_BUCKET=browseros-eval
export EVAL_R2_CDN_BASE_URL=https://eval.browseros.com
```
Published runs are available at `EVAL_R2_CDN_BASE_URL/viewer.html?run=<run-id>`.
### BrowserOS infrastructure
```json
@@ -179,34 +196,29 @@ The `apiKey` field supports two formats:
}
```
Each worker gets its own Chrome instance. Worker N uses `base_port + N` for CDP and server ports. `base_extension_port` is still reserved as a legacy BrowserOS launch argument for compatibility with Chromium builds that still pass it.
Each worker gets its own Chrome instance. Worker N uses `base_port + N` for CDP and server ports.
### Execution settings
| Field | Description | Default |
|-------|-------------|---------|
| `num_workers` | Parallel workers (each gets its own Chrome) | `1` |
| `timeout_ms` | Per-task timeout in ms | `900000` (15 min) |
| `timeout_ms` | Per-task timeout in ms | `1800000` (30 min) |
| `restart_server_per_task` | Restart Chrome between tasks (cleaner state, slower) | `false` |
### Grading
Results are auto-graded after each task. The grader uses an LLM judge.
| Field | Description |
|-------|-------------|
| `grader_model` | Model for grading (e.g., `openai/gpt-4.1`) |
| `grader_api_key_env` | Env var name for grader API key |
| `grader_base_url` | API endpoint (e.g., `https://openrouter.ai/api/v1`) |
## Datasets
| File | Tasks | Description |
|------|-------|-------------|
| `webvoyager_e2e_test.jsonl` | 10 | WebVoyager test subset (quick smoke test) |
| `agisdk-daily-10.jsonl` | 10 | Daily AGI SDK / REAL Bench subset |
| `webvoyager.jsonl` | 643 | Full WebVoyager benchmark |
| `mind2web_e2e_test.jsonl` | 10 | Mind2Web test subset |
| `mind2web.jsonl` | 300 | Full Mind2Web benchmark |
| `mind2web.jsonl` | 300 | Online-Mind2Web |
| `webbench-{0,1,2}of4-50.jsonl` | 50 each | WebBench shards (50-task subsets) |
| `agisdk-real-smoke.jsonl` | 1 | AGI SDK / REAL Bench smoke task |
| `agisdk-real.jsonl` | 36 | AGI SDK / REAL Bench (action-only tasks) |
| `webarena-infinity-hard-50.jsonl` | 50 | WebArena-Infinity hard set |
| `browsecomp-medium-hard-50.jsonl` | 50 | BrowseComp medium-hard |
| `browsecomp-very-hard-50.jsonl` | 50 | BrowseComp very-hard |
Task format (JSONL, one per line):
@@ -215,7 +227,7 @@ Task format (JSONL, one per line):
"query_id": "Amazon--0",
"dataset": "webvoyager",
"query": "Search an Xbox Wireless controller with green color and rated above 4 stars.",
"graders": ["webvoyager_grader", "fara_combined"],
"graders": ["performance_grader"],
"start_url": "https://www.amazon.com/",
"metadata": { "original_task_id": "Amazon--0", "website": "Amazon" }
}
@@ -227,24 +239,30 @@ Results are saved to `output_dir`:
```
results/
oe-clado-openai/
Amazon--0/
metadata.json # Task result, timing, grader scores
messages.jsonl # Full message log
screenshots/
001.png # Step-by-step screenshots
002.png
summary.json # Aggregate pass rates
browseros-agent-weekly/
2026-04-29-1430/
Amazon--0/
attempt.json # Stable attempt summary for viewer/reporting
metadata.json # Task result, timing, grader scores
grades.json # Compact grader results
messages.jsonl # Full message log
grader-artifacts/ # Grader-specific inputs/outputs/stderr
screenshots/
001.png # Step-by-step screenshots
002.png
summary.json # Aggregate pass rates
```
R2 publishing preserves the same task files under `runs/<run-id>/...`, writes `runs/<run-id>/manifest.json`, and uploads `viewer.html` at the bucket root. The viewer URL is `EVAL_R2_CDN_BASE_URL/viewer.html?run=<run-id>`.
## Troubleshooting
**BrowserOS not found**: Expects `/Applications/BrowserOS.app/Contents/MacOS/BrowserOS`. Make sure it's installed.
**BrowserOS not found**: Expects `/Applications/BrowserOS.app/Contents/MacOS/BrowserOS`. Set `BROWSEROS_BINARY` to override.
**Port conflicts**: Each worker uses `base_port + workerIndex`. 3 workers on base 9110 → ports 9110, 9111, 9112. Stop other BrowserOS instances first.
**API key not resolving**: If your config has `"apiKey": "OPENAI_API_KEY"`, ensure the env var is set in `.env.development`.
**Tasks timing out**: Increase `timeout_ms`. Default is 15 minutes; complex tasks may need 20+ minutes.
**Tasks timing out**: Increase `timeout_ms`. Default is 30 minutes.
**Headless vs headed**: Set `"headless": false` to watch Chrome in real-time. Useful for debugging.
**Headless vs headed**: Set `"headless": false` to watch Chrome in real time.

View File

@@ -1,18 +0,0 @@
{
"agent": {
"type": "single",
"provider": "openrouter",
"model": "openai/gpt-4o",
"apiKey": "OPENROUTER_API_KEY"
},
"dataset": "data/webvoyager_e2e_test.jsonl",
"output_dir": "results",
"num_workers": 5,
"browseros": {
"server_url": "http://127.0.0.1:9110"
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 300000
}

View File

@@ -0,0 +1,26 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "moonshotai/kimi-k2.5",
"apiKey": "OPENROUTER_API_KEY",
"baseUrl": "https://openrouter.ai/api/v1",
"supportsImages": true
},
"dataset": "../../data/agisdk-real-smoke.jsonl",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["agisdk_state_diff"],
"timeout_ms": 1800000
}

View File

@@ -7,8 +7,8 @@
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
},
"dataset": "../data/webbench-2of4-50.jsonl",
"num_workers": 10,
"dataset": "../../data/agisdk-real.jsonl",
"num_workers": 4,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
@@ -21,9 +21,6 @@
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"graders": ["agisdk_state_diff"],
"timeout_ms": 1800000
}

View File

@@ -0,0 +1,26 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "moonshotai/kimi-k2.5",
"apiKey": "OPENROUTER_API_KEY",
"baseUrl": "https://openrouter.ai/api/v1",
"supportsImages": true
},
"dataset": "../../data/webbench-2of4-50.jsonl",
"num_workers": 10,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"timeout_ms": 1800000
}

View File

@@ -14,7 +14,7 @@
"baseUrl": "https://api.fireworks.ai/inference/v1"
}
},
"dataset": "../data/webbench-2of4-50.jsonl",
"dataset": "../../data/webbench-2of4-50.jsonl",
"num_workers": 10,
"restart_server_per_task": true,
"browseros": {
@@ -29,8 +29,5 @@
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1800000
}

View File

@@ -9,12 +9,12 @@
},
"executor": {
"provider": "clado-action",
"model": "qwen3-vl-30b-a3b-instruct",
"model": "Qwen3.5-35B-A3B-action-000159-merged",
"apiKey": "",
"baseUrl": "https://clado-ai--clado-browseros-action-actionmodel-generate.modal.run"
"baseUrl": "https://clado-ai--clado-browseros-action-000159-merged-actionmod-f4a6ef.modal.run"
}
},
"dataset": "../data/webbench-2of4-50.jsonl",
"dataset": "../../data/agisdk-real.jsonl",
"num_workers": 10,
"restart_server_per_task": true,
"browseros": {
@@ -23,14 +23,11 @@
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
"headless": true
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"graders": ["agisdk_state_diff"],
"timeout_ms": 1800000
}

View File

@@ -0,0 +1,26 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "moonshotai/kimi-k2.5",
"apiKey": "OPENROUTER_API_KEY",
"baseUrl": "https://openrouter.ai/api/v1",
"supportsImages": true
},
"dataset": "../../data/webarena-infinity-hard-50.jsonl",
"num_workers": 10,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["infinity_state"],
"timeout_ms": 1800000
}

View File

@@ -5,7 +5,7 @@
"model": "openai/gpt-4.1",
"apiKey": "OPENROUTER_API_KEY"
},
"dataset": "../data/mind2web.jsonl",
"dataset": "../../data/mind2web.jsonl",
"num_workers": 5,
"restart_server_per_task": true,
"browseros": {
@@ -20,8 +20,5 @@
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 300000
}

View File

@@ -7,7 +7,7 @@
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
},
"dataset": "../data/webvoyager.jsonl",
"dataset": "../../data/webvoyager.jsonl",
"num_workers": 3,
"restart_server_per_task": true,
"browseros": {
@@ -22,8 +22,5 @@
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -0,0 +1,22 @@
{
"id": "agisdk-daily-10",
"dataset": "../../data/agisdk-daily-10.jsonl",
"agent": {
"type": "single"
},
"graders": ["agisdk_state_diff"],
"workers": 1,
"restartBrowserPerTask": true,
"timeoutMs": 1800000,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
}
}

View File

@@ -0,0 +1,22 @@
{
"id": "agisdk-real-smoke",
"dataset": "../../data/agisdk-real-smoke.jsonl",
"agent": {
"type": "single"
},
"graders": ["agisdk_state_diff"],
"workers": 1,
"restartBrowserPerTask": true,
"timeoutMs": 1800000,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
}
}

View File

@@ -0,0 +1,22 @@
{
"id": "agisdk-real",
"dataset": "../../data/agisdk-real.jsonl",
"agent": {
"type": "single"
},
"graders": ["agisdk_state_diff"],
"workers": 1,
"restartBrowserPerTask": true,
"timeoutMs": 1800000,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
}
}

View File

@@ -1,30 +0,0 @@
{
"agent": {
"type": "gemini-computer-use",
"apiKey": "GOOGLE_AI_API_KEY",
"screenSize": {
"width": 1440,
"height": 900
},
"turnLimit": 100
},
"dataset": "../data/test-set.jsonl",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -1,30 +0,0 @@
{
"agent": {
"type": "yutori-navigator",
"apiKey": "YUTORI_API_KEY",
"screenSize": {
"width": 1280,
"height": 800
},
"turnLimit": 100
},
"dataset": "../data/test-set.jsonl",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -0,0 +1,10 @@
{"query_id": "agisdk-dashdish-10", "dataset": "agisdk-real", "query": "Place an order from \"Souvla\" for a \"Medium Classic Cheeseburger\" and a \"Small Bacon Double Cheeseburger\" with \"Standard Delivery\" as the method with the default charged options.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-10", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}
{"query_id": "agisdk-fly-unified-5", "dataset": "agisdk-real", "query": "Find me the cheapest fare for a flight from Orlando to Milwaukee on December 5th, 2024 and book it.\nPassenger: John Doe\nDate of Birth: 01/01/1990\nSex: Male\nSeat Selection: No\nPayment: Credit Card (378342143523967), Exp: 12/30, Security Code: 420 Address: 123 Main St, San Francisco, CA, 94105, USA, Phone: 555-123-4567, Email: johndoe@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-5", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "United Airlines"}}}
{"query_id": "agisdk-udriver-10", "dataset": "agisdk-real", "query": "Order me a ride for 4pm, I'll be at the de Young muesum headed to the Waterbar, fanciest option possible please.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-10", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-udriver-9", "dataset": "agisdk-real", "query": "Book me a ride from the thai restaurant I last took a ride to for later today at 2pm, I'll be at 333 Apartments on Fremont", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-9", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-9", "challenge_type": "retrieval-action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-topwork-4", "dataset": "agisdk-real", "query": "Create a job post for a UI/UX Designer with expertise in Figma, Sketch, and Adobe Creative Suite, including project details, timeline, and required skills (Wireframing, Prototyping, Responsive Design).", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-4", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
{"query_id": "agisdk-gocalendar-4", "dataset": "agisdk-real", "query": "Change the \"Team Check-In\" event on July 18, 2024, name to \"Project Kickoff\" and update the location to \"Zoom\"", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-4", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Google Calendar"}}}
{"query_id": "agisdk-staynb-6", "dataset": "agisdk-real", "query": "Find and book the stay with the best value for money (cheapest stay with the best reviews) for 1 day. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-6", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-6", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Airbnb"}}}
{"query_id": "agisdk-udriver-11", "dataset": "agisdk-real", "query": "I need to go from Pacific Catch on Chestnut back home to 333 Fremont now. If the fancy version is within ten dollars of the regular one, book that.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-11", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-11", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-networkin-5", "dataset": "agisdk-real", "query": "Send a connection request to John Smith.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-5", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-5", "challenge_type": "action", "difficulty": "easy", "similar_to": "LinkedIn"}}}
{"query_id": "agisdk-zilloft-6", "dataset": "agisdk-real", "query": "Select a property listed in San Francisco as \"Condos\" within a price range under $300,000 and request a tour for tomorrow at 4:00 PM. Use these contact details: Name: Sarah Brown, Email: sarahbrown@example.com, Phone: 555-987-6543.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-zilloft.vercel.app", "metadata": {"original_task_id": "zilloft-6", "website": "Zilloft", "category": "agisdk-real", "additional": {"agisdk_task_id": "zilloft-6", "challenge_type": "action", "difficulty": "medium", "similar_to": "Zillow"}}}

View File

@@ -0,0 +1 @@
{"query_id": "agisdk-dashdish-10", "dataset": "agisdk-real", "query": "Place an order from \"Souvla\" for a \"Medium Classic Cheeseburger\" and a \"Small Bacon Double Cheeseburger\" with \"Standard Delivery\" as the method with the default charged options.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-10", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}

View File

@@ -0,0 +1,36 @@
{"query_id": "agisdk-dashdish-10", "dataset": "agisdk-real", "query": "Place an order from \"Souvla\" for a \"Medium Classic Cheeseburger\" and a \"Small Bacon Double Cheeseburger\" with \"Standard Delivery\" as the method with the default charged options.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-10", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}
{"query_id": "agisdk-fly-unified-5", "dataset": "agisdk-real", "query": "Find me the cheapest fare for a flight from Orlando to Milwaukee on December 5th, 2024 and book it.\nPassenger: John Doe\nDate of Birth: 01/01/1990\nSex: Male\nSeat Selection: No\nPayment: Credit Card (378342143523967), Exp: 12/30, Security Code: 420 Address: 123 Main St, San Francisco, CA, 94105, USA, Phone: 555-123-4567, Email: johndoe@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-5", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "United Airlines"}}}
{"query_id": "agisdk-udriver-10", "dataset": "agisdk-real", "query": "Order me a ride for 4pm, I'll be at the de Young muesum headed to the Waterbar, fanciest option possible please.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-10", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-udriver-9", "dataset": "agisdk-real", "query": "Book me a ride from the thai restaurant I last took a ride to for later today at 2pm, I'll be at 333 Apartments on Fremont", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-9", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-9", "challenge_type": "retrieval-action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-topwork-4", "dataset": "agisdk-real", "query": "Create a job post for a UI/UX Designer with expertise in Figma, Sketch, and Adobe Creative Suite, including project details, timeline, and required skills (Wireframing, Prototyping, Responsive Design).", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-4", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
{"query_id": "agisdk-gocalendar-4", "dataset": "agisdk-real", "query": "Change the \"Team Check-In\" event on July 18, 2024, name to \"Project Kickoff\" and update the location to \"Zoom\"", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-4", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Google Calendar"}}}
{"query_id": "agisdk-staynb-6", "dataset": "agisdk-real", "query": "Find and book the stay with the best value for money (cheapest stay with the best reviews) for 1 day. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-6", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-6", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Airbnb"}}}
{"query_id": "agisdk-udriver-11", "dataset": "agisdk-real", "query": "I need to go from Pacific Catch on Chestnut back home to 333 Fremont now. If the fancy version is within ten dollars of the regular one, book that.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-11", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-11", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-networkin-5", "dataset": "agisdk-real", "query": "Send a connection request to John Smith.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-5", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-5", "challenge_type": "action", "difficulty": "easy", "similar_to": "LinkedIn"}}}
{"query_id": "agisdk-zilloft-6", "dataset": "agisdk-real", "query": "Select a property listed in San Francisco as \"Condos\" within a price range under $300,000 and request a tour for tomorrow at 4:00 PM. Use these contact details: Name: Sarah Brown, Email: sarahbrown@example.com, Phone: 555-987-6543.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-zilloft.vercel.app", "metadata": {"original_task_id": "zilloft-6", "website": "Zilloft", "category": "agisdk-real", "additional": {"agisdk_task_id": "zilloft-6", "challenge_type": "action", "difficulty": "medium", "similar_to": "Zillow"}}}
{"query_id": "agisdk-topwork-2", "dataset": "agisdk-real", "query": "Create a job posting for a Backend Developer specializing in Python, Django, and Flask to develop a high-performance web application. Include project details such as required skills (PostgreSQL, Docker, AWS, CI/CD), estimated project timeline, and budget.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-2", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-2", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
{"query_id": "agisdk-gocalendar-3", "dataset": "agisdk-real", "query": "Delete the event titled \"Breakfast Meeting with Client\" scheduled for July 19, 2024", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-3", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-3", "challenge_type": "action", "difficulty": "easy", "similar_to": "Google Calendar"}}}
{"query_id": "agisdk-topwork-3", "dataset": "agisdk-real", "query": "Create a job listing for a Full-Stack Developer with expertise in Java, Spring Boot, and Angular, outlining the project scope, estimated duration, and required skills (MySQL, Docker, Kubernetes, and Jenkins). The ideal candidate should have experience in enterprise-level applications and building scalable microservices. After creating the job post, please describe what you included in the job listing.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-3", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-3", "challenge_type": "retrieval", "difficulty": "medium", "similar_to": "Upwork"}}}
{"query_id": "agisdk-dashdish-7", "dataset": "agisdk-real", "query": "Select \"Express Delivery\" for an order from \"DragonEats\" of \"Mushroom Swiss Burger\" and complete the checkout with the pre-loaded Visa card.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-7", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-7", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}
{"query_id": "agisdk-networkin-3", "dataset": "agisdk-real", "query": "Write a post inviting users to a networking event, including details about the event's purpose, date, and target audience.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-3", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-3", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
{"query_id": "agisdk-gomail-7", "dataset": "agisdk-real", "query": "Delete the email with the subject \"New Leadership Articles You Can't Miss\" from the Inbox.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-7", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-7", "challenge_type": "retrieval-action", "difficulty": "hard", "similar_to": "Gmail"}}}
{"query_id": "agisdk-opendining-8", "dataset": "agisdk-real", "query": "Identify and book the restaurant with the lowest rating. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-8", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-8", "challenge_type": "retrieval-action", "difficulty": "easy", "similar_to": "OpenTable"}}}
{"query_id": "agisdk-udriver-1", "dataset": "agisdk-real", "query": "Book a ride from Fitness Urbano to Pacific Cafe", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-1", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-1", "challenge_type": "action", "difficulty": "easy", "similar_to": "Uber"}}}
{"query_id": "agisdk-staynb-2", "dataset": "agisdk-real", "query": "Click on one of the stays displayed on the homepage and book it for a family of 4 (2 adults and 2 children). For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-2", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Airbnb"}}}
{"query_id": "agisdk-opendining-10", "dataset": "agisdk-real", "query": "Check the menus of all restaurants for vegetarian options and make a reservation at the one with the most vegetarian choices. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-10", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-10", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "OpenTable"}}}
{"query_id": "agisdk-opendining-4", "dataset": "agisdk-real", "query": "Use the search bar to search for a restaurant on September 2nd at 4:30 PM for 7 people, using \"Japanese\" as the search term, and book the first result. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-4", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-4", "challenge_type": "action", "difficulty": "hard", "similar_to": "OpenTable"}}}
{"query_id": "agisdk-dashdish-4", "dataset": "agisdk-real", "query": "Schedule a delivery order from \"Taco Bell\" adding a \"Classic Cheeseburger\" large size for later and add the note \"Leave at the front door\".", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-4", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Doordash"}}}
{"query_id": "agisdk-networkin-1", "dataset": "agisdk-real", "query": "Create a new text post for the feed with a professional update about AI trends in 2025, mentioning three key advancements and their impact on the job market.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-1", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-1", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
{"query_id": "agisdk-dashdish-5", "dataset": "agisdk-real", "query": "Add three \"Loaded Bacon Cheese Fries\" to the shopping cart from \"Man vs. Fries\". Proceed to checkout and select \"Pickup\" as the delivery method.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-5", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Doordash"}}}
{"query_id": "agisdk-opendining-5", "dataset": "agisdk-real", "query": "Scroll through the homepage carousel until \"Ocean Breeze\" is visible, select the second available time slot, and complete the reservation. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-5", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-5", "challenge_type": "action", "difficulty": "medium", "similar_to": "OpenTable"}}}
{"query_id": "agisdk-gocalendar-1", "dataset": "agisdk-real", "query": "Create a new event titled \"Team Meeting\" on July 19, 2024, from 2 PM to 2:30 PM, and include \"Conference Room A\" as the location", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-1", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-1", "challenge_type": "action", "difficulty": "medium", "similar_to": "Google Calendar"}}}
{"query_id": "agisdk-gomail-5", "dataset": "agisdk-real", "query": "Schedule an email to jane.doe@example.com with the subject \"Weekly Update\" to be sent next Monday at 9:00 AM.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-5", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Gmail"}}}
{"query_id": "agisdk-staynb-4", "dataset": "agisdk-real", "query": "Book a stay for 2 children with 1 adult. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-4", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Airbnb"}}}
{"query_id": "agisdk-dashdish-2", "dataset": "agisdk-real", "query": "Add a \"Medium Pepperoni Pizza\" from the restaurant \"Papa Johns Pizza\" to the shopping cart and purchase it.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-2", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Doordash"}}}
{"query_id": "agisdk-staynb-8", "dataset": "agisdk-real", "query": "Scroll through the homepage and book the last stay located in Paris.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-8", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-8", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Airbnb"}}}
{"query_id": "agisdk-gomail-2", "dataset": "agisdk-real", "query": "Mark the first email in the Inbox as \"read\".", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-2", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Gmail"}}}
{"query_id": "agisdk-networkin-10", "dataset": "agisdk-real", "query": "Generate a polite follow-up message for a previous unanswered chat, starting with \"Following up on\".", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-10", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-10", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
{"query_id": "agisdk-gomail-3", "dataset": "agisdk-real", "query": "Compose a new email to jonathan.smith@example.com with the subject \"Meeting Notes\" and body \"Please find the meeting notes attached.\"", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-3", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-3", "challenge_type": "action", "difficulty": "easy", "similar_to": "Gmail"}}}
{"query_id": "agisdk-udriver-6", "dataset": "agisdk-real", "query": "Me and 4 friends need a ride from the Palace Hotel to dinner at Osha Thai leaving now", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-6", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-6", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
{"query_id": "agisdk-zilloft-3", "dataset": "agisdk-real", "query": "Find a home in San Diego priced under $150,000 with at least 2 bedrooms and request a tour. Use these details: Contact Name: John Doe, Email: johndoe@example.com, Phone: 555-123-4567, Tour Time: 2:00 PM, Tour Date: First available.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-zilloft.vercel.app", "metadata": {"original_task_id": "zilloft-3", "website": "Zilloft", "category": "agisdk-real", "additional": {"agisdk_task_id": "zilloft-3", "challenge_type": "retrieval-action", "difficulty": "easy", "similar_to": "Zillow"}}}
{"query_id": "agisdk-fly-unified-6", "dataset": "agisdk-real", "query": "Reserve me a seat for the flight from Austin to Pittsburgh departing on December 11th, 2024 at 8:00 in Basic Economy.\nPassenger: Alice Brown\nDate of Birth: 05/20/1992\nSex: Female\nSeat Selection: Yes (Aisle seat)\nPayment: Credit Card (378342143523967), Exp: 09/27, security code: 332 Address: 789 Pine St, Los Angeles, CA, 90012, USA, Phone: 555-456-7890, Email: alicebrown@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-6", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-6", "challenge_type": "action", "difficulty": "medium", "similar_to": "United Airlines"}}}

View File

@@ -1,5 +0,0 @@
{"query_id": "CoordClick--1", "dataset": "coordinate-click", "query": "Click on circle A located at the top-left corner of the page.", "graders": ["webvoyager_grader"], "start_url": "http://localhost:3100", "metadata": {"original_task_id": "CoordClick--1", "website": "eval-target", "category": "coordinate-prediction", "additional": {"ground_truth": "Circle A is clicked and shows data-clicked=true", "answer_type": "golden"}}}
{"query_id": "CoordClick--2", "dataset": "coordinate-click", "query": "Click on circle B located at the top-right corner of the page.", "graders": ["webvoyager_grader"], "start_url": "http://localhost:3100", "metadata": {"original_task_id": "CoordClick--2", "website": "eval-target", "category": "coordinate-prediction", "additional": {"ground_truth": "Circle B is clicked and shows data-clicked=true", "answer_type": "golden"}}}
{"query_id": "CoordClick--3", "dataset": "coordinate-click", "query": "Click on circle C located at the bottom-left corner of the page.", "graders": ["webvoyager_grader"], "start_url": "http://localhost:3100", "metadata": {"original_task_id": "CoordClick--3", "website": "eval-target", "category": "coordinate-prediction", "additional": {"ground_truth": "Circle C is clicked and shows data-clicked=true", "answer_type": "golden"}}}
{"query_id": "CoordClick--4", "dataset": "coordinate-click", "query": "Click on circle D located at the bottom-right corner of the page.", "graders": ["webvoyager_grader"], "start_url": "http://localhost:3100", "metadata": {"original_task_id": "CoordClick--4", "website": "eval-target", "category": "coordinate-prediction", "additional": {"ground_truth": "Circle D is clicked and shows data-clicked=true", "answer_type": "golden"}}}
{"query_id": "CoordClick--5", "dataset": "coordinate-click", "query": "Click on all four circles A, B, C, and D on the page.", "graders": ["webvoyager_grader"], "start_url": "http://localhost:3100", "metadata": {"original_task_id": "CoordClick--5", "website": "eval-target", "category": "coordinate-prediction", "additional": {"ground_truth": "All four circles are clicked and page shows ALL TARGETS HIT", "answer_type": "golden"}}}

View File

@@ -0,0 +1,50 @@
{"query_id": "infinity-elation-prescriptions-task_h69", "dataset": "webarena-infinity", "query": "Approve all pending refill requests except for any medication that is involved in a major drug-drug interaction with another of the patient's active medications. Deny those with the reason 'Drug interaction \u2014 needs provider review before renewal'.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h69", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h69.py", "app_base_port": 8020}}}
{"query_id": "infinity-elation-clinical-records-task_h52", "dataset": "webarena-infinity", "query": "Add the document tag 'Provider-Reviewed' to every visit note template that was created by the current logged-in provider. Do not modify templates created by other providers.", "graders": ["infinity_state"], "start_url": "http://localhost:8000", "metadata": {"original_task_id": "elation-clinical-records-task_h52", "website": "elation-clinical-records", "category": "webarena-infinity", "additional": {"app_name": "elation-clinical-records", "difficulty": "hard", "verifier_path": "real-tasks/task_h52.py", "app_base_port": 8000}}}
{"query_id": "infinity-gmail-accounts-and-contacts-task_h44", "dataset": "webarena-infinity", "query": "Your sister's husband is one of your contacts. Find him, star his entry, and add the Friends label.", "graders": ["infinity_state"], "start_url": "http://localhost:8070", "metadata": {"original_task_id": "gmail-accounts-and-contacts-task_h44", "website": "gmail-accounts-and-contacts", "category": "webarena-infinity", "additional": {"app_name": "gmail-accounts-and-contacts", "difficulty": "hard", "verifier_path": "real-tasks/task_h44.py", "app_base_port": 8070}}}
{"query_id": "infinity-gmail-task_h2", "dataset": "webarena-infinity", "query": "Update the Datadog alerts filter to also archive matching emails and forward them to priya.sharma@cloudnine.dev instead of nate.patel@devops.tools.", "graders": ["infinity_state"], "start_url": "http://localhost:8060", "metadata": {"original_task_id": "gmail-task_h2", "website": "gmail", "category": "webarena-infinity", "additional": {"app_name": "gmail", "difficulty": "hard", "verifier_path": "real-tasks/task_h2.py", "app_base_port": 8060}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h58", "dataset": "webarena-infinity", "query": "The Performance Initiative epic has two child epics. For the child epic with more open issues, set the weight of every issue in it to 13. For the other child epic, close all its open issues.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h58", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h58.py", "app_base_port": 8050}}}
{"query_id": "infinity-figma-slides-task_h46", "dataset": "webarena-infinity", "query": "There are two slides with tables in the deck. Lock the table that compares competitors, and change the font size to 16 on the table that tracks quarterly feature adoption.", "graders": ["infinity_state"], "start_url": "http://localhost:8030", "metadata": {"original_task_id": "figma-slides-task_h46", "website": "figma-slides", "category": "webarena-infinity", "additional": {"app_name": "figma-slides", "difficulty": "hard", "verifier_path": "real-tasks/task_h46.py", "app_base_port": 8030}}}
{"query_id": "infinity-elation-prescriptions-task_h50", "dataset": "webarena-infinity", "query": "Deny the pending refill for the patient's cholesterol medication because his lipid panel is overdue. Then deny the Lisinopril refill as well \u2014 he needs a follow-up blood pressure check first.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h50", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h50.py", "app_base_port": 8020}}}
{"query_id": "infinity-elation-prescriptions-task_h19", "dataset": "webarena-infinity", "query": "Discontinue the Omeprazole and prescribe Famotidine 20mg tablet twice daily as a replacement for GERD \u2014 qty 60, 3 refills, send to CVS #4521.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h19", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h19.py", "app_base_port": 8020}}}
{"query_id": "infinity-paypal-my-wallet-task_h25", "dataset": "webarena-infinity", "query": "Convert all of my Australian dollars to euros.", "graders": ["infinity_state"], "start_url": "http://localhost:8100", "metadata": {"original_task_id": "paypal-my-wallet-task_h25", "website": "paypal-my-wallet", "category": "webarena-infinity", "additional": {"app_name": "paypal-my-wallet", "difficulty": "hard", "verifier_path": "real-tasks/task_h25.py", "app_base_port": 8100}}}
{"query_id": "infinity-elation-clinical-records-task_h66", "dataset": "webarena-infinity", "query": "Create a new template called 'Anxiety Management' with HPI and Assessment sections, and billing code 99213 with description 'Office visit, established, low complexity'. Then create a visit note for Emily Nakamura using that new template and the Telehealth category, add a Psychological Status block to the note, and sign it.", "graders": ["infinity_state"], "start_url": "http://localhost:8000", "metadata": {"original_task_id": "elation-clinical-records-task_h66", "website": "elation-clinical-records", "category": "webarena-infinity", "additional": {"app_name": "elation-clinical-records", "difficulty": "hard", "verifier_path": "real-tasks/task_h66.py", "app_base_port": 8000}}}
{"query_id": "infinity-elation-clinical-records-task_h62", "dataset": "webarena-infinity", "query": "Look up which template is assigned to the COVID Vaccine appointment type. Remove all its existing document tags and replace them with the single tag 'COVID-Protocol'. Then also assign that same template to the Urgent Same-Day appointment type.", "graders": ["infinity_state"], "start_url": "http://localhost:8000", "metadata": {"original_task_id": "elation-clinical-records-task_h62", "website": "elation-clinical-records", "category": "webarena-infinity", "additional": {"app_name": "elation-clinical-records", "difficulty": "hard", "verifier_path": "real-tasks/task_h62.py", "app_base_port": 8000}}}
{"query_id": "infinity-elation-prescriptions-task_h32", "dataset": "webarena-infinity", "query": "The patient has a medication that's being dispensed as written (brand name only). Discontinue that prescription and replace it with a new one \u2014 same medication, same sig, same pharmacy \u2014 but allow generic substitution this time. Qty 30, 3 refills, 30 days supply.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h32", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h32.py", "app_base_port": 8020}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h48", "dataset": "webarena-infinity", "query": "Add the 'breaking-change' label to every open issue in the API v3 Migration epic and remove any existing workflow-scoped labels from those issues.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h48", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h48.py", "app_base_port": 8050}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h77", "dataset": "webarena-infinity", "query": "Rename the 'UX' label to 'user-experience', change its type to 'group', and then add it to every open issue in the Frontend Modernization epic that doesn't already have it.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h77", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h77.py", "app_base_port": 8050}}}
{"query_id": "infinity-xero-invoicing-task_h15", "dataset": "webarena-infinity", "query": "Create a new invoice for Summit Health Group for an annual software license and 12 months of support with a 10% discount on support.", "graders": ["infinity_state"], "start_url": "http://localhost:8120", "metadata": {"original_task_id": "xero-invoicing-task_h15", "website": "xero-invoicing", "category": "webarena-infinity", "additional": {"app_name": "xero-invoicing", "difficulty": "hard", "verifier_path": "real-tasks/task_h15.py", "app_base_port": 8120}}}
{"query_id": "infinity-elation-clinical-records-task_h55", "dataset": "webarena-infinity", "query": "Resolve every problem across all patients in the system that currently has a status of Controlled.", "graders": ["infinity_state"], "start_url": "http://localhost:8000", "metadata": {"original_task_id": "elation-clinical-records-task_h55", "website": "elation-clinical-records", "category": "webarena-infinity", "additional": {"app_name": "elation-clinical-records", "difficulty": "hard", "verifier_path": "real-tasks/task_h55.py", "app_base_port": 8000}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h8", "dataset": "webarena-infinity", "query": "Create a confidential issue titled 'Emergency security patch' with priority::critical and the 'security' label, assigned to James O'Brien and Oliver Schmidt, with weight 2 in the Security Hardening milestone.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h8", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h8.py", "app_base_port": 8050}}}
{"query_id": "infinity-paypal-my-wallet-task_h20", "dataset": "webarena-infinity", "query": "Make a $200 payment on PayPal Credit and change autopay to pay the full balance.", "graders": ["infinity_state"], "start_url": "http://localhost:8100", "metadata": {"original_task_id": "paypal-my-wallet-task_h20", "website": "paypal-my-wallet", "category": "webarena-infinity", "additional": {"app_name": "paypal-my-wallet", "difficulty": "hard", "verifier_path": "real-tasks/task_h20.py", "app_base_port": 8100}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h52", "dataset": "webarena-infinity", "query": "Create a new board called 'Performance Tracker' with lists for the priority::critical, priority::high, and priority::medium labels. Then add the 'priority::high' label to every open issue in the v4.1 milestone that has the 'performance' label.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h52", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h52.py", "app_base_port": 8050}}}
{"query_id": "infinity-paypal-my-wallet-task_h80", "dataset": "webarena-infinity", "query": "Save all available Food & Drink offers, buy a $25 DoorDash gift card for yourself, and switch currency conversion to use my card issuer.", "graders": ["infinity_state"], "start_url": "http://localhost:8100", "metadata": {"original_task_id": "paypal-my-wallet-task_h80", "website": "paypal-my-wallet", "category": "webarena-infinity", "additional": {"app_name": "paypal-my-wallet", "difficulty": "hard", "verifier_path": "real-tasks/task_h80.py", "app_base_port": 8100}}}
{"query_id": "infinity-gmail-accounts-and-contacts-task_h50", "dataset": "webarena-infinity", "query": "Add the Emergency label to every contact who is currently listed as a delegate (active, pending, or expired). Then remove all delegates whose status is not 'active'.", "graders": ["infinity_state"], "start_url": "http://localhost:8070", "metadata": {"original_task_id": "gmail-accounts-and-contacts-task_h50", "website": "gmail-accounts-and-contacts", "category": "webarena-infinity", "additional": {"app_name": "gmail-accounts-and-contacts", "difficulty": "hard", "verifier_path": "real-tasks/task_h50.py", "app_base_port": 8070}}}
{"query_id": "infinity-elation-clinical-records-task_h14", "dataset": "webarena-infinity", "query": "Add the tag 'Flu-Season' to every patient whose primary provider is Dr. Sarah Chen.", "graders": ["infinity_state"], "start_url": "http://localhost:8000", "metadata": {"original_task_id": "elation-clinical-records-task_h14", "website": "elation-clinical-records", "category": "webarena-infinity", "additional": {"app_name": "elation-clinical-records", "difficulty": "hard", "verifier_path": "real-tasks/task_h14.py", "app_base_port": 8000}}}
{"query_id": "infinity-figma-text-and-typography-task_h7", "dataset": "webarena-infinity", "query": "Remove all list formatting from every layer.", "graders": ["infinity_state"], "start_url": "http://localhost:8040", "metadata": {"original_task_id": "figma-text-and-typography-task_h7", "website": "figma-text-and-typography", "category": "webarena-infinity", "additional": {"app_name": "figma-text-and-typography", "difficulty": "hard", "verifier_path": "real-tasks/task_h7.py", "app_base_port": 8040}}}
{"query_id": "infinity-paypal-my-wallet-task_h26", "dataset": "webarena-infinity", "query": "Send a $50 Amazon gift card to sarah.chen@email.com with 'Thank you!' as the message, and save the Amazon cashback offer.", "graders": ["infinity_state"], "start_url": "http://localhost:8100", "metadata": {"original_task_id": "paypal-my-wallet-task_h26", "website": "paypal-my-wallet", "category": "webarena-infinity", "additional": {"app_name": "paypal-my-wallet", "difficulty": "hard", "verifier_path": "real-tasks/task_h26.py", "app_base_port": 8100}}}
{"query_id": "infinity-handshake-career-exploration-task_h97", "dataset": "webarena-infinity", "query": "Find the single most helpful answer across all Q&A questions and mark it helpful. Then find the most-viewed question and submit your own answer to it.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h97", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h97.py", "app_base_port": 8080}}}
{"query_id": "infinity-figma-slides-task_h79", "dataset": "webarena-infinity", "query": "In the adoption table, find the feature with the highest Target Q4 percentage. In the competitive table, change DesignCraft's entry for that same feature to 'Market Leader'. Then update that feature's Target Q4 to '95%'.", "graders": ["infinity_state"], "start_url": "http://localhost:8030", "metadata": {"original_task_id": "figma-slides-task_h79", "website": "figma-slides", "category": "webarena-infinity", "additional": {"app_name": "figma-slides", "difficulty": "hard", "verifier_path": "real-tasks/task_h79.py", "app_base_port": 8030}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h41", "dataset": "webarena-infinity", "query": "For every open issue in the v4.2 - Security Hardening milestone: if it is already confidential, set its health status to 'at risk'. If it is not confidential, make it confidential and set its health status to 'needs attention'.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h41", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h41.py", "app_base_port": 8050}}}
{"query_id": "infinity-handshake-career-exploration-task_h90", "dataset": "webarena-infinity", "query": "A student in the feed mentioned attending the NSBE conference. That student also answered a Q&A question about diversity programs in tech. Submit your own answer to that same question sharing your experience, then bookmark that student's feed post.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h90", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h90.py", "app_base_port": 8080}}}
{"query_id": "infinity-elation-prescriptions-task_h30", "dataset": "webarena-infinity", "query": "The patient has three temporary medications. Discontinue the corticosteroid taper and the penicillin antibiotic \u2014 the patient completed both courses. Move the remaining temporary medication to permanent Rx.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h30", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h30.py", "app_base_port": 8020}}}
{"query_id": "infinity-linear-account-settings-task_h19", "dataset": "webarena-infinity", "query": "Turn off all desktop application settings: open in desktop app, notification badge, and spell check.", "graders": ["infinity_state"], "start_url": "http://localhost:8090", "metadata": {"original_task_id": "linear-account-settings-task_h19", "website": "linear-account-settings", "category": "webarena-infinity", "additional": {"app_name": "linear-account-settings", "difficulty": "hard", "verifier_path": "real-tasks/task_h19.py", "app_base_port": 8090}}}
{"query_id": "infinity-elation-prescriptions-task_h39", "dataset": "webarena-infinity", "query": "Change the default pharmacy to Express Scripts Mail Pharmacy for mail-order prescriptions. Then document that the patient takes Magnesium Citrate 400mg tablet as an OTC supplement \u2014 once daily at bedtime, 30-day supply.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h39", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h39.py", "app_base_port": 8020}}}
{"query_id": "infinity-handshake-career-exploration-task_h136", "dataset": "webarena-infinity", "query": "Your earliest completed appointment was a specific type. Schedule a follow-up appointment of the same category and type with the same staff member, for March 28, 2026 at 9:00 AM, in person.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h136", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h136.py", "app_base_port": 8080}}}
{"query_id": "infinity-handshake-career-exploration-task_h105", "dataset": "webarena-infinity", "query": "Find the second-most-viewed question in Q&A. It has two answers \u2014 mark the one with fewer helpful votes as helpful.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h105", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h105.py", "app_base_port": 8080}}}
{"query_id": "infinity-gmail-accounts-and-contacts-task_h22", "dataset": "webarena-infinity", "query": "The Engineering Manager at TechCorp is listed as one of your delegates. Remove her delegation and unstar her contact.", "graders": ["infinity_state"], "start_url": "http://localhost:8070", "metadata": {"original_task_id": "gmail-accounts-and-contacts-task_h22", "website": "gmail-accounts-and-contacts", "category": "webarena-infinity", "additional": {"app_name": "gmail-accounts-and-contacts", "difficulty": "hard", "verifier_path": "real-tasks/task_h22.py", "app_base_port": 8070}}}
{"query_id": "infinity-elation-patient-communication-task_h9", "dataset": "webarena-infinity", "query": "Acknowledge all unacknowledged reminders in the system.", "graders": ["infinity_state"], "start_url": "http://localhost:8010", "metadata": {"original_task_id": "elation-patient-communication-task_h9", "website": "elation-patient-communication", "category": "webarena-infinity", "additional": {"app_name": "elation-patient-communication", "difficulty": "hard", "verifier_path": "real-tasks/task_h9.py", "app_base_port": 8010}}}
{"query_id": "infinity-superhuman-general-task_h1", "dataset": "webarena-infinity", "query": "Label the FinancePlus partnership email and the QuantumLab prototype email as 'Clients'.", "graders": ["infinity_state"], "start_url": "http://localhost:8110", "metadata": {"original_task_id": "superhuman-general-task_h1", "website": "superhuman-general", "category": "webarena-infinity", "additional": {"app_name": "superhuman-general", "difficulty": "hard", "verifier_path": "real-tasks/task_h1.py", "app_base_port": 8110}}}
{"query_id": "infinity-xero-invoicing-task_h79", "dataset": "webarena-infinity", "query": "Change the invoice prefix to 'AUS-' and the next number to 100, then create a new invoice for CloudNine Analytics for 8 hours of UI/UX design work.", "graders": ["infinity_state"], "start_url": "http://localhost:8120", "metadata": {"original_task_id": "xero-invoicing-task_h79", "website": "xero-invoicing", "category": "webarena-infinity", "additional": {"app_name": "xero-invoicing", "difficulty": "hard", "verifier_path": "real-tasks/task_h79.py", "app_base_port": 8120}}}
{"query_id": "infinity-figma-slides-task_h16", "dataset": "webarena-infinity", "query": "Enable slide numbers on every slide using the 'with total' format and change the aspect ratio to 4:3.", "graders": ["infinity_state"], "start_url": "http://localhost:8030", "metadata": {"original_task_id": "figma-slides-task_h16", "website": "figma-slides", "category": "webarena-infinity", "additional": {"app_name": "figma-slides", "difficulty": "hard", "verifier_path": "real-tasks/task_h16.py", "app_base_port": 8030}}}
{"query_id": "infinity-linear-account-settings-task_h16", "dataset": "webarena-infinity", "query": "Revoke all API keys that have an expiration date.", "graders": ["infinity_state"], "start_url": "http://localhost:8090", "metadata": {"original_task_id": "linear-account-settings-task_h16", "website": "linear-account-settings", "category": "webarena-infinity", "additional": {"app_name": "linear-account-settings", "difficulty": "hard", "verifier_path": "real-tasks/task_h16.py", "app_base_port": 8090}}}
{"query_id": "infinity-elation-prescriptions-task_h2", "dataset": "webarena-infinity", "query": "Prescribe Buspirone 10mg for the patient's anxiety \u2014 once daily in the morning, qty 30, 5 refills. Send it to the same pharmacy that fills his Sertraline.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h2", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h2.py", "app_base_port": 8020}}}
{"query_id": "infinity-handshake-career-exploration-task_h1", "dataset": "webarena-infinity", "query": "Follow all consulting firms on Handshake.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h1", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h1.py", "app_base_port": 8080}}}
{"query_id": "infinity-handshake-career-exploration-task_h141", "dataset": "webarena-infinity", "query": "Some of your saved jobs are from employers you haven't followed yet. Find and follow each of those employers.", "graders": ["infinity_state"], "start_url": "http://localhost:8080", "metadata": {"original_task_id": "handshake-career-exploration-task_h141", "website": "handshake-career-exploration", "category": "webarena-infinity", "additional": {"app_name": "handshake-career-exploration", "difficulty": "hard", "verifier_path": "real-tasks/task_h141.py", "app_base_port": 8080}}}
{"query_id": "infinity-figma-text-and-typography-task_h74", "dataset": "webarena-infinity", "query": "Set the spelling language to Japanese, the big nudge amount to 50, and the default horizontal alignment to right.", "graders": ["infinity_state"], "start_url": "http://localhost:8040", "metadata": {"original_task_id": "figma-text-and-typography-task_h74", "website": "figma-text-and-typography", "category": "webarena-infinity", "additional": {"app_name": "figma-text-and-typography", "difficulty": "hard", "verifier_path": "real-tasks/task_h74.py", "app_base_port": 8040}}}
{"query_id": "infinity-elation-patient-communication-task_h63", "dataset": "webarena-infinity", "query": "Check the visit summaries to find the patient whose BNP level improved. Reply to their most recent message confirming they can resume light activity, then update their emergency contact's phone number to (650) 555-0001.", "graders": ["infinity_state"], "start_url": "http://localhost:8010", "metadata": {"original_task_id": "elation-patient-communication-task_h63", "website": "elation-patient-communication", "category": "webarena-infinity", "additional": {"app_name": "elation-patient-communication", "difficulty": "hard", "verifier_path": "real-tasks/task_h63.py", "app_base_port": 8010}}}
{"query_id": "infinity-elation-patient-communication-task_h14", "dataset": "webarena-infinity", "query": "Change Dr. Torres's notification timeframe to 'Do not notify me' and remove Dr. Torres from Dr. Chen's General Question routing.", "graders": ["infinity_state"], "start_url": "http://localhost:8010", "metadata": {"original_task_id": "elation-patient-communication-task_h14", "website": "elation-patient-communication", "category": "webarena-infinity", "additional": {"app_name": "elation-patient-communication", "difficulty": "hard", "verifier_path": "real-tasks/task_h14.py", "app_base_port": 8010}}}
{"query_id": "infinity-gitlab-plan-and-track-task_h67", "dataset": "webarena-infinity", "query": "Delete all time entries from the GraphQL gateway issue, add a single new entry of 16 hours with summary 'Complete rewrite estimate', and set its time estimate to 40 hours.", "graders": ["infinity_state"], "start_url": "http://localhost:8050", "metadata": {"original_task_id": "gitlab-plan-and-track-task_h67", "website": "gitlab-plan-and-track", "category": "webarena-infinity", "additional": {"app_name": "gitlab-plan-and-track", "difficulty": "hard", "verifier_path": "real-tasks/task_h67.py", "app_base_port": 8050}}}
{"query_id": "infinity-gmail-accounts-and-contacts-task_h73", "dataset": "webarena-infinity", "query": "Among the individual people in your other contacts (those with a first and last name), find the one who was saved most recently. Move them to your main contacts, set their company to 'Salesforce', job title to 'Account Executive', and add the Work label.", "graders": ["infinity_state"], "start_url": "http://localhost:8070", "metadata": {"original_task_id": "gmail-accounts-and-contacts-task_h73", "website": "gmail-accounts-and-contacts", "category": "webarena-infinity", "additional": {"app_name": "gmail-accounts-and-contacts", "difficulty": "hard", "verifier_path": "real-tasks/task_h73.py", "app_base_port": 8070}}}
{"query_id": "infinity-elation-prescriptions-task_h4", "dataset": "webarena-infinity", "query": "Run a medication reconciliation and mark the Calcium+D3 supplement for discontinuation during the review.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h4", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h4.py", "app_base_port": 8020}}}
{"query_id": "infinity-elation-prescriptions-task_h47", "dataset": "webarena-infinity", "query": "The patient's SSRI is currently dispensed at a different pharmacy than most of his other medications. Prescribe a refill of the same SSRI at the same dose and sig, but send it to CVS #4521 instead \u2014 qty 30, 5 refills, 30 days supply.", "graders": ["infinity_state"], "start_url": "http://localhost:8020", "metadata": {"original_task_id": "elation-prescriptions-task_h47", "website": "elation-prescriptions", "category": "webarena-infinity", "additional": {"app_name": "elation-prescriptions", "difficulty": "hard", "verifier_path": "real-tasks/task_h47.py", "app_base_port": 8020}}}
{"query_id": "infinity-paypal-my-wallet-task_h89", "dataset": "webarena-infinity", "query": "If your USD PayPal balance is above $2,500, convert $500 to Japanese Yen. If it is $2,500 or below, first add $500 from your Chase bank account, then convert $500 to JPY. Either way, set the debit card cash back category to Fuel.", "graders": ["infinity_state"], "start_url": "http://localhost:8100", "metadata": {"original_task_id": "paypal-my-wallet-task_h89", "website": "paypal-my-wallet", "category": "webarena-infinity", "additional": {"app_name": "paypal-my-wallet", "difficulty": "hard", "verifier_path": "real-tasks/task_h89.py", "app_base_port": 8100}}}

View File

@@ -1,147 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Coordinate Click Test</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
width: 100vw;
height: 100vh;
overflow: hidden;
background: #1a1a2e;
font-family: system-ui, -apple-system, sans-serif;
}
.circle {
position: fixed;
border-radius: 50%;
background: #e94560;
display: flex;
align-items: center;
justify-content: center;
color: #fff;
font-weight: 700;
cursor: pointer;
user-select: none;
transition: background 0.2s, transform 0.15s;
clip-path: circle(50%);
}
.circle:hover { transform: scale(1.08); }
.circle[data-clicked="true"] {
background: #0f3460;
pointer-events: none;
}
/* A — top-left area, large */
.circle-a {
width: 80px;
height: 80px;
font-size: 24px;
top: 15%;
left: 10%;
}
/* B — right side, upper-middle, medium */
.circle-b {
width: 50px;
height: 50px;
font-size: 18px;
top: 30%;
right: 18%;
}
/* C — center-left, lower area, small */
.circle-c {
width: 30px;
height: 30px;
font-size: 13px;
bottom: 25%;
left: 35%;
}
/* D — bottom-right area, very small */
.circle-d {
width: 16px;
height: 16px;
font-size: 9px;
bottom: 12%;
right: 30%;
}
#status {
position: fixed;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
color: #eee;
font-size: 16px;
text-align: center;
pointer-events: none;
}
#status .count {
font-size: 48px;
font-weight: 700;
color: #0f3460;
}
.success-flash {
animation: flash 0.4s ease-out;
}
@keyframes flash {
0% { background: #16c79a; transform: scale(1.3); }
100% { background: #0f3460; transform: scale(1); }
}
</style>
</head>
<body>
<div id="circle-a" class="circle circle-a" data-target="A" data-clicked="false">A</div>
<div id="circle-b" class="circle circle-b" data-target="B" data-clicked="false">B</div>
<div id="circle-c" class="circle circle-c" data-target="C" data-clicked="false">C</div>
<div id="circle-d" class="circle circle-d" data-target="D" data-clicked="false">D</div>
<div id="status">
<div class="count" id="clicked-count">0</div>
<div>of 4 targets clicked</div>
</div>
<script>
const circles = document.querySelectorAll('.circle')
const countEl = document.getElementById('clicked-count')
let clicked = 0
circles.forEach(circle => {
circle.addEventListener('click', (e) => {
if (circle.dataset.clicked === 'true') return
const rect = circle.getBoundingClientRect()
const centerX = rect.left + rect.width / 2
const centerY = rect.top + rect.height / 2
const radius = rect.width / 2
const dx = e.clientX - centerX
const dy = e.clientY - centerY
if (dx * dx + dy * dy > radius * radius) return
circle.dataset.clicked = 'true'
circle.classList.add('success-flash')
clicked++
countEl.textContent = clicked
if (clicked === 4) {
document.getElementById('status').innerHTML =
'<div class="count" style="color:#16c79a">ALL TARGETS HIT</div>' +
'<div>4 of 4 targets clicked</div>'
document.body.dataset.allClicked = 'true'
}
})
})
</script>
</body>
</html>

View File

@@ -1,16 +0,0 @@
const server = Bun.serve({
port: 3100,
async fetch(req) {
const url = new URL(req.url)
const path = url.pathname === '/' ? '/index.html' : url.pathname
const file = Bun.file(import.meta.dir + path)
if (await file.exists()) {
return new Response(file)
}
return new Response('Not Found', { status: 404 })
},
})
console.log(`Coordinate click test running at http://localhost:${server.port}`)

Some files were not shown because too many files have changed in this diff Show More