LLM/BrowserOS
1
0
Fork 0
mirror of https://github.com/browseros-ai/BrowserOS.git synced 2026-05-19 11:31:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,421 Commits 368 Branches 66 Tags
fix/eval-3
Commit Graph

2421 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nikhil Sonti
6956434b4f fix(eval): default agisdk smoke to openrouter 2026-04-29 13:38:21 -07:00
Nikhil Sonti
565aea10f3 fix(eval): split agisdk smoke and full configs 2026-04-29 13:34:47 -07:00
shivammittal274
df0f45dd29 Feat: eval debug dev ci (#869) 
* chore(eval): instrument server startup to root-cause dev CI health-check timeouts

Three diagnostics + one config swap to investigate why the eval-weekly
workflow has been failing on dev since 2026-04-25 with "Server health
check timed out" (every worker, every retry).

Background:
- Last successful weekly eval on dev: 2026-04-18 (sha f5a2b73)
- Since then, ~30 server commits landed including Lima/VM runtime,
  OpenClaw service, ACL system, ACP SDK — 108 server files changed,
  ~13K LOC added.
- Server process spawns cleanly in CI (PID logged) but never binds
  /health within the 30s eval-side timeout. Static analysis finds no
  obvious blocker; we need runtime evidence.

Changes:

1. apps/server/package.json — add `start:ci` script (no `--watch`).
   The default `start` uses `bun --watch` which forks a child process
   that watches every file in the import graph. Dev's graph is ~108
   files larger than main's; on a cold CI runner the watcher setup is a
   plausible source of multi-second startup overhead.

2. apps/eval/src/runner/browseros-app-manager.ts:
   - Use `start:ci` when `process.env.CI` is set (true on
     GitHub-hosted runners by default), else `start`.
   - Capture per-worker server stderr to /tmp/browseros-server-logs/
     instead of ignoring it. Without this we have no visibility into
     why the server is hung pre-/health.
   - Bump SERVER_HEALTH_TIMEOUT_MS 30s -> 90s. Dev's larger module
     graph may simply need more cold-start time on CI.

3. .github/workflows/eval-weekly.yml — upload the server logs dir as a
   workflow artifact (always, not just on success) so we can post-mortem
   any startup failure on the next run.

4. configs/agisdk-real-smoke.json — swap K2.5 from OpenRouter ->
   Fireworks (bypasses the OpenRouter per-key spend cap that has been
   eating recent runs) and drop num_workers 10 -> 4 (well below the
   Fireworks per-account TPM threshold that overwhelmed the original
   2026-04-23 run).

Plan: trigger the eval-weekly workflow on this branch with the agisdk
config and observe (a) whether it gets past server startup, and
(b) if it doesn't, what the captured server stderr says.

* fix(eval): capture stdout too — pino logger writes to stdout, not stderr

Previous diagnostic patch only redirected stderr; the captured per-worker
log files came back as 0 bytes because the server uses pino which writes
all log output to stdout (fd 1), not stderr (fd 2). Capture both into
the same file.

* fix(server): catch sync throw from OpenClaw constructor on Linux

The container runtime constructor in OpenClawService throws synchronously
on non-darwin platforms, e.g. GitHub Actions Linux runners. The existing
.catch() on tryAutoStart() only handles async throws inside auto-start —
the sync throw from configureOpenClawService(...) itself propagates up
through Application.start() and crashes the process via index.ts:48
(process.exit(EXIT_CODES.GENERAL_ERROR)).

This is what's been killing dev's eval-weekly CI: the server crashes in
milliseconds, the eval client polls /health, gets nothing, times out.

Fix: wrap the configureOpenClawService call in try/catch matching the
existing .catch() intent (best-effort, don't crash). Server continues
without OpenClaw on platforms where it can't initialize.

Verified by reading captured server stdout from run 25123195126:
  Failed to start server: error: browseros-vm currently supports macOS only
    at buildContainerRuntime (container-runtime-factory.ts:54:11)
    at new OpenClawService (openclaw-service.ts:652:15)
    at configureOpenClawService (openclaw-service.ts:1527:19)
    at start (main.ts:127:5)

* fix(server): defer OpenClaw chat client port lookup to request time

apps/server/src/api/server.ts:149 was calling getOpenClawService().getPort()
synchronously when constructing the OpenClawGatewayChatClient inside the
createHttpServer object literal. On non-darwin platforms this throws via
the OpenClawService constructor → buildContainerRuntime, escaping the
try/catch added in 5cf7b765 (which only protected the configureOpenClawService
call further down in main.ts).

Every other getOpenClawService() reference in server.ts is already wrapped
in an arrow function. This was the lone holdout. Make it lazy too: change
the chat client constructor to take getHostPort: () => number instead of
hostPort: number, evaluate it inside streamTurn at request time. Behavior
on darwin is unchanged.

This unblocks dev's eval-weekly CI on Linux runners where OpenClaw isn't
available — the chat endpoint isn't exercised by the eval, so a deferred
throw is acceptable.

* fix(server): allow Linux to skip OpenClaw via BROWSEROS_SKIP_OPENCLAW=1

Earlier surgical fixes (try/catch in main.ts, lazy chat client port) didn't
unblock dev's Linux CI — same throw kept reproducing. Whether this is bun
caching stale stack frames or a missed eager call site, the safer move is
to fix it at the root: make buildContainerRuntime never throw on Linux
when the runner has explicitly opted out.

Adds BROWSEROS_SKIP_OPENCLAW env check alongside the existing NODE_ENV=test
escape hatch in container-runtime-factory.ts. When set, returns the existing
UnsupportedPlatformTestRuntime stub — server boots normally, /health binds,
any actual OpenClaw API call still fails loudly at request time.

eval-weekly.yml sets the flag for the Linux runner. Darwin behavior and
non-CI Linux behavior unchanged (without the flag they still throw).

* feat(eval): align Clado action executor with new endpoint contract

David Shan shared the updated Clado BrowserOS Action Model spec.
Changes to match it:

- Bump endpoint URL + model id to the 000159-merged checkpoint
  (clado-ai--clado-browseros-action-000159-merged-actionmod-f4a6ef)
  in browseros-oe-clado-weekly.json and the README example.
- CLADO_REQUEST_TIMEOUT_MS 120s → 360s. Cold start can take ~5 min;
  the 2-min ceiling was failing every cold-start request.
- Treat HTTP 200 with action=null / parse_error as an INVALID step
  instead of aborting the executor loop. The model can self-correct
  on the next call. Cap consecutive parse failures at 3 to avoid
  infinite loops.
- Capture final_answer from end actions. Surface it in the observation
  back to the orchestrator so its task answer can use the model's
  declared result.
- Add macOS Cmd-* key mappings (M-a, M-c, M-v, M-x → Meta+A/C/V/X).
- Switch screenshot format from webp → png to match the documented
  "PNG or JPEG" contract.

* chore(eval): refresh test-clado-api script for new Clado contract

Updated the local smoke-test to match the new Clado endpoint and
response contract:

- New action + health URLs (000159-merged checkpoint).
- Drop the grounding-model branch (orchestrator-executor doesn't
  use it; the README David shared only documents the action model).
- Health-check waits up to 6 minutes for cold start with a 30s
  warning so the operator knows it's spinning up.
- Print every documented response field (action, x/y, text, key,
  direction, amount, drag start/end, time, final_answer, thinking,
  parse_error, inference_time_seconds).
- Three-step run that exercises a click, a typing continuation
  with formatted history, and an end+final_answer probe.

* chore(eval): point clado weekly config at agisdk-real

Switches the orchestrator-executor + Clado weekly config to run on the
AGI SDK / REAL Bench task set with the deterministic agisdk_state_diff
grader. Matches the orchestrator-executor smoke target (Fireworks K2.5
orchestrator + Clado action executor) we want to track week-over-week.

* chore(eval): run clado weekly headless

Default to headless so the weekly job (and local repros) don't pop ten
visible Chrome windows. Set headless=false locally if you need to watch
a worker.

* fix(eval): address Greptile P1+P2 on server log fd handling

P1: openSync was outside the mkdirSync try/catch, so a swallowed mkdir
failure (e.g. unwritable custom BROWSEROS_SERVER_LOG_DIR) would leave the
log directory missing and crash the server spawn with ENOENT. Move openSync
into the same try block; fall back to /dev/null so spawn always succeeds.

P2: the log fd was opened on every server start but never closed. Each
restart attempt leaked one fd across all workers — over a long eval run
that could exhaust the process fd limit. Track the fd on the manager and
closeSync it in killApp() right after the server process exits (the child's
dup keeps the file open until it exits, so we don't truncate output).
 2026-04-30 01:33:49 +05:30
Nikhil
edfc5c751c fix: align OpenClaw gateway image with VM cache (#868) 
* fix: load OpenClaw gateway image from VM cache

* fix: use container port for OpenClaw ACP bridge

* fix: address review feedback for PR #868
 2026-04-29 12:11:00 -07:00
Nikhil
471256f31c fix: stop passing native permission flags to ACP adapters (#867) 2026-04-29 11:07:51 -07:00
Nikhil
4c90ca696b fix(agents): connect OpenClaw ACP inside gateway container (#866) 2026-04-29 11:07:29 -07:00
Nikhil
f2ac87d7c3 feat: show created agents in sidepanel (#865) 
* feat(agent): list created agents in sidepanel target catalog

* feat(agent): show created agents in sidepanel selector

* feat(server): add sidepanel chat route for created agents

* feat(agent): route sidepanel agent sends by agent id

* chore(agent): retire virtual sidepanel acp targets

* fix: address review feedback for PR #865
 2026-04-29 10:15:58 -07:00
shivammittal274
231bd6821d fix(eval): pin agisdk version + exclude 4 invalid tasks (Phase 2 dataset hygiene) (#844) 
* chore(eval): pin agisdk version to prevent silent dataset drift

`pip install agisdk` previously fetched whatever version pip resolved at
CI time. If agisdk publishes a new version with changed task definitions
or grader behavior, the weekly eval silently shifts under our feet —
making "did the score move because of code or data?" unanswerable.

Pin to agisdk==0.3.5 (the version we currently develop against). Bump
intentionally with a documented re-baseline run.

* fix(eval): exclude 4 more tasks identified by 8-trial never-passing audit

After 8 trials across K2.5 + Opus 4.6 (Phase 1 and Phase 2), 5 tasks
never passed. Per-task root-cause investigation via parallel deep-dive
subagents flagged 4 of them as fundamentally unfixable in the eval
pipeline as it stands; the 5th (`dashdish-5`) is a prompt-rule fix
that stays in.

- gocalendar-7: goal/grader contradiction. Goal says "move event to
  July 19, 10 AM"; grader expects `eventsDiff.updated.*.start ==
  "2024-07-18T17:00Z"` (= July 18, 10 AM PDT — same day, 1 hour shift).
  Even after the Phase 2 HTML5 dnd dispatch fix correctly populates
  `eventsDiff.updated`, the values are July 19 (matching the goal),
  which the grader rejects.

- staynb-5: grader hardcodes literal `'Oct 13 2025'` and `'Oct 23 2025'`
  year strings. The staynb date picker interprets bare "Oct 13" as the
  most-recent-past instance (currently 2024 since today is 2026), not
  2025. No agent can produce a persisted date string containing 2025.

- staynb-9: under-specified task. Goal says "maximum number of guests
  supported"; grader requires the very specific string "32 Guests, 16
  Infants" — encoding UI knowledge (Adults+Children=Guests display,
  Infants render separately, per-category cap=16, Pets excluded) that
  isn't in the prompt. Even Opus 4.6 stopped at 16 across 3 trials.

- opendining-3: grader requires `contains(booking.date, '2024-07-20')`
  but the React-controlled date textbox flakily no-ops on `fill`. 3/8
  trial pass rate is essentially coin-flip noise driven by tool-fidelity
  variance rather than agent capability. Removing to reduce score noise;
  Phase 2 fill post-validate warning helps when it does work, but the
  task's signal-to-noise is too low for the eval set.

Dataset goes from 40 -> 36 tasks. Total EXCLUDED_TASKS now 11 entries.

Validated by 8-trial pass-record audit; deep-dive notes saved to
plans/audits/.
 2026-04-29 22:07:53 +05:30
Dani Akash
a228c278c6 feat(agents): background-resilient chat — turns survive tab disconnect (#863) 
* feat(agents): decouple chat turn lifecycle from SSE response

Introduce a per-process ActiveTurnRegistry that owns each agent turn's
lifecycle and a ring-buffered event stream, so chat tabs that close,
refresh, or navigate away no longer cancel the in-flight turn. New
endpoints:

  POST   /agents/:id/chat          starts a turn (now returns 409 when
                                   one is already running, with the
                                   active turnId for attaching)
  GET    /agents/:id/chat/active   reports the running turn for a UI
                                   that just mounted
  GET    /agents/:id/chat/stream   subscribes to a turn; supports
                                   Last-Event-ID resume via per-event
                                   seq ids
  POST   /agents/:id/chat/cancel   explicit cancel — fetch abort no
                                   longer affects the underlying turn

The chat hook now captures X-Turn-Id, tracks lastSeq from SSE id lines,
re-attaches on mount when the server still has an active turn, and
routes Stop through the cancel endpoint. The runtime call uses the
registry's per-turn AbortController instead of the HTTP request signal,
which is the core decoupling that lets turns outlive their initiator.

* feat(agents): add ActiveTurnRegistry primitive backing the new chat lifecycle

The previous commit referenced these files in tests and the harness
service but global gitignore swallowed them on the first add.

The registry owns the per-turn ring buffer (drop-oldest, terminal frame
preserved), the per-turn AbortController, and subscriber fan-out used
by /chat/stream resume.
 2026-04-29 21:01:06 +05:30
Dani Akash
e2ec1991cf feat(agents): redesign the agent command center for multi-adapter use (#861) 
* feat(agents): redesign agent rail to match the rest of the app

Reshape the `/agents` page so it reads as a sibling of `/scheduled`
and `/soul` and adapts to the multi-adapter world (OpenClaw, Claude
Code, Codex). Visual scaffolding only in this commit — per-agent
liveness state ships as `unknown` until the server-side activity
tracker lands.

  - New `AgentsHeader` mirrors `SoulHeader`/`ScheduledTasksHeader`:
    accent bot tile, title, descriptive subtitle, "+ New Agent"
    button. Replaces the loose top toolbar that mixed page-level and
    OpenClaw-lifecycle controls.
  - New `GatewayStatusBar` collects the OpenClaw lifecycle pills
    (running, control plane connected) plus the Terminal/Refresh
    affordances into a single labeled bar that only renders when the
    gateway is running AND there is at least one OpenClaw agent in
    the merged list.
  - New `AgentRowCard` per agent: adapter tile with liveness dot,
    name + status badge, adapter/model/reasoning chips, last-used
    relative time + truncated workspace path, primary "Chat" button,
    overflow menu (Copy id / Rename* / Reset history* / Delete).
    Rename + Reset are disabled with "coming soon" tooltips until
    the corresponding endpoints ship; Delete is hidden for the
    protected `main` agent.
  - New `AgentsEmptyState` mirrors the scheduled-tasks empty card.
  - New `AdapterIcon` + `LivenessDot` + `agent-display.helpers.ts`
    keep the row card focused on layout; helpers cover display name
    fallbacks for legacy `oc-<uuid>` titles, workspace label rules,
    and a tiny relative-time formatter.
  - `AgentList` now sorts by `lastUsedAt` desc with `null`s falling
    to the bottom; the gateway's `main` agent is pinned to the top
    only while it has zero turns so a fresh install has an obvious
    starting point. The list also threads a per-agent activity map
    so future commits can light up working/idle/asleep without
    reshuffling the API.
  - `AgentsPage` swaps to the standard `fade-in slide-in-from-bottom-5
    animate-in space-y-6 duration-500` shell and threads a
    `harnessAgentLookup` Map down to the row card so adapter chips
    and reasoning effort render correctly without a re-fetch.

* feat(agents): wire per-agent liveness end-to-end into the rail

Closes the placeholder `unknown` dot from the redesign's first
commit. The rail now shows real working / idle / asleep / error
states per agent, with `lastUsedAt` driving the recency sort.

Server side:
  - `AgentHarnessService` keeps an in-memory activity tracker keyed
    by agentId. `notifyTurnStarted` flips an entry to `working`,
    `notifyTurnEnded({ok})` either drops it (success) or pins it to
    `error` (failure / error event).
  - `send()` wraps the runtime stream so the lifecycle hook fires
    exactly once on natural close, error event, downstream cancel,
    or thrown setup. The runtime itself stays unchanged — fork is
    contained at the harness layer.
  - New `listAgentsWithActivity()` method enriches every agent with
    `{ status, lastUsedAt }`. lastUsedAt is read from the acpx
    session record's last persisted item via `runtime.getHistory`,
    so it survives server restart even though the activity map
    doesn't.
  - Status derivation: `working`/`error` take precedence; otherwise
    timestamp-based — `idle` until 15 min of silence, then `asleep`.
    Never-used agents resolve to `idle` (asleep implies "was active,
    went quiet").
  - `GET /agents` returns the enriched shape.

Client side:
  - `HarnessAgent` UI type extended with optional `status` +
    `lastUsedAt` so older deployments still typecheck.
  - `useHarnessAgents` flips on `refetchInterval: 5_000` (with
    `refetchIntervalInBackground: false` so hidden tabs go quiet)
    so the per-row dots and last-used copy stay fresh without a
    websocket.
  - `AgentsPage` builds an activity map from the harness listing
    response and threads it into `AgentList` → `AgentRowCard`. The
    sort by `lastUsedAt` desc (already in the row card) now has
    real data to operate on.

Tests:
  - New `marks an agent working while a turn streams and idle once
    it ends` exercises the wrap; uses a held upstream stream so
    the in-flight `working` state is observable.
  - New `flips to error when a turn emits an error event`.

* fix(agents): dedupe agent rail when /claw/agents and /agents share an id

The agents page was rendering every OpenClaw agent twice — once from
the legacy `/claw/agents` listing (`useOpenClawAgents`) and once from
the harness `/agents` listing (`useHarnessAgents`). Post Step 9
backfill the harness store contains every gateway agent, so the
overlap is the rule, not the exception.

Mirror the dedup the chat-panel layout already does: when a gateway
agent's id appears in the harness listing, drop the legacy entry and
keep the harness one (it has adapter/model/reasoning/status/lastUsedAt
the chat path actually consumes).

* feat(agents): swap GatewayStatusBar refresh icon for a Restart Gateway button + tooltips

The manual refresh became redundant once `useHarnessAgents` and
`useOpenClawStatus` started polling on a 5s interval — every visible
field self-refreshes within seconds. The previous AgentsPageHeader
had a real Restart action that the redesign dropped; reinstate it on
the bar so a wedged gateway is one click away again.

  - GatewayStatusBar: dropped the `RotateCcw` refresh icon and the
    `onRefresh` prop. Added `onRestart` + `actionInProgress` props;
    the button shows a spinner while a gateway lifecycle mutation is
    in flight.
  - Both Terminal and Restart Gateway buttons get tooltips explaining
    what they do — Terminal as a power-user shell escape hatch,
    Restart for unsticking a wedged gateway or after manual config
    edits.
  - AgentsPage: drop the now-unused `refreshAll` helper and the
    `refetchStatus`/`refetchAdapters`/`refetchOpenClawAgents`
    destructures it depended on. Wire `restartOpenClaw` (already
    pulled from `useOpenClawMutations`) through
    `runWithPageErrorHandling` like the legacy header did.

* feat(agents): consolidate gateway status into the /agents listing

Folds the gateway lifecycle snapshot into the harness listing so the
agents page polls one endpoint instead of two. Drops the dead
`/claw/status` call from the command center while keeping every UI
affordance the page already shipped (Running / Control plane
connected pills, GatewayStateCards setup/start prompts,
ControlPlaneAlert for degraded states).

Server side:
  - `OpenClawProvisioner.getStatus()` (optional) — when wired, returns
    the same `GatewayStatusSnapshot` shape `/claw/status` does.
  - `AgentHarnessService.getGatewayStatus()` — best-effort wrapper
    around the provisioner method; logs and swallows errors so a
    transient gateway issue doesn't 500 the listing endpoint.
  - `GET /agents` now returns `{agents, gateway}` in a single
    `Promise.all`. Both fields are independent — agents enrichment
    succeeds even if the gateway snapshot is null.
  - `server.ts` wires `getOpenClawService().getStatus()` into the
    provisioner accessor object alongside `createAgent` /
    `removeAgent` / `listAgents`.

Client side:
  - `useHarnessAgents` returns `{harnessAgents, gateway}` (plus the
    legacy `agents` mapping). Same 5s `refetchInterval` as before —
    one round-trip drives the per-row liveness AND the gateway pills.
  - `AgentsPage` drops `useOpenClawStatus` entirely; `status` comes
    from the harness query. Loader + error/lifecycle plumbing
    rewired around the harness query's loading/error.
  - `agents-page-utils.getInlineError` and `getAgentsLoading` lose
    the now-redundant `statusError` / `statusLoading` /
    `openClawAgentsEnabled` params.

The chat-panel layout (`agent-command-layout.tsx`) still consumes
`useOpenClawStatus(5000)` for now — left intact per the user's "only
the command center" scope. Folding that one in is a separate,
smaller pass once we're sure no regression slipped here.

* test(agents): teach the route fake service about the new listing shape

PR #861 CI surfaced two failures in tests/api/routes/agents.test.ts:
both call \`GET /agents\` and the route handler now invokes
\`service.listAgentsWithActivity()\` + \`service.getGatewayStatus()\`
which the fake created here didn't implement. Add both methods to
the fake (returning idle / null) and update the empty-list assertion
to expect the new \`{agents, gateway}\` envelope.
 2026-04-29 19:03:29 +05:30
Dani Akash
0c84547e8f feat(agents): migrate OpenClaw chat onto the unified harness/ACP path (#859) 
* chore(acp): smoke-test ACP capabilities against running gateway

Adds apps/server/scripts/acp-smoke.ts which spawns `openclaw acp`
inside the gateway container and exercises every method we plan to
depend on: initialize, newSession, prompt (text + image), cancel,
listSessions, loadSession.

SDK pinned to 0.19.1 (Bun's minimum-release-age policy blocks 0.20+
which were released < 7 days ago).

Findings (full notes in plan outcomes):
- promptCapabilities advertises image:true but the model does NOT see
  image bytes — silently dropped at the bridge.
- sessionCapabilities advertises {list:{}} but session/list throws
  "Method not found": stale capability advertising.
- loadSession works; replays user/assistant/thought text and
  session_info/usage/commands updates. No tool_call replay, as
  documented.
- cancel works end-to-end: stopReason=cancelled.
- closeSession/resumeSession are not on ClientSideConnection in
  0.19.1; kill child to close, use loadSession for rebind.

Plan revisions triggered by spike are recorded in
plans/browseros-ai/BrowserOS/features/2026-04-28-2310-claude-code-acp-implementation-roadmap.md.

* chore(acp): re-run smoke on SDK 0.21.0 and add mode/config/auth scenarios

After bypassing Bun's minimum-release-age and upgrading the SDK to
0.21.0, restore the previously-skipped resume/close paths and add
three new scenarios: mode (setSessionMode), config (setSessionConfigOption,
correct configId field), and auth (authenticate noop).

Findings, all bridge-side (independent of SDK):
- session/list, session/resume, session/close all throw -32601 on
  OpenClaw 2026.4.12 — capability advertising is stale.
- Image content blocks silently dropped; model never sees the bytes.
- setSessionMode and setSessionConfigOption work; latter requires
  `configId` (not `optionId`) per the schema.
- loadSession replays user/assistant/thought text + session_info +
  usage + available_commands; no tool_call replay (documented).
- authenticate is a noop on OpenClaw (no authMethods advertised).

Plan outcomes updated with full method-support matrix.

* chore(deps): promote @agentclientprotocol/sdk to a runtime dependency

The smoke script in apps/server/scripts/acp-smoke.ts used the SDK as
devDependency. The upcoming ACP bridge (apps/server/src/api/services/acp/)
needs it at runtime, not just for tooling. Move the entry from
devDependencies to dependencies, alphabetically first under @a*.

Pinned to 0.21.0 — same version the smoke script validated against.
README gains a small Dependencies note pointing at the future bridge
location.

No code changes yet. The bridge wiring lands in subsequent commits.

* fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config

When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.

Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
  side (useOpenClaw.ts) and the route body schema + SetupInput +
  createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
  LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
  `input: ['text', 'image']` on the model entry when the flag is
  truthy; otherwise emits the explicit `['text']` so the value is
  always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
  bootstrap batch so the gateway's image-understanding pipeline is
  always wired up — per-model `input` still gates which models see
  images, this just enables the global path.

ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.

Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.

* feat(agents): add OpenClaw to AgentAdapter union and catalog

Extends AgentAdapter to 'claude' | 'codex' | 'openclaw' and adds the
OpenClaw entry to AGENT_ADAPTER_CATALOG. The new entry has:

- defaultModelId: 'default' — OpenClaw's ACP bridge does not surface
  per-session model selection (verified during the ACP spike), so
  models live in the OpenClawService config, not in the adapter
  catalog. AgentDefinition.modelId carries the gateway-side model
  name for display only.
- models: [] — empty list signals "no per-session model picker" in
  the UI; isSupportedAgentModel('openclaw', undefined|'default')
  returns true via the existing fallback path.
- reasoningEfforts mirror OpenClaw's session-level `thought_level`
  config option (off / minimal / low / medium / high / adaptive).

Also extends:
- isAgentAdapter type guard recognizes 'openclaw'
- HarnessAgentAdapter union on the extension side
- agents.test.ts createAgent fake type
- agent-catalog.test.ts asserts on the new entry, empty model list
  passthrough behavior, and OpenClaw's reasoning effort set

Lockfile delta is the workspace SDK pin reconciling 0.20.0 (taken
from dev's lock) up to our package.json's 0.21.0 (added in
c1d987ea). acpx still uses 0.20.0 transitively — both are present.

No runtime wiring yet — the registry override and AcpxRuntime
plumbing land in subsequent commits.

* feat(agents): plumb OpenClaw gateway accessors into AcpxRuntime

Adds an optional `openclawGateway` accessor to AcpxRuntime so the
upcoming registry override (Step 4) can spawn `openclaw acp` inside
the gateway container with the right port, token, and container/VM
identity. All accessors are getter-shaped so values stay live across
gateway restarts (port can change, token can rotate).

The accessor is threaded:
  server.ts → createAgentRoutes → AgentHarnessService → AcpxRuntime
                            ↘ sidepanel lazy AcpxRuntime

Also adds OpenClawService.getGatewayToken() returning the in-memory
token string. We pass it via OPENCLAW_GATEWAY_TOKEN env var on the
spawn (per OpenClaw's documented env-var precedence) instead of via
`--token` flag (which leaks to ps aux) or `--token-file` path (no
discrete token file lives inside the container — the token is nested
inside openclaw.json).

Wiring is dormant — the registry override that consumes these
accessors lands in Step 4. Typecheck + existing acpx/harness/routes
tests pass unchanged.

* refactor(agents): scrub local plan-step references from code comments

Replaces forward-looking comments that referenced internal plan
steps (e.g. "Step 4 wires this into…") with comments that justify
the code on its own merits. Plan files live locally on the
contributor's machine, so cross-references are noise to the rest of
the project.

No behavior change.

* feat(agents): spawn openclaw ACP adapter inside the gateway container

When the harness resolves the `openclaw` adapter, it now returns a
command that runs `openclaw acp` inside the bundled gateway container
via `limactl shell <vm> -- nerdctl exec -i ... openclaw acp --url
ws://127.0.0.1:<port>`. This reuses the openclaw binary already
installed alongside the gateway — no host-side openclaw install is
required.

Auth: the gateway token is injected via OPENCLAW_GATEWAY_TOKEN on
the container exec rather than `--token` on the openclaw CLI, so
the secret never appears in `ps aux`.

Banner output: OPENCLAW_HIDE_BANNER=1 and OPENCLAW_SUPPRESS_NOTES=1
keep stdout JSON-RPC-clean.

LIMA_HOME: prefixed via `env LIMA_HOME=<path>` on the resolved
command so the spawned limactl finds the BrowserOS-owned VM (the
server doesn't set LIMA_HOME on its own process env).

When the gateway accessor is absent, falls through to acpx's
built-in openclaw adapter which assumes a host-side install — that
branch will fail at spawn time with a descriptive error.

Verified end-to-end via the existing acp-smoke script during the
Step 0 spike.

* feat(agents): dual-create OpenClaw harness agents on the gateway

When the harness creates an `openclaw` adapter agent, it now also
provisions a matching agent on the OpenClaw gateway via the existing
CLI path (OpenClawService.createAgent). Symmetric on delete: gateway
removeAgent runs alongside the harness-store delete.

- Adds an OpenClawProvisioner interface (decoupled from OpenClawService
  for testability) and injects it through AgentHarnessService.
- createAgent rolls back the harness record if gateway provisioning
  fails; deleteAgent tolerates gateway-side failures so harness
  identity stays consistent with the user-facing UI.
- New OpenClawProvisionerUnavailableError surfaces as a 503 when an
  openclaw create request lands on a harness with no provisioner
  wired in (instead of a generic 500).
- FileAgentStore mints openclaw agent ids with an 'oc-' prefix so
  the id satisfies the gateway's `^[a-z][a-z0-9-]*$` agent name
  pattern. Other adapters keep raw UUIDs to preserve compatibility.
- POST /agents body schema accepts providerType / providerName /
  baseUrl / apiKey / supportsImages, forwarded to the provisioner
  when adapter='openclaw'.

The agents-page UI still routes openclaw create through the legacy
/claw/agents flow; switching that path to the harness is a separate
UI cutover.

Tests cover: gateway dual-create on success, rollback on gateway
failure, 503 when provisioner is missing, and tolerant delete on
gateway-side failure.

* fix(agents): skip catalog model validation for OpenClaw adapter

OpenClaw agents resolve their model from the gateway-side provider
config (set at agent-create time via OpenClawService) rather than
from the harness catalog, which has an empty `models: []` entry by
design. Without this carve-out, every OpenClaw create body fails
parsing with "Invalid modelId" because no concrete model id can
satisfy isSupportedAgentModel('openclaw', ...).

The reasoning-effort check still runs against the catalog (those
values map directly to OpenClaw's session `thought_level` config
option).

* fix(agents): pass --session to openclaw bridge so newSession routes correctly

acpx's AcpClient.createSession calls connection.newSession with cwd
and mcpServers but never forwards the sessionKey. Without it, the
openclaw bridge falls back to a synthetic acp:<uuid> session that
doesn't resolve to any provisioned gateway agent — every harness
chat returns a generic "Internal error" from -32603.

Fix: bake `--session <key>` into the resolved spawn command. The
bridge then uses that as the default session key for any newSession
the bridge receives, routing the turn to the matching gateway agent.

Per-session keying means each openclaw agent gets its own
AcpxCoreRuntime instance (cached by sessionKey on top of the
existing cwd/permissionMode key). This adds one extra runtime per
active openclaw session — claude/codex are unaffected.

Test asserts the resolved command includes the right --session arg.

* fix(agents): suppress BrowserOS MCP for openclaw bridge

The openclaw ACP bridge rejects newSession when mcpServers is non-empty
because its provider tooling comes from the gateway, not from ACP-side
MCP servers. Forwarding the BrowserOS HTTP MCP made every harness chat
fail with a JSON-RPC -32603 "Internal error" before the session was even
opened. Claude/codex still need the BrowserOS MCP for browser tooling,
so the carve-out is keyed off whether the runtime is for an openclaw
session.

* feat(agents): route OpenClaw chat through the harness behind a flag

Adds the `feature.useAcpxForOpenClaw` extension storage flag. When on,
OpenClaw agents in the agent-command chat panel use the harness
/agents/<id>/chat SSE and harness history hook instead of the legacy
/claw/agents/<id>/chat. When off, behavior is unchanged.

Also dedupes the agent rail when the same id appears in both stores
(dual-created agents from /claw/agents and /agents) by preferring the
harness entry — without this, every dual-created OpenClaw agent shows
up twice after Step 5.

Image attachments are temporarily disabled when the harness path is
active; the carve-out lands in the next commit.

* fix(agents): keep legacy OpenClaw agents on ClawChat

The previous commit's flag-gated branch routed every `source='openclaw'`
agent through `/agents/<id>/chat` when the flag was on, but the layout
dedup means the only agents that ever reach that branch are legacy
gateway-only entries (`main`, orphan agents from rolled-back creates) —
which by definition have no harness record, so the harness path 404s
and chat is unusable. Source is the only routing signal again: harness
agents go through the harness, legacy agents stay on ClawChat. The
storage flag stays for Step 9/10's migration story.

* feat(agents): expose OpenClaw in sidepanel and route through gateway main

`buildSidepanelChatTargets` now emits a single default ACP target for
adapters with no per-session model picker (OpenClaw, whose model is
configured on the gateway-side agent). Without this, OpenClaw never
appeared in the sidepanel target picker because the catalog entry has
`models: []`.

Sidepanel sessions don't have a dedicated provisioned gateway agent.
The openclaw bridge `--session` flag previously got the raw sidepanel
key (`sidepanel:<convId>:openclaw:...`), which doesn't match any
gateway agent — newSession was accepted but every prompt hung
forever. The bridge command now rewrites non-harness session keys
onto the always-present `main` gateway agent, encoding the original
key as a channel suffix to keep state segregated per conversation.
Verified end-to-end via curl: sidepanel openclaw chat streams
`text-delta` + `finish: stop`.

* feat(agents): backfill harness records for legacy gateway agents

Reframes Step 9 of the OpenClaw-on-acpx migration. The plan's literal
Step 9 (route OpenClaw history through the harness when the flag is on)
was already a no-op after the Step 6 walkback — history is routed by
source today. The actual blocker for Steps 10–13 was that legacy
gateway-only agents (e.g. `main`, orphans from rolled-back creates) had
no harness record, so they could never migrate to the harness path
without breaking chat.

`AgentHarnessService.reconcileWithGateway()` now lists every gateway
agent and upserts a matching harness record for any that are missing.
The pass runs lazily on first `listAgents()` call (memoized on success,
retried on failure so a gateway-down boot doesn't permanently disable
backfill). Verified end-to-end: the legacy `agent` agent now streams
`text_delta` + `done(end_turn)` through `/agents/agent/chat`, with the
bridge resolving to the gateway's `agent` record via the existing
`agent:<name>:main` session-key format.

After this, every OpenClaw agent surfaces as `source='agent-harness'`
post-dedup, the legacy `useClawChatHistory` hook becomes unreachable
for OpenClaw, and Steps 11–13 (delete legacy chat/history paths) are
unblocked.

* fix(agents): drop duplicate OpenClaw entry from NewAgentDialog adapter list

The adapter Select hardcoded an `<SelectItem value="openclaw">OpenClaw</SelectItem>`
on top of iterating `adapters`, which now includes OpenClaw post the
catalog change. The dropdown rendered "OpenClaw" twice — once at the
top, once at the bottom of the list. The literal was a pre-catalog
artifact; removing it leaves a single OpenClaw entry sourced from the
catalog. Routing into `handleOpenClawCreate` is unchanged because
the value (`'openclaw'`) is identical either way.

* fix(agents): always reconcile harness with gateway on list, just dedupe concurrent calls

Memoizing the first successful reconcile meant new gateway agents (created
via the legacy /claw/agents path or out-of-band CLI) never appeared in the
harness until server restart. The Promise now serves as a concurrent-call
dedupe only — cleared on settle — so every listAgents call picks up the
current gateway state. Reconcile is one cheap idempotent CLI call.

* chore(agents): remove dormant useAcpxForOpenClaw flag

The flag was scaffolded in Step 6 but its routing effect was walked
back the same day after it broke chat for legacy gateway-only agents.
After the Step 9 backfill, every OpenClaw agent has a harness record
and routes through the harness path purely from `source='agent-harness'`
— no flag is consulted anywhere. Remove the dead storage item, hook,
and stale comment.

* refactor(agents): drop legacy /claw/agents/:id/history endpoint

The harness /agents/:id/sessions/main/history endpoint replaced this
once every OpenClaw agent got a harness record (Step 9 backfill).
Routing is fully source-driven now, so the UI's useClawChatHistory
hook is never enabled today — verified live: legacy URL returns 404,
harness history hydrates correctly for the same agent.

Removes the GET /claw/agents/:id/history route, OpenClawService's
getAgentHistoryPage method plus its cursor/limit helpers and the
history-only types it owned (BrowserOSOpenClawHistoryPageResponse,
HistoryPageInput, normalizeHistoryLimit, encodeHistoryCursor,
decodeHistoryCursor, jsonlEventsToHistoryItems), and the route +
service tests that covered the dropped endpoint.

OpenClawJsonlReader stays alive — still feeds /claw/dashboard,
/claw/agents/:id/sessions, and the boot-time clawSession seed.
Removing those is its own follow-up since the dashboard would need
a harness-side replacement first.

* feat(agents): wire image attachments through the harness ACP path

Composer attachments now flow into the ACP `prompt` request as
spec-compliant `image` content blocks alongside the user's text. End
to end:

  composer → chatWithHarnessAgent({attachments}) →
  POST /agents/:id/chat {message, attachments} →
  parseChatBody decodes data: URLs to {mediaType, base64} →
  AgentHarnessService.send forwards →
  AcpxRuntime.send forwards →
  acpx startTurn({attachments}) → ACP image blocks

UI no longer disables the attach button on harness agents — the
gating was just a placeholder before this commit landed. Verified
end to end with a 1×1 red PNG against a Claude harness agent: model
replies "Red." correctly.

OpenClaw's `acp` bridge still drops image content blocks upstream
(verified by the same probe — Kimi-k2p5 reports "I don't see an
image"). That's an upstream openclaw limitation, not a harness-side
gap; Claude/Codex agents work as advertised today.

* chore(openclaw): delete OpenClawJsonlReader and JSONL-backed routes

* chore(openclaw): remove legacy /claw/agents/:id/chat and /queue routes

* chore(agents): collapse chat panel to harness-only path

* feat(agents): route OpenClaw image turns through the gateway HTTP client

The OpenClaw `acp` bridge silently drops ACP `image` content blocks
(verified during dogfood — model says "I don't see an image"). When
the user attaches images to an OpenClaw agent, the harness now diverts
that turn to the gateway's HTTP `/v1/chat/completions` endpoint, which
accepts OpenAI-style `image_url` parts and forwards them natively to
the provider.

  - New `OpenClawGatewayChatClient` translates an OpenAI streaming
    response into the same `AgentStreamEvent` shape the rest of the
    harness already consumes, so the chat panel renders identically
    whether a turn went through ACP or the gateway carve-out.
  - `AcpxRuntime.send` forks at the top: openclaw + any image
    attachment + a wired gateway client → `sendOpenclawViaGateway`.
    Other turns (text-only openclaw, claude, codex) take the existing
    ACP path unchanged.
  - The diverted path reads the prior turn history from the acpx
    session record so context is preserved, builds the OpenAI
    multimodal user message with text + image_url parts, and pumps
    the gateway SSE back to the caller through a tee that accumulates
    the assistant text. On natural completion, persists a synthetic
    user+assistant message pair to the acpx session record so reload
    shows the image turn in history.
  - Wired `OpenClawGatewayChatClient` into `AgentHarnessService` via
    `server.ts` (gateway port + token accessor, just like the existing
    `openclawGateway`).

Persistence note: the acpx record requires User messages to carry an
`id` and Agent messages to carry `tool_results` — without them the
record fails to round-trip through `parseSessionRecord`. The persist
helper now sets both.

Limitation by design: image recognition only works if the OpenClaw
agent's provider supports vision (e.g. Claude-via-OpenClaw, GPT-4o).
The pipeline routes images correctly to the provider regardless;
text-only providers like Kimi-k2p5 will reply "I don't see an image"
because the model itself has no vision capability — that's a provider
config issue, not a routing bug. The unit test asserts the image_url
part is present in the OpenAI request the gateway client sends.

The wider plan (background-resilient chat, queue, replay) remains in
`plans/.../2026-04-29-1527-...-background-resilient-chat-and-image-uploads.md`
as Phases 3–12; this commit ships only Phases 1–2.

* feat(agents): validate inbound image attachments on /agents/:id/chat

The harness chat body parser was accepting any mediaType and any
dataUrl length. The composer enforces these caps client-side but the
endpoint also serves direct curl/script callers, so the server has to
defend itself.

Restores the same caps the legacy /claw/agents/:id/chat parser had
before it was deleted in the migration:

  - 10 attachments per message
  - 5 MB raw image bytes (≈ 6.7 MB once base64-encoded plus prefix)
  - PNG / JPEG / WebP / GIF only
  - Must start with `data:`

Each violation returns 400 with a specific error message instead of
silently dropping or forwarding the payload.
 2026-04-29 16:37:03 +05:30
Nikhil
2ff5c12840 feat: add sidepanel ACP chat targets (#857) 
* feat(agent): add sidepanel chat target catalog

* feat(agent): show acp models in sidepanel selector

* feat(server): adapt acp events to ui message streams

* feat(server): add sidepanel acp chat route

* feat(agent): route sidepanel chat through acp targets

* chore: self-review fixes

* fix: address review feedback for PR #857
 2026-04-28 18:23:38 -07:00
Nikhil
d87422eea1 fix: hide BrowserOS ACP wrapper in history (#856) 2026-04-28 17:31:11 -07:00
Nikhil
1946ca0cf8 chore: clean up unused agent sdk (#855) 2026-04-28 17:21:46 -07:00
Nikhil
754f7d0e1d test: cover terminal limactl resolver errors (#854) 2026-04-28 17:12:08 -07:00
Nikhil
85bb3f7b42 fix: avoid eager limactl resolution in server tests (#853) 2026-04-28 16:56:41 -07:00
Nikhil
cb32b8191d fix: show rich ACP harness history from ACPX (#852) 
* fix: load ACP harness history from ACPX

* fix: address ACP history review comments
 2026-04-28 16:40:22 -07:00
Nikhil
7a92654abc feat: add BrowserOS MCP to ACP agents (#851) 
* feat: add BrowserOS MCP to ACP agents

* fix: bypass ACP agent permissions

* fix: address review feedback for PR #851
 2026-04-28 16:30:20 -07:00
Nikhil
91d3285aa0 feat: add ACP agent harness (#849) 
* feat: add acp agent runtime spike

* feat: add agent harness catalog

* feat: persist harness agents in json

* feat: persist agent transcripts

* feat: route harness service through agent records

* feat: expose generic agent harness routes

* feat: add harness agent frontend api

* feat: create harness agents from agents page

* feat: chat with persisted harness agents

* chore: remove obsolete agent profile spike

* chore: self-review fixes

* fix: combine openclaw and harness agents UI

* refactor: split agents page components

* fix: hide persisted harness turns
 2026-04-28 15:29:38 -07:00
Nikhil
7bb6dac949 fix(dogfood): copy extension state into dev profile (#850) 
* fix(dogfood): copy extension state into dev profile

* fix(dogfood): address profile import review feedback

* fix(dogfood): clarify refresh profile in-use error
 2026-04-28 15:25:38 -07:00
shivammittal274
d9c254053e refactor(eval): drop unused agents/graders, collapse registries (#847) 
* refactor(eval): drop unused agents/graders, collapse registries

Sweep of dead code in the eval app: deleted gemini-computer-use and
yutori-navigator agents, fara/webvoyager/mind2web graders, eight
debug/analyze/test scripts, three stale planning docs, and the orphaned
eval-targets/coordinate-click testbed.

With two agents and three graders left, the Map-backed plugin registries
were over-engineered — collapsed both into plain switches. Removed the
now-dead GraderOptions plumbing (no remaining grader takes API keys),
dropped grader_api_key_env/grader_base_url/grader_model from the schema
and configs, and de-duped PASS_FAIL_GRADER_ORDER (was defined in three
places). Replaced the URL-parsing extractCdpPort hack in single-agent
and orchestrator-executor with workerIndex passed cleanly through
AgentContext.

README and --help text rewritten to match reality. Renamed
configs/test_*.json to test-*.json for kebab-case consistency.

Net: ~10,460 LOC removed across 60 files. Typecheck clean, all tests
pass.

* ci(eval): pull BrowserOS from rolling stable CDN URL

The pinned v0.44.0.1 .deb on GitHub releases regressed on Linux —
servers start but never become healthy. Switch to the canonical rolling
URL at cdn.browseros.com/download/BrowserOS.deb so CI tracks the same
stable channel users get from the marketing site.
 2026-04-29 02:14:47 +05:30
Nikhil
6b9945f933 feat(dev): use dev dock icon for browser launches (#848) 2026-04-28 13:28:19 -07:00
Dani Akash
6a5a7775a9 fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config (#846) 
When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.

Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
  side (useOpenClaw.ts) and the route body schema + SetupInput +
  createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
  LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
  `input: ['text', 'image']` on the model entry when the flag is
  truthy; otherwise emits the explicit `['text']` so the value is
  always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
  bootstrap batch so the gateway's image-understanding pipeline is
  always wired up — per-model `input` still gates which models see
  images, this just enables the global path.

ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.

Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.
 2026-04-29 00:23:45 +05:30
shivammittal274
af48a2110c feat(eval): Phase 1 — exclude broken tasks, freshen card dates, add grader leniency (#841) 
* fix(eval): exclude broken tasks + freshen expired card dates

Two AGISDK tasks are unsolvable today for non-model reasons:

- topwork-1: evals-topwork.vercel.app throws Minified React error #185
  ("Maximum update depth exceeded") on every form submit. The page renders
  "Application error: a client-side exception has occurred" instead of saving.
  Whole-task failure, every model affected.

- fly-unified-2: hardcodes Exp: 12/25 in both the goal text AND a jmespath
  grader criterion. Today is 2026-04, so the eval-site rejects the card.
  Freshening the goal alone leaves the grader expecting the original value;
  freshening both would require monkey-patching agisdk's TaskConfig at
  runtime — too fragile to maintain.

Adds these to a new EXCLUDED_TASKS set alongside the existing
EXCLUDED_WEBSITES (omnizon).

Also adds freshen_goal_dates(): for AGISDK fly-unified tasks whose goal
contains an `Exp: MM/YY` within 6 months of today (or past), rewrites it
to a far-future date (12/30). This rescues fly-unified-5 (had Exp 12/25,
no card-exp grader criterion) and protects fly-unified-4 (had Exp 06/26,
2 months from expiring) from the next eval run hitting the same trap.

Dataset goes from 47 -> 45 tasks; 2 freshened.

* feat(eval): add lenient-strings grader softening

The agisdk grader compares jmespath-extracted values via strict equality.
For tasks where the model adds harmless decoration to a free-text field
(e.g. topwork-3 expects title "Full-Stack Developer" but model produces
"Full-Stack Developer - Enterprise Microservices Platform"), this fails
every other criterion would pass.

Adds a substring fallback in the wrapper: a failed criterion is re-marked
as a softened pass when both actual_value and expected_value are strings
and the (stripped, lower-cased) expected_value is contained in the
actual_value. Numbers/bools/dates/None stay strict.

- Default-on. Set AGISDK_STRICT_STRINGS=1 to recover the strict score.
- Softened criteria are tagged with `softened: true` in per_criterion
  output for transparency in run manifests.
- Aggregate `pass`/`reward` are recomputed after softening.

Expected to rescue 4 tasks in our 45-set: topwork-3, topwork-4 (both pure
title-decoration), gomail-8 (grader contradicts goal), and networkin-6
(grader hardcodes profile id).

* fix(eval): exclude 5 more tasks where pipeline (not agent) fails

Extends EXCLUDED_TASKS to 7 entries based on the K2.5 + Opus 4.6
head-to-head deep-dive on the 2026-04-28 runs. The exclusion rule:
remove a task only if it is unsolvable for any agent — either the task
data is invalid, the eval site is broken, or the grader penalizes
correct work. Tasks that fail because of our agent's tool fidelity
(drag, custom-widget fill, click on React submit, etc.) STAY in — those
are real capability gaps the team should see in the score.

New exclusions:

- fly-unified-9: goal references "Dec 18 2024 at 10:00" but the live
  eval site has only 2025 inventory and no 10:00 slot. Both models
  successfully booked the closest available flight and were penalized
  on a grader expectation that can never be met.

- fly-unified-4: eval site stores wall-clock flight times as bare UTC
  (T08:00:00.000Z) while the grader expects them shifted by 8h
  (T16:00:00.000Z = 8 AM PST). Opus 4.6 completed the entire booking
  correctly. Eval-site TZ-storage bug.

- gomail-8: goal says "Clear all emails from GitHub in the inbox", but
  criterion 3 expects exactly 1 email updated. Both K2.5 and Opus
  correctly cleared all 4 GitHub emails. Grader contradicts goal.

- networkin-6: goal says "Choose a random person you haven't connected
  with"; grader hardcodes profilesDiff.updated."4".connectionGrade.
  Both models randomized correctly and missed id 4. Grader contradicts
  goal.

- networkin-9: eval site's searchHistoryDiff doesn't record queries
  submitted via the autocomplete + Enter path. Opus 4.6 completed the
  task end-to-end (Stanford alum, connection request, message); only
  failed because the search-history criterion was never written
  server-side. Eval-site bug.

Dataset goes from 45 -> 40 tasks. Score impact (same K2.5/Opus runs,
recomputed against the cleaned 40-task denominator):

  K2.5:     21/45 (46.7%) -> 21/40 (52.5%)
  Opus 4.6: 28/45 (62.2%) -> 28/40 (70.0%)
  Δ:        15.6 pp -> 17.5 pp (real model gap, less pipeline noise)
 2026-04-28 23:19:31 +05:30
Nikhil
c5ff8d75bc fix(dogfood): clarify init prompts (#839) 2026-04-28 07:48:42 -07:00
Nikhil
445a6a6c45 fix(dogfood): use alpha dock icon (#837) 2026-04-27 21:47:10 -07:00
Nikhil
72d39b9a0f docs(dogfood): simplify alpha workflow readme (#838) 2026-04-27 21:44:03 -07:00
Nikhil
3b47f330f5 fix(dogfood): separate BrowserOS state root (#836) 2026-04-27 17:38:15 -07:00
Nikhil
15a82ff9cb feat: add dogfood background daemon mode (#833) 2026-04-27 17:15:50 -07:00
Nikhil
427549f081 feat: Add BrowserOS Dock icon variants (#835) 2026-04-27 17:10:36 -07:00
Nikhil
a11f9caa64 fix(dogfood): colorize cli output (#834) 
* fix(dogfood): colorize cli output

* fix: address dogfood cli review comments
 2026-04-27 16:29:25 -07:00
Nikhil
da1397900b refactor: rename internal BrowserOS CLIs (#832) 
* refactor: rename internal BrowserOS CLIs

* fix: update dogfood binary gitignore
 2026-04-27 16:18:45 -07:00
Nikhil
368c7dcfe8 fix(alpha): write balpha process logs (#830) 
* fix(alpha): write balpha process logs

* fix(alpha): address log review feedback
 2026-04-27 15:48:40 -07:00
Nikhil
599f8b6b9c fix: address balpha CLI dogfooding feedback (#831) 2026-04-27 15:43:22 -07:00
Nikhil
27834b1d31 fix: udpate readme (#829) 2026-04-27 15:27:16 -07:00
Nikhil
aa30eb3aaa feat: add balpha dogfooding CLI (#828) 
* feat(alpha): scaffold balpha cli

* fix(alpha): address scaffold review

* feat(alpha): add balpha config

* feat(alpha): parse browseros profiles

* feat(alpha): import browseros profile

* feat(alpha): add browser launch helpers

* feat(alpha): add repo build and env pipeline

* feat(alpha): add process supervision

* feat(alpha): add balpha commands

* docs(alpha): document balpha setup

* fix(alpha): reuse dev setup script

* fix(alpha): address review feedback

* fix(alpha): normalize imported browser profile

* fix(alpha): use generic profile fixture names
 2026-04-27 15:03:37 -07:00
shivammittal274
e045e34b73 fix(eval): switch weekly eval configs from Fireworks to OpenRouter (#827) 
The 2026-04-23 weekly run had 42% of AGISDK and 46% of Infinity tasks
fail with `AI_RetryError: ... the service is overloaded` from Fireworks
(20 concurrent kimi-k2p5 streams across both runs at 10 workers each).

Switching to OpenRouter (which fronts the same Moonshot K2.5 weights
and falls back across providers) for the three weekly configs:
- browseros-agent-weekly.json
- agisdk-real-smoke.json
- infinity-hard-50.json

Model accounts/fireworks/models/kimi-k2p5 -> moonshotai/kimi-k2.5
(same weights, same 262K context). API key env var, base URL updated.

OPENROUTER_API_KEY is already wired into .github/workflows/eval-weekly.yml
and present in repo secrets — no GH config changes needed.

Orchestrator-executor configs and test_webvoyager left on Fireworks
intentionally; can switch later if needed.
 2026-04-27 21:52:26 +05:30
shivammittal274
01d649da9a feat(eval): bring deterministic graders to dev + drop omnizon (#824) 
* feat: deterministic eval graders (AGI SDK + WebArena-Infinity) (#664)

* feat: add deterministic eval graders (AGI SDK + WebArena-Infinity)

Two new benchmark integrations with programmatic grading — no LLM judge.

AGI SDK / REAL Bench (52 tasks):
- 11 React/Next.js clones of consumer apps (DoorDash, Amazon, Gmail, etc.)
- Grader navigates browser to /finish, extracts state diff from <pre> tag
- Python verifier checks exact values via jmespath queries

WebArena-Infinity (50 hard tasks):
- 13 LLM-generated SaaS clones (Gmail, GitLab, Linear, Figma, etc.)
- InfinityAppManager starts fresh app server per task per worker
- Python verifier calls /api/state and asserts on JSON state

Infrastructure:
- GraderInput extended with mcpUrl + infinityAppUrl for parallel workers
- Each worker gets isolated ports (no cross-worker state contamination)
- CI workflow: pip install agisdk, clone webarena-infinity repo

* chore: switch eval configs back to kimi-k2p5

* fix: register deterministic graders in pass rate calculation

Add agisdk_state_diff and infinity_state to PASS_FAIL_GRADER_ORDER
in both runner types and weekly report script, so scores show correctly
in the dashboard.

* chore: temp switch to opus 4.6 for eval run

* chore: restore kimi-k2p5 as default eval config

* ci: add timeout and continue-on-error for trend report step

* fix(eval): drop omnizon from AGISDK dataset (DMCA takedown)

evals-omnizon.vercel.app returns HTTP 451 ("This content has been
blocked for legal reasons / DMCA_TAKEDOWN"). All 5 omnizon-* tasks
fail grading with "Failed to fetch /finish endpoint: JSON Parse error".

Adds an EXCLUDED_WEBSITES set to the dataset builder and regenerates
agisdk-real.jsonl (52 → 47 tasks).

* fix(eval): correct Infinity port-assignment bugs

Two related bugs in the Infinity eval runner that cause silent port
collisions / fallbacks under parallel execution:

1. build-infinity-dataset.py emitted "app_port" but task-executor and
   the committed JSONL both read "app_base_port". Re-running the build
   script would silently make every task fall back to the 8000 default,
   ignoring per-app port assignments. Renamed the key to match.

2. task-executor derived workerIndex as `base_server_port - 9110`, but
   parallel-executor doesn't override base_server_port per worker —
   only server_url. Every worker computed workerIndex = 0, causing all
   parallel workers to spawn Infinity app servers on the same port.
   Threading workerIndex explicitly through TaskExecutor instead.

Also drops an unused app_name parameter from load_tasks().
 2026-04-27 21:35:43 +05:30
Dani Akash
ddbb2cf492 feat(agent): composer attachments + server-side outbound message queue (#826) 
* feat(agent): attach images and text files to chat messages

Adds end-to-end support for image and text file attachments in the chat
composer, with the staged files round-tripping through the OpenClaw
gateway as OpenAI-compatible content blocks and persisting in the JSONL
so they show up in the historical view.

Server
- HTTP client: new OpenClawChatContentPart union and a buildUserContent
  helper that emits multimodal content arrays when messageParts is
  supplied, falls back to the legacy string content otherwise.
- Service: chatStream takes an optional messageParts array and forwards
  it; BrowserOSChatHistoryItem gains an attachments field.
- JSONL reader: PiContentBlock learns the OpenAI image_url and Anthropic
  image source/data shapes; user messages now emit user.attachment
  events that the history mapper accumulates onto the next user item.
- Route: validates an inbound attachments[] (kind/mime/size/count),
  inlines text-shaped files as <attachment> blocks in the message body,
  attaches images via image_url parts. Replaces the immediate 409 on
  active monitoring session with a 30s waitForSessionFree(agentId) wait
  (registry now exposes onSessionEnd) so cron/hook contention does not
  reject a user-chat send outright. Returns 503 if the wait times out.

Client
- New lib/attachments.ts: validateAttachment / compressImageIfNeeded
  (canvas downscale to 2048px long edge, JPEG 0.85 re-encode for >1.5
  MB inputs) / stageAttachment / stageAttachments that produces the
  staged-attachment shape the composer renders and the payload the
  server accepts.
- ConversationInput: drag-and-drop, paperclip button, clipboard paste,
  staged attachment chip strip with thumbnails for images and a
  paperclip+name chip for text files. Send button enables on either
  text or attachments. Drop-zone overlay during drag.
- chatWithAgent forwards attachments[]; useAgentConversation.send
  accepts a SendInput shape and renders user attachments on the
  optimistic streaming turn via MessageAttachments / MessageAttachment.
- ClawChatMessage groups historical attachment parts into a single
  MessageAttachments strip, ordered before reasoning/tools/text.
- claw-chat-types adds an attachment ClawChatMessagePart variant; the
  history mapper emits attachment parts first and skips the text part
  when the user only sent media.
- AgentCommandHome forwards the new SendInput shape — home composer
  drops attachments at the boundary in v1 (the conversation page is
  where staging is most useful; carrying bytes through the URL bar
  is not sensible).

Limits: 10 attachments per message, 5 MB per image (post compression),
1 MB per text file, mime types png/jpeg/webp/gif and text/* +
application/json. PDFs and other binaries are deferred to v2.

* feat(agent): outbound message queue for chats while agent is mid-turn

Lets users keep typing and submitting messages while the agent is still
streaming a previous turn. Each press is appended to a single-flight
queue and dispatched as soon as `streaming` flips false; the queued
state renders as a strip above the composer so the user sees what's
pending vs. what's already sending.

- New `useOutboundQueue` hook owns the queue, the worker effect, and
  cancel/retry actions. Single-flight by design — a re-entrancy ref
  guard prevents two simultaneous dispatches when `streaming` flickers.
- Composer (`ConversationInput`) accepts optional `outboundQueue`,
  `onCancelQueued`, `onRetryQueued` props. When the queue is provided
  the send-button gate stops blocking on `streaming`; the spinner stays
  as the visual cue that the agent is still busy. Legacy direct-send
  callers keep the old streaming-blocks-send semantic.
- Renders an OutboundQueueStrip above the staged-attachment strip with
  per-item status (queued / sending / failed), a cancel button on
  queued items, and retry + discard on failed items.
- AgentCommandConversation wires `onSend` to `queue.enqueue` and routes
  the home composer's `?q=` initial-message handoff through the queue
  too, so it inherits the same single-flight serialization.

The server-side `waitForSessionFree` (added with attachments) and this
client-side queue together cover both contention sources: cron / hook
turns and back-to-back user sends. Persistence across reloads is
intentionally out of scope for v1 — losing the queue on extension
reload is documented as a known limitation.

* feat(server): server-side outbound message queue

Replaces the client-only React-state queue from 123ef21d with a
proper server-owned queue. Closing the tab is now safe — the server
holds queued messages and dispatches them through the existing
chatStream path the moment the agent's ClawSession status flips to
idle.

Server
- New OutboundQueueService (apps/server/src/api/services/queue) — per
  agent FIFO, in-memory. Subscribes to ClawSession.onStateChange
  through OpenClawService.onAgentStatusChange, and dispatches via
  OpenClawService.chatStream so attachments / history / monitoring
  all behave identically to the existing /chat route. The worker
  drains the SSE response server-side so the gateway run finalizes
  cleanly even with no client connected.
- Four new routes under /claw/agents/:id/queue:
  POST   /queue            enqueue
  DELETE /queue/:itemId    cancel a queued item
  POST   /queue/:itemId/retry  re-queue a failed item
  GET    /queue/stream     SSE feed of the per-agent queue state.
  Validation reuses validateChatAttachments and
  buildMessagePartsFromAttachments from the existing chat route.
- Singleton wired in apps/server/src/main.ts; shutdown on SIGTERM.
- New OpenClawService.getAgentState getter for the queue worker's
  pre-dispatch sanity check.

Client
- useOutboundQueue rewritten as an SSE-backed projection over server
  state. Public API unchanged so the composer still works.
- enqueue POSTs to /queue and shows an optimistic local entry until
  the server's SSE snapshot reflects it; local-only entries get a
  `local-` id prefix so cancel can short-circuit them without
  hitting the server.
- AgentCommandConversation watches the queue for sending items
  dropping out and refetches history so the new assistant turn shows
  up in the conversation view (the server worker streams the
  dispatched turn into OpenClaw without exposing per-turn SSE to
  the client).

Out of scope (documented in the plan as v2 follow-ups): disk
persistence (server restart loses queue), per-turn live streaming
of queued sends in the conversation view, and switching the
underlying dispatch from /v1/chat/completions to the chat.send RPC
(which would also fix the multimodal attachment routing problem).

* fix(server): outbound queue must reuse existing session, not spawn UUIDs

The queue worker was generating a fresh randomUUID() as the sessionKey
when the queued item didn't carry one — and the client wasn't sending
one. Result: every queued message kicked off a brand-new OpenClaw
session, orphaning the user's active conversation behind the new
"most recent" entry in sessions.json. The history endpoint then
resolved to the orphan and the chat appeared to disappear.

Fix is layered:
- Client (useOutboundQueue): forward the current resolvedSessionKey
  in the POST /queue body so every queued message targets the same
  conversation the user is viewing. AgentCommandConversation passes
  resolvedSessionKey into the hook.
- Server (OutboundQueueService): the worker now resolves to the
  agent's existing user-chat session when no sessionKey is provided
  on the queued item, via OpenClawService.resolveAgentSession. UUID
  fallback is now reserved for the first-ever message on a brand
  new agent — same semantic the existing /chat route has implicitly
  through the catalog of historical sessions.

No JSONL data was lost by the original bug (the prior conversations
are intact on disk); the orphan sessions just shadowed the original
in sessions.json.

* fix(agent,server): address PR review feedback for chat queue

- Tighten image data URL cap to base64-aware ~6.7 MB (was ~7.5 MB
  through `MAX_IMAGE_BYTES * 2`).
- Forward chat history from useOutboundQueue.enqueue so queued sends
  preserve conversation context like direct sends do.
- Match local attachment previews to server snapshots by id (not by
  message text), and prune the preview map as items drain.
- Pass an AbortSignal into chatStream so a queue shutdown cancels the
  initial OpenClaw handshake, not just the SSE drain loop.
- Track previously gitignored apps/agent/lib/attachments.ts (was caught
  by global lib/ ignore) so CI typecheck can resolve @/lib/attachments.
- Update server-api openclaw route tests to the new chatStream signature
  and the waitForSessionFree-based busy-agent path.

* fix(agent): dedupe optimistic queue entries for text-only sends

The localId↔serverId map was only populated when the message had
attachments, so plain-text sends left the optimistic local entry in
place after the server snapshot arrived — the user saw the same
message rendered twice in the queue strip.

* fix(agent): prune optimistic queue entry on POST ack, not just SSE

The server broadcasts the new queue snapshot before its POST response
returns, so the SSE handler often runs first — at that point the
localId↔serverId map has no entry for the new server id yet, so the
SSE-based dedupe path can't drop the optimistic local entry. Pruning
on POST success closes the race deterministically.

* fix(agent): hand off optimistic queue entry without a render gap

Pruning the local entry on POST success only worked when the SSE
snapshot had already overwritten it; if the POST response landed
first, the optimistic row disappeared for a frame before the SSE
snapshot brought back the server-keyed row, producing a visible
flicker. Gate the POST-side prune on the SSE snapshot already
carrying the server id, and rely on the SSE-based dedupe (now
guaranteed to find the localId↔serverId link in the map) to clean
up when SSE arrives later.

* fix(agent,server): client-generated queue id eliminates render flicker

The server used to assign its own UUID when an item was enqueued, so
the optimistic client row carried a `local-` id while the SSE snapshot
carried a server UUID — the client had to wait for the POST response
to learn the mapping before it could dedupe, and during that window
both rows rendered.

Now the browser generates the id, sends it in the POST body, and the
server uses it verbatim (falling back to a fresh UUID only if the id
collides with an existing item). The client collapses to a single
id-keyed list, so the optimistic row and the SSE row reconcile on the
same key from the very first render.
 2026-04-27 21:31:03 +05:30
Dani Akash
711934555d feat(agent): enrich chat UI with tool activity, reasoning duration, and cost (#825) 
* feat: pass per-turn cost and token data through chat history items

- Add costUsd, tokensIn, tokensOut to BrowserOSChatHistoryItem (server)
- Pass through from JSONL agent.message events in jsonlEventsToHistoryItems()
- Add same fields to client-side BrowserOSChatHistoryItem and ClawChatMessage
- Map cost/token data in mapHistoryItemToClawMessage()

Data flows: JSONL message.usage → server history item → API response →
client ClawChatMessage. Available for rendering in ClawChatMessage
component (message toolbar, cost badges).

* feat: add message toolbar with copy button and per-turn cost display

Add MessageToolbar to historical assistant messages in ClawChatMessage:
- Copy button copies message text to clipboard via MessageAction
- Per-turn token count (22.7K → 238) and cost ($0.003) shown as muted
  tabular-nums text on the right side of the toolbar
- Toolbar appears on hover (opacity transition via group-hover)
- Only shown when the message has text content
- Cost/token display only shown when data is available from JSONL

* fix: toolbar only on assistant messages, always visible, cost only

- Only render toolbar on assistant messages (not user messages)
- Remove hover-only opacity — toolbar is always visible
- Remove token counts (22.7K → 238 is meaningless to users)
- Show only cost as a budget signal ($0.003)

* feat: group all tool activity into single Task collapsible per turn

Replace flat tool rows with a single ai-elements Task collapsible per
assistant turn that lists every tool/MCP call in sequence.

Live streaming (ConversationMessage):
- Aggregate all tool-batch parts into one Task
- Title: "Working… (N actions)" while running, "Agent activity (N actions)" when done
- Default open while turn is in progress
- Wrench icon in trigger

Historical (ClawChatMessage):
- Group all tool-call parts into one Task
- Title includes failed count if any tools errored
- Default collapsed — expandable on click
- Tool name + status icon + error text per row

Both views show one clean collapsible per turn instead of N individual
tool cards. Collapsed reads "5 actions"; expanded shows the timeline.

* feat: include tool calls in chat history responses

Server: jsonlEventsToHistoryItems() now walks ALL events (not just
messages) and pairs agent.tool_use with agent.tool_result by toolCallId.
The resulting tool call list is attached to the next assistant text
message as toolCalls[]. Each entry includes status, input arguments,
output text, error string, and duration computed from event timestamps.

Client:
- BrowserOSChatHistoryItem gets optional toolCalls field
- Tool-call message part type gets durationMs field
- mapHistoryItemToClawMessage() emits tool-call parts BEFORE the text
  part (the order the agent produced them)
- ClawChatMessage Task view now shows tool duration in seconds

Result: historical messages now display the full tool activity
timeline grouped into the single Task collapsible per turn (designed
in step 3), instead of showing only the final text response.

* feat: render activity rows as human verbs sourced from tool registry

Tool calls in the chat activity view now read as sentences:
"Opened tab · news.ycombinator.com" instead of "browseros__new_page".

Server (tool-label-registry.ts):
- Curated verb override map for ~70 BrowserOS first-party tools
- Per-tool subject extractors that pull the meaningful argument from
  input (URL → host, query → quoted, element → ID, etc.)
- Generic fallback humanizes snake_case for any unmapped tool
- Strips MCP namespace prefixes (browseros__, mcp_)

Server (openclaw-service.ts):
- jsonlEventsToHistoryItems calls buildToolLabel for each tool_use,
  attaches label and subject to the BrowserOSChatHistoryToolCall

Client:
- Mirrored label module at lib/tool-labels.ts
- useAgentConversation tool-start handler computes label/subject
  from the SSE tool args
- ClawChatMessage and ConversationMessage render label · subject
  with foreground/muted styling, no font-mono
- ToolEntry, BrowserOSChatHistoryToolCall, and tool-call message
  part types all carry label and optional subject

* fix: drop meaningless tab N subject from page-read tool rows

Page IDs are internal numbers, not URLs. 'Took screenshot · tab 4'
tells the user nothing. Removed subject extractors for take_snapshot,
take_enhanced_snapshot, get_page_content, get_page_links, get_dom,
and take_screenshot. The verb alone is the right signal.

* fix: gate initial loading on historyQuery.isFetched not isLoading

The session and history queries are sequential: the history query is
disabled until session resolves. After session resolves, there's a render
frame where historyQuery.isLoading is still false (the query hasn't
been kicked off yet). isInitialLoading flipped to false during that
window, exposing an empty chat shell with just Task collapsibles and
copy buttons before the messages filled in.

Switching the guard to isFetched closes that window — the loading state
stays true until the first history fetch actually completes.

* fix: render historical messages immediately instead of through Streamdown's idle-callback debounce

Streamdown defaults to mode="streaming" which uses requestIdleCallback (300ms
debounce, 500ms idle timeout) and lazy/Suspense to optimize for token-by-token
live streams. For finalized historical messages this caused tool collapsibles
and copy buttons to paint while text bodies stayed blank for ~300-500ms after
load. Pass mode="static" + parseIncompleteMarkdown=false on the historical
MessageResponse so completed text paints in the same frame as the surrounding
chrome. Live streaming turns still use the default streaming mode.

Also collapse the redundant /agents/:id/session round-trip into the existing
/history endpoint (server already resolves the most recent user-chat session
when sessionKey is omitted) and tighten the initial-loading gate to stay true
across the render frame where the query is enabled but hasn't started fetching.

* feat: surface thinking duration on historical reasoning collapsibles

Server accumulates agent.thinking events per turn from JSONL and attaches a
single reasoning block (joined text + durationMs from first thinking event
to the closing agent.message) on each assistant history item. Reasoning
buffer resets on user.message alongside the tool-call buffer.

Client mirrors the type, emits the reasoning part before tool calls in
mapHistoryItemToClawMessage (chronological: think → act → answer), and
passes duration in seconds to <Reasoning> so the trigger reads "Thought
for N seconds" instead of just "Thinking" on collapsed historical turns.

* fix: read thinking blocks from the correct JSONL field name

OpenClaw stores reasoning blocks as {type:'thinking', thinking:'...'} but
the JSONL parser was reading block.text, so every thinking event was
silently dropped before it ever reached jsonlEventsToHistoryItems. As a
result the reasoning field on history items was always empty even though
the new accumulator was wired up correctly.

Also guard the client mapping: when durationMs is 0 (think + answer
emitted in the same JSONL line, no real elapsed wall-clock) pass
undefined to <Reasoning> so it renders the static "Thinking" trigger
instead of the streaming shimmer / "Thought for 0 seconds".

* fix: reset reasoning buffer on discarded turns and drop dead session hook

Two cleanups from PR review:

1. jsonlEventsToHistoryItems: when an agent.message is discarded (the
   "[Chat messages since your last reply" wrapper without a current-message
   marker) the tool buffers were already reset but the reasoning buffer
   was not. Accumulated thinking from the discarded turn would bleed onto
   the next assistant message. Reset pendingReasoningTexts and
   pendingReasoningFirstAt alongside the tool buffers.

2. useClawAgentSession, the AgentSessionResponse type, and the unused
   session entry in CLAW_CHAT_QUERY_KEYS became dead code after the
   session round-trip was folded into the history endpoint. Removed.
 2026-04-27 18:29:15 +05:30
Nikhil
5125dffbf3 fix: sign limactl with VZ entitlement (#822) 2026-04-26 13:30:09 -07:00
Dani Akash
0035893f33 feat: dashboard API, JSONL reader, and OpenClaw observer for enriched home page (#810) 
* feat: draft agent chat ui exploration

* feat: refine agent chat ui draft

* feat: remove outer frame from agent chat workspace

* fix: offset agent chat for app sidebar

* fix: simplify agent conversation shell

* fix: remove redundant chat header actions

* fix: unify agent conversation headers

* fix: tighten agent chat spacing

* fix: bound agent chat composer height

* fix: remove agent chat page inset

* fix: align agent header height with sidepanel

* fix: center agent composer resting state

* fix: anchor multiline composer controls

* fix: remove focus grid from agent home

* fix: remove redundant agent home header

* fix: constrain home agent composer

* fix: match home composer default posture

* feat: add openclaw chat history APIs

* feat: add claw chat history hydration

* fix: stabilize claw chat viewport layout

* fix: use conversation scroll base for claw chat

* refactor: split claw chat controller responsibilities

* fix: keep active agent turns in memory

* fix: normalize openclaw chat sessions

* refactor: use HTTP client for agent history instead of CLI client

Replace the CLI-based getChatHistory() call in getAgentHistoryPage()
with the HTTP client's getSessionHistory() from PR #795. This uses
the direct HTTP transport to OpenClaw's /sessions/<key>/history
endpoint instead of shelling out through the CLI.

- Add filterHttpSessionHistoryMessages() for flat-string content format
- Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape
- Update getAgentHistoryPage() to call getSessionHistory() via httpClient
- Remove unused getChatHistory(), filterOpenClawSystemMessages(),
  normalizeChatHistoryMessages(), and getTextContent()
- Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory
- Update MutableOpenClawService type: chatClient -> httpClient

* fix: fetch all session messages by iterating OpenClaw pagination

OpenClaw's HTTP history endpoint returns a limited page by default.
When called without a limit, only the first ~27 messages were returned,
causing all newer conversation messages to be silently dropped.

Add fetchAllSessionMessages() that iterates through OpenClaw's cursor-
based pagination (200 messages per page) until hasMore is false, then
feeds the complete message list into the existing BrowserOS normalization
and in-memory pagination layer.

* refactor: migrate chat history from HTTP gateway to direct JSONL file reads

Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw
gateway /sessions/:key/history pagination loop) with direct JSONL file reads
from the host filesystem via Lima's virtiofs mount.

- Add OpenClawJsonlReader that reads session JSONL files directly from
  ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/
- Replace fetchAllSessionMessages() HTTP pagination with single file read
- Replace CLI-based listSessions() with sessions.json file reads
- Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous
- Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages,
  normalizeHttpHistoryMessages helpers
- Update route handlers to drop unnecessary async/await
- Update tests to use temp JSONL files instead of mocked HTTP/CLI clients

* fix: restore async route handlers for test compatibility with mocked service

* fix: address review feedback — path traversal guard, lazy reader, exists flag

- Add safePath() to OpenClawJsonlReader that validates resolved paths stay
  within stateRoot, preventing path traversal via crafted agentId values
- Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients)
  instead of creating a new instance per property access
- Return exists: false from resolveSpecificAgentSession when no session
  matches instead of fabricating a ghost session with sessionId: ''

* feat: add dashboard API and enrich home page agent cards

Server:
- Add summarizeToolActivity() that converts tool events into natural
  language descriptions ("Browsed 3 pages, took 2 screenshots")
- Add getDashboard() to OpenClawService that aggregates per-agent stats
  from JSONL: latest message, activity summary, cost, session count
- Add GET /claw/dashboard endpoint

Client:
- Add useAgentDashboard() React Query hook (10s refetch, 5s stale)
- Rewrite useAgentCardData from async IndexedDB hook to pure
  buildAgentCardData() function merging agent entries with dashboard data
- Add activity summary and cost to AgentCardExpanded footer
- Add activitySummary and costUsd fields to AgentCardData type
- Remove IndexedDB dependency from the home page

* feat: add OpenClawObserver for real-time per-agent status via gateway WS

- Add OpenClawObserver that connects to the OpenClaw gateway WebSocket
  control plane and subscribes to chat broadcast events
- Track per-agent status in real time: working (streaming), idle (turn
  complete), error (run failed), with current tool name
- Auto-connect when gateway control plane becomes available, auto-
  reconnect on disconnect with 5s backoff
- Disconnect observer on stop/shutdown
- Wire live status + currentTool into getDashboard() response
- Update client: AgentOverview includes status + currentTool, card shows
  spinning loader + tool name when agent is working
- Status resolution: per-agent WS status takes precedence over gateway-
  level status for working/error states

* feat: add SSE dashboard stream for real-time agent status on home page

Server:
- Add GET /claw/dashboard/stream SSE endpoint that sends an initial
  snapshot then pushes per-agent status events as they arrive from
  the OpenClaw observer
- Add onAgentStatusChange() to OpenClawService exposing the observer's
  listener for the route layer
- Heartbeat every 15s to keep connections alive

Client:
- useAgentDashboard() now subscribes to EventSource at /claw/dashboard/stream
- SSE snapshot event hydrates the React Query cache immediately
- SSE status events patch individual agent status + currentTool in the
  cache without refetching — agent cards update instantly
- Polling fallback raised to 30s since SSE handles real-time

* fix: observer WS handshake — wait for challenge before sending connect

The OpenClaw gateway sends a connect.challenge event before accepting
the connect request. The observer was sending the connect request on
ws.open which raced with the challenge. Now waits for the challenge
event before sending the handshake.

Also add dangerouslyDisableDeviceAuth to the gateway setup config
batch so the observer can connect without device identity on new
installs.

* fix: JSONL reader falls back to most recent file when sessions.json is stale

OpenClaw's sessions.json can record a Pi session ID that doesn't match
the actual JSONL filename on disk. This happens after context compaction
or session restart — the JSONL file gets a new UUID but sessions.json
keeps the old one.

Previously this caused history to silently disappear (the reader tried
to open a non-existent file and returned empty). Now resolveJsonlPath()
checks if the mapped file exists and, when it doesn't, scans the
sessions directory for the most recently modified .jsonl file as a
fallback.

* feat: add ClawSession state machine for reliable per-agent status

The OpenClawObserver only knows about status changes it witnesses via
WS events. If an agent was already running when the observer connected,
or after a reconnect, statuses were stuck at "unknown".

ClawSession is an in-memory state machine that solves this:

1. Seeds from JSONL on first control plane call — reads the latest
   events for each agent and infers working/idle. A session is "working"
   if the last event is a user.message with no subsequent agent.message,
   or an agent.tool_use with no matching agent.tool_result.

2. Receives live transitions from the WS observer — the observer now
   delegates all state management to ClawSession instead of maintaining
   its own status map.

3. Applies a 5-minute staleness threshold — if the last JSONL event
   is older than 5 minutes, assume idle (handles agent crashes).

Consumers (SSE stream, dashboard endpoint) read from ClawSession and
get correct state from the first call — no "unknown" period.

* fix: remove staleTime so dashboard refetches on every mount

* fix: reset stale working status on WS disconnect, eliminate redundant JSONL reads

- Observer resets all "working" agents to "unknown" when the WS closes,
  preventing agents from appearing stuck as Working indefinitely after
  a gateway restart. ClawSession re-seeds correct state on reconnect.

- getDashboard() now derives latestAgentMessage and cost from the
  already-loaded events array for the latest session instead of calling
  latestAgentMessage() and getSessionStats() which each re-read the
  same JSONL file. Reduces file reads from 3x to 1x per agent.
 2026-04-25 19:03:03 +05:30
Neel Gupta
4284e88625 feat: Implement lazy LLM judge for passive monitoring (#777) 
* fix: double close on stream controller

* feat: initial lazy llm judge impl

* feat: added regex-based matching to insert button context

* fix: tests & bugfix

fix: redundant truthiness check

* fix(tests): stabilize server suites on dev
 2026-04-25 12:52:41 +01:00
Nikhil
0b91c735ab chore: bump server version, offset and patch for release (#814) 2026-04-24 12:05:47 -07:00
Nikhil
d189b50b03 fix: package bundled Lima guest agent (#813) 
* fix(build): upload Lima runtime files

* fix(build): stage Lima prefix resources

* fix(vm): resolve bundled Lima prefix

* docs(build): document Lima runtime packaging

* chore: self-review fixes

* fix: address review feedback for PR #813
 2026-04-24 12:03:26 -07:00
Nikhil
a407e48209 Prefetch runtime VM cache (#811) 
* feat: add runtime vm cache sync

* feat: configure runtime vm cache sync

* feat: prefetch vm cache on startup

* feat: await vm cache before vm startup

* fix: recheck vm cache after prefetch wait

* fix: address vm cache review feedback

* build(server): require VM cache manifest env
 2026-04-24 10:41:20 -07:00
shivammittal274
1f75b91fba feat(openclaw): add Claude CLI as a CLI-backed provider (#791) 
* feat(openclaw): add Claude CLI as a CLI-backed provider

Extensible registry of "OpenClaw CLI-backed providers" — tools that run
as subprocesses inside the gateway container rather than via an API key.
Claude CLI is the first entry; Gemini CLI / Codex CLI / etc. are
one-line additions in the same shape.

Backend:
- New openclaw-cli-providers/ module: types, registry, claude-cli entry.
- OpenClawService: generic ensureAllCliProvidersInstalled() (runs on
  setup/start/restart/auto-start) and getCliProviderAuthStatus(provider).
- Provider dispatch: resolveProviderForAgent() short-circuits CLI
  providers (no env var, no custom-provider merge) before falling
  through to the API-key resolver. No changes to openclaw-provider-map.
- Container runtime: PATH + NPM_CONFIG_PREFIX env so tools installed
  under /home/node/.npm-global/bin (mounted) are discoverable by
  OpenClaw's child-process spawns and persist across restarts.
- New route: GET /claw/providers/:providerId/auth-status returns
  installed / loggedIn / account / plan / error.

Frontend:
- New openclaw-cli-providers.tsx: mirrors backend registry (id, models,
  authLoginCommand), useOpenClawCliProviderAuthStatus hook (2-s poll
  while enabled), OpenClawCliProviderStatusPanel component.
- AgentsPage: synthesized CLI-provider options merged into the Create
  Agent dropdown, inline status panel, auth modal mounting the existing
  AgentTerminal with provider.authLoginCommand, auto-close on loggedIn.
- AgentTerminal: new optional initialCommand + onSessionExit props
  (ref-based so parent re-renders don't rebuild the PTY).

No global ProviderType changes. No custom container image — runtime
install into the mounted home dir persists across restarts.

* fix(openclaw): address review comments for claude-cli provider

- Drop redundant providerId field from OpenClawCliProviderOption (type
  already carries the same value).
- Reuse SetupInput type in resolveProviderForAgent instead of inlining.
- Split ensureCliProviderInstalled into probe + install so logs
  distinguish "already present" from "freshly installed".
- Narrow union in handleCreate via explicit LlmProviderConfig cast; the
  'in'-based narrowing stopped working once the two option shapes
  overlapped on required fields.

* fix: green up server-api tests after claude-cli additions

- Update container-runtime.test.ts snapshot to include the new
  PATH + NPM_CONFIG_PREFIX env args.
- Add a defensive guard in ensureAllCliProvidersInstalled so test
  mocks that swap runtime for a partial stub without execInContainer
  simply skip the install step; production runtime always provides it.

No production behavior change.

* fix(openclaw): use claude /login for auth flow and render terminal full-page

`claude auth login` in 2.1.x silently discards stdin, so the pasted OAuth
code never reaches claude. Switch to the REPL's `/login` slash command,
which does accept a pasted token. Also render the auth terminal
full-page instead of inside a Radix Dialog — the focus trap was hiding
keyboard events from xterm's helper textarea. Finally, guard the async
WebSocket in AgentTerminal against React 18 StrictMode's double-invoke
so the first mount's orphaned WS doesn't leak a second live session.

- terminal-session: pass PATH on podman exec so user-installed CLIs
  resolve in interactive sessions without manual re-exports.
- claude-cli parseAuthStatus: treat exit-code-1 as a valid "not logged
  in" JSON payload instead of a hard error.

* fix(openclaw): drop unnecessary PATH override on podman exec

`podman exec` inherits the container's run-time env (PATH includes
/home/node/.npm-global/bin via `podman run -e PATH=…`), so the extra
`-e PATH` on the exec call was redundant. Reverts the export of
GATEWAY_PATH and the exec flag added in the previous commit.

* feat(openclaw): show CLI-backed providers in Set Up dialog

The Set Up OpenClaw dialog previously listed only API-key LLM
providers. Add the CLI-backed ones (currently just Claude CLI) so
users can bootstrap the gateway with a Claude.ai-subscription-backed
agent without round-tripping through the Create Agent flow first.

When the user picks a CLI provider at setup, skip the apiKey/baseUrl
fields and open the auth terminal immediately after the gateway comes
up, so /login runs in one click.

* fix(openclaw): robust claude auth-status parsing and cleaner CLI UX

parseClaudeAuthStatus was doing JSON.parse on the entire stdout, which
fails when Lima/nerdctl appends a stderr line like `level=fatal
msg="exec failed with exit code 1"` whenever the inner command exits
non-zero (claude auth status exits 1 when not logged in). The panel
then surfaced the raw output as an error. Switch to a line-by-line
scan that picks the first parseable JSON object — handles trailing
noise and nested JSON fields cleanly.

UI polish around the Setup dialog:
- Hide the "uses your API key" hint when the selected provider is
  CLI-backed — it is inaccurate and confusing.
- When a CLI provider is picked in Setup, show a short helper line
  instead of the status panel (the /auth-status poll would be
  pre-gateway and would always fail). Set Up & Start boots the
  gateway and then auto-opens the auth terminal in one click.
- Track the active CLI provider across both Setup and Create dialogs
  so the auth terminal opens for the right provider regardless of
  which dialog triggered it.

* feat(terminal): make selection + copy work under TUI mouse tracking

Interactive TUIs like `claude /login` enable xterm mouse-tracking,
which forwards every click to the app and disables click-drag text
selection. Our terminal had no escape hatch, so users couldn't grab
the OAuth URL.

Three general-purpose fixes (none CLI-specific):
- macOptionClickForcesSelection: Opt+drag always selects on Mac,
  regardless of what the running program does with mouse events.
- Cmd/Ctrl+A and Cmd/Ctrl+C custom key handler: select-all and copy
  to clipboard via navigator.clipboard, even when the TUI would
  swallow the keys.
- Copy button in the terminal header: writes the current selection
  to the clipboard, or the full visible viewport if nothing is
  selected. One-click escape hatch that works in every state.

Applies to any interactive CLI in our terminal (sudo, vim, claude,
gh auth, etc.), not just the claude login flow.

* fix(terminal): make xterm selection actually visible

Selection was registering internally (xterm-selection layer had
correct width/height rects), but the rectangles rendered in
rgb(252,252,251) — practically invisible against the white
background — so users concluded selection was broken.

Root cause: the theme derived selectionBackground from
`withAlpha(resolveCssColor('--accent-orange'), 0.2)`. When the CSS
var failed to resolve it fell back near-white, and the alpha
compositing against the page background made the result
indistinguishable from the background.

Switch to solid terminal-standard selection colors (VSCode-like
light-blue / dark-indigo). Also set selectionInactiveBackground so
the selection persists when focus moves away (useful while copying).
Drop the now-unused withAlpha helper.

* fix(openclaw): handle pretty-printed JSON in claude auth status parser

claude auth status --json emits multi-line pretty-printed JSON. The previous line-by-line parser never matched, so the UI treated every response as an error and surfaced the raw JSON — even when loggedIn was true. Replace with a brace-matching JSON extractor (string- and escape-aware) that tolerates multi-line JSON, leading banners, trailing lima/nerdctl stderr, and nested objects.

* refactor(openclaw): separate exec streams, argv installs, cleaner async cleanup

Audit-driven cleanup. Net -42 lines, four concrete issues fixed:

1. ContainerRuntime.runInContainer() exposes {exitCode, stdout, stderr}
   from the nerdctl exec (ContainerCli.runCommand already tracked them
   separately; we were just throwing stderr into the same string). The
   40-line hand-rolled brace-matching JSON extractor in claude-cli.ts
   existed only because the prior merged-stream output had lima/
   nerdctl's 'level=fatal' line fused with claude's JSON. parser is
   now JSON.parse(stdout.trim()).

2. Replace shell-based 'sh -lc "npm install -g ${pkg}@latest"' with
   argv: execInContainer(['npm','install','-g','${pkg}@${version}']).
   Registry values no longer flow through a shell (removes injection
   surface from future CLI providers). Pinned version instead of
   @latest (adds npmPackageVersion to the provider type).

3. AgentTerminal: replace the 'let cancelled' + out-of-effect
   disposeSocketBindings pattern with an AbortController scoped to
   the effect and a cleanups[] array. Matches the canonical React 18
   async-effect pattern — no partial-cleanup race if StrictMode
   unmounts between the async await and the resolve.

4. AgentTerminal: drop the full-buffer fallback in the Copy button
   (was copying all 8000 scrollback lines when nothing selected —
   surprising). Button now only copies the actual xterm selection,
   or no-ops silently. Users who want everything can Cmd+A first.
 2026-04-24 20:13:18 +05:30
Dani Akash
752f42d1fe refactor: migrate chat history to direct JSONL file reads via Lima filesystem (#808) 
* feat: draft agent chat ui exploration

* feat: refine agent chat ui draft

* feat: remove outer frame from agent chat workspace

* fix: offset agent chat for app sidebar

* fix: simplify agent conversation shell

* fix: remove redundant chat header actions

* fix: unify agent conversation headers

* fix: tighten agent chat spacing

* fix: bound agent chat composer height

* fix: remove agent chat page inset

* fix: align agent header height with sidepanel

* fix: center agent composer resting state

* fix: anchor multiline composer controls

* fix: remove focus grid from agent home

* fix: remove redundant agent home header

* fix: constrain home agent composer

* fix: match home composer default posture

* feat: add openclaw chat history APIs

* feat: add claw chat history hydration

* fix: stabilize claw chat viewport layout

* fix: use conversation scroll base for claw chat

* refactor: split claw chat controller responsibilities

* fix: keep active agent turns in memory

* fix: normalize openclaw chat sessions

* refactor: use HTTP client for agent history instead of CLI client

Replace the CLI-based getChatHistory() call in getAgentHistoryPage()
with the HTTP client's getSessionHistory() from PR #795. This uses
the direct HTTP transport to OpenClaw's /sessions/<key>/history
endpoint instead of shelling out through the CLI.

- Add filterHttpSessionHistoryMessages() for flat-string content format
- Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape
- Update getAgentHistoryPage() to call getSessionHistory() via httpClient
- Remove unused getChatHistory(), filterOpenClawSystemMessages(),
  normalizeChatHistoryMessages(), and getTextContent()
- Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory
- Update MutableOpenClawService type: chatClient -> httpClient

* fix: fetch all session messages by iterating OpenClaw pagination

OpenClaw's HTTP history endpoint returns a limited page by default.
When called without a limit, only the first ~27 messages were returned,
causing all newer conversation messages to be silently dropped.

Add fetchAllSessionMessages() that iterates through OpenClaw's cursor-
based pagination (200 messages per page) until hasMore is false, then
feeds the complete message list into the existing BrowserOS normalization
and in-memory pagination layer.

* refactor: migrate chat history from HTTP gateway to direct JSONL file reads

Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw
gateway /sessions/:key/history pagination loop) with direct JSONL file reads
from the host filesystem via Lima's virtiofs mount.

- Add OpenClawJsonlReader that reads session JSONL files directly from
  ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/
- Replace fetchAllSessionMessages() HTTP pagination with single file read
- Replace CLI-based listSessions() with sessions.json file reads
- Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous
- Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages,
  normalizeHttpHistoryMessages helpers
- Update route handlers to drop unnecessary async/await
- Update tests to use temp JSONL files instead of mocked HTTP/CLI clients

* fix: restore async route handlers for test compatibility with mocked service

* fix: address review feedback — path traversal guard, lazy reader, exists flag

- Add safePath() to OpenClawJsonlReader that validates resolved paths stay
  within stateRoot, preventing path traversal via crafted agentId values
- Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients)
  instead of creating a new instance per property access
- Return exists: false from resolveSpecificAgentSession when no session
  matches instead of fabricating a ghost session with sessionId: ''
 2026-04-24 13:19:46 +05:30
Nikhil
2f8e36546f fix: resize BrowserOS VM resources (#807) 2026-04-23 18:24:49 -07:00
Nikhil
461dcd29e8 fix: upload Lima resources under vendor prefix (#805) 2026-04-23 17:19:45 -07:00