mirror of
https://github.com/browseros-ai/BrowserOS.git
synced 2026-05-18 19:16:22 +00:00
0c84547e8f
* chore(acp): smoke-test ACP capabilities against running gateway
Adds apps/server/scripts/acp-smoke.ts which spawns `openclaw acp`
inside the gateway container and exercises every method we plan to
depend on: initialize, newSession, prompt (text + image), cancel,
listSessions, loadSession.
SDK pinned to 0.19.1 (Bun's minimum-release-age policy blocks 0.20+
which were released < 7 days ago).
Findings (full notes in plan outcomes):
- promptCapabilities advertises image:true but the model does NOT see
image bytes — silently dropped at the bridge.
- sessionCapabilities advertises {list:{}} but session/list throws
"Method not found": stale capability advertising.
- loadSession works; replays user/assistant/thought text and
session_info/usage/commands updates. No tool_call replay, as
documented.
- cancel works end-to-end: stopReason=cancelled.
- closeSession/resumeSession are not on ClientSideConnection in
0.19.1; kill child to close, use loadSession for rebind.
Plan revisions triggered by spike are recorded in
plans/browseros-ai/BrowserOS/features/2026-04-28-2310-claude-code-acp-implementation-roadmap.md.
* chore(acp): re-run smoke on SDK 0.21.0 and add mode/config/auth scenarios
After bypassing Bun's minimum-release-age and upgrading the SDK to
0.21.0, restore the previously-skipped resume/close paths and add
three new scenarios: mode (setSessionMode), config (setSessionConfigOption,
correct configId field), and auth (authenticate noop).
Findings, all bridge-side (independent of SDK):
- session/list, session/resume, session/close all throw -32601 on
OpenClaw 2026.4.12 — capability advertising is stale.
- Image content blocks silently dropped; model never sees the bytes.
- setSessionMode and setSessionConfigOption work; latter requires
`configId` (not `optionId`) per the schema.
- loadSession replays user/assistant/thought text + session_info +
usage + available_commands; no tool_call replay (documented).
- authenticate is a noop on OpenClaw (no authMethods advertised).
Plan outcomes updated with full method-support matrix.
* chore(deps): promote @agentclientprotocol/sdk to a runtime dependency
The smoke script in apps/server/scripts/acp-smoke.ts used the SDK as
devDependency. The upcoming ACP bridge (apps/server/src/api/services/acp/)
needs it at runtime, not just for tooling. Move the entry from
devDependencies to dependencies, alphabetically first under @a*.
Pinned to 0.21.0 — same version the smoke script validated against.
README gains a small Dependencies note pointing at the future bridge
location.
No code changes yet. The bridge wiring lands in subsequent commits.
* fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config
When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.
Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
side (useOpenClaw.ts) and the route body schema + SetupInput +
createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
`input: ['text', 'image']` on the model entry when the flag is
truthy; otherwise emits the explicit `['text']` so the value is
always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
bootstrap batch so the gateway's image-understanding pipeline is
always wired up — per-model `input` still gates which models see
images, this just enables the global path.
ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.
Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.
* feat(agents): add OpenClaw to AgentAdapter union and catalog
Extends AgentAdapter to 'claude' | 'codex' | 'openclaw' and adds the
OpenClaw entry to AGENT_ADAPTER_CATALOG. The new entry has:
- defaultModelId: 'default' — OpenClaw's ACP bridge does not surface
per-session model selection (verified during the ACP spike), so
models live in the OpenClawService config, not in the adapter
catalog. AgentDefinition.modelId carries the gateway-side model
name for display only.
- models: [] — empty list signals "no per-session model picker" in
the UI; isSupportedAgentModel('openclaw', undefined|'default')
returns true via the existing fallback path.
- reasoningEfforts mirror OpenClaw's session-level `thought_level`
config option (off / minimal / low / medium / high / adaptive).
Also extends:
- isAgentAdapter type guard recognizes 'openclaw'
- HarnessAgentAdapter union on the extension side
- agents.test.ts createAgent fake type
- agent-catalog.test.ts asserts on the new entry, empty model list
passthrough behavior, and OpenClaw's reasoning effort set
Lockfile delta is the workspace SDK pin reconciling 0.20.0 (taken
from dev's lock) up to our package.json's 0.21.0 (added in
c1d987ea). acpx still uses 0.20.0 transitively — both are present.
No runtime wiring yet — the registry override and AcpxRuntime
plumbing land in subsequent commits.
* feat(agents): plumb OpenClaw gateway accessors into AcpxRuntime
Adds an optional `openclawGateway` accessor to AcpxRuntime so the
upcoming registry override (Step 4) can spawn `openclaw acp` inside
the gateway container with the right port, token, and container/VM
identity. All accessors are getter-shaped so values stay live across
gateway restarts (port can change, token can rotate).
The accessor is threaded:
server.ts → createAgentRoutes → AgentHarnessService → AcpxRuntime
↘ sidepanel lazy AcpxRuntime
Also adds OpenClawService.getGatewayToken() returning the in-memory
token string. We pass it via OPENCLAW_GATEWAY_TOKEN env var on the
spawn (per OpenClaw's documented env-var precedence) instead of via
`--token` flag (which leaks to ps aux) or `--token-file` path (no
discrete token file lives inside the container — the token is nested
inside openclaw.json).
Wiring is dormant — the registry override that consumes these
accessors lands in Step 4. Typecheck + existing acpx/harness/routes
tests pass unchanged.
* refactor(agents): scrub local plan-step references from code comments
Replaces forward-looking comments that referenced internal plan
steps (e.g. "Step 4 wires this into…") with comments that justify
the code on its own merits. Plan files live locally on the
contributor's machine, so cross-references are noise to the rest of
the project.
No behavior change.
* feat(agents): spawn openclaw ACP adapter inside the gateway container
When the harness resolves the `openclaw` adapter, it now returns a
command that runs `openclaw acp` inside the bundled gateway container
via `limactl shell <vm> -- nerdctl exec -i ... openclaw acp --url
ws://127.0.0.1:<port>`. This reuses the openclaw binary already
installed alongside the gateway — no host-side openclaw install is
required.
Auth: the gateway token is injected via OPENCLAW_GATEWAY_TOKEN on
the container exec rather than `--token` on the openclaw CLI, so
the secret never appears in `ps aux`.
Banner output: OPENCLAW_HIDE_BANNER=1 and OPENCLAW_SUPPRESS_NOTES=1
keep stdout JSON-RPC-clean.
LIMA_HOME: prefixed via `env LIMA_HOME=<path>` on the resolved
command so the spawned limactl finds the BrowserOS-owned VM (the
server doesn't set LIMA_HOME on its own process env).
When the gateway accessor is absent, falls through to acpx's
built-in openclaw adapter which assumes a host-side install — that
branch will fail at spawn time with a descriptive error.
Verified end-to-end via the existing acp-smoke script during the
Step 0 spike.
* feat(agents): dual-create OpenClaw harness agents on the gateway
When the harness creates an `openclaw` adapter agent, it now also
provisions a matching agent on the OpenClaw gateway via the existing
CLI path (OpenClawService.createAgent). Symmetric on delete: gateway
removeAgent runs alongside the harness-store delete.
- Adds an OpenClawProvisioner interface (decoupled from OpenClawService
for testability) and injects it through AgentHarnessService.
- createAgent rolls back the harness record if gateway provisioning
fails; deleteAgent tolerates gateway-side failures so harness
identity stays consistent with the user-facing UI.
- New OpenClawProvisionerUnavailableError surfaces as a 503 when an
openclaw create request lands on a harness with no provisioner
wired in (instead of a generic 500).
- FileAgentStore mints openclaw agent ids with an 'oc-' prefix so
the id satisfies the gateway's `^[a-z][a-z0-9-]*$` agent name
pattern. Other adapters keep raw UUIDs to preserve compatibility.
- POST /agents body schema accepts providerType / providerName /
baseUrl / apiKey / supportsImages, forwarded to the provisioner
when adapter='openclaw'.
The agents-page UI still routes openclaw create through the legacy
/claw/agents flow; switching that path to the harness is a separate
UI cutover.
Tests cover: gateway dual-create on success, rollback on gateway
failure, 503 when provisioner is missing, and tolerant delete on
gateway-side failure.
* fix(agents): skip catalog model validation for OpenClaw adapter
OpenClaw agents resolve their model from the gateway-side provider
config (set at agent-create time via OpenClawService) rather than
from the harness catalog, which has an empty `models: []` entry by
design. Without this carve-out, every OpenClaw create body fails
parsing with "Invalid modelId" because no concrete model id can
satisfy isSupportedAgentModel('openclaw', ...).
The reasoning-effort check still runs against the catalog (those
values map directly to OpenClaw's session `thought_level` config
option).
* fix(agents): pass --session to openclaw bridge so newSession routes correctly
acpx's AcpClient.createSession calls connection.newSession with cwd
and mcpServers but never forwards the sessionKey. Without it, the
openclaw bridge falls back to a synthetic acp:<uuid> session that
doesn't resolve to any provisioned gateway agent — every harness
chat returns a generic "Internal error" from -32603.
Fix: bake `--session <key>` into the resolved spawn command. The
bridge then uses that as the default session key for any newSession
the bridge receives, routing the turn to the matching gateway agent.
Per-session keying means each openclaw agent gets its own
AcpxCoreRuntime instance (cached by sessionKey on top of the
existing cwd/permissionMode key). This adds one extra runtime per
active openclaw session — claude/codex are unaffected.
Test asserts the resolved command includes the right --session arg.
* fix(agents): suppress BrowserOS MCP for openclaw bridge
The openclaw ACP bridge rejects newSession when mcpServers is non-empty
because its provider tooling comes from the gateway, not from ACP-side
MCP servers. Forwarding the BrowserOS HTTP MCP made every harness chat
fail with a JSON-RPC -32603 "Internal error" before the session was even
opened. Claude/codex still need the BrowserOS MCP for browser tooling,
so the carve-out is keyed off whether the runtime is for an openclaw
session.
* feat(agents): route OpenClaw chat through the harness behind a flag
Adds the `feature.useAcpxForOpenClaw` extension storage flag. When on,
OpenClaw agents in the agent-command chat panel use the harness
/agents/<id>/chat SSE and harness history hook instead of the legacy
/claw/agents/<id>/chat. When off, behavior is unchanged.
Also dedupes the agent rail when the same id appears in both stores
(dual-created agents from /claw/agents and /agents) by preferring the
harness entry — without this, every dual-created OpenClaw agent shows
up twice after Step 5.
Image attachments are temporarily disabled when the harness path is
active; the carve-out lands in the next commit.
* fix(agents): keep legacy OpenClaw agents on ClawChat
The previous commit's flag-gated branch routed every `source='openclaw'`
agent through `/agents/<id>/chat` when the flag was on, but the layout
dedup means the only agents that ever reach that branch are legacy
gateway-only entries (`main`, orphan agents from rolled-back creates) —
which by definition have no harness record, so the harness path 404s
and chat is unusable. Source is the only routing signal again: harness
agents go through the harness, legacy agents stay on ClawChat. The
storage flag stays for Step 9/10's migration story.
* feat(agents): expose OpenClaw in sidepanel and route through gateway main
`buildSidepanelChatTargets` now emits a single default ACP target for
adapters with no per-session model picker (OpenClaw, whose model is
configured on the gateway-side agent). Without this, OpenClaw never
appeared in the sidepanel target picker because the catalog entry has
`models: []`.
Sidepanel sessions don't have a dedicated provisioned gateway agent.
The openclaw bridge `--session` flag previously got the raw sidepanel
key (`sidepanel:<convId>:openclaw:...`), which doesn't match any
gateway agent — newSession was accepted but every prompt hung
forever. The bridge command now rewrites non-harness session keys
onto the always-present `main` gateway agent, encoding the original
key as a channel suffix to keep state segregated per conversation.
Verified end-to-end via curl: sidepanel openclaw chat streams
`text-delta` + `finish: stop`.
* feat(agents): backfill harness records for legacy gateway agents
Reframes Step 9 of the OpenClaw-on-acpx migration. The plan's literal
Step 9 (route OpenClaw history through the harness when the flag is on)
was already a no-op after the Step 6 walkback — history is routed by
source today. The actual blocker for Steps 10–13 was that legacy
gateway-only agents (e.g. `main`, orphans from rolled-back creates) had
no harness record, so they could never migrate to the harness path
without breaking chat.
`AgentHarnessService.reconcileWithGateway()` now lists every gateway
agent and upserts a matching harness record for any that are missing.
The pass runs lazily on first `listAgents()` call (memoized on success,
retried on failure so a gateway-down boot doesn't permanently disable
backfill). Verified end-to-end: the legacy `agent` agent now streams
`text_delta` + `done(end_turn)` through `/agents/agent/chat`, with the
bridge resolving to the gateway's `agent` record via the existing
`agent:<name>:main` session-key format.
After this, every OpenClaw agent surfaces as `source='agent-harness'`
post-dedup, the legacy `useClawChatHistory` hook becomes unreachable
for OpenClaw, and Steps 11–13 (delete legacy chat/history paths) are
unblocked.
* fix(agents): drop duplicate OpenClaw entry from NewAgentDialog adapter list
The adapter Select hardcoded an `<SelectItem value="openclaw">OpenClaw</SelectItem>`
on top of iterating `adapters`, which now includes OpenClaw post the
catalog change. The dropdown rendered "OpenClaw" twice — once at the
top, once at the bottom of the list. The literal was a pre-catalog
artifact; removing it leaves a single OpenClaw entry sourced from the
catalog. Routing into `handleOpenClawCreate` is unchanged because
the value (`'openclaw'`) is identical either way.
* fix(agents): always reconcile harness with gateway on list, just dedupe concurrent calls
Memoizing the first successful reconcile meant new gateway agents (created
via the legacy /claw/agents path or out-of-band CLI) never appeared in the
harness until server restart. The Promise now serves as a concurrent-call
dedupe only — cleared on settle — so every listAgents call picks up the
current gateway state. Reconcile is one cheap idempotent CLI call.
* chore(agents): remove dormant useAcpxForOpenClaw flag
The flag was scaffolded in Step 6 but its routing effect was walked
back the same day after it broke chat for legacy gateway-only agents.
After the Step 9 backfill, every OpenClaw agent has a harness record
and routes through the harness path purely from `source='agent-harness'`
— no flag is consulted anywhere. Remove the dead storage item, hook,
and stale comment.
* refactor(agents): drop legacy /claw/agents/:id/history endpoint
The harness /agents/:id/sessions/main/history endpoint replaced this
once every OpenClaw agent got a harness record (Step 9 backfill).
Routing is fully source-driven now, so the UI's useClawChatHistory
hook is never enabled today — verified live: legacy URL returns 404,
harness history hydrates correctly for the same agent.
Removes the GET /claw/agents/:id/history route, OpenClawService's
getAgentHistoryPage method plus its cursor/limit helpers and the
history-only types it owned (BrowserOSOpenClawHistoryPageResponse,
HistoryPageInput, normalizeHistoryLimit, encodeHistoryCursor,
decodeHistoryCursor, jsonlEventsToHistoryItems), and the route +
service tests that covered the dropped endpoint.
OpenClawJsonlReader stays alive — still feeds /claw/dashboard,
/claw/agents/:id/sessions, and the boot-time clawSession seed.
Removing those is its own follow-up since the dashboard would need
a harness-side replacement first.
* feat(agents): wire image attachments through the harness ACP path
Composer attachments now flow into the ACP `prompt` request as
spec-compliant `image` content blocks alongside the user's text. End
to end:
composer → chatWithHarnessAgent({attachments}) →
POST /agents/:id/chat {message, attachments} →
parseChatBody decodes data: URLs to {mediaType, base64} →
AgentHarnessService.send forwards →
AcpxRuntime.send forwards →
acpx startTurn({attachments}) → ACP image blocks
UI no longer disables the attach button on harness agents — the
gating was just a placeholder before this commit landed. Verified
end to end with a 1×1 red PNG against a Claude harness agent: model
replies "Red." correctly.
OpenClaw's `acp` bridge still drops image content blocks upstream
(verified by the same probe — Kimi-k2p5 reports "I don't see an
image"). That's an upstream openclaw limitation, not a harness-side
gap; Claude/Codex agents work as advertised today.
* chore(openclaw): delete OpenClawJsonlReader and JSONL-backed routes
* chore(openclaw): remove legacy /claw/agents/:id/chat and /queue routes
* chore(agents): collapse chat panel to harness-only path
* feat(agents): route OpenClaw image turns through the gateway HTTP client
The OpenClaw `acp` bridge silently drops ACP `image` content blocks
(verified during dogfood — model says "I don't see an image"). When
the user attaches images to an OpenClaw agent, the harness now diverts
that turn to the gateway's HTTP `/v1/chat/completions` endpoint, which
accepts OpenAI-style `image_url` parts and forwards them natively to
the provider.
- New `OpenClawGatewayChatClient` translates an OpenAI streaming
response into the same `AgentStreamEvent` shape the rest of the
harness already consumes, so the chat panel renders identically
whether a turn went through ACP or the gateway carve-out.
- `AcpxRuntime.send` forks at the top: openclaw + any image
attachment + a wired gateway client → `sendOpenclawViaGateway`.
Other turns (text-only openclaw, claude, codex) take the existing
ACP path unchanged.
- The diverted path reads the prior turn history from the acpx
session record so context is preserved, builds the OpenAI
multimodal user message with text + image_url parts, and pumps
the gateway SSE back to the caller through a tee that accumulates
the assistant text. On natural completion, persists a synthetic
user+assistant message pair to the acpx session record so reload
shows the image turn in history.
- Wired `OpenClawGatewayChatClient` into `AgentHarnessService` via
`server.ts` (gateway port + token accessor, just like the existing
`openclawGateway`).
Persistence note: the acpx record requires User messages to carry an
`id` and Agent messages to carry `tool_results` — without them the
record fails to round-trip through `parseSessionRecord`. The persist
helper now sets both.
Limitation by design: image recognition only works if the OpenClaw
agent's provider supports vision (e.g. Claude-via-OpenClaw, GPT-4o).
The pipeline routes images correctly to the provider regardless;
text-only providers like Kimi-k2p5 will reply "I don't see an image"
because the model itself has no vision capability — that's a provider
config issue, not a routing bug. The unit test asserts the image_url
part is present in the OpenAI request the gateway client sends.
The wider plan (background-resilient chat, queue, replay) remains in
`plans/.../2026-04-29-1527-...-background-resilient-chat-and-image-uploads.md`
as Phases 3–12; this commit ships only Phases 1–2.
* feat(agents): validate inbound image attachments on /agents/:id/chat
The harness chat body parser was accepting any mediaType and any
dataUrl length. The composer enforces these caps client-side but the
endpoint also serves direct curl/script callers, so the server has to
defend itself.
Restores the same caps the legacy /claw/agents/:id/chat parser had
before it was deleted in the migration:
- 10 attachments per message
- 5 MB raw image bytes (≈ 6.7 MB once base64-encoded plus prefix)
- PNG / JPEG / WebP / GIF only
- Must start with `data:`
Each violation returns 400 with a specific error message instead of
silently dropping or forwarding the payload.