mirror of
https://github.com/browseros-ai/BrowserOS.git
synced 2026-05-20 20:39:10 +00:00
fix/patch-cli-sync
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Dani Akash
|
ce4bb44083 |
feat(agent): /home composer parity with image attachments (#930)
* feat(agent): /home composer parity with image attachments
The /home composer used the same ConversationInput component as the
chat screen but passed attachmentsEnabled={false}, and the home →
chat handoff was a URL search param `?q=<text>` that physically
can't carry binary attachments. Pasting a screenshot at /home did
nothing.
Add a small in-memory registry (pending-initial-message.ts) as the
rich-data side channel for the same navigation: the home composer
writes { agentId, text, attachments } there before navigating; the
chat screen consumes it on mount and replays through the existing
harness send() path that already supports attachments. URL `?q=`
stays for shareable text-only prompts; the registry wins when both
are present. Module-scope, 10s TTL, destructive consume.
Net: home is now flagged attachmentsEnabled={true}; users can paste,
drag, or pick image files at /home and they survive the navigation
into the chat screen with previews intact.
* docs(agent): clarify why initial-message ref reset is safe post-registry-fire
|
||
|
Dani Akash
|
84e2739663 |
feat(agent): rich rail + header on /agents/:agentId chat (#908)
* feat(agent): rich rail + header on /agents/:agentId chat Replace the chat screen's legacy AgentEntry rail and binary READY header with the same rich data the /agents page already exposes: adapter glyph, liveness dot, pin star, status badge, adapter · model · reasoning chip line, last-used time, lifetime tokens, queue count, and the Adapter Unavailable warning. Source of truth flips from the merged AgentEntry list to useHarnessAgents() directly. Sort order matches /agents (pinned → recency) — not /home (active-first → recency) — because chat is index-shaped and shuffling rows every 5s as turns transition would be jarring while reading. Lift the inline pin-then-recency comparator out of /agents AgentList.tsx into a shared agents-list-order.ts so both surfaces stay on identical sort semantics. * fix(agent): chat header height + composer sticking to bottom Header was clipping descenders because the strip was vertical-content sized at min-h-14 with tight py-2.5; bump padding and lean on natural content height. Drop the AgentTile glyph (the rail row already shows adapter identity) and the cwd path (too long, pushed the meta line off-screen). Header is now name + pin star + status pill, then adapter · model · reasoning, then last-used · tokens · queued. Composer was floating mid-screen on short chats because the chat grid had no grid-template-rows — the implicit auto row collapsed to content height, so the right-column flex wrapper never received the full container height. Add grid-rows-[minmax(0,1fr)] so the single row claims 100% and ClawChat's flex-1 expands to push the composer flush to the bottom. * fix(agent): composer flush to bottom on short chats Match the sidepanel chat's nested-flex pattern. The right-column wrapper got h-full so it expands to the grid row; the conversation controller's root added flex-1 so ClawChat's existing flex-1 has something to actually fill against. Without these, the grid cell stretched but the inner flex columns shrank to content height, leaving the composer floating mid-screen. * fix(agent): align rail header with chat header in shared top band Pull the rail's "Agents" + back-button into the same horizontal strip as the agent identity header. The two halves now sit on a single row that spans both columns, so they can't drift in height as the chat header gains/loses meta lines (last-used, tokens, queued). The rail below the band keeps its scrollable list only; the chat column below holds the conversation + composer. Border-bottom moves from ConversationHeader to the band wrapper so we don't get a double-rule on the boundary. * fix(agent): reserve header height to prevent layout shift on data load The chat header grew from a single line to three lines once the useHarnessAgents() poll resolved (adapter chips + meta line populate asynchronously), shoving the rail and conversation body downward. Lock min-h-[84px] on both the band's left "Agents" cell and the ConversationHeader root, and always render the meta line slot (non-breaking space when empty) so the typographic frame is stable regardless of data state. * refactor(agent): pull status pill + meta to right side of chat header Two-column header layout instead of three stacked rows: name + pin star + adapter chips on the left, status pill stacked on top of the last-used / tokens / queued meta line on the right. Drops min-h from 84px → 60px so the band reclaims ~24px of vertical space and the chat body starts higher on screen. Band's left "Agents" cell matches the new height. |
||
|
Dani Akash
|
8712f89f18 |
feat(agents): durable per-agent chat message queue + composer Stop (#880)
* feat(agents): durable per-agent chat message queue + composer Stop button
* fix(agents): tighten queue UI — smaller Stop, drop empty indicator, live drain attach
User feedback round 1 on the message-queue UX:
1) The Stop button matched the send/voice mics at h-10 w-10 with a
solid destructive fill, which read as alarming. Shrunk to h-8 w-8,
ghost variant with a soft destructive/10 background, smaller
filled square glyph. Reads as a calm 'stop' affordance instead of
a panic button.
2) The QueueItem's leading <QueueItemIndicator> dot was decorative
only — no state, no interaction. Dropped it from QueuePanel along
with the import; queue items now render as a clean preview line
with the trailing X remove action.
3) When the server drained the queue and started the next turn, the
chat panel didn't pick up the live stream until the user
navigated away and back. The hook's resume effect previously
only fired on agent change, not on listing-observed activeTurnId
change. Surface activeTurnId from useHarnessAgents into
useAgentConversation; effect now re-runs when the id changes,
calls /chat/active, and attaches to the new turn — so a queued
message starts streaming the moment the server drain pops it.
* fix(agents): don't reset streaming state from the resume effect's no-op paths
The Stop button was disappearing while the agent was actively
streaming, even though events were still flowing into the chat. Root
cause: the resume effect's `finally` block reset `streaming`,
`turnIdRef`, and `lastSeqRef` unconditionally — including on the
early-return paths (no active turn, or another mechanism already
owns the stream).
Sequence that triggered it:
1) User sends a message → send() sets streamAbortRef + streaming=true
and starts consuming the SSE.
2) User enqueues another message → enqueue mutation invalidates the
listing query.
3) Listing refetches with the live activeTurnId → the resume
effect re-fires (deps include activeTurnIdDep).
4) attemptResume hits `if (streamAbortRef.current) return` because
send() owns it.
5) The finally clause fires anyway and calls setStreaming(false),
clobbering the live state set by send(). The SSE consumer keeps
running (refs are intact) so text keeps streaming, but the React
flag is wrong, so the Stop button gates off.
Fix: track whether *this* run actually started a stream
(`weStartedStream`). The finally only resets state when it does.
Early-return / no-active-turn paths now leave streaming/turnIdRef/
lastSeqRef alone for whoever does own them.
Also widens the Stop button's visibility (`canStop` prop on
ConversationInput) so it stays steady across the brief gap between
turns when a queue drain is mid-flight; the parent computes
`streaming || activeTurnId !== null || queue.length > 0`. The
visibility widening is independent of the streaming-state fix above
— both are now in place.
* revert: drop canStop widening — Stop only shows while streaming
Reverts the canStop prop on ConversationInput and the OR-with-queue
visibility from AgentCommandConversation. Stop is gated solely on
`streaming` again. Between turns (queue draining) the button stays
hidden — only the actively-streaming turn is interruptible from the
composer, which matches what the user actually expects.
* fix(agents): persist the kicking-off prompt on active turns so the resume placeholder isn't empty
When a queued message drained and started a new turn, the chat
panel's resume effect staged a placeholder turn with userText: ''
because the hook had no way to know what message kicked off the
turn — only the agent-side stream was visible, and the user bubble
above it was blank until the user navigated away and back (at which
point the session record's history loaded normally).
Fix: ActiveTurnRegistry.register now accepts an optional `prompt`
that's stashed on the turn and surfaced via describe() / the
ActiveTurnInfo response. AgentHarnessService.startTurn passes the
incoming message into register. /chat/active returns it. The chat
hook's resume effect uses active.prompt as the placeholder
turn's userText, so the user bubble shows the queued message text
the moment streaming begins. Falls back to '' for older clients
that haven't been refetched yet.
* fix(agents): always release streamAbortRef on resume cleanup, even when cancelled
Greptile P1 follow-up. The previous `weStartedStream` guard correctly
stopped the resume effect's no-op early-returns from clobbering an
in-flight `send()` stream — but it also stopped a *cancelled*
mid-stream resume from clearing its own `streamAbortRef`. When the
cleanup fires (e.g. the 5s listing poll captures a new queue-drain
turn id while the SSE for the prior turn is still finishing), the
next effect run hits the `if (streamAbortRef.current) return` guard
against the now-aborted controller and never reattaches, leaving
`streaming === true` with no live stream until the user navigates
away.
Split the finally block: always release `streamAbortRef` when we
owned the controller (so the next run can take over), but only
reset the streaming flag / turn id / lastSeq on a clean exit (the
new run will set those itself, so resetting on cancel would just
flicker).
|
||
|
Dani Akash
|
ba60bf466f |
feat(agents): rich command-center rows + home grid + dead-code sweep (#879)
* feat(agents): rich-info command center rows + pin/PATCH/adapter-health backbone
Splits AgentRowCard from a 271-line monolith into a shallow tree of
single-responsibility sub-components under `agent-row/`:
AgentTile, AdapterHealthDot, PinToggle, AgentTitleRow,
AgentSparkline, AgentSummaryChips, AgentLastMessage, CwdChip,
AgentTokenSummary, AgentMetaRow, AgentErrorPanel, AgentActions
Adds the data each row consumes:
- pinned: boolean field on AgentDefinition + FileAgentStore.update
+ new PATCH /agents/:id route. useUpdateHarnessAgent mutation
optimistically updates the listing cache so the star flips
instantly; rolls back on error.
- Listing payload extended with lastUserMessage, cwd, tokens
(cumulative + last7d shape — last7d zero-filled until the
activity ledger lands), turnsByDay/failedByDay (zero-filled),
lastError/lastErrorAt, activeTurnId. AcpxRuntime grows a
getRowSnapshot() that reads cwd + cumulative tokens + last user
message from the session record in one pass.
- Adapter health: in-memory AdapterHealthChecker probes
`claude --version` / `codex --version` with a 2s timeout and
caches results for 5 min. /adapters response carries
{ healthy, reason?, checkedAt }. Tile-corner dot exposes the
state via HoverCard; openclaw inherits health from the gateway
snapshot already on the page.
Sub-components are pure: card itself owns no state. Sort order
becomes pinned-first, then recency. HoverCard is the workhorse for
keeping rows compact while exposing depth (full message, token
breakdown, daily turn list, error stack, adapter reason).
* refactor(agents): tighten command-center row design + cut redundant affordances
User feedback round 1:
1) Two green dots on the tile (health + liveness) was confusing. Health
moves out of the tile entirely and surfaces as an inline 'Unavailable'
chip in the model line — silent when the adapter is healthy, with a
warning amber chip + HoverCard reason when not. The tile now shows
one signal: liveness.
2) The last-user-message HoverCard wasn't telegraphing intent. Drop the
HoverCard. The line is informational, italic, with a leading quote
glyph so the row reads like a conversation snippet. To see the full
message the user opens the chat (which is the action they want next
anyway).
3) Resume + Chat were duplicate CTAs. Single primary action per row:
Resume (filled, accent-orange, with a pulsing dot) replaces Chat
when there's an active turn. Both navigate to /agents/:id but the
row tells the user which action they're taking.
4) Tokens weren't visible because the row gated on last7d.requestCount,
which is zero until the activity ledger ships. Switch to lifetime
tokens (which we have today). Drop the '7d stats:' framing — talking
about a window we can't compute would be misleading. The HoverCard
surfaces input/output split + a footnote that per-window stats land
in a follow-up.
5) CWD was rendering the server's own running directory, which is
meaningless to users. Hide it from the row entirely. The cwd field
still rides in the listing payload for future surfaces (chat panel,
debug view) — only the row stops rendering it.
Aesthetic refinements while we're here:
- Whole card carries state, not just the tile: working rows get an
accent-orange tinted border with a soft glow, error rows tint
destructive, idle rows lift on hover.
- Pin star fades in on hover (group-hover) when unpinned and stays
solid amber when pinned — keeps the rail calm by default.
- Tabular-nums on token figures so columns visually align across rows.
- Drop CwdChip and AdapterHealthDot files: no callers left.
* fix(agents): align row title flush-left whether pinned or not
Pin star moved from leading the title to trailing the badges, and
hidden from layout entirely (`hidden group-hover:inline-flex`) when
unpinned. The previous `opacity-0` rule kept the star reserving its
`size-6` slot, which left every unpinned title indented relative to
the model / preview / meta lines underneath it. Title now flushes
left in both states; pinned star stays solid amber so the signal
isn't hidden, and unpinned reveals an outline star on row hover for
the toggle affordance.
* fix(agents): keep pin-toggle slot reserved so row height is constant
Switching the unpinned star from `hidden group-hover:inline-flex`
to `opacity-0 group-hover:opacity-100`. The hidden/show variant was
collapsing the title row's height when the star wasn't rendered,
which made every card below visibly shift on hover. Always rendering
the button (with opacity-only visibility) keeps the row's vertical
metrics constant; the title still flushes left because the slot is
trailing, not leading.
Card hover effect (-translate-y + shadow-md) restored — the layout
shift wasn't coming from the card hover; it was the pin slot
appearing and disappearing.
* fix(agents): quieten row hover — border-tint only, no lift, no shadow
Drop the `-translate-y-px` and `hover:shadow-md` from the row card
plus the working-state inner ring. The translate + shadow grow
combination was visibly noisy as the cursor moved through the rail —
each row 'lifted' as you passed over it. Hover now just tints the
border in accent-orange/30; working and error states keep their
distinct border colours but no inner ring. Card height and shadow
stay constant in every state, so the rail reads as a calm vertical
list of cards.
* feat(home): rich Recent Agents grid + dead-code sweep
The /home Recent Agents grid was a placeholder shell. Every 'rich'
field on the card (lastMessage, lastMessageTimestamp, activitySummary,
currentTool, costUsd) was wired to undefined because AgentCommandHome
called `buildAgentCardData(agents, status?.status, undefined)` — the
dashboard arg has been hard-coded undefined since the harness
migration. Repointing the grid at `useHarnessAgents` + `useAgentAdapters`
gives every card the same enriched data the rail uses.
What the new card shows per agent:
• Adapter glyph tile + liveness dot (working pulses; asleep is
hollow; error is red)
• Name + Working pill (when active)
• Adapter · model · reasoning summary line, with an inline
Unavailable chip + HoverCard reason when the adapter binary
isn't on $PATH
• Italic last-user-message preview (line-clamp-2, leading quote
glyph) — same visual language as the rail
• Footer: 'X ago' + state chip (Asleep / Attention) OR a Resume
button (orange, with pulsing dot) when activeTurnId is non-null
Sort on the home grid is active-turn → recency. Pinning is NOT a
sort key here (and there's no pin indicator on the card) — pinning
belongs to the rail at /agents; the home page is action-oriented
and trusts active-turn + recency to surface the right agent.
Dead code removed:
• useAgentDashboard.ts (96 lines, no callers; subscribed to the
dead /claw/dashboard/stream from the OpenClaw-only era)
• useAgentCardData.ts (the dashboard-merge shim; passed undefined
every call so all enriched fields landed as undefined)
• AgentCard.tsx (AgentCardExpanded replaced by HomeAgentCard;
AgentCardCompact had no callers — the dock's compact mode was
never used)
• AgentCardData interface dropped from lib/agent-conversations/
types.ts; the new card consumes HarnessAgent directly
Visual language stays continuous between rail and grid: same
<AgentTile>, same <LivenessDot>, same italic-quote message
preview, same orange Resume button with a pulsing dot.
|
||
|
Dani Akash
|
a228c278c6 |
feat(agents): background-resilient chat — turns survive tab disconnect (#863)
* feat(agents): decouple chat turn lifecycle from SSE response
Introduce a per-process ActiveTurnRegistry that owns each agent turn's
lifecycle and a ring-buffered event stream, so chat tabs that close,
refresh, or navigate away no longer cancel the in-flight turn. New
endpoints:
POST /agents/:id/chat starts a turn (now returns 409 when
one is already running, with the
active turnId for attaching)
GET /agents/:id/chat/active reports the running turn for a UI
that just mounted
GET /agents/:id/chat/stream subscribes to a turn; supports
Last-Event-ID resume via per-event
seq ids
POST /agents/:id/chat/cancel explicit cancel — fetch abort no
longer affects the underlying turn
The chat hook now captures X-Turn-Id, tracks lastSeq from SSE id lines,
re-attaches on mount when the server still has an active turn, and
routes Stop through the cancel endpoint. The runtime call uses the
registry's per-turn AbortController instead of the HTTP request signal,
which is the core decoupling that lets turns outlive their initiator.
* feat(agents): add ActiveTurnRegistry primitive backing the new chat lifecycle
The previous commit referenced these files in tests and the harness
service but global gitignore swallowed them on the first add.
The registry owns the per-turn ring buffer (drop-oldest, terminal frame
preserved), the per-turn AbortController, and subscriber fan-out used
by /chat/stream resume.
|
||
|
Dani Akash
|
0c84547e8f |
feat(agents): migrate OpenClaw chat onto the unified harness/ACP path (#859)
* chore(acp): smoke-test ACP capabilities against running gateway
Adds apps/server/scripts/acp-smoke.ts which spawns `openclaw acp`
inside the gateway container and exercises every method we plan to
depend on: initialize, newSession, prompt (text + image), cancel,
listSessions, loadSession.
SDK pinned to 0.19.1 (Bun's minimum-release-age policy blocks 0.20+
which were released < 7 days ago).
Findings (full notes in plan outcomes):
- promptCapabilities advertises image:true but the model does NOT see
image bytes — silently dropped at the bridge.
- sessionCapabilities advertises {list:{}} but session/list throws
"Method not found": stale capability advertising.
- loadSession works; replays user/assistant/thought text and
session_info/usage/commands updates. No tool_call replay, as
documented.
- cancel works end-to-end: stopReason=cancelled.
- closeSession/resumeSession are not on ClientSideConnection in
0.19.1; kill child to close, use loadSession for rebind.
Plan revisions triggered by spike are recorded in
plans/browseros-ai/BrowserOS/features/2026-04-28-2310-claude-code-acp-implementation-roadmap.md.
* chore(acp): re-run smoke on SDK 0.21.0 and add mode/config/auth scenarios
After bypassing Bun's minimum-release-age and upgrading the SDK to
0.21.0, restore the previously-skipped resume/close paths and add
three new scenarios: mode (setSessionMode), config (setSessionConfigOption,
correct configId field), and auth (authenticate noop).
Findings, all bridge-side (independent of SDK):
- session/list, session/resume, session/close all throw -32601 on
OpenClaw 2026.4.12 — capability advertising is stale.
- Image content blocks silently dropped; model never sees the bytes.
- setSessionMode and setSessionConfigOption work; latter requires
`configId` (not `optionId`) per the schema.
- loadSession replays user/assistant/thought text + session_info +
usage + available_commands; no tool_call replay (documented).
- authenticate is a noop on OpenClaw (no authMethods advertised).
Plan outcomes updated with full method-support matrix.
* chore(deps): promote @agentclientprotocol/sdk to a runtime dependency
The smoke script in apps/server/scripts/acp-smoke.ts used the SDK as
devDependency. The upcoming ACP bridge (apps/server/src/api/services/acp/)
needs it at runtime, not just for tooling. Move the entry from
devDependencies to dependencies, alphabetically first under @a*.
Pinned to 0.21.0 — same version the smoke script validated against.
README gains a small Dependencies note pointing at the future bridge
location.
No code changes yet. The bridge wiring lands in subsequent commits.
* fix(openclaw): wire LlmProvider.supportsImages through to OpenClaw model config
When BrowserOS sets up a custom OpenAI-compat provider on the gateway,
the agent UI's "Supports Image" flag (LlmProviderConfig.supportsImages)
was being dropped on the floor. As a result the persisted model entry
had no `input` field, OpenClaw defaulted it to ['text'], and image_url
content parts were silently stripped before the model saw them.
Fix:
- Extend OpenClawSetupInput / OpenClawAgentMutationInput on the agent
side (useOpenClaw.ts) and the route body schema + SetupInput +
createAgent input on the server side with `supportsImages?: boolean`.
- AgentsPage forwards `llmOption?.supportsImages` from the selected
LlmProviderConfig in both handleSetup and handleCreate.
- provider-map.resolveSupportedOpenClawProvider emits
`input: ['text', 'image']` on the model entry when the flag is
truthy; otherwise emits the explicit `['text']` so the value is
always pinned (avoids relying on OpenClaw's implicit default).
- applyBrowserosConfig adds `tools.media.image.enabled = true` to the
bootstrap batch so the gateway's image-understanding pipeline is
always wired up — per-model `input` still gates which models see
images, this just enables the global path.
ACP image content blocks are still dropped by the OpenClaw bridge —
that's a separate bridge bug, not addressed here. This commit
restores image support for the OpenAI-compat /v1/chat/completions
path that the upcoming ACP chat panel will use as a carve-out for
image-bearing prompts.
Existing custom-provider configs are NOT auto-migrated; users will
re-acquire image support either by re-running setup or by editing
their model entries' `input` field manually. A migration pass for
legacy installs is not in scope for this commit because the
"supportsImages" intent isn't recoverable from the persisted config
alone — the source of truth is the LlmProvider record on the agent
side.
* feat(agents): add OpenClaw to AgentAdapter union and catalog
Extends AgentAdapter to 'claude' | 'codex' | 'openclaw' and adds the
OpenClaw entry to AGENT_ADAPTER_CATALOG. The new entry has:
- defaultModelId: 'default' — OpenClaw's ACP bridge does not surface
per-session model selection (verified during the ACP spike), so
models live in the OpenClawService config, not in the adapter
catalog. AgentDefinition.modelId carries the gateway-side model
name for display only.
- models: [] — empty list signals "no per-session model picker" in
the UI; isSupportedAgentModel('openclaw', undefined|'default')
returns true via the existing fallback path.
- reasoningEfforts mirror OpenClaw's session-level `thought_level`
config option (off / minimal / low / medium / high / adaptive).
Also extends:
- isAgentAdapter type guard recognizes 'openclaw'
- HarnessAgentAdapter union on the extension side
- agents.test.ts createAgent fake type
- agent-catalog.test.ts asserts on the new entry, empty model list
passthrough behavior, and OpenClaw's reasoning effort set
Lockfile delta is the workspace SDK pin reconciling 0.20.0 (taken
from dev's lock) up to our package.json's 0.21.0 (added in
|
||
|
Nikhil
|
cb32b8191d |
fix: show rich ACP harness history from ACPX (#852)
* fix: load ACP harness history from ACPX * fix: address ACP history review comments |
||
|
Nikhil
|
91d3285aa0 |
feat: add ACP agent harness (#849)
* feat: add acp agent runtime spike * feat: add agent harness catalog * feat: persist harness agents in json * feat: persist agent transcripts * feat: route harness service through agent records * feat: expose generic agent harness routes * feat: add harness agent frontend api * feat: create harness agents from agents page * feat: chat with persisted harness agents * chore: remove obsolete agent profile spike * chore: self-review fixes * fix: combine openclaw and harness agents UI * refactor: split agents page components * fix: hide persisted harness turns |
||
|
Dani Akash
|
ddbb2cf492 |
feat(agent): composer attachments + server-side outbound message queue (#826)
* feat(agent): attach images and text files to chat messages
Adds end-to-end support for image and text file attachments in the chat
composer, with the staged files round-tripping through the OpenClaw
gateway as OpenAI-compatible content blocks and persisting in the JSONL
so they show up in the historical view.
Server
- HTTP client: new OpenClawChatContentPart union and a buildUserContent
helper that emits multimodal content arrays when messageParts is
supplied, falls back to the legacy string content otherwise.
- Service: chatStream takes an optional messageParts array and forwards
it; BrowserOSChatHistoryItem gains an attachments field.
- JSONL reader: PiContentBlock learns the OpenAI image_url and Anthropic
image source/data shapes; user messages now emit user.attachment
events that the history mapper accumulates onto the next user item.
- Route: validates an inbound attachments[] (kind/mime/size/count),
inlines text-shaped files as <attachment> blocks in the message body,
attaches images via image_url parts. Replaces the immediate 409 on
active monitoring session with a 30s waitForSessionFree(agentId) wait
(registry now exposes onSessionEnd) so cron/hook contention does not
reject a user-chat send outright. Returns 503 if the wait times out.
Client
- New lib/attachments.ts: validateAttachment / compressImageIfNeeded
(canvas downscale to 2048px long edge, JPEG 0.85 re-encode for >1.5
MB inputs) / stageAttachment / stageAttachments that produces the
staged-attachment shape the composer renders and the payload the
server accepts.
- ConversationInput: drag-and-drop, paperclip button, clipboard paste,
staged attachment chip strip with thumbnails for images and a
paperclip+name chip for text files. Send button enables on either
text or attachments. Drop-zone overlay during drag.
- chatWithAgent forwards attachments[]; useAgentConversation.send
accepts a SendInput shape and renders user attachments on the
optimistic streaming turn via MessageAttachments / MessageAttachment.
- ClawChatMessage groups historical attachment parts into a single
MessageAttachments strip, ordered before reasoning/tools/text.
- claw-chat-types adds an attachment ClawChatMessagePart variant; the
history mapper emits attachment parts first and skips the text part
when the user only sent media.
- AgentCommandHome forwards the new SendInput shape — home composer
drops attachments at the boundary in v1 (the conversation page is
where staging is most useful; carrying bytes through the URL bar
is not sensible).
Limits: 10 attachments per message, 5 MB per image (post compression),
1 MB per text file, mime types png/jpeg/webp/gif and text/* +
application/json. PDFs and other binaries are deferred to v2.
* feat(agent): outbound message queue for chats while agent is mid-turn
Lets users keep typing and submitting messages while the agent is still
streaming a previous turn. Each press is appended to a single-flight
queue and dispatched as soon as `streaming` flips false; the queued
state renders as a strip above the composer so the user sees what's
pending vs. what's already sending.
- New `useOutboundQueue` hook owns the queue, the worker effect, and
cancel/retry actions. Single-flight by design — a re-entrancy ref
guard prevents two simultaneous dispatches when `streaming` flickers.
- Composer (`ConversationInput`) accepts optional `outboundQueue`,
`onCancelQueued`, `onRetryQueued` props. When the queue is provided
the send-button gate stops blocking on `streaming`; the spinner stays
as the visual cue that the agent is still busy. Legacy direct-send
callers keep the old streaming-blocks-send semantic.
- Renders an OutboundQueueStrip above the staged-attachment strip with
per-item status (queued / sending / failed), a cancel button on
queued items, and retry + discard on failed items.
- AgentCommandConversation wires `onSend` to `queue.enqueue` and routes
the home composer's `?q=` initial-message handoff through the queue
too, so it inherits the same single-flight serialization.
The server-side `waitForSessionFree` (added with attachments) and this
client-side queue together cover both contention sources: cron / hook
turns and back-to-back user sends. Persistence across reloads is
intentionally out of scope for v1 — losing the queue on extension
reload is documented as a known limitation.
* feat(server): server-side outbound message queue
Replaces the client-only React-state queue from
|
||
|
Dani Akash
|
711934555d |
feat(agent): enrich chat UI with tool activity, reasoning duration, and cost (#825)
* feat: pass per-turn cost and token data through chat history items
- Add costUsd, tokensIn, tokensOut to BrowserOSChatHistoryItem (server)
- Pass through from JSONL agent.message events in jsonlEventsToHistoryItems()
- Add same fields to client-side BrowserOSChatHistoryItem and ClawChatMessage
- Map cost/token data in mapHistoryItemToClawMessage()
Data flows: JSONL message.usage → server history item → API response →
client ClawChatMessage. Available for rendering in ClawChatMessage
component (message toolbar, cost badges).
* feat: add message toolbar with copy button and per-turn cost display
Add MessageToolbar to historical assistant messages in ClawChatMessage:
- Copy button copies message text to clipboard via MessageAction
- Per-turn token count (22.7K → 238) and cost ($0.003) shown as muted
tabular-nums text on the right side of the toolbar
- Toolbar appears on hover (opacity transition via group-hover)
- Only shown when the message has text content
- Cost/token display only shown when data is available from JSONL
* fix: toolbar only on assistant messages, always visible, cost only
- Only render toolbar on assistant messages (not user messages)
- Remove hover-only opacity — toolbar is always visible
- Remove token counts (22.7K → 238 is meaningless to users)
- Show only cost as a budget signal ($0.003)
* feat: group all tool activity into single Task collapsible per turn
Replace flat tool rows with a single ai-elements Task collapsible per
assistant turn that lists every tool/MCP call in sequence.
Live streaming (ConversationMessage):
- Aggregate all tool-batch parts into one Task
- Title: "Working… (N actions)" while running, "Agent activity (N actions)" when done
- Default open while turn is in progress
- Wrench icon in trigger
Historical (ClawChatMessage):
- Group all tool-call parts into one Task
- Title includes failed count if any tools errored
- Default collapsed — expandable on click
- Tool name + status icon + error text per row
Both views show one clean collapsible per turn instead of N individual
tool cards. Collapsed reads "5 actions"; expanded shows the timeline.
* feat: include tool calls in chat history responses
Server: jsonlEventsToHistoryItems() now walks ALL events (not just
messages) and pairs agent.tool_use with agent.tool_result by toolCallId.
The resulting tool call list is attached to the next assistant text
message as toolCalls[]. Each entry includes status, input arguments,
output text, error string, and duration computed from event timestamps.
Client:
- BrowserOSChatHistoryItem gets optional toolCalls field
- Tool-call message part type gets durationMs field
- mapHistoryItemToClawMessage() emits tool-call parts BEFORE the text
part (the order the agent produced them)
- ClawChatMessage Task view now shows tool duration in seconds
Result: historical messages now display the full tool activity
timeline grouped into the single Task collapsible per turn (designed
in step 3), instead of showing only the final text response.
* feat: render activity rows as human verbs sourced from tool registry
Tool calls in the chat activity view now read as sentences:
"Opened tab · news.ycombinator.com" instead of "browseros__new_page".
Server (tool-label-registry.ts):
- Curated verb override map for ~70 BrowserOS first-party tools
- Per-tool subject extractors that pull the meaningful argument from
input (URL → host, query → quoted, element → ID, etc.)
- Generic fallback humanizes snake_case for any unmapped tool
- Strips MCP namespace prefixes (browseros__, mcp_)
Server (openclaw-service.ts):
- jsonlEventsToHistoryItems calls buildToolLabel for each tool_use,
attaches label and subject to the BrowserOSChatHistoryToolCall
Client:
- Mirrored label module at lib/tool-labels.ts
- useAgentConversation tool-start handler computes label/subject
from the SSE tool args
- ClawChatMessage and ConversationMessage render label · subject
with foreground/muted styling, no font-mono
- ToolEntry, BrowserOSChatHistoryToolCall, and tool-call message
part types all carry label and optional subject
* fix: drop meaningless tab N subject from page-read tool rows
Page IDs are internal numbers, not URLs. 'Took screenshot · tab 4'
tells the user nothing. Removed subject extractors for take_snapshot,
take_enhanced_snapshot, get_page_content, get_page_links, get_dom,
and take_screenshot. The verb alone is the right signal.
* fix: gate initial loading on historyQuery.isFetched not isLoading
The session and history queries are sequential: the history query is
disabled until session resolves. After session resolves, there's a render
frame where historyQuery.isLoading is still false (the query hasn't
been kicked off yet). isInitialLoading flipped to false during that
window, exposing an empty chat shell with just Task collapsibles and
copy buttons before the messages filled in.
Switching the guard to isFetched closes that window — the loading state
stays true until the first history fetch actually completes.
* fix: render historical messages immediately instead of through Streamdown's idle-callback debounce
Streamdown defaults to mode="streaming" which uses requestIdleCallback (300ms
debounce, 500ms idle timeout) and lazy/Suspense to optimize for token-by-token
live streams. For finalized historical messages this caused tool collapsibles
and copy buttons to paint while text bodies stayed blank for ~300-500ms after
load. Pass mode="static" + parseIncompleteMarkdown=false on the historical
MessageResponse so completed text paints in the same frame as the surrounding
chrome. Live streaming turns still use the default streaming mode.
Also collapse the redundant /agents/:id/session round-trip into the existing
/history endpoint (server already resolves the most recent user-chat session
when sessionKey is omitted) and tighten the initial-loading gate to stay true
across the render frame where the query is enabled but hasn't started fetching.
* feat: surface thinking duration on historical reasoning collapsibles
Server accumulates agent.thinking events per turn from JSONL and attaches a
single reasoning block (joined text + durationMs from first thinking event
to the closing agent.message) on each assistant history item. Reasoning
buffer resets on user.message alongside the tool-call buffer.
Client mirrors the type, emits the reasoning part before tool calls in
mapHistoryItemToClawMessage (chronological: think → act → answer), and
passes duration in seconds to <Reasoning> so the trigger reads "Thought
for N seconds" instead of just "Thinking" on collapsed historical turns.
* fix: read thinking blocks from the correct JSONL field name
OpenClaw stores reasoning blocks as {type:'thinking', thinking:'...'} but
the JSONL parser was reading block.text, so every thinking event was
silently dropped before it ever reached jsonlEventsToHistoryItems. As a
result the reasoning field on history items was always empty even though
the new accumulator was wired up correctly.
Also guard the client mapping: when durationMs is 0 (think + answer
emitted in the same JSONL line, no real elapsed wall-clock) pass
undefined to <Reasoning> so it renders the static "Thinking" trigger
instead of the streaming shimmer / "Thought for 0 seconds".
* fix: reset reasoning buffer on discarded turns and drop dead session hook
Two cleanups from PR review:
1. jsonlEventsToHistoryItems: when an agent.message is discarded (the
"[Chat messages since your last reply" wrapper without a current-message
marker) the tool buffers were already reset but the reasoning buffer
was not. Accumulated thinking from the discarded turn would bleed onto
the next assistant message. Reset pendingReasoningTexts and
pendingReasoningFirstAt alongside the tool buffers.
2. useClawAgentSession, the AgentSessionResponse type, and the unused
session entry in CLAW_CHAT_QUERY_KEYS became dead code after the
session round-trip was folded into the history endpoint. Removed.
|
||
|
Dani Akash
|
0035893f33 |
feat: dashboard API, JSONL reader, and OpenClaw observer for enriched home page (#810)
* feat: draft agent chat ui exploration * feat: refine agent chat ui draft * feat: remove outer frame from agent chat workspace * fix: offset agent chat for app sidebar * fix: simplify agent conversation shell * fix: remove redundant chat header actions * fix: unify agent conversation headers * fix: tighten agent chat spacing * fix: bound agent chat composer height * fix: remove agent chat page inset * fix: align agent header height with sidepanel * fix: center agent composer resting state * fix: anchor multiline composer controls * fix: remove focus grid from agent home * fix: remove redundant agent home header * fix: constrain home agent composer * fix: match home composer default posture * feat: add openclaw chat history APIs * feat: add claw chat history hydration * fix: stabilize claw chat viewport layout * fix: use conversation scroll base for claw chat * refactor: split claw chat controller responsibilities * fix: keep active agent turns in memory * fix: normalize openclaw chat sessions * refactor: use HTTP client for agent history instead of CLI client Replace the CLI-based getChatHistory() call in getAgentHistoryPage() with the HTTP client's getSessionHistory() from PR #795. This uses the direct HTTP transport to OpenClaw's /sessions/<key>/history endpoint instead of shelling out through the CLI. - Add filterHttpSessionHistoryMessages() for flat-string content format - Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape - Update getAgentHistoryPage() to call getSessionHistory() via httpClient - Remove unused getChatHistory(), filterOpenClawSystemMessages(), normalizeChatHistoryMessages(), and getTextContent() - Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory - Update MutableOpenClawService type: chatClient -> httpClient * fix: fetch all session messages by iterating OpenClaw pagination OpenClaw's HTTP history endpoint returns a limited page by default. When called without a limit, only the first ~27 messages were returned, causing all newer conversation messages to be silently dropped. Add fetchAllSessionMessages() that iterates through OpenClaw's cursor- based pagination (200 messages per page) until hasMore is false, then feeds the complete message list into the existing BrowserOS normalization and in-memory pagination layer. * refactor: migrate chat history from HTTP gateway to direct JSONL file reads Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw gateway /sessions/:key/history pagination loop) with direct JSONL file reads from the host filesystem via Lima's virtiofs mount. - Add OpenClawJsonlReader that reads session JSONL files directly from ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/ - Replace fetchAllSessionMessages() HTTP pagination with single file read - Replace CLI-based listSessions() with sessions.json file reads - Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous - Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages, normalizeHttpHistoryMessages helpers - Update route handlers to drop unnecessary async/await - Update tests to use temp JSONL files instead of mocked HTTP/CLI clients * fix: restore async route handlers for test compatibility with mocked service * fix: address review feedback — path traversal guard, lazy reader, exists flag - Add safePath() to OpenClawJsonlReader that validates resolved paths stay within stateRoot, preventing path traversal via crafted agentId values - Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients) instead of creating a new instance per property access - Return exists: false from resolveSpecificAgentSession when no session matches instead of fabricating a ghost session with sessionId: '' * feat: add dashboard API and enrich home page agent cards Server: - Add summarizeToolActivity() that converts tool events into natural language descriptions ("Browsed 3 pages, took 2 screenshots") - Add getDashboard() to OpenClawService that aggregates per-agent stats from JSONL: latest message, activity summary, cost, session count - Add GET /claw/dashboard endpoint Client: - Add useAgentDashboard() React Query hook (10s refetch, 5s stale) - Rewrite useAgentCardData from async IndexedDB hook to pure buildAgentCardData() function merging agent entries with dashboard data - Add activity summary and cost to AgentCardExpanded footer - Add activitySummary and costUsd fields to AgentCardData type - Remove IndexedDB dependency from the home page * feat: add OpenClawObserver for real-time per-agent status via gateway WS - Add OpenClawObserver that connects to the OpenClaw gateway WebSocket control plane and subscribes to chat broadcast events - Track per-agent status in real time: working (streaming), idle (turn complete), error (run failed), with current tool name - Auto-connect when gateway control plane becomes available, auto- reconnect on disconnect with 5s backoff - Disconnect observer on stop/shutdown - Wire live status + currentTool into getDashboard() response - Update client: AgentOverview includes status + currentTool, card shows spinning loader + tool name when agent is working - Status resolution: per-agent WS status takes precedence over gateway- level status for working/error states * feat: add SSE dashboard stream for real-time agent status on home page Server: - Add GET /claw/dashboard/stream SSE endpoint that sends an initial snapshot then pushes per-agent status events as they arrive from the OpenClaw observer - Add onAgentStatusChange() to OpenClawService exposing the observer's listener for the route layer - Heartbeat every 15s to keep connections alive Client: - useAgentDashboard() now subscribes to EventSource at /claw/dashboard/stream - SSE snapshot event hydrates the React Query cache immediately - SSE status events patch individual agent status + currentTool in the cache without refetching — agent cards update instantly - Polling fallback raised to 30s since SSE handles real-time * fix: observer WS handshake — wait for challenge before sending connect The OpenClaw gateway sends a connect.challenge event before accepting the connect request. The observer was sending the connect request on ws.open which raced with the challenge. Now waits for the challenge event before sending the handshake. Also add dangerouslyDisableDeviceAuth to the gateway setup config batch so the observer can connect without device identity on new installs. * fix: JSONL reader falls back to most recent file when sessions.json is stale OpenClaw's sessions.json can record a Pi session ID that doesn't match the actual JSONL filename on disk. This happens after context compaction or session restart — the JSONL file gets a new UUID but sessions.json keeps the old one. Previously this caused history to silently disappear (the reader tried to open a non-existent file and returned empty). Now resolveJsonlPath() checks if the mapped file exists and, when it doesn't, scans the sessions directory for the most recently modified .jsonl file as a fallback. * feat: add ClawSession state machine for reliable per-agent status The OpenClawObserver only knows about status changes it witnesses via WS events. If an agent was already running when the observer connected, or after a reconnect, statuses were stuck at "unknown". ClawSession is an in-memory state machine that solves this: 1. Seeds from JSONL on first control plane call — reads the latest events for each agent and infers working/idle. A session is "working" if the last event is a user.message with no subsequent agent.message, or an agent.tool_use with no matching agent.tool_result. 2. Receives live transitions from the WS observer — the observer now delegates all state management to ClawSession instead of maintaining its own status map. 3. Applies a 5-minute staleness threshold — if the last JSONL event is older than 5 minutes, assume idle (handles agent crashes). Consumers (SSE stream, dashboard endpoint) read from ClawSession and get correct state from the first call — no "unknown" period. * fix: remove staleTime so dashboard refetches on every mount * fix: reset stale working status on WS disconnect, eliminate redundant JSONL reads - Observer resets all "working" agents to "unknown" when the WS closes, preventing agents from appearing stuck as Working indefinitely after a gateway restart. ClawSession re-seeds correct state on reconnect. - getDashboard() now derives latestAgentMessage and cost from the already-loaded events array for the latest session instead of calling latestAgentMessage() and getSessionStats() which each re-read the same JSONL file. Reduces file reads from 3x to 1x per agent. |
||
|
Dani Akash
|
752f42d1fe |
refactor: migrate chat history to direct JSONL file reads via Lima filesystem (#808)
* feat: draft agent chat ui exploration * feat: refine agent chat ui draft * feat: remove outer frame from agent chat workspace * fix: offset agent chat for app sidebar * fix: simplify agent conversation shell * fix: remove redundant chat header actions * fix: unify agent conversation headers * fix: tighten agent chat spacing * fix: bound agent chat composer height * fix: remove agent chat page inset * fix: align agent header height with sidepanel * fix: center agent composer resting state * fix: anchor multiline composer controls * fix: remove focus grid from agent home * fix: remove redundant agent home header * fix: constrain home agent composer * fix: match home composer default posture * feat: add openclaw chat history APIs * feat: add claw chat history hydration * fix: stabilize claw chat viewport layout * fix: use conversation scroll base for claw chat * refactor: split claw chat controller responsibilities * fix: keep active agent turns in memory * fix: normalize openclaw chat sessions * refactor: use HTTP client for agent history instead of CLI client Replace the CLI-based getChatHistory() call in getAgentHistoryPage() with the HTTP client's getSessionHistory() from PR #795. This uses the direct HTTP transport to OpenClaw's /sessions/<key>/history endpoint instead of shelling out through the CLI. - Add filterHttpSessionHistoryMessages() for flat-string content format - Add normalizeHttpHistoryMessages() for OpenClawSessionHistoryMessage shape - Update getAgentHistoryPage() to call getSessionHistory() via httpClient - Remove unused getChatHistory(), filterOpenClawSystemMessages(), normalizeChatHistoryMessages(), and getTextContent() - Update test mocks from cliClient.getChatHistory to httpClient.getSessionHistory - Update MutableOpenClawService type: chatClient -> httpClient * fix: fetch all session messages by iterating OpenClaw pagination OpenClaw's HTTP history endpoint returns a limited page by default. When called without a limit, only the first ~27 messages were returned, causing all newer conversation messages to be silently dropped. Add fetchAllSessionMessages() that iterates through OpenClaw's cursor- based pagination (200 messages per page) until hasMore is false, then feeds the complete message list into the existing BrowserOS normalization and in-memory pagination layer. * refactor: migrate chat history from HTTP gateway to direct JSONL file reads Replace the HTTP-based chat history pipeline (BrowserOS server → OpenClaw gateway /sessions/:key/history pagination loop) with direct JSONL file reads from the host filesystem via Lima's virtiofs mount. - Add OpenClawJsonlReader that reads session JSONL files directly from ~/.browseros/vm/openclaw/.openclaw/agents/<id>/sessions/ - Replace fetchAllSessionMessages() HTTP pagination with single file read - Replace CLI-based listSessions() with sessions.json file reads - Make listSessions, resolveAgentSession, getAgentHistoryPage synchronous - Remove unused toBrowserOSSession, filterHttpSessionHistoryMessages, normalizeHttpHistoryMessages helpers - Update route handlers to drop unnecessary async/await - Update tests to use temp JSONL files instead of mocked HTTP/CLI clients * fix: restore async route handlers for test compatibility with mocked service * fix: address review feedback — path traversal guard, lazy reader, exists flag - Add safePath() to OpenClawJsonlReader that validates resolved paths stay within stateRoot, preventing path traversal via crafted agentId values - Use lazy initialization for jsonlReader (nulled on rebuildRuntimeClients) instead of creating a new instance per property access - Return exists: false from resolveSpecificAgentSession when no session matches instead of fabricating a ghost session with sessionId: '' |
||
|
Nikhil
|
6e37742a5a | feat: reuse agent command chat for agents page (#803) | ||
|
Dani Akash
|
e80ec467f4 |
feat: wire lazy monitoring to OpenClaw chat handoff (#768)
* feat: add lazy monitoring substrate * feat: wire lazy monitoring to openclaw chat handoff * test: cover openclaw chat history handoff * fix: reject concurrent monitored chats |
||
|
Dani Akash
|
0c96002cf5 |
fix: complete openclaw gateway recovery UX (#703)
* fix: complete openclaw gateway recovery ui * fix: guard unknown gateway ui state * fix: guard unknown openclaw status badge |
||
|
Nikhil
|
ce7c209ba6 |
feat: add OpenClaw agent command center and terminal (#692)
* feat: agent command center new tab with OpenClaw conversation history * feat: add web terminal for Podman container shell access * feat: align agent command center with new tab * fix: simplify agent command center styling * style: polish agent terminal layout and theming * style: simplify agent terminal styling * fix: address PR review comments for OpenClaw routes * fix: handle OpenClaw client start and error states * fix: resolve remaining OpenClaw review comments |