mirror of
https://github.com/browseros-ai/BrowserOS.git
synced 2026-05-14 08:03:58 +00:00
Compare commits
16 Commits
feat/eval
...
feat/voice
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8c62697a47 | ||
|
|
b038d772e8 | ||
|
|
c3229bd5f3 | ||
|
|
62f8107405 | ||
|
|
45da6526ab | ||
|
|
41e8461f01 | ||
|
|
62440de783 | ||
|
|
94e3f99adb | ||
|
|
e2069bc999 | ||
|
|
2d51c82722 | ||
|
|
29056226bb | ||
|
|
d1d2074abc | ||
|
|
41c9b1547c | ||
|
|
8b0e6dbfd3 | ||
|
|
07a2d13f16 | ||
|
|
46031ed573 |
286
packages/browseros-agent/.claude/skills/test-ui/SKILL.md
Normal file
286
packages/browseros-agent/.claude/skills/test-ui/SKILL.md
Normal file
@@ -0,0 +1,286 @@
|
||||
---
|
||||
name: test-ui
|
||||
description: Test the BrowserOS agent extension UI by starting the dev environment and visually verifying changes via CDP. Covers the new tab page (left sidebar — Home, Scheduled Tasks, Settings, etc.) and the right side panel (chat interface). Use after making UI changes to apps/agent/.
|
||||
argument-hint: [what to test, e.g. "verify the new settings page renders correctly"]
|
||||
---
|
||||
|
||||
# Test Agent UI
|
||||
|
||||
Visually test the BrowserOS agent extension UI — both the new tab page (left sidebar) and the right side panel (chat) — by starting the dev environment and inspecting via CDP.
|
||||
|
||||
## When to use
|
||||
|
||||
After making code changes to `apps/agent/` (the Chrome extension), use this skill to:
|
||||
- Verify new UI components render correctly
|
||||
- Check navigation between views works
|
||||
- Confirm layout/styling changes look right
|
||||
- Test interactive elements (buttons, inputs, forms)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Go** must be installed (`brew install go`) — the dev tool is written in Go
|
||||
- **BrowserOS.app** must be installed at `/Applications/BrowserOS.app/`
|
||||
- The `scripts/dev/inspect-ui.ts` utility must exist (CDP inspector script)
|
||||
|
||||
## Step 1: Start the dev environment
|
||||
|
||||
```bash
|
||||
bun run dev:watch -- --new
|
||||
```
|
||||
|
||||
This single command handles everything:
|
||||
- Builds the Go dev CLI tool
|
||||
- Picks random available ports (avoids conflicts)
|
||||
- Creates a fresh browser profile
|
||||
- Builds controller-ext
|
||||
- Runs GraphQL codegen if `apps/agent/generated/graphql/` doesn't exist
|
||||
- Starts the agent extension with WXT HMR (hot module replacement)
|
||||
- Waits for CDP to be ready
|
||||
- Starts the MCP server
|
||||
|
||||
Run it in the background and **read the output to find the CDP port**:
|
||||
|
||||
```
|
||||
[info] Ports: CDP=9552 Server=9065 Extension=9929
|
||||
```
|
||||
|
||||
The CDP port is randomized. You MUST extract it from the output and set it for all subsequent commands:
|
||||
|
||||
```bash
|
||||
export BROWSEROS_CDP_PORT=<port from output>
|
||||
```
|
||||
|
||||
Wait for these messages before proceeding:
|
||||
1. `[server] CDP ready`
|
||||
2. `[server] HTTP server listening`
|
||||
|
||||
## Step 2: Discover targets
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts targets
|
||||
```
|
||||
|
||||
You will see targets like:
|
||||
- `[service_worker]` — extension background scripts (not directly testable for UI)
|
||||
- `[page] chrome-extension://bflpfmnmnokmjhmgnolecpppdbdophmk/app.html#/...` — **New tab page (left sidebar)**
|
||||
- `[page] sidepanel.html` — **Right side panel (chat)**
|
||||
|
||||
The two main testable surfaces:
|
||||
- **`app.html`** — the new tab page with left sidebar (Home, Connect Apps, Scheduled Tasks, Skills, Memory, Soul, Settings)
|
||||
- **`sidepanel.html`** — the right side panel chat interface
|
||||
|
||||
## Step 3: Navigate to the main UI
|
||||
|
||||
A fresh profile opens the **onboarding page** (`app.html#/onboarding`). Navigate to the home page first:
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval app.html "window.location.hash = '#/home'"
|
||||
```
|
||||
|
||||
Verify with a snapshot (not screenshot — snapshot is faster and sufficient for structural checks):
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts snapshot app.html
|
||||
```
|
||||
|
||||
## Snapshot vs Screenshot
|
||||
|
||||
**Prefer `snapshot` for most checks** — it's fast, text-based, and tells you what elements exist, their text, and their IDs. Use it after every navigation or interaction to verify state.
|
||||
|
||||
**Use `screenshot` only when you need visual verification** — layout changes, CSS/styling, colors, images, or a final "does it look right" check. Screenshots are expensive (capture → save → read image).
|
||||
|
||||
| Check | Use |
|
||||
|-------|-----|
|
||||
| Did the page navigate? | `snapshot` — look for new elements |
|
||||
| Does my new component render? | `snapshot` — look for its text/role |
|
||||
| Did a click change state? | `snapshot` — check element names/values |
|
||||
| Is the layout correct? | `screenshot` — visual check needed |
|
||||
| Do CSS changes look right? | `screenshot` — visual check needed |
|
||||
| Final verification before committing | `screenshot` — one visual confirmation |
|
||||
|
||||
## Step 4: Test the new tab page (left sidebar)
|
||||
|
||||
### Get element IDs
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts snapshot app.html
|
||||
```
|
||||
|
||||
Output shows interactive elements with IDs:
|
||||
```
|
||||
[52] link "Home"
|
||||
[57] link "Connect Apps"
|
||||
[65] link "Scheduled Tasks"
|
||||
[74] link "Skills"
|
||||
[103] link "Settings"
|
||||
```
|
||||
|
||||
### Navigate via click or hash routing
|
||||
|
||||
**Click-based** (use element IDs from snapshot):
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts click app.html 65 # Click "Scheduled Tasks"
|
||||
```
|
||||
|
||||
**Hash routing** (faster, no snapshot needed):
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval app.html "window.location.hash = '#/settings'"
|
||||
bun scripts/dev/inspect-ui.ts eval app.html "window.location.hash = '#/scheduled-tasks'"
|
||||
bun scripts/dev/inspect-ui.ts eval app.html "window.location.hash = '#/home'"
|
||||
```
|
||||
|
||||
### Verify navigation
|
||||
|
||||
```bash
|
||||
# Snapshot to confirm the page changed (fast, preferred)
|
||||
bun scripts/dev/inspect-ui.ts snapshot app.html
|
||||
|
||||
# Screenshot only if you need to check visual layout
|
||||
bun scripts/dev/inspect-ui.ts screenshot app.html /tmp/settings.png
|
||||
```
|
||||
|
||||
### CRITICAL: Re-snapshot after every navigation
|
||||
|
||||
React re-renders change element IDs. **Always run snapshot again** before clicking/filling after navigating to a new view. Using stale IDs will fail.
|
||||
|
||||
## Step 5: Open and test the right side panel
|
||||
|
||||
The side panel starts **disabled** in a fresh profile. Open it using BrowserOS-specific APIs:
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts open-sidepanel
|
||||
```
|
||||
|
||||
Wait 2 seconds for it to appear as a target, then:
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts screenshot sidepanel /tmp/panel.png
|
||||
bun scripts/dev/inspect-ui.ts snapshot sidepanel
|
||||
```
|
||||
|
||||
### Interact with the side panel
|
||||
|
||||
```bash
|
||||
# Get element IDs
|
||||
bun scripts/dev/inspect-ui.ts snapshot sidepanel
|
||||
# Output: [37] textbox "What should I do?"
|
||||
# [124] button "Send"
|
||||
# [60] link "Chat history"
|
||||
# [99] button "Agent Mode ON"
|
||||
|
||||
# Fill the chat input and press Enter to send
|
||||
bun scripts/dev/inspect-ui.ts fill sidepanel 37 "Hello world"
|
||||
bun scripts/dev/inspect-ui.ts press_key sidepanel Enter
|
||||
|
||||
# Or click the Send button
|
||||
bun scripts/dev/inspect-ui.ts click sidepanel 124
|
||||
|
||||
# Wait for a response to appear
|
||||
bun scripts/dev/inspect-ui.ts wait_for sidepanel text "response text"
|
||||
|
||||
# Scroll down to see more content
|
||||
bun scripts/dev/inspect-ui.ts scroll sidepanel down 3
|
||||
|
||||
# Hover over an element to test hover states
|
||||
bun scripts/dev/inspect-ui.ts hover sidepanel 99
|
||||
|
||||
# Snapshot to verify state changed (fast, preferred)
|
||||
bun scripts/dev/inspect-ui.ts snapshot sidepanel
|
||||
|
||||
# Screenshot only for visual/layout verification
|
||||
bun scripts/dev/inspect-ui.ts screenshot sidepanel /tmp/result.png
|
||||
```
|
||||
|
||||
## Step 6: Verify and iterate
|
||||
|
||||
### The core loop
|
||||
|
||||
```
|
||||
snapshot → identify element IDs → click/fill/press_key → snapshot → verify
|
||||
```
|
||||
|
||||
Use `screenshot` only when visual layout verification is needed (CSS changes, final check).
|
||||
|
||||
### After making code changes
|
||||
|
||||
1. Fix the code in `apps/agent/`
|
||||
2. WXT HMR will hot-reload the extension automatically (watch mode)
|
||||
3. Wait 2-3 seconds for the reload to complete
|
||||
4. **Re-snapshot** — element IDs WILL change after HMR reload
|
||||
5. Verify the fix with snapshot (or screenshot if visual)
|
||||
|
||||
### Check server logs
|
||||
|
||||
The dev server output (running in background) contains useful diagnostics:
|
||||
- `[agent]` — WXT build/HMR status, compilation errors
|
||||
- `[server]` — MCP server logs, tool execution, errors
|
||||
- `[build]` — Extension build output
|
||||
|
||||
If the UI isn't rendering, check for build errors in the `[agent]` output.
|
||||
|
||||
### Check for JavaScript errors
|
||||
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval sidepanel "JSON.stringify(window.__errors || 'no errors')"
|
||||
```
|
||||
|
||||
Or check the console for React errors:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval app.html "document.querySelector('#root')?.innerHTML?.substring(0, 200)"
|
||||
```
|
||||
|
||||
### Verify API connectivity
|
||||
|
||||
The extension talks to the MCP server. Verify the server is reachable:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval sidepanel "fetch('http://127.0.0.1:<serverPort>/health').then(r => r.ok).catch(() => false)"
|
||||
```
|
||||
|
||||
### Common issues
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Blank page after navigation | React render error | Check `eval` for JS errors |
|
||||
| Element IDs don't match | Page re-rendered (HMR/navigation) | Re-run `snapshot` before interacting |
|
||||
| `open-sidepanel` fails | Extension not fully loaded | Wait longer after dev server starts |
|
||||
| Click does nothing | Element not visible (below fold) | Use `scroll` first, then re-snapshot |
|
||||
| `wait_for` times out | Content hasn't loaded yet | Check server logs for API errors |
|
||||
|
||||
## Available commands reference
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `targets` | List all CDP targets, marks extension pages with `[EXTENSION]` |
|
||||
| `screenshot <target> [file]` | Capture PNG screenshot (default: `screenshot.png`) |
|
||||
| `snapshot <target>` | Print accessibility tree with `[elementId] role "name"` |
|
||||
| `click <target> <elementId>` | Click element by ID (3-tier coordinate fallback + JS click) |
|
||||
| `fill <target> <elementId> <text>` | Focus element, clear, type text |
|
||||
| `press_key <target> <key>` | Press key or combo: `Enter`, `Escape`, `Tab`, `Control+A`, `Meta+Shift+P` |
|
||||
| `scroll <target> <dir> [amount]` | Scroll `up`/`down`/`left`/`right`, amount in ticks (default 3) |
|
||||
| `hover <target> <elementId>` | Hover over element (for tooltips, hover states) |
|
||||
| `select_option <target> <id> <val>` | Select dropdown option by value or visible text |
|
||||
| `wait_for <target> text\|selector <v>` | Wait up to 10s for text or CSS selector to appear |
|
||||
| `eval <target> <expression>` | Run JavaScript in the target's context |
|
||||
| `open-sidepanel` | Enable and open the right side panel |
|
||||
|
||||
`<target>` is a URL substring (e.g., `sidepanel`, `app.html`) or numeric index from `targets` output.
|
||||
|
||||
## Known app.html routes
|
||||
|
||||
These can be used with `eval app.html "window.location.hash = '#/<route>'"`:
|
||||
|
||||
| Route | View |
|
||||
|-------|------|
|
||||
| `/home` | Home page with search bar and top sites |
|
||||
| `/settings` | Settings (LLM providers, customization, workflows, MCP) |
|
||||
| `/scheduled-tasks` | Scheduled Tasks management |
|
||||
| `/onboarding` | Onboarding flow (first-run experience) |
|
||||
|
||||
## Gotchas learned from real testing
|
||||
|
||||
1. **Ports are randomized** with `--new` — always extract from dev server output
|
||||
2. **Fresh profile = onboarding page** — navigate to `#/home` to see the main UI
|
||||
3. **Element IDs change after navigation** — always re-snapshot before clicking
|
||||
4. **Side panel starts disabled** — `open-sidepanel` handles the BrowserOS-specific enable + toggle API
|
||||
5. **`Input.enable` does not exist** — the CDP Input domain has no enable method (already handled in the script)
|
||||
6. **`DOM.getDocument` required** — must be called before DOM operations like `pushNodesByBackendIdsToFrontend` (already handled in the script)
|
||||
7. **Settings sub-navigation** — the settings page has its own left sidebar (BrowserOS AI, Chat & Council Provider, Search Provider, Customize BrowserOS, BrowserOS as MCP, Workflows) — use snapshot + click to navigate within settings
|
||||
@@ -165,3 +165,68 @@ Tests are in `apps/server/tests/`:
|
||||
- `agent/` - Agent tests (compaction, rate limiter)
|
||||
- `sdk/` - Agent SDK tests
|
||||
- `__helpers__/` - Test utilities and fixtures
|
||||
|
||||
## Self-Testing UI Changes
|
||||
|
||||
After making UI changes to the agent extension (`apps/agent/`), you can visually verify them using the CDP inspector script. This connects directly to the browser via Chrome DevTools Protocol and can inspect extension pages (side panel, new tab, etc.) that the agent's own tools cannot see.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
The dev server must be running:
|
||||
```bash
|
||||
bun run dev:watch -- --new
|
||||
```
|
||||
Read the output to find the randomized CDP port, then:
|
||||
```bash
|
||||
export BROWSEROS_CDP_PORT=<port from output>
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
1. **List all targets** to see what's available:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts targets
|
||||
```
|
||||
|
||||
2. **Open the side panel** if it's not already open:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts open-sidepanel
|
||||
```
|
||||
|
||||
3. **Take a screenshot** of the side panel:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts screenshot sidepanel /tmp/panel.png
|
||||
```
|
||||
Then read `/tmp/panel.png` to view the result.
|
||||
|
||||
4. **Get the accessibility tree** for structural verification:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts snapshot sidepanel
|
||||
```
|
||||
|
||||
5. **Click an element** by its ID from the snapshot:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts click sidepanel 142
|
||||
```
|
||||
|
||||
6. **Fill a text input** by its ID from the snapshot:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts fill sidepanel 85 "search query"
|
||||
```
|
||||
|
||||
7. **Evaluate JavaScript** in the extension context:
|
||||
```bash
|
||||
bun scripts/dev/inspect-ui.ts eval sidepanel "document.title"
|
||||
```
|
||||
|
||||
### Interaction workflow
|
||||
|
||||
The typical loop is: snapshot → identify element IDs → click/fill → screenshot to verify.
|
||||
Element IDs come from the `[number]` in snapshot output (these are `backendDOMNodeId` values).
|
||||
This uses the same element resolution as the server's MCP tools — no coordinate guessing.
|
||||
|
||||
### Target selection
|
||||
|
||||
The `<target>` argument can be:
|
||||
- An **index** from the `targets` output (e.g., `3`)
|
||||
- A **URL substring** (e.g., `sidepanel`, `newtab`, `chrome-extension://`)
|
||||
|
||||
@@ -4,8 +4,8 @@ import { MessageResponse } from '@/components/ai-elements/message'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { Textarea } from '@/components/ui/textarea'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { useVoiceInput } from '@/lib/voice/useVoiceInput'
|
||||
import type { Message } from './useSurveyChat'
|
||||
import { useVoiceInput } from './useVoiceInput'
|
||||
import { VoiceInputButton } from './VoiceInputButton'
|
||||
|
||||
interface Props {
|
||||
@@ -81,6 +81,7 @@ export const Chat: FC<Props> = ({
|
||||
}, [messagesLength])
|
||||
|
||||
// Insert transcript into input when transcription completes
|
||||
// biome-ignore lint/correctness/useExhaustiveDependencies: only trigger on transcript/transcribing change
|
||||
useEffect(() => {
|
||||
if (voice.transcript && !voice.isTranscribing) {
|
||||
setInput((prev) => {
|
||||
@@ -89,7 +90,7 @@ export const Chat: FC<Props> = ({
|
||||
})
|
||||
voice.clearTranscript()
|
||||
}
|
||||
}, [voice])
|
||||
}, [voice.transcript, voice.isTranscribing])
|
||||
|
||||
const handleSubmit = (e: FormEvent) => {
|
||||
e.preventDefault()
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
import { zodResolver } from '@hookform/resolvers/zod'
|
||||
import { ChevronDown } from 'lucide-react'
|
||||
import type { FC } from 'react'
|
||||
import { useEffect } from 'react'
|
||||
import { useEffect, useState } from 'react'
|
||||
import { useForm } from 'react-hook-form'
|
||||
import { z } from 'zod/v3'
|
||||
import { ChatProviderSelector } from '@/components/chat/ChatProviderSelector'
|
||||
import type { Provider } from '@/components/chat/chatComponentTypes'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { Checkbox } from '@/components/ui/checkbox'
|
||||
import {
|
||||
@@ -31,6 +34,12 @@ import {
|
||||
SelectValue,
|
||||
} from '@/components/ui/select'
|
||||
import { Textarea } from '@/components/ui/textarea'
|
||||
import { BrowserOSIcon, ProviderIcon } from '@/lib/llm-providers/providerIcons'
|
||||
import {
|
||||
defaultProviderIdStorage,
|
||||
providersStorage,
|
||||
} from '@/lib/llm-providers/storage'
|
||||
import type { LlmProviderConfig, ProviderType } from '@/lib/llm-providers/types'
|
||||
import type { ScheduledJob } from './types'
|
||||
|
||||
const formSchema = z
|
||||
@@ -43,6 +52,7 @@ const formSchema = z
|
||||
scheduleType: z.enum(['daily', 'hourly', 'minutes']),
|
||||
scheduleTime: z.string().optional(),
|
||||
scheduleInterval: z.number().int().min(1).max(60).optional(),
|
||||
providerId: z.string().optional(),
|
||||
enabled: z.boolean(),
|
||||
})
|
||||
.superRefine((data, ctx) => {
|
||||
@@ -81,6 +91,8 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
onSave,
|
||||
}) => {
|
||||
const isEditing = !!initialValues
|
||||
const [providers, setProviders] = useState<LlmProviderConfig[]>([])
|
||||
const [defaultProviderId, setDefaultProviderId] = useState<string>('')
|
||||
|
||||
const form = useForm<FormValues>({
|
||||
resolver: zodResolver(formSchema),
|
||||
@@ -90,11 +102,25 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
scheduleType: 'daily',
|
||||
scheduleTime: '09:00',
|
||||
scheduleInterval: 1,
|
||||
providerId: undefined,
|
||||
enabled: true,
|
||||
},
|
||||
})
|
||||
|
||||
const scheduleType = form.watch('scheduleType')
|
||||
const selectedProviderId = form.watch('providerId')
|
||||
|
||||
// Load providers from storage
|
||||
useEffect(() => {
|
||||
if (!open) return
|
||||
Promise.all([
|
||||
providersStorage.getValue(),
|
||||
defaultProviderIdStorage.getValue(),
|
||||
]).then(([providerList, defId]) => {
|
||||
setProviders(providerList ?? [])
|
||||
setDefaultProviderId(defId ?? '')
|
||||
})
|
||||
}, [open])
|
||||
|
||||
useEffect(() => {
|
||||
if (open) {
|
||||
@@ -105,6 +131,7 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
scheduleType: initialValues.scheduleType,
|
||||
scheduleTime: initialValues.scheduleTime || '09:00',
|
||||
scheduleInterval: initialValues.scheduleInterval || 1,
|
||||
providerId: initialValues.providerId,
|
||||
enabled: initialValues.enabled,
|
||||
})
|
||||
} else {
|
||||
@@ -114,12 +141,33 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
scheduleType: 'daily',
|
||||
scheduleTime: '09:00',
|
||||
scheduleInterval: 1,
|
||||
providerId: undefined,
|
||||
enabled: true,
|
||||
})
|
||||
}
|
||||
}
|
||||
}, [open, initialValues, form])
|
||||
|
||||
// Resolve the currently selected provider for the selector display
|
||||
const resolvedProvider: Provider | null = (() => {
|
||||
const id = selectedProviderId ?? defaultProviderId
|
||||
const found = providers.find((p) => p.id === id)
|
||||
if (found) return { id: found.id, name: found.name, type: found.type }
|
||||
if (providers[0])
|
||||
return {
|
||||
id: providers[0].id,
|
||||
name: providers[0].name,
|
||||
type: providers[0].type,
|
||||
}
|
||||
return null
|
||||
})()
|
||||
|
||||
const providerOptions: Provider[] = providers.map((p) => ({
|
||||
id: p.id,
|
||||
name: p.name,
|
||||
type: p.type,
|
||||
}))
|
||||
|
||||
const onSubmit = (values: FormValues) => {
|
||||
onSave({
|
||||
name: values.name.trim(),
|
||||
@@ -129,6 +177,7 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
values.scheduleType === 'daily' ? values.scheduleTime : undefined,
|
||||
scheduleInterval:
|
||||
values.scheduleType !== 'daily' ? values.scheduleInterval : undefined,
|
||||
providerId: values.providerId,
|
||||
enabled: values.enabled,
|
||||
})
|
||||
form.reset()
|
||||
@@ -185,6 +234,43 @@ export const NewScheduledTaskDialog: FC<NewScheduledTaskDialogProps> = ({
|
||||
)}
|
||||
/>
|
||||
|
||||
{providers.length > 0 && resolvedProvider && (
|
||||
<FormItem>
|
||||
<FormLabel>AI Provider</FormLabel>
|
||||
<ChatProviderSelector
|
||||
providers={providerOptions}
|
||||
selectedProvider={resolvedProvider}
|
||||
onSelectProvider={(provider) =>
|
||||
form.setValue('providerId', provider.id)
|
||||
}
|
||||
>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
className="w-full justify-between"
|
||||
>
|
||||
<span className="flex items-center gap-2">
|
||||
<span className="text-muted-foreground">
|
||||
{resolvedProvider.type === 'browseros' ? (
|
||||
<BrowserOSIcon size={16} />
|
||||
) : (
|
||||
<ProviderIcon
|
||||
type={resolvedProvider.type as ProviderType}
|
||||
size={16}
|
||||
/>
|
||||
)}
|
||||
</span>
|
||||
{resolvedProvider.name}
|
||||
</span>
|
||||
<ChevronDown className="h-4 w-4 opacity-50" />
|
||||
</Button>
|
||||
</ChatProviderSelector>
|
||||
<FormDescription>
|
||||
The AI provider used to run this task
|
||||
</FormDescription>
|
||||
</FormItem>
|
||||
)}
|
||||
|
||||
<div className="grid gap-4 sm:grid-cols-2">
|
||||
<FormField
|
||||
control={form.control}
|
||||
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
Trash2,
|
||||
XCircle,
|
||||
} from 'lucide-react'
|
||||
import { type FC, useMemo, useState } from 'react'
|
||||
import { type FC, useEffect, useMemo, useState } from 'react'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import {
|
||||
Collapsible,
|
||||
@@ -20,6 +20,9 @@ import {
|
||||
CollapsibleTrigger,
|
||||
} from '@/components/ui/collapsible'
|
||||
import { Switch } from '@/components/ui/switch'
|
||||
import { BrowserOSIcon, ProviderIcon } from '@/lib/llm-providers/providerIcons'
|
||||
import { providersStorage } from '@/lib/llm-providers/storage'
|
||||
import type { ProviderType } from '@/lib/llm-providers/types'
|
||||
import { useScheduledJobRuns } from '@/lib/schedules/scheduleStorage'
|
||||
import type { ScheduledJob, ScheduledJobRun } from './types'
|
||||
|
||||
@@ -80,9 +83,25 @@ export const ScheduledTaskCard: FC<ScheduledTaskCardProps> = ({
|
||||
onRetryRun,
|
||||
}) => {
|
||||
const [isOpen, setIsOpen] = useState(false)
|
||||
const [providerInfo, setProviderInfo] = useState<{
|
||||
name: string
|
||||
type: ProviderType
|
||||
} | null>(null)
|
||||
|
||||
const { jobRuns } = useScheduledJobRuns()
|
||||
|
||||
// Load provider info for display
|
||||
useEffect(() => {
|
||||
if (!job.providerId) {
|
||||
setProviderInfo(null)
|
||||
return
|
||||
}
|
||||
providersStorage.getValue().then((providers) => {
|
||||
const match = providers?.find((p) => p.id === job.providerId)
|
||||
setProviderInfo(match ? { name: match.name, type: match.type } : null)
|
||||
})
|
||||
}, [job.providerId])
|
||||
|
||||
const runs = useMemo(
|
||||
() =>
|
||||
jobRuns
|
||||
@@ -117,6 +136,19 @@ export const ScheduledTaskCard: FC<ScheduledTaskCardProps> = ({
|
||||
</p>
|
||||
<div className="flex items-center gap-2 text-muted-foreground text-xs">
|
||||
<span>{formatSchedule(job)}</span>
|
||||
{providerInfo && (
|
||||
<>
|
||||
<span>•</span>
|
||||
<span className="flex items-center gap-1">
|
||||
{providerInfo.type === 'browseros' ? (
|
||||
<BrowserOSIcon size={12} />
|
||||
) : (
|
||||
<ProviderIcon type={providerInfo.type} size={12} />
|
||||
)}
|
||||
{providerInfo.name}
|
||||
</span>
|
||||
</>
|
||||
)}
|
||||
{job.lastRunAt && (
|
||||
<>
|
||||
<span>•</span>
|
||||
|
||||
@@ -117,6 +117,7 @@ export const scheduledJobRuns = async () => {
|
||||
const response = await getChatServerResponse({
|
||||
message: job.query,
|
||||
signal: abortController.signal,
|
||||
providerId: job.providerId,
|
||||
})
|
||||
|
||||
await updateJobRun(jobRun.id, {
|
||||
|
||||
@@ -8,9 +8,14 @@ import {
|
||||
SIDEPANEL_SUGGESTION_CLICKED_EVENT,
|
||||
SIDEPANEL_TAB_REMOVED_EVENT,
|
||||
SIDEPANEL_TAB_TOGGLED_EVENT,
|
||||
SIDEPANEL_VOICE_ERROR_EVENT,
|
||||
SIDEPANEL_VOICE_RECORDING_STARTED_EVENT,
|
||||
SIDEPANEL_VOICE_RECORDING_STOPPED_EVENT,
|
||||
SIDEPANEL_VOICE_TRANSCRIPTION_COMPLETED_EVENT,
|
||||
} from '@/lib/constants/analyticsEvents'
|
||||
import { useJtbdPopup } from '@/lib/jtbd-popup/useJtbdPopup'
|
||||
import { track } from '@/lib/metrics/track'
|
||||
import { useVoiceInput } from '@/lib/voice/useVoiceInput'
|
||||
import { useChatSessionContext } from '../layout/ChatSessionContext'
|
||||
import { ChatEmptyState } from './ChatEmptyState'
|
||||
import { ChatError } from './ChatError'
|
||||
@@ -48,6 +53,8 @@ export const Chat = () => {
|
||||
onDismiss: onDismissJtbdPopup,
|
||||
} = useJtbdPopup()
|
||||
|
||||
const voice = useVoiceInput()
|
||||
|
||||
const [input, setInput] = useState('')
|
||||
const [attachedTabs, setAttachedTabs] = useState<chrome.tabs.Tab[]>([])
|
||||
const [mounted, setMounted] = useState(false)
|
||||
@@ -83,6 +90,26 @@ export const Chat = () => {
|
||||
previousChatStatus.current = status
|
||||
}, [status])
|
||||
|
||||
// Insert transcript into input when transcription completes
|
||||
// biome-ignore lint/correctness/useExhaustiveDependencies: only trigger on transcript/transcribing change
|
||||
useEffect(() => {
|
||||
if (voice.transcript && !voice.isTranscribing) {
|
||||
setInput((prev) => {
|
||||
const separator = prev.trim() ? ' ' : ''
|
||||
return prev + separator + voice.transcript
|
||||
})
|
||||
track(SIDEPANEL_VOICE_TRANSCRIPTION_COMPLETED_EVENT)
|
||||
voice.clearTranscript()
|
||||
}
|
||||
}, [voice.transcript, voice.isTranscribing])
|
||||
|
||||
// Track voice errors
|
||||
useEffect(() => {
|
||||
if (voice.error) {
|
||||
track(SIDEPANEL_VOICE_ERROR_EVENT, { error: voice.error })
|
||||
}
|
||||
}, [voice.error])
|
||||
|
||||
const handleModeChange = (newMode: ChatMode) => {
|
||||
track(SIDEPANEL_MODE_CHANGED_EVENT, { from: mode, to: newMode })
|
||||
setMode(newMode)
|
||||
@@ -147,6 +174,27 @@ export const Chat = () => {
|
||||
executeMessage(suggestion)
|
||||
}
|
||||
|
||||
const handleStartRecording = async () => {
|
||||
const started = await voice.startRecording()
|
||||
if (started) {
|
||||
track(SIDEPANEL_VOICE_RECORDING_STARTED_EVENT)
|
||||
}
|
||||
}
|
||||
|
||||
const handleStopRecording = async () => {
|
||||
await voice.stopRecording()
|
||||
track(SIDEPANEL_VOICE_RECORDING_STOPPED_EVENT)
|
||||
}
|
||||
|
||||
const voiceState = {
|
||||
isRecording: voice.isRecording,
|
||||
isTranscribing: voice.isTranscribing,
|
||||
audioLevels: voice.audioLevels,
|
||||
error: voice.error,
|
||||
onStartRecording: handleStartRecording,
|
||||
onStopRecording: handleStopRecording,
|
||||
}
|
||||
|
||||
return (
|
||||
<>
|
||||
<main className="mt-4 flex h-full flex-1 flex-col space-y-4 overflow-y-auto">
|
||||
@@ -190,6 +238,7 @@ export const Chat = () => {
|
||||
attachedTabs={attachedTabs}
|
||||
onToggleTab={toggleTabSelection}
|
||||
onRemoveTab={removeTab}
|
||||
voice={voiceState}
|
||||
/>
|
||||
</>
|
||||
)
|
||||
|
||||
@@ -10,6 +10,7 @@ import { useCapabilities } from '@/lib/browseros/useCapabilities'
|
||||
import { useMcpServers } from '@/lib/mcp/mcpServerStorage'
|
||||
import { useSyncRemoteIntegrations } from '@/lib/mcp/useSyncRemoteIntegrations'
|
||||
import { cn } from '@/lib/utils'
|
||||
import type { VoiceInputState } from '@/lib/voice/useVoiceInput'
|
||||
import { useWorkspace } from '@/lib/workspace/use-workspace'
|
||||
import { ChatAttachedTabs } from './ChatAttachedTabs'
|
||||
import { ChatInput, type ChatInputHandle } from './ChatInput'
|
||||
@@ -27,6 +28,7 @@ interface ChatFooterProps {
|
||||
attachedTabs: chrome.tabs.Tab[]
|
||||
onToggleTab: (tab: chrome.tabs.Tab) => void
|
||||
onRemoveTab: (tabId?: number) => void
|
||||
voice?: VoiceInputState
|
||||
}
|
||||
|
||||
export const ChatFooter: FC<ChatFooterProps> = ({
|
||||
@@ -40,6 +42,7 @@ export const ChatFooter: FC<ChatFooterProps> = ({
|
||||
attachedTabs,
|
||||
onToggleTab,
|
||||
onRemoveTab,
|
||||
voice,
|
||||
}) => {
|
||||
const { selectedFolder } = useWorkspace()
|
||||
const { supports } = useCapabilities()
|
||||
@@ -172,6 +175,10 @@ export const ChatFooter: FC<ChatFooterProps> = ({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{voice?.error && (
|
||||
<div className="mt-1 text-destructive text-xs">{voice.error}</div>
|
||||
)}
|
||||
|
||||
<ChatInput
|
||||
input={input}
|
||||
status={status}
|
||||
@@ -182,6 +189,7 @@ export const ChatFooter: FC<ChatFooterProps> = ({
|
||||
selectedTabs={attachedTabs}
|
||||
onToggleTab={onToggleTab}
|
||||
onTabMentionOpenChange={setIsTabMentionOpen}
|
||||
voice={voice}
|
||||
ref={chatInputRef}
|
||||
/>
|
||||
</div>
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { Send, SquareStop } from 'lucide-react'
|
||||
import { Loader2, Mic, Send, Square, SquareStop } from 'lucide-react'
|
||||
import type { FormEvent, KeyboardEvent } from 'react'
|
||||
import {
|
||||
forwardRef,
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
} from 'react'
|
||||
import { TabPickerPopover } from '@/components/elements/tab-picker-popover'
|
||||
import { cn } from '@/lib/utils'
|
||||
import type { VoiceInputState } from '@/lib/voice/useVoiceInput'
|
||||
import type { ChatMode } from './chatTypes'
|
||||
|
||||
interface MentionState {
|
||||
@@ -28,6 +29,7 @@ interface ChatInputProps {
|
||||
selectedTabs: chrome.tabs.Tab[]
|
||||
onToggleTab: (tab: chrome.tabs.Tab) => void
|
||||
onTabMentionOpenChange?: (isOpen: boolean) => void
|
||||
voice?: VoiceInputState
|
||||
}
|
||||
|
||||
export interface ChatInputHandle {
|
||||
@@ -49,6 +51,7 @@ export const ChatInput = forwardRef<ChatInputHandle, ChatInputProps>(
|
||||
selectedTabs,
|
||||
onToggleTab,
|
||||
onTabMentionOpenChange,
|
||||
voice,
|
||||
},
|
||||
ref,
|
||||
) => {
|
||||
@@ -259,6 +262,70 @@ export const ChatInput = forwardRef<ChatInputHandle, ChatInputProps>(
|
||||
return () => document.removeEventListener('mousedown', handleClickOutside)
|
||||
}, [mentionState.isOpen, closeMention])
|
||||
|
||||
const renderVoiceButton = () => {
|
||||
if (!voice) return null
|
||||
|
||||
if (voice.isRecording) {
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
onClick={voice.onStopRecording}
|
||||
className="cursor-pointer rounded-full bg-red-600 p-2 text-white shadow-sm transition-all duration-200 hover:bg-red-900"
|
||||
>
|
||||
<Square className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Stop recording</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
if (voice.isTranscribing) {
|
||||
return (
|
||||
<button type="button" disabled className="rounded-full p-2 text-muted-foreground">
|
||||
<Loader2 className="h-3.5 w-3.5 animate-spin" />
|
||||
<span className="sr-only">Transcribing</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
onClick={voice.onStartRecording}
|
||||
disabled={isBusy}
|
||||
className="cursor-pointer rounded-full p-2 text-muted-foreground transition-all duration-200 hover:bg-muted hover:text-foreground disabled:cursor-not-allowed disabled:opacity-50"
|
||||
>
|
||||
<Mic className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Voice input</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
const renderSendButton = () => {
|
||||
if (isBusy) {
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
onClick={onStop}
|
||||
className="cursor-pointer rounded-full bg-red-600 p-2 text-white shadow-sm transition-all duration-200 hover:bg-red-900"
|
||||
>
|
||||
<SquareStop className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Stop</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<button
|
||||
type="submit"
|
||||
disabled={!input.trim() || voice?.isRecording || voice?.isTranscribing}
|
||||
className="cursor-pointer rounded-full bg-[var(--accent-orange)] p-2 text-white shadow-sm transition-all duration-200 hover:bg-[var(--accent-orange-bright)] disabled:cursor-not-allowed disabled:opacity-50"
|
||||
>
|
||||
<Send className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Send</span>
|
||||
</button>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<form
|
||||
onSubmit={handleSubmit}
|
||||
@@ -273,38 +340,45 @@ export const ChatInput = forwardRef<ChatInputHandle, ChatInputProps>(
|
||||
onClose={closeMention}
|
||||
anchorRef={textareaRef}
|
||||
/>
|
||||
<textarea
|
||||
ref={textareaRef}
|
||||
className={cn(
|
||||
'field-sizing-content max-h-60 min-h-[42px] flex-1 resize-none overflow-hidden rounded-2xl border border-border/50 bg-muted/50 px-4 py-2.5 pr-11 text-sm outline-none transition-colors placeholder:text-muted-foreground/70 hover:border-border focus:border-[var(--accent-orange)]',
|
||||
)}
|
||||
value={input}
|
||||
onChange={(e) => handleInputChange(e.target.value)}
|
||||
onKeyDown={handleKeyDown}
|
||||
placeholder={
|
||||
mode === 'chat' ? 'Ask about this page...' : 'What should I do?'
|
||||
}
|
||||
rows={1}
|
||||
/>
|
||||
{isBusy ? (
|
||||
<button
|
||||
type="button"
|
||||
onClick={onStop}
|
||||
className="absolute right-1.5 bottom-1.5 cursor-pointer rounded-full bg-red-600 p-2 text-white shadow-sm transition-all duration-200 hover:bg-red-900 disabled:cursor-not-allowed disabled:opacity-50"
|
||||
{voice?.isRecording ? (
|
||||
<div
|
||||
className="flex min-h-[42px] flex-1 items-center justify-center gap-1 rounded-2xl border border-red-500/50 bg-muted/50 px-4 py-2.5 pr-[4.5rem]"
|
||||
>
|
||||
<SquareStop className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Stop</span>
|
||||
</button>
|
||||
{voice.audioLevels.map((level, i) => (
|
||||
<div
|
||||
key={i}
|
||||
className="w-1 rounded-full bg-red-500 transition-all duration-75"
|
||||
style={{
|
||||
height: `${Math.max(4, Math.min(20, level * 0.6))}px`,
|
||||
}}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
) : (
|
||||
<button
|
||||
type="submit"
|
||||
disabled={!input.trim()}
|
||||
className="absolute right-1.5 bottom-1.5 cursor-pointer rounded-full bg-[var(--accent-orange)] p-2 text-white shadow-sm transition-all duration-200 hover:bg-[var(--accent-orange-bright)] disabled:cursor-not-allowed disabled:opacity-50"
|
||||
>
|
||||
<Send className="h-3.5 w-3.5" />
|
||||
<span className="sr-only">Send</span>
|
||||
</button>
|
||||
<textarea
|
||||
ref={textareaRef}
|
||||
className={cn(
|
||||
'field-sizing-content max-h-60 min-h-[42px] flex-1 resize-none overflow-hidden rounded-2xl border border-border/50 bg-muted/50 px-4 py-2.5 text-sm outline-none transition-colors placeholder:text-muted-foreground/70 hover:border-border focus:border-[var(--accent-orange)]',
|
||||
voice ? 'pr-[4.5rem]' : 'pr-11',
|
||||
)}
|
||||
value={input}
|
||||
onChange={(e) => handleInputChange(e.target.value)}
|
||||
onKeyDown={handleKeyDown}
|
||||
placeholder={
|
||||
voice?.isTranscribing
|
||||
? 'Transcribing...'
|
||||
: mode === 'chat'
|
||||
? 'Ask about this page...'
|
||||
: 'What should I do?'
|
||||
}
|
||||
disabled={voice?.isTranscribing}
|
||||
rows={1}
|
||||
/>
|
||||
)}
|
||||
<div className="absolute right-1.5 bottom-1.5 flex items-center gap-1">
|
||||
{renderVoiceButton()}
|
||||
{renderSendButton()}
|
||||
</div>
|
||||
</form>
|
||||
)
|
||||
},
|
||||
|
||||
@@ -209,7 +209,8 @@ export const useChatSession = (options?: ChatSessionOptions) => {
|
||||
})
|
||||
const activeTab = activeTabsList?.[0] ?? undefined
|
||||
const message = getLastMessageText(messages)
|
||||
const provider = selectedLlmProviderRef.current ?? createDefaultBrowserOSProvider()
|
||||
const provider =
|
||||
selectedLlmProviderRef.current ?? createDefaultBrowserOSProvider()
|
||||
const currentMode = modeRef.current
|
||||
const enabledMcpServers = enabledMcpServersRef.current
|
||||
const customMcpServers = enabledCustomServersRef.current
|
||||
|
||||
@@ -251,3 +251,18 @@ export const KIMI_RATE_LIMIT_DOCS_CLICKED_EVENT =
|
||||
/** @public */
|
||||
export const KIMI_RATE_LIMIT_PLATFORM_CLICKED_EVENT =
|
||||
'ui.rate_limit.moonshot_platform_clicked'
|
||||
|
||||
/** @public */
|
||||
export const SIDEPANEL_VOICE_RECORDING_STARTED_EVENT =
|
||||
'sidepanel.voice.recording_started'
|
||||
|
||||
/** @public */
|
||||
export const SIDEPANEL_VOICE_RECORDING_STOPPED_EVENT =
|
||||
'sidepanel.voice.recording_stopped'
|
||||
|
||||
/** @public */
|
||||
export const SIDEPANEL_VOICE_TRANSCRIPTION_COMPLETED_EVENT =
|
||||
'sidepanel.voice.transcription_completed'
|
||||
|
||||
/** @public */
|
||||
export const SIDEPANEL_VOICE_ERROR_EVENT = 'sidepanel.voice.error'
|
||||
|
||||
@@ -158,9 +158,7 @@ export function useLlmProviders(): UseLlmProvidersReturn {
|
||||
// Fall back to first provider if defaultProviderId is stale/invalid
|
||||
const selectedProvider = useMemo(
|
||||
() =>
|
||||
providers.find((p) => p.id === defaultProviderId) ??
|
||||
providers[0] ??
|
||||
null,
|
||||
providers.find((p) => p.id === defaultProviderId) ?? providers[0] ?? null,
|
||||
[providers, defaultProviderId],
|
||||
)
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ interface ChatServerRequest {
|
||||
windowId?: number
|
||||
activeTab?: ActiveTab
|
||||
signal?: AbortSignal
|
||||
providerId?: string
|
||||
}
|
||||
|
||||
interface ChatServerResponse {
|
||||
@@ -75,11 +76,23 @@ const getDefaultProvider = async (): Promise<LlmProviderConfig | null> => {
|
||||
return defaultProvider ?? providers[0] ?? null
|
||||
}
|
||||
|
||||
// Resolve provider by ID, falling back to global default
|
||||
const resolveProvider = async (
|
||||
providerId?: string,
|
||||
): Promise<LlmProviderConfig> => {
|
||||
if (providerId) {
|
||||
const providers = await providersStorage.getValue()
|
||||
const match = providers?.find((p) => p.id === providerId)
|
||||
if (match) return match
|
||||
}
|
||||
return (await getDefaultProvider()) ?? createDefaultBrowserOSProvider()
|
||||
}
|
||||
|
||||
export async function getChatServerResponse(
|
||||
request: ChatServerRequest,
|
||||
): Promise<ChatServerResponse> {
|
||||
const agentServerUrl = await getAgentServerUrl()
|
||||
const provider = (await getDefaultProvider()) ?? createDefaultBrowserOSProvider()
|
||||
const provider = await resolveProvider(request.providerId)
|
||||
const conversationId = request.conversationId ?? crypto.randomUUID()
|
||||
const personalization = await personalizationStorage.getValue()
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ export const GetScheduledJobsByProfileIdDocument = graphql(`
|
||||
scheduleTime
|
||||
scheduleInterval
|
||||
enabled
|
||||
llmProviderId
|
||||
createdAt
|
||||
updatedAt
|
||||
lastRunAt
|
||||
|
||||
@@ -6,6 +6,7 @@ export interface ScheduledJob {
|
||||
scheduleTime?: string
|
||||
scheduleInterval?: number
|
||||
enabled: boolean
|
||||
providerId?: string
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
lastRunAt?: string
|
||||
|
||||
@@ -19,6 +19,7 @@ type RemoteScheduledJob = {
|
||||
scheduleTime: string | null
|
||||
scheduleInterval: number | null
|
||||
enabled: boolean
|
||||
llmProviderId: string | null
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
lastRunAt: string | null
|
||||
@@ -32,6 +33,7 @@ function toComparable(job: ScheduledJob) {
|
||||
...data,
|
||||
scheduleTime: data.scheduleTime ?? null,
|
||||
scheduleInterval: data.scheduleInterval ?? null,
|
||||
providerId: data.providerId ?? null,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,6 +45,7 @@ function remoteToComparable(job: RemoteScheduledJob) {
|
||||
scheduleTime: job.scheduleTime,
|
||||
scheduleInterval: job.scheduleInterval,
|
||||
enabled: job.enabled,
|
||||
providerId: job.llmProviderId,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,6 +62,7 @@ function remoteToLocal(remote: RemoteScheduledJob): ScheduledJob {
|
||||
scheduleTime: remote.scheduleTime ?? undefined,
|
||||
scheduleInterval: remote.scheduleInterval ?? undefined,
|
||||
enabled: remote.enabled,
|
||||
providerId: remote.llmProviderId ?? undefined,
|
||||
createdAt: normalizeTimestamp(remote.createdAt),
|
||||
updatedAt: normalizeTimestamp(remote.updatedAt),
|
||||
lastRunAt: remote.lastRunAt
|
||||
@@ -163,6 +167,7 @@ export async function syncSchedulesToBackend(
|
||||
scheduleTime: job.scheduleTime ?? null,
|
||||
scheduleInterval: job.scheduleInterval ?? null,
|
||||
enabled: job.enabled,
|
||||
llmProviderId: job.providerId ?? null,
|
||||
lastRunAt: job.lastRunAt
|
||||
? new Date(job.lastRunAt).toISOString()
|
||||
: null,
|
||||
@@ -182,6 +187,7 @@ export async function syncSchedulesToBackend(
|
||||
scheduleTime: job.scheduleTime ?? null,
|
||||
scheduleInterval: job.scheduleInterval ?? null,
|
||||
enabled: job.enabled,
|
||||
llmProviderId: job.providerId ?? null,
|
||||
createdAt: new Date(job.createdAt).toISOString(),
|
||||
updatedAt: job.updatedAt || new Date().toISOString(),
|
||||
lastRunAt: job.lastRunAt
|
||||
|
||||
@@ -1,18 +1,35 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
|
||||
const GATEWAY_URL = 'https://llm.browseros.com'
|
||||
const WAVEFORM_BAND_COUNT = 5
|
||||
|
||||
interface UseVoiceInputReturn {
|
||||
export interface VoiceInputState {
|
||||
isRecording: boolean
|
||||
isTranscribing: boolean
|
||||
audioLevels: number[]
|
||||
error: string | null
|
||||
onStartRecording: () => void
|
||||
onStopRecording: () => void
|
||||
}
|
||||
|
||||
export interface UseVoiceInputReturn {
|
||||
isRecording: boolean
|
||||
isTranscribing: boolean
|
||||
transcript: string
|
||||
audioLevel: number
|
||||
audioLevels: number[]
|
||||
error: string | null
|
||||
startRecording: () => Promise<void>
|
||||
startRecording: () => Promise<boolean>
|
||||
stopRecording: () => Promise<void>
|
||||
clearTranscript: () => void
|
||||
}
|
||||
|
||||
const EMPTY_LEVELS = Array(WAVEFORM_BAND_COUNT).fill(0)
|
||||
|
||||
interface TranscribeResponse {
|
||||
text: string
|
||||
}
|
||||
|
||||
async function transcribeAudio(audioBlob: Blob): Promise<string> {
|
||||
const formData = new FormData()
|
||||
formData.append('file', audioBlob, 'recording.webm')
|
||||
@@ -21,16 +38,17 @@ async function transcribeAudio(audioBlob: Blob): Promise<string> {
|
||||
const response = await fetch(`${GATEWAY_URL}/api/transcribe`, {
|
||||
method: 'POST',
|
||||
body: formData,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const error = await response
|
||||
const errorBody: { error?: string } = await response
|
||||
.json()
|
||||
.catch(() => ({ error: 'Transcription failed' }))
|
||||
throw new Error(error.error || `Transcription failed: ${response.status}`)
|
||||
throw new Error(errorBody.error || `Transcription failed: ${response.status}`)
|
||||
}
|
||||
|
||||
const result = await response.json()
|
||||
const result: TranscribeResponse = await response.json()
|
||||
return result.text || ''
|
||||
}
|
||||
|
||||
@@ -39,6 +57,7 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
const [isTranscribing, setIsTranscribing] = useState(false)
|
||||
const [transcript, setTranscript] = useState('')
|
||||
const [audioLevel, setAudioLevel] = useState(0)
|
||||
const [audioLevels, setAudioLevels] = useState<number[]>(EMPTY_LEVELS)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
|
||||
const mediaRecorderRef = useRef<MediaRecorder | null>(null)
|
||||
@@ -48,7 +67,7 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
const analyserRef = useRef<AnalyserNode | null>(null)
|
||||
const animationFrameRef = useRef<number | null>(null)
|
||||
|
||||
const stopAudioLevelMonitoring = useCallback(() => {
|
||||
const stopAudioLevelMonitoring = () => {
|
||||
if (animationFrameRef.current) {
|
||||
cancelAnimationFrame(animationFrameRef.current)
|
||||
animationFrameRef.current = null
|
||||
@@ -59,7 +78,8 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
audioContextRef.current = null
|
||||
analyserRef.current = null
|
||||
setAudioLevel(0)
|
||||
}, [])
|
||||
setAudioLevels(EMPTY_LEVELS)
|
||||
}
|
||||
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
@@ -71,9 +91,9 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
}
|
||||
stopAudioLevelMonitoring()
|
||||
}
|
||||
}, [stopAudioLevelMonitoring])
|
||||
}, [])
|
||||
|
||||
const startAudioLevelMonitoring = useCallback((stream: MediaStream) => {
|
||||
const startAudioLevelMonitoring = (stream: MediaStream) => {
|
||||
const audioContext = new AudioContext()
|
||||
const analyser = audioContext.createAnalyser()
|
||||
analyser.fftSize = 256
|
||||
@@ -87,20 +107,36 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
const updateLevel = () => {
|
||||
if (!analyserRef.current) return
|
||||
|
||||
const dataArray = new Uint8Array(analyserRef.current.frequencyBinCount)
|
||||
analyserRef.current.getByteFrequencyData(dataArray)
|
||||
const dataArray = new Uint8Array(analyserRef.current.fftSize)
|
||||
analyserRef.current.getByteTimeDomainData(dataArray)
|
||||
|
||||
const average = dataArray.reduce((a, b) => a + b, 0) / dataArray.length
|
||||
const normalized = Math.min(100, (average / 128) * 100)
|
||||
setAudioLevel(Math.round(normalized))
|
||||
const binCount = dataArray.length
|
||||
const levels: number[] = []
|
||||
let totalPeak = 0
|
||||
|
||||
for (let band = 0; band < WAVEFORM_BAND_COUNT; band++) {
|
||||
const start = Math.floor((band / WAVEFORM_BAND_COUNT) * binCount)
|
||||
const end = Math.floor(((band + 1) / WAVEFORM_BAND_COUNT) * binCount)
|
||||
let peak = 0
|
||||
for (let j = start; j < end; j++) {
|
||||
const amplitude = Math.abs(dataArray[j] - 128)
|
||||
if (amplitude > peak) peak = amplitude
|
||||
}
|
||||
const normalized = Math.round(Math.min(100, (peak / 50) * 100))
|
||||
levels.push(normalized)
|
||||
totalPeak += normalized
|
||||
}
|
||||
|
||||
setAudioLevels(levels)
|
||||
setAudioLevel(Math.round(totalPeak / WAVEFORM_BAND_COUNT))
|
||||
|
||||
animationFrameRef.current = requestAnimationFrame(updateLevel)
|
||||
}
|
||||
|
||||
updateLevel()
|
||||
}, [])
|
||||
}
|
||||
|
||||
const startRecording = useCallback(async () => {
|
||||
const startRecording = async (): Promise<boolean> => {
|
||||
try {
|
||||
setError(null)
|
||||
setTranscript('')
|
||||
@@ -133,7 +169,12 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
|
||||
mediaRecorder.start(250)
|
||||
setIsRecording(true)
|
||||
return true
|
||||
} catch (err) {
|
||||
streamRef.current?.getTracks().forEach((track) => track.stop())
|
||||
streamRef.current = null
|
||||
stopAudioLevelMonitoring()
|
||||
|
||||
if (err instanceof Error) {
|
||||
if (err.name === 'NotAllowedError') {
|
||||
setError('Microphone permission denied')
|
||||
@@ -145,10 +186,11 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
} else {
|
||||
setError('Failed to start recording')
|
||||
}
|
||||
return false
|
||||
}
|
||||
}, [startAudioLevelMonitoring])
|
||||
}
|
||||
|
||||
const stopRecording = useCallback(async () => {
|
||||
const stopRecording = async () => {
|
||||
const mediaRecorder = mediaRecorderRef.current
|
||||
|
||||
if (!mediaRecorder || mediaRecorder.state === 'inactive') {
|
||||
@@ -188,18 +230,19 @@ export function useVoiceInput(): UseVoiceInputReturn {
|
||||
} finally {
|
||||
setIsTranscribing(false)
|
||||
}
|
||||
}, [stopAudioLevelMonitoring])
|
||||
}
|
||||
|
||||
const clearTranscript = useCallback(() => {
|
||||
const clearTranscript = () => {
|
||||
setTranscript('')
|
||||
setError(null)
|
||||
}, [])
|
||||
}
|
||||
|
||||
return {
|
||||
isRecording,
|
||||
isTranscribing,
|
||||
transcript,
|
||||
audioLevel,
|
||||
audioLevels,
|
||||
error,
|
||||
startRecording,
|
||||
stopRecording,
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@browseros/server",
|
||||
"version": "0.0.74",
|
||||
"version": "0.0.75",
|
||||
"description": "BrowserOS server",
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
/**
|
||||
* @license
|
||||
* Copyright 2025 BrowserOS
|
||||
* SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
*/
|
||||
|
||||
import type { UIMessage } from 'ai'
|
||||
|
||||
/**
|
||||
* Checks whether a UIMessage has meaningful content that can be sent
|
||||
* to the AI provider without causing validation errors.
|
||||
*
|
||||
* Two layers of validation can reject messages:
|
||||
*
|
||||
* 1. **AI SDK** (`validate-ui-messages.ts`):
|
||||
* - `parts` array must be `.nonempty()` — rejects `parts: []`
|
||||
*
|
||||
* 2. **Provider API** (e.g. Gemini `generateContent`, Anthropic, OpenAI):
|
||||
* - Assistant messages with only empty-string text are rejected
|
||||
* as semantically empty, even though the SDK schema allows it
|
||||
*
|
||||
* This function guards against both layers so callers can filter
|
||||
* messages before passing them to `createAgentUIStreamResponse`.
|
||||
*/
|
||||
export function hasMessageContent(message: UIMessage): boolean {
|
||||
if (message.parts.length === 0) return false
|
||||
|
||||
// A message that contains any non-text part (tool invocation, reasoning,
|
||||
// file, step-start, etc.) is always considered valid — those part types
|
||||
// carry meaning regardless of text content.
|
||||
const hasNonTextPart = message.parts.some((p) => p.type !== 'text')
|
||||
if (hasNonTextPart) return true
|
||||
|
||||
// All parts are text — at least one must have non-whitespace content.
|
||||
return message.parts.some(
|
||||
(p) => p.type === 'text' && p.text.trim().length > 0,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Filters a UIMessage array, removing messages that would fail
|
||||
* SDK validation or provider-level content checks.
|
||||
*/
|
||||
export function filterValidMessages(messages: UIMessage[]): UIMessage[] {
|
||||
return messages.filter(hasMessageContent)
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import { mkdir, utimes } from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { createAgentUIStreamResponse, type UIMessage } from 'ai'
|
||||
import { AiSdkAgent } from '../../agent/ai-sdk-agent'
|
||||
import { filterValidMessages } from '../../agent/message-validation'
|
||||
import { formatUserMessage } from '../../agent/format-message'
|
||||
import type { SessionStore } from '../../agent/session-store'
|
||||
import type { ResolvedAgentConfig } from '../../agent/types'
|
||||
@@ -139,6 +140,7 @@ export class ChatService {
|
||||
|
||||
if (isNewSession && request.previousConversation?.length) {
|
||||
for (const msg of request.previousConversation) {
|
||||
if (!msg.content.trim()) continue
|
||||
session.agent.messages.push({
|
||||
id: crypto.randomUUID(),
|
||||
role: msg.role === 'assistant' ? 'assistant' : 'user',
|
||||
@@ -168,10 +170,10 @@ export class ChatService {
|
||||
|
||||
return createAgentUIStreamResponse({
|
||||
agent: session.agent.toolLoopAgent,
|
||||
uiMessages: session.agent.messages,
|
||||
uiMessages: filterValidMessages(session.agent.messages),
|
||||
abortSignal,
|
||||
onFinish: async ({ messages }: { messages: UIMessage[] }) => {
|
||||
session.agent.messages = messages
|
||||
session.agent.messages = filterValidMessages(messages)
|
||||
logger.info('Agent execution complete', {
|
||||
conversationId: request.conversationId,
|
||||
totalMessages: messages.length,
|
||||
|
||||
@@ -36,6 +36,10 @@ class CdpBackend implements ICdpBackend {
|
||||
private reconnecting = false
|
||||
private reconnectRequested = false
|
||||
private eventHandlers = new Map<string, ((params: unknown) => void)[]>()
|
||||
private sessionEventHandlers = new Map<
|
||||
string,
|
||||
((params: unknown, sessionId: string) => void)[]
|
||||
>()
|
||||
private sessionCache = new Map<string, ProtocolApi>()
|
||||
private keepaliveTimer: ReturnType<typeof setInterval> | null = null
|
||||
private preferredDiscoveryHost: LoopbackDiscoveryHost | null = null
|
||||
@@ -432,6 +436,26 @@ class CdpBackend implements ICdpBackend {
|
||||
}
|
||||
}
|
||||
|
||||
onSessionEvent(
|
||||
event: string,
|
||||
handler: (params: unknown, sessionId: string) => void,
|
||||
): () => void {
|
||||
if (!this.sessionEventHandlers.has(event)) {
|
||||
this.sessionEventHandlers.set(event, [])
|
||||
}
|
||||
const handlers = this.sessionEventHandlers.get(event)
|
||||
if (handlers) {
|
||||
handlers.push(handler)
|
||||
}
|
||||
return () => {
|
||||
const list = this.sessionEventHandlers.get(event)
|
||||
if (list) {
|
||||
const idx = list.indexOf(handler)
|
||||
if (idx !== -1) list.splice(idx, 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private handleMessage(data: string): void {
|
||||
const message = JSON.parse(data) as {
|
||||
id?: number
|
||||
@@ -439,8 +463,10 @@ class CdpBackend implements ICdpBackend {
|
||||
params?: unknown
|
||||
result?: unknown
|
||||
error?: { message: string; code: number }
|
||||
sessionId?: string
|
||||
}
|
||||
|
||||
// Route responses to pending requests
|
||||
if (message.id !== undefined) {
|
||||
const pending = this.pending.get(message.id)
|
||||
if (pending) {
|
||||
@@ -453,12 +479,23 @@ class CdpBackend implements ICdpBackend {
|
||||
}
|
||||
}
|
||||
} else if (message.method) {
|
||||
// Dispatch to global event handlers
|
||||
const handlers = this.eventHandlers.get(message.method)
|
||||
if (handlers) {
|
||||
for (const handler of handlers) {
|
||||
handler(message.params)
|
||||
}
|
||||
}
|
||||
|
||||
// Dispatch to session-aware handlers when sessionId is present
|
||||
if (message.sessionId) {
|
||||
const sessionHandlers = this.sessionEventHandlers.get(message.method)
|
||||
if (sessionHandlers) {
|
||||
for (const handler of sessionHandlers) {
|
||||
handler(message.params, message.sessionId)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,10 @@ export interface CdpBackend extends ProtocolApi {
|
||||
isConnected(): boolean
|
||||
getTargets(): Promise<CdpTarget[]>
|
||||
session(sessionId: string): ProtocolApi
|
||||
onSessionEvent(
|
||||
event: string,
|
||||
handler: (params: unknown, sessionId: string) => void,
|
||||
): () => void
|
||||
}
|
||||
|
||||
export interface ControllerBackend {
|
||||
|
||||
@@ -3,6 +3,11 @@ import { logger } from '../lib/logger'
|
||||
import type { CdpBackend, ControllerBackend } from './backends/types'
|
||||
import type { BookmarkNode } from './bookmarks'
|
||||
import * as bookmarks from './bookmarks'
|
||||
import {
|
||||
ConsoleCollector,
|
||||
type GetConsoleLogsOptions,
|
||||
type GetConsoleLogsResult,
|
||||
} from './console-collector'
|
||||
import {
|
||||
buildContentMarkdownExpression,
|
||||
type ContentMarkdownOptions,
|
||||
@@ -84,6 +89,7 @@ export class Browser {
|
||||
private cdp: CdpBackend
|
||||
// biome-ignore lint/correctness/noUnusedPrivateClassMembers: kept for later removal
|
||||
private controller: ControllerBackend
|
||||
private consoleCollector: ConsoleCollector
|
||||
private pages = new Map<number, PageInfo>()
|
||||
private sessions = new Map<string, string>()
|
||||
private nextPageId = 1
|
||||
@@ -91,6 +97,7 @@ export class Browser {
|
||||
constructor(cdp: CdpBackend, controller: ControllerBackend) {
|
||||
this.cdp = cdp
|
||||
this.controller = controller
|
||||
this.consoleCollector = new ConsoleCollector(cdp)
|
||||
this.setupEventHandlers()
|
||||
}
|
||||
|
||||
@@ -123,11 +130,14 @@ export class Browser {
|
||||
throw new Error(
|
||||
`Unknown page ${page}. Use list_pages to see available pages.`,
|
||||
)
|
||||
const sessionId = await this.attachToPage(info.targetId)
|
||||
const sessionId = await this.attachToPage(info.targetId, page)
|
||||
return this.cdp.session(sessionId)
|
||||
}
|
||||
|
||||
private async attachToPage(targetId: string): Promise<string> {
|
||||
private async attachToPage(
|
||||
targetId: string,
|
||||
pageId: number,
|
||||
): Promise<string> {
|
||||
const cached = this.sessions.get(targetId)
|
||||
if (cached) return cached
|
||||
|
||||
@@ -143,10 +153,13 @@ export class Browser {
|
||||
session.Page.enable(),
|
||||
session.DOM.enable(),
|
||||
session.Runtime.enable(),
|
||||
session.Log.enable(),
|
||||
session.Accessibility.enable(),
|
||||
])
|
||||
|
||||
this.sessions.set(targetId, sessionId)
|
||||
this.consoleCollector.attach(pageId, sessionId)
|
||||
|
||||
return sessionId
|
||||
}
|
||||
|
||||
@@ -204,6 +217,7 @@ export class Browser {
|
||||
|
||||
for (const [pageId, info] of this.pages) {
|
||||
if (!seenTargetIds.has(info.targetId)) {
|
||||
this.consoleCollector.detach(pageId)
|
||||
this.pages.delete(pageId)
|
||||
}
|
||||
}
|
||||
@@ -290,6 +304,7 @@ export class Browser {
|
||||
`Unknown page ${page}. Use list_pages to see available pages.`,
|
||||
)
|
||||
await this.cdp.Browser.closeTab({ tabId: info.tabId })
|
||||
this.consoleCollector.detach(page)
|
||||
this.pages.delete(page)
|
||||
this.sessions.delete(info.targetId)
|
||||
}
|
||||
@@ -389,7 +404,30 @@ export class Browser {
|
||||
const session = await this.resolveSession(page)
|
||||
const nodes = await this.fetchAXTree(session)
|
||||
if (nodes.length === 0) return ''
|
||||
return snapshot.buildInteractiveTree(nodes).join('\n')
|
||||
|
||||
const lines = snapshot.buildInteractiveTree(nodes)
|
||||
|
||||
try {
|
||||
const cursorElements =
|
||||
await snapshot.findCursorInteractiveElements(session)
|
||||
|
||||
if (cursorElements.length > 0) {
|
||||
const includedIds = new Set<number>()
|
||||
for (const line of lines) {
|
||||
const match = line.match(/^\[(\d+)\]/)
|
||||
if (match) includedIds.add(Number(match[1]))
|
||||
}
|
||||
|
||||
for (const el of cursorElements) {
|
||||
if (includedIds.has(el.backendNodeId)) continue
|
||||
lines.push(`[${el.backendNodeId}] clickable "${el.text}"`)
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// cursor detection is best-effort; AX tree results are still returned
|
||||
}
|
||||
|
||||
return lines.join('\n')
|
||||
}
|
||||
|
||||
async getPageLinks(
|
||||
@@ -441,15 +479,15 @@ export class Browser {
|
||||
await snapshot.findCursorInteractiveElements(session)
|
||||
|
||||
if (cursorElements.length > 0) {
|
||||
const existingIds = new Set<number>()
|
||||
for (const node of nodes) {
|
||||
if (node.backendDOMNodeId !== undefined)
|
||||
existingIds.add(node.backendDOMNodeId)
|
||||
const includedIds = new Set<number>()
|
||||
for (const line of treeLines) {
|
||||
const match = line.match(/\[(\d+)\]/)
|
||||
if (match) includedIds.add(Number(match[1]))
|
||||
}
|
||||
|
||||
const extras: string[] = []
|
||||
for (const el of cursorElements) {
|
||||
if (existingIds.has(el.backendNodeId)) continue
|
||||
if (includedIds.has(el.backendNodeId)) continue
|
||||
extras.push(
|
||||
`[${el.backendNodeId}] clickable "${el.text}" (${el.reasons.join(', ')})`,
|
||||
)
|
||||
@@ -1263,4 +1301,14 @@ export class Browser {
|
||||
async closeTabGroup(groupId: string): Promise<void> {
|
||||
return tabGroups.closeTabGroup(this.cdp, groupId)
|
||||
}
|
||||
|
||||
// --- Console ---
|
||||
|
||||
async getConsoleLogs(
|
||||
page: number,
|
||||
opts?: GetConsoleLogsOptions,
|
||||
): Promise<GetConsoleLogsResult> {
|
||||
await this.resolveSession(page)
|
||||
return this.consoleCollector.getLogs(page, opts)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import type { EntryAddedEvent } from '@browseros/cdp-protocol/domains/log'
|
||||
import type {
|
||||
ConsoleAPICalledEvent,
|
||||
ExceptionThrownEvent,
|
||||
RemoteObject,
|
||||
} from '@browseros/cdp-protocol/domains/runtime'
|
||||
import { CONTENT_LIMITS } from '@browseros/shared/constants/limits'
|
||||
import type { CdpBackend } from './backends/types'
|
||||
|
||||
export type ConsoleLevel = 'error' | 'warning' | 'info' | 'debug'
|
||||
|
||||
export interface ConsoleEntry {
|
||||
source: 'console' | 'exception' | 'browser'
|
||||
level: ConsoleLevel
|
||||
text: string
|
||||
url?: string
|
||||
lineNumber?: number
|
||||
timestamp: number
|
||||
}
|
||||
|
||||
export interface GetConsoleLogsOptions {
|
||||
level?: ConsoleLevel
|
||||
search?: string
|
||||
limit?: number
|
||||
clear?: boolean
|
||||
}
|
||||
|
||||
export interface GetConsoleLogsResult {
|
||||
entries: ConsoleEntry[]
|
||||
totalCount: number
|
||||
}
|
||||
|
||||
// Lower number = higher severity
|
||||
const LEVEL_PRIORITY: Record<ConsoleLevel, number> = {
|
||||
error: 0,
|
||||
warning: 1,
|
||||
info: 2,
|
||||
debug: 3,
|
||||
}
|
||||
|
||||
const CONSOLE_TYPE_TO_LEVEL: Record<string, ConsoleLevel> = {
|
||||
error: 'error',
|
||||
assert: 'error',
|
||||
warning: 'warning',
|
||||
log: 'info',
|
||||
info: 'info',
|
||||
dir: 'info',
|
||||
dirxml: 'info',
|
||||
table: 'info',
|
||||
count: 'info',
|
||||
timeEnd: 'info',
|
||||
debug: 'debug',
|
||||
trace: 'debug',
|
||||
clear: 'debug',
|
||||
startGroup: 'debug',
|
||||
startGroupCollapsed: 'debug',
|
||||
endGroup: 'debug',
|
||||
profile: 'debug',
|
||||
profileEnd: 'debug',
|
||||
}
|
||||
|
||||
const LOG_LEVEL_MAP: Record<string, ConsoleLevel> = {
|
||||
error: 'error',
|
||||
warning: 'warning',
|
||||
info: 'info',
|
||||
verbose: 'debug',
|
||||
}
|
||||
|
||||
export class ConsoleCollector {
|
||||
private readonly buffers = new Map<number, ConsoleEntry[]>()
|
||||
private readonly sessionToPage = new Map<string, number>()
|
||||
private readonly pageToSession = new Map<number, string>()
|
||||
private readonly maxEntries = CONTENT_LIMITS.CONSOLE_BUFFER_MAX_ENTRIES
|
||||
|
||||
constructor(cdp: CdpBackend) {
|
||||
// Single handler per event type — O(1) routing via sessionToPage lookup
|
||||
cdp.onSessionEvent('Runtime.consoleAPICalled', (params, sessionId) => {
|
||||
const pageId = this.sessionToPage.get(sessionId)
|
||||
if (pageId === undefined) return
|
||||
this.handleConsoleAPI(pageId, params as ConsoleAPICalledEvent)
|
||||
})
|
||||
|
||||
cdp.onSessionEvent('Runtime.exceptionThrown', (params, sessionId) => {
|
||||
const pageId = this.sessionToPage.get(sessionId)
|
||||
if (pageId === undefined) return
|
||||
this.handleException(pageId, params as ExceptionThrownEvent)
|
||||
})
|
||||
|
||||
cdp.onSessionEvent('Log.entryAdded', (params, sessionId) => {
|
||||
const pageId = this.sessionToPage.get(sessionId)
|
||||
if (pageId === undefined) return
|
||||
this.handleLogEntry(pageId, params as EntryAddedEvent)
|
||||
})
|
||||
|
||||
// Clear buffer on main-frame navigation
|
||||
cdp.onSessionEvent('Page.frameNavigated', (params, sessionId) => {
|
||||
const pageId = this.sessionToPage.get(sessionId)
|
||||
if (pageId === undefined) return
|
||||
const frame = (params as { frame: { parentId?: string } }).frame
|
||||
if (!frame.parentId) {
|
||||
this.buffers.set(pageId, [])
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
attach(pageId: number, sessionId: string): void {
|
||||
if (!this.buffers.has(pageId)) {
|
||||
this.buffers.set(pageId, [])
|
||||
}
|
||||
// Clean up old session mapping if session changed (re-attach after detach)
|
||||
const oldSession = this.pageToSession.get(pageId)
|
||||
if (oldSession && oldSession !== sessionId) {
|
||||
this.sessionToPage.delete(oldSession)
|
||||
}
|
||||
this.sessionToPage.set(sessionId, pageId)
|
||||
this.pageToSession.set(pageId, sessionId)
|
||||
}
|
||||
|
||||
detach(pageId: number): void {
|
||||
const sessionId = this.pageToSession.get(pageId)
|
||||
if (sessionId) this.sessionToPage.delete(sessionId)
|
||||
this.pageToSession.delete(pageId)
|
||||
this.buffers.delete(pageId)
|
||||
}
|
||||
|
||||
getLogs(pageId: number, opts?: GetConsoleLogsOptions): GetConsoleLogsResult {
|
||||
const buffer = this.buffers.get(pageId) ?? []
|
||||
const levelThreshold = LEVEL_PRIORITY[opts?.level ?? 'info']
|
||||
|
||||
// Filter by level
|
||||
let filtered = buffer.filter(
|
||||
(e) => LEVEL_PRIORITY[e.level] <= levelThreshold,
|
||||
)
|
||||
|
||||
// Filter by search text
|
||||
if (opts?.search) {
|
||||
const term = opts.search.toLowerCase()
|
||||
filtered = filtered.filter((e) => e.text.toLowerCase().includes(term))
|
||||
}
|
||||
|
||||
// Return most recent entries up to limit
|
||||
const totalCount = filtered.length
|
||||
const limit = Math.min(
|
||||
opts?.limit ?? CONTENT_LIMITS.CONSOLE_DEFAULT_LIMIT,
|
||||
CONTENT_LIMITS.CONSOLE_MAX_LIMIT,
|
||||
)
|
||||
const entries = filtered.slice(-limit)
|
||||
|
||||
if (opts?.clear) {
|
||||
this.buffers.set(pageId, [])
|
||||
}
|
||||
|
||||
return { entries, totalCount }
|
||||
}
|
||||
|
||||
private addEntry(pageId: number, entry: ConsoleEntry): void {
|
||||
const buffer = this.buffers.get(pageId)
|
||||
if (!buffer) return
|
||||
|
||||
// FIFO eviction when buffer is full
|
||||
if (buffer.length >= this.maxEntries) {
|
||||
buffer.shift()
|
||||
}
|
||||
buffer.push(entry)
|
||||
}
|
||||
|
||||
private handleConsoleAPI(pageId: number, event: ConsoleAPICalledEvent): void {
|
||||
const level = CONSOLE_TYPE_TO_LEVEL[event.type] ?? 'info'
|
||||
const text = serializeArgs(event.args)
|
||||
const frame = event.stackTrace?.callFrames[0]
|
||||
|
||||
this.addEntry(pageId, {
|
||||
source: 'console',
|
||||
level,
|
||||
text,
|
||||
url: frame?.url,
|
||||
lineNumber: frame?.lineNumber,
|
||||
timestamp: event.timestamp,
|
||||
})
|
||||
}
|
||||
|
||||
private handleException(pageId: number, event: ExceptionThrownEvent): void {
|
||||
const details = event.exceptionDetails
|
||||
const text = details.exception?.description ?? details.text
|
||||
|
||||
this.addEntry(pageId, {
|
||||
source: 'exception',
|
||||
level: 'error',
|
||||
text,
|
||||
url: details.url ?? details.stackTrace?.callFrames[0]?.url,
|
||||
lineNumber: details.lineNumber,
|
||||
timestamp: event.timestamp,
|
||||
})
|
||||
}
|
||||
|
||||
private handleLogEntry(pageId: number, event: EntryAddedEvent): void {
|
||||
const entry = event.entry
|
||||
const level = LOG_LEVEL_MAP[entry.level] ?? 'info'
|
||||
|
||||
this.addEntry(pageId, {
|
||||
source: 'browser',
|
||||
level,
|
||||
text: entry.text,
|
||||
url: entry.url,
|
||||
lineNumber: entry.lineNumber,
|
||||
timestamp: entry.timestamp,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function serializeArgs(args: RemoteObject[]): string {
|
||||
return args
|
||||
.map((arg) => {
|
||||
if (arg.type === 'string') return arg.value as string
|
||||
if (arg.value !== undefined) return String(arg.value)
|
||||
return arg.description ?? `[${arg.type}]`
|
||||
})
|
||||
.join(' ')
|
||||
}
|
||||
@@ -41,6 +41,7 @@ const INTERACTIVE_ROLES = new Set([
|
||||
'option',
|
||||
'treeitem',
|
||||
'listbox',
|
||||
'DisclosureTriangle',
|
||||
])
|
||||
|
||||
const NAMED_CONTENT_ROLES = new Set([
|
||||
@@ -196,6 +197,7 @@ const CURSOR_INTERACTIVE_JS = `(function() {
|
||||
if (parent && getComputedStyle(parent).cursor === 'pointer') continue;
|
||||
}
|
||||
var text = (el.textContent || '').trim().slice(0, 100);
|
||||
if (!text) text = (el.getAttribute('aria-label') || '').trim();
|
||||
if (!text) continue;
|
||||
var rect = el.getBoundingClientRect();
|
||||
if (rect.width === 0 || rect.height === 0) continue;
|
||||
|
||||
95
packages/browseros-agent/apps/server/src/tools/console.ts
Normal file
95
packages/browseros-agent/apps/server/src/tools/console.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
import { CONTENT_LIMITS } from '@browseros/shared/constants/limits'
|
||||
import { z } from 'zod'
|
||||
import type { ConsoleLevel } from '../browser/console-collector'
|
||||
import { defineTool } from './framework'
|
||||
|
||||
const pageParam = z.number().describe('Page ID (from list_pages)')
|
||||
|
||||
export const get_console_logs = defineTool({
|
||||
name: 'get_console_logs',
|
||||
description:
|
||||
'Get browser console output (logs, warnings, errors, exceptions) for a page. Use to debug JavaScript errors, failed network requests, or unexpected page behavior.',
|
||||
input: z.object({
|
||||
page: pageParam,
|
||||
level: z
|
||||
.enum(['error', 'warning', 'info', 'debug'])
|
||||
.default('info')
|
||||
.describe(
|
||||
'Minimum severity level. "error" = errors only, "warning" = errors + warnings, "info" = errors + warnings + logs (default), "debug" = everything',
|
||||
),
|
||||
search: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe('Filter entries containing this text (case-insensitive)'),
|
||||
limit: z
|
||||
.number()
|
||||
.min(1)
|
||||
.max(CONTENT_LIMITS.CONSOLE_MAX_LIMIT)
|
||||
.optional()
|
||||
.describe(
|
||||
`Max entries to return (default ${CONTENT_LIMITS.CONSOLE_DEFAULT_LIMIT}, max ${CONTENT_LIMITS.CONSOLE_MAX_LIMIT}). Returns most recent entries.`,
|
||||
),
|
||||
clear: z
|
||||
.boolean()
|
||||
.default(false)
|
||||
.describe('Clear the console buffer after reading'),
|
||||
}),
|
||||
output: z.object({
|
||||
entries: z.array(
|
||||
z.object({
|
||||
source: z.enum(['console', 'exception', 'browser']),
|
||||
level: z.enum(['error', 'warning', 'info', 'debug']),
|
||||
text: z.string(),
|
||||
url: z.string().optional(),
|
||||
lineNumber: z.number().optional(),
|
||||
timestamp: z.number(),
|
||||
}),
|
||||
),
|
||||
totalCount: z.number(),
|
||||
returnedCount: z.number(),
|
||||
}),
|
||||
handler: async (args, ctx, response) => {
|
||||
const result = await ctx.browser.getConsoleLogs(args.page, {
|
||||
level: args.level as ConsoleLevel,
|
||||
search: args.search,
|
||||
limit: args.limit,
|
||||
clear: args.clear,
|
||||
})
|
||||
|
||||
// Empty results
|
||||
if (result.entries.length === 0) {
|
||||
response.text(
|
||||
result.totalCount === 0
|
||||
? `No console output for page ${args.page}.`
|
||||
: `No entries match the filter (${result.totalCount} total entries in buffer).`,
|
||||
)
|
||||
response.data({
|
||||
entries: [],
|
||||
totalCount: result.totalCount,
|
||||
returnedCount: 0,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Format each entry as [level] text — url:line
|
||||
const lines = result.entries.map((e) => {
|
||||
const location = e.url
|
||||
? ` — ${e.url}${e.lineNumber !== undefined ? `:${e.lineNumber}` : ''}`
|
||||
: ''
|
||||
return `[${e.level}] ${e.text}${location}`
|
||||
})
|
||||
|
||||
// Build header with count info
|
||||
const header =
|
||||
result.entries.length < result.totalCount
|
||||
? `Console logs for page ${args.page} (showing ${result.entries.length} of ${result.totalCount}, level ≥ ${args.level}):`
|
||||
: `Console logs for page ${args.page} (${result.entries.length} entries, level ≥ ${args.level}):`
|
||||
|
||||
response.text(`${header}\n\n${lines.join('\n')}`)
|
||||
response.data({
|
||||
entries: result.entries,
|
||||
totalCount: result.totalCount,
|
||||
returnedCount: result.entries.length,
|
||||
})
|
||||
},
|
||||
})
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
update_bookmark,
|
||||
} from './bookmarks'
|
||||
import { browseros_info } from './browseros-info'
|
||||
import { get_console_logs } from './console'
|
||||
import { get_dom, search_dom } from './dom'
|
||||
import {
|
||||
delete_history_range,
|
||||
@@ -83,7 +84,7 @@ export const registry = createRegistry([
|
||||
close_page,
|
||||
// wait_for, // temporarily disabled
|
||||
|
||||
// Observation (8)
|
||||
// Observation (9)
|
||||
take_snapshot,
|
||||
take_enhanced_snapshot,
|
||||
get_page_content,
|
||||
@@ -92,6 +93,7 @@ export const registry = createRegistry([
|
||||
search_dom,
|
||||
take_screenshot,
|
||||
evaluate_script,
|
||||
get_console_logs,
|
||||
|
||||
// Input (17)
|
||||
click,
|
||||
|
||||
@@ -167,7 +167,7 @@
|
||||
},
|
||||
"apps/server": {
|
||||
"name": "@browseros/server",
|
||||
"version": "0.0.74",
|
||||
"version": "0.0.75",
|
||||
"bin": {
|
||||
"browseros-server": "./src/index.ts",
|
||||
},
|
||||
|
||||
@@ -82,4 +82,7 @@ export const CONTENT_LIMITS = {
|
||||
BODY_CONTEXT_SIZE: 10_000,
|
||||
MAX_QUEUE_SIZE: 1_000,
|
||||
CONSOLE_META_CHAR: 1_000,
|
||||
CONSOLE_BUFFER_MAX_ENTRIES: 500,
|
||||
CONSOLE_DEFAULT_LIMIT: 50,
|
||||
CONSOLE_MAX_LIMIT: 200,
|
||||
} as const
|
||||
|
||||
1060
packages/browseros-agent/scripts/dev/inspect-ui.ts
Normal file
1060
packages/browseros-agent/scripts/dev/inspect-ui.ts
Normal file
File diff suppressed because it is too large
Load Diff
@@ -127,6 +127,14 @@
|
||||
"created_at": "2026-03-06T16:18:27Z",
|
||||
"repoId": 985839104,
|
||||
"pullRequestNo": 400
|
||||
},
|
||||
{
|
||||
"name": "shivammittal274",
|
||||
"id": 56757235,
|
||||
"comment_id": 4062893235,
|
||||
"created_at": "2026-03-15T12:26:54Z",
|
||||
"repoId": 985839104,
|
||||
"pullRequestNo": 448
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user