Compare commits

...

17 Commits

Author SHA1 Message Date
shivammittal274
3cd7fa2c06 ci(cli): change release workflow to manual dispatch from main
- Trigger via Actions UI with a version input (e.g. "0.1.0")
- Only runs on main branch
- Creates git tag cli/v<version> automatically
- Then GoReleaser builds all 6 binaries and creates the GitHub Release
2026-03-27 00:29:23 +05:30
shivammittal274
ac900b6b07 fix(cli): platform-native detection, launch, and install for all OSes
Detection (isBrowserOSInstalled):
- macOS: uses `open -Ra` to query Launch Services (no hardcoded paths)
- Linux: checks /usr/bin/browseros (.deb), browseros.desktop, AppImage search
- Windows: checks %LOCALAPPDATA%\BrowserOS\Application\BrowserOS.exe
  and HKCU/HKLM uninstall registry keys

Launch (startBrowserOS):
- macOS: `open -b com.browseros.BrowserOS` (bundle ID, not path)
- Linux: `browseros` binary, AppImage, or `gtk-launch browseros`
  (fixed: was using xdg-open which opens by MIME type, not desktop files)
- Windows: runs BrowserOS.exe from known Chromium per-user install path
  (fixed: was using `cmd /c start BrowserOS` which doesn't resolve)

Install (runPostInstall):
- macOS: hdiutil attach → cp -R to /Applications → hdiutil detach
- Linux: chmod +x for AppImage, dpkg -i instruction for .deb
- Windows: launches installer exe
- --deb flag now errors on non-Linux platforms

Removed auto-launch from newClient() — CLI never does surprising things.

Sources verified from:
- packages/browseros/build/common/context.py (binary names per platform)
- packages/browseros/build/modules/package/linux.py (.deb structure, .desktop file)
- packages/browseros/chromium_patches/chrome/install_static/chromium_install_modes.h
  (Windows base_app_name="BrowserOS", registry GUID, install paths)
- /Applications/BrowserOS.app/Contents/Info.plist (bundle ID)
2026-03-26 02:00:01 +05:30
shivammittal274
cb54d3aa7a refactor(cli): make launch an explicit command, remove auto-launch from newClient
- launch: new explicit command to find and open BrowserOS app
- launch: probes server.json, config, and common ports before launching
- launch: if already running, reports URL instead of launching again
- init --auto: uses port probing to find running servers
- install --deb: errors on non-Linux instead of silently downloading DMG
- error messages: guide users to launch/install/init explicitly
- removed: auto-launch from newClient() — CLI never does something surprising
2026-03-26 01:36:27 +05:30
shivammittal274
46ce8755c3 fix(cli): check health status code and add progress dots during launch
- Health check in newClient() now verifies HTTP 200, not just no error
- waitForServer prints dots during the 30s poll so users know it's working
2026-03-26 01:19:13 +05:30
shivammittal274
d80167d806 feat(cli): production-ready CLI with auto-launch, install, and cross-platform builds
- init: accept URL argument and --auto flag for non-interactive setup
- install: new command to download BrowserOS app for current platform
- launch: auto-detect and launch BrowserOS when server is not running
- discovery: prefer server.json (live) over config.yaml (may be stale)
- errors: actionable messages guiding users to init/install
- goreleaser: cross-platform builds for 6 targets (darwin/linux/windows × amd64/arm64)
- ci: GitHub Actions workflow to release CLI binaries on cli/v* tag push
2026-03-26 00:55:41 +05:30
Dani Akash
7f20319272 docs: add OAuth provider setup guides for ChatGPT Pro, GitHub Copilot, and Qwen Code (#545)
* docs: add setup guides for ChatGPT Pro, GitHub Copilot, and Qwen Code

Add individual OAuth setup guide pages with step-by-step screenshots
for each provider. Add "Use Your Existing Subscription" section to the
Bring Your Own LLM page with card links to each guide. Register pages
in docs navigation.

* docs: add ChatGPT Pro setup screenshots

* docs: use custom provider icons for OAuth setup cards

* docs: inline SVG icons in provider cards for dark mode support

* docs: place provider icons above card titles
2026-03-24 18:29:20 +05:30
shivammittal274
c8204efab6 feat: improve rate limit UX, usage page, and provider selector (#544)
* feat: improve rate limit UX, usage page, and provider selector

- Show "Add your own provider for unlimited usage" CTA when BrowserOS
  credits are exhausted or daily limit is reached
- Fix credit exhaustion detection to match actual error message
- Improve Usage page: remove disabled Add Credits button, add "Coming
  soon" badge, add "Want unlimited usage?" section linking to providers
- Add "+ Add Provider" button at bottom of chat provider selector dropdown

* fix: use asChild pattern for Button+anchor in usage page

Replace nested <a><Button> (invalid HTML) with Button asChild
pattern per shadcn/ui convention.
2026-03-24 18:01:42 +05:30
shivammittal274
fb5143b563 feat: UI improvements for OAuth dialog, provider badges, and events docs (#543)
* feat: UI improvements for OAuth dialog, provider badges, and events docs

- Replace OAuth device code toast with a proper Dialog showing the code
  prominently with a copy button (GitHub Copilot, Qwen Code, ChatGPT Pro)
- Add "New" badge on provider template cards for ChatGPT Plus/Pro,
  GitHub Copilot, and Qwen Code with orange border highlight
- Add events.md documenting all analytics events across the platform

* fix: add verificationUri to DeviceCodeDialog for popup-blocked fallback

Add verificationUri to PendingDeviceCode interface and pass it from
both handleClientAuth and handleServerAuth. Render a fallback "Open
verification page" link in DeviceCodeDialog so users can navigate
to the auth page if the popup was blocked.
2026-03-24 17:27:27 +05:30
Dani Akash
fe257cd8d1 feat: only parse browseros provider errors (#542) 2026-03-24 14:43:05 +05:30
shivammittal274
890d3406dd feat: promote BrowserOS as MCP with UI improvements (#541)
- Add MCP promo banner on AI providers page with "New" badge and
  "66+ tools" highlight, linking to /settings/mcp
- Add Quick Setup section on MCP settings page with copy-paste
  commands for Claude Code, Gemini CLI, Codex, Claude Desktop, OpenClaw
- Consolidate MCP settings: move restart button inline with server URL,
  remove separate MCP Server Settings card
- Add analytics event for promo banner clicks
2026-03-24 03:08:08 +05:30
shivammittal274
c316e09c11 feat: add source tag to tool_executed PostHog events (#538)
Add `source: 'mcp' | 'chat'` property to all `tool_executed` metrics
events so we can distinguish tool calls from external MCP clients
(Claude Code, Cursor) vs the built-in BrowserOS agent in PostHog.

- register-mcp.ts: source='mcp' (browser tools via MCP endpoint)
- register-klavis-mcp.ts: source='mcp' (Klavis tools via MCP endpoint)
- tool-adapter.ts: source='chat' (browser tools via chat agent)
- ai-sdk-agent.ts: source='chat' (Klavis/external MCP tools via chat agent, previously untracked)
- filesystem/utils.ts: source='chat' (filesystem tools via chat agent)
2026-03-24 02:03:18 +05:30
shivammittal274
65547c60c0 fix(eval): clean up eval configs and add test-clado-api script (#540)
Consolidate 13 configs down to 7 with uniform settings:
- 3 weekly (CI): browseros-agent, browseros-oe-agent, browseros-oe-clado
- 4 test (local): test_gemini-computer-use, test_yutori-navigator, test_webvoyager, test_mind2web
- All configs: headless=false, captcha block, full browseros ports, restart_server_per_task

Deleted: debug-test, mind2web-test, tool-loop-test, orchestrator-executor-test,
orchestrator-executor-clado-test, fireworks-minimax-m2, webvoyager-test

Added: test-clado-api.ts script, browseros-oe-agent-weekly.json (OE with AI SDK executor)
2026-03-24 01:28:05 +05:30
shivammittal274
0babc05077 feat(eval): NopeCHA CAPTCHA solver integration (#537)
* feat(eval): show mean score instead of pass/fail in report and viewer

* feat(eval): integrate NopeCHA CAPTCHA solver into eval pipeline

Add CAPTCHA detection and waiting so screenshots capture post-solve state.
Run headed with xvfb on CI since headless breaks extension content scripts.

- Add CaptchaWaiter module (detect reCAPTCHA/hCaptcha/Turnstile, poll until solved)
- Add optional `captcha` config block to EvalConfigSchema
- Wait for CAPTCHA solve before screenshot in single-agent and orchestrator-executor
- Patch NopeCHA manifest with API key before launching workers
- Fix CAPTCHA_EXT_DIR path (was pointing one level too high)
- Remove --incognito (extensions don't run in incognito; fresh user-data-dir isolates)
- CI: install xvfb, run headed via xvfb-run, pass NOPECHA_API_KEY secret
2026-03-24 00:14:16 +05:30
Nikhil
1270b5b55c feat: new manifest perms (#536)
* feat: new manifest perms

* fix: minor

* fix: minor
2026-03-23 09:31:07 -07:00
Nikhil
e97d8bc1cb fix: remove daily rate-limit middleware (#535)
* fix: remove daily rate-limit middleware

The daily conversation rate limit is no longer needed. Remove the
middleware, RateLimiter class, fetch-config, error type, shared
constants, DB schema table, and integration tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove unused getDb() method

No longer needed after rate-limiter removal.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:31:20 -07:00
Dani Akash
5109ca4347 feat: added scope in server error logs (#533)
* feat: added scope in server error logs

* fix: prevent double capture on chat request
2026-03-23 20:47:28 +05:30
shivammittal274
f14942c6f9 feat(eval): show mean score instead of pass/fail in report and viewer (#534) 2026-03-23 20:28:34 +05:30
88 changed files with 2581 additions and 759 deletions

View File

@@ -43,6 +43,9 @@ jobs:
working-directory: packages/browseros-agent
run: bun install --ignore-scripts && bun run build:agent-sdk
- name: Install xvfb
run: sudo apt-get update && sudo apt-get install -y xvfb
- name: Install captcha solver extension
working-directory: packages/browseros-agent/apps/eval
run: |
@@ -55,11 +58,12 @@ jobs:
env:
FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
NOPECHA_API_KEY: ${{ secrets.NOPECHA_API_KEY }}
BROWSEROS_BINARY: /usr/bin/browseros
EVAL_CONFIG: ${{ github.event.inputs.config || 'configs/browseros-agent-weekly.json' }}
run: |
echo "Running eval with config: $EVAL_CONFIG"
bun run src/index.ts -c "$EVAL_CONFIG"
xvfb-run --auto-servernum --server-args="-screen 0 1440x900x24" bun run src/index.ts -c "$EVAL_CONFIG"
- name: Upload runs to R2
if: success()

50
.github/workflows/release-cli.yml vendored Normal file
View File

@@ -0,0 +1,50 @@
name: Release CLI
on:
workflow_dispatch:
inputs:
version:
description: "Release version (e.g. 0.1.0)"
required: true
type: string
permissions:
contents: write
jobs:
release:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
defaults:
run:
working-directory: packages/browseros-agent/apps/cli
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: packages/browseros-agent/apps/cli/go.mod
- name: Run tests
run: go test ./... -v
- name: Run vet
run: go vet ./...
- name: Create tag
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -a "cli/v${{ inputs.version }}" -m "browseros-cli v${{ inputs.version }}"
git push origin "cli/v${{ inputs.version }}"
- uses: goreleaser/goreleaser-action@v6
with:
version: "~> v2"
args: release --clean
workdir: packages/browseros-agent/apps/cli
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -23,6 +23,9 @@
"group": "Core Features",
"pages": [
"features/bring-your-own-llm",
"features/chatgpt-pro-oauth",
"features/github-copilot-oauth",
"features/qwen-code-oauth",
"features/local-models",
"features/workflows",
"features/scheduled-tasks",

View File

@@ -13,6 +13,33 @@ See how to connect your own LLM in under a minute:
src="https://pub-80f8a01e6e8b4239ae53a7652ef85877.r2.dev/resources/feature-videos/1-bring-your-own-LLM.mov"
></video>
## Use Your Existing Subscription
Already paying for ChatGPT Pro, GitHub Copilot, or Qwen Code? Connect your existing account to BrowserOS with a single sign-in — no API keys, no extra cost.
<CardGroup cols={3}>
<Card href="/features/chatgpt-pro-oauth">
<svg fill="currentColor" fillRule="evenodd" height="24" width="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M9.205 8.658v-2.26c0-.19.072-.333.238-.428l4.543-2.616c.619-.357 1.356-.523 2.117-.523 2.854 0 4.662 2.212 4.662 4.566 0 .167 0 .357-.024.547l-4.71-2.759a.797.797 0 00-.856 0l-5.97 3.473zm10.609 8.8V12.06c0-.333-.143-.57-.429-.737l-5.97-3.473 1.95-1.118a.433.433 0 01.476 0l4.543 2.617c1.309.76 2.189 2.378 2.189 3.948 0 1.808-1.07 3.473-2.76 4.163zM7.802 12.703l-1.95-1.142c-.167-.095-.239-.238-.239-.428V5.899c0-2.545 1.95-4.472 4.591-4.472 1 0 1.927.333 2.712.928L8.23 5.067c-.285.166-.428.404-.428.737v6.898zM12 15.128l-2.795-1.57v-3.33L12 8.658l2.795 1.57v3.33L12 15.128zm1.796 7.23c-1 0-1.927-.332-2.712-.927l4.686-2.712c.285-.166.428-.404.428-.737v-6.898l1.974 1.142c.167.095.238.238.238.428v5.233c0 2.545-1.974 4.472-4.614 4.472zm-5.637-5.303l-4.544-2.617c-1.308-.761-2.188-2.378-2.188-3.948A4.482 4.482 0 014.21 6.327v5.423c0 .333.143.571.428.738l5.947 3.449-1.95 1.118a.432.432 0 01-.476 0zm-.262 3.9c-2.688 0-4.662-2.021-4.662-4.519 0-.19.024-.38.047-.57l4.686 2.71c.286.167.571.167.856 0l5.97-3.448v2.26c0 .19-.07.333-.237.428l-4.543 2.616c-.619.357-1.356.523-2.117.523zm5.899 2.83a5.947 5.947 0 005.827-4.756C22.287 18.339 24 15.84 24 13.296c0-1.665-.713-3.282-1.998-4.448.119-.5.19-.999.19-1.498 0-3.401-2.759-5.947-5.946-5.947-.642 0-1.26.095-1.88.31A5.962 5.962 0 0010.205 0a5.947 5.947 0 00-5.827 4.757C1.713 5.447 0 7.945 0 10.49c0 1.666.713 3.283 1.998 4.448-.119.5-.19 1-.19 1.499 0 3.401 2.759 5.946 5.946 5.946.642 0 1.26-.095 1.88-.309a5.96 5.96 0 004.162 1.713z"></path></svg>
**ChatGPT Pro / Plus**
Sign in with your OpenAI account. Access GPT-5 Codex, GPT-5.4, and the full Codex lineup with up to 400K context.
</Card>
<Card href="/features/github-copilot-oauth">
<svg fill="currentColor" fillRule="evenodd" height="24" width="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M19.245 5.364c1.322 1.36 1.877 3.216 2.11 5.817.622 0 1.2.135 1.592.654l.73.964c.21.278.323.61.323.955v2.62c0 .339-.173.669-.453.868C20.239 19.602 16.157 21.5 12 21.5c-4.6 0-9.205-2.583-11.547-4.258-.28-.2-.452-.53-.453-.868v-2.62c0-.345.113-.679.321-.956l.73-.963c.392-.517.974-.654 1.593-.654l.029-.297c.25-2.446.81-4.213 2.082-5.52 2.461-2.54 5.71-2.851 7.146-2.864h.198c1.436.013 4.685.323 7.146 2.864zm-7.244 4.328c-.284 0-.613.016-.962.05-.123.447-.305.85-.57 1.108-1.05 1.023-2.316 1.18-2.994 1.18-.638 0-1.306-.13-1.851-.464-.516.165-1.012.403-1.044.996a65.882 65.882 0 00-.063 2.884l-.002.48c-.002.563-.005 1.126-.013 1.69.002.326.204.63.51.765 2.482 1.102 4.83 1.657 6.99 1.657 2.156 0 4.504-.555 6.985-1.657a.854.854 0 00.51-.766c.03-1.682.006-3.372-.076-5.053-.031-.596-.528-.83-1.046-.996-.546.333-1.212.464-1.85.464-.677 0-1.942-.157-2.993-1.18-.266-.258-.447-.661-.57-1.108-.32-.032-.64-.049-.96-.05zm-2.525 4.013c.539 0 .976.426.976.95v1.753c0 .525-.437.95-.976.95a.964.964 0 01-.976-.95v-1.752c0-.525.437-.951.976-.951zm5 0c.539 0 .976.426.976.95v1.753c0 .525-.437.95-.976.95a.964.964 0 01-.976-.95v-1.752c0-.525.437-.951.976-.951zM7.635 5.087c-1.05.102-1.935.438-2.385.906-.975 1.037-.765 3.668-.21 4.224.405.394 1.17.657 1.995.657h.09c.649-.013 1.785-.176 2.73-1.11.435-.41.705-1.433.675-2.47-.03-.834-.27-1.52-.63-1.813-.39-.336-1.275-.482-2.265-.394zm6.465.394c-.36.292-.6.98-.63 1.813-.03 1.037.24 2.06.675 2.47.968.957 2.136 1.104 2.776 1.11h.044c.825 0 1.59-.263 1.995-.657.555-.556.765-3.187-.21-4.224-.45-.468-1.335-.804-2.385-.906-.99-.088-1.875.058-2.265.394zM12 7.615c-.24 0-.525.015-.84.044.03.16.045.336.06.526l-.001.159a2.94 2.94 0 01-.014.25c.225-.022.425-.027.612-.028h.366c.187 0 .387.006.612.028-.015-.146-.015-.277-.015-.409.015-.19.03-.365.06-.526a9.29 9.29 0 00-.84-.044z"></path></svg>
**GitHub Copilot**
Sign in with your GitHub account. Access 19+ models including Claude, GPT-5, and Gemini through one subscription.
</Card>
<Card href="/features/qwen-code-oauth">
<svg fill="currentColor" fillRule="evenodd" height="24" width="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M12.604 1.34c.393.69.784 1.382 1.174 2.075a.18.18 0 00.157.091h5.552c.174 0 .322.11.446.327l1.454 2.57c.19.337.24.478.024.837-.26.43-.513.864-.76 1.3l-.367.658c-.106.196-.223.28-.04.512l2.652 4.637c.172.301.111.494-.043.77-.437.785-.882 1.564-1.335 2.34-.159.272-.352.375-.68.37-.777-.016-1.552-.01-2.327.016a.099.099 0 00-.081.05 575.097 575.097 0 01-2.705 4.74c-.169.293-.38.363-.725.364-.997.003-2.002.004-3.017.002a.537.537 0 01-.465-.271l-1.335-2.323a.09.09 0 00-.083-.049H4.982c-.285.03-.553-.001-.805-.092l-1.603-2.77a.543.543 0 01-.002-.54l1.207-2.12a.198.198 0 000-.197 550.951 550.951 0 01-1.875-3.272l-.79-1.395c-.16-.31-.173-.496.095-.965.465-.813.927-1.625 1.387-2.436.132-.234.304-.334.584-.335a338.3 338.3 0 012.589-.001.124.124 0 00.107-.063l2.806-4.895a.488.488 0 01.422-.246c.524-.001 1.053 0 1.583-.006L11.704 1c.341-.003.724.032.9.34zm-3.432.403a.06.06 0 00-.052.03L6.254 6.788a.157.157 0 01-.135.078H3.253c-.056 0-.07.025-.041.074l5.81 10.156c.025.042.013.062-.034.063l-2.795.015a.218.218 0 00-.2.116l-1.32 2.31c-.044.078-.021.118.068.118l5.716.008c.046 0 .08.02.104.061l1.403 2.454c.046.081.092.082.139 0l5.006-8.76.783-1.382a.055.055 0 01.096 0l1.424 2.53a.122.122 0 00.107.062l2.763-.02a.04.04 0 00.035-.02.041.041 0 000-.04l-2.9-5.086a.108.108 0 010-.113l.293-.507 1.12-1.977c.024-.041.012-.062-.035-.062H9.2c-.059 0-.073-.026-.043-.077l1.434-2.505a.107.107 0 000-.114L9.225 1.774a.06.06 0 00-.053-.031zm6.29 8.02c.046 0 .058.02.034.06l-.832 1.465-2.613 4.585a.056.056 0 01-.05.029.058.058 0 01-.05-.029L8.498 9.841c-.02-.034-.01-.052.028-.054l.216-.012 6.722-.012z"></path></svg>
**Qwen Code**
Sign in with your Qwen account. Access Qwen 3 Coder with a 1 million token context window.
</Card>
</CardGroup>
---
## Which Model Should I Use?
| Mode | What works | Recommendation |

View File

@@ -0,0 +1,56 @@
---
title: "ChatGPT Pro / Plus"
description: "Use your ChatGPT subscription to power BrowserOS"
---
Connect your ChatGPT Pro or Plus subscription to BrowserOS and access GPT-5 Codex, GPT-5.4, and the full lineup of OpenAI's most advanced models — with up to 400K context. No API keys needed.
## Setup
**1.** Open BrowserOS and go to **Settings** (`chrome://browseros/settings`). You'll see the AI Providers section.
![AI Settings screen](/images/setting-up-chatgpt/llm-screen.png)
**2.** Click **USE** on the **ChatGPT Plus/Pro** card. You'll be prompted to sign in with your OpenAI account.
![Login screen](/images/setting-up-chatgpt/login-screen.png)
**3.** Sign in with the OpenAI account that has your ChatGPT Pro or Plus subscription active, and accept the authorization.
![Accept authorization](/images/setting-up-chatgpt/accept-screen.png)
**4.** Once authorized, ChatGPT will appear as a provider in your settings. Select a model and start using it.
## Available Models
| Model | Context Window |
|-------|---------------|
| `gpt-5.4` | 400K |
| `gpt-5.3-codex` | 400K |
| `gpt-5.2-codex` | 400K |
| `gpt-5.2` | 200K |
| `gpt-5.1-codex` | 400K |
| `gpt-5.1-codex-max` | 400K |
| `gpt-5.1-codex-mini` | 400K |
| `gpt-5.1` | 200K |
<Info>
ChatGPT Pro subscribers have access to the full model lineup. ChatGPT Plus subscribers can access a subset of models depending on their plan. The available models will be shown automatically after you connect.
</Info>
<Tip>
The Codex models (e.g., `gpt-5.3-codex`) are optimized for code and reasoning tasks — ideal for complex browser automation workflows that involve form filling, data extraction, and multi-step navigation.
</Tip>
## Reasoning Settings
ChatGPT Pro includes additional settings for models that support reasoning:
- **Reasoning Effort** — Control how much the model "thinks" before responding. Options: none, low, medium, high.
- **Reasoning Summary** — Choose how reasoning is displayed. Options: auto, concise, detailed.
These settings are available in the provider configuration after connecting.
## Disconnecting
To disconnect your OpenAI account, go to **Settings**, find the ChatGPT Plus/Pro provider, and click **Disconnect**. Your OAuth tokens will be immediately deleted from your machine.

View File

@@ -0,0 +1,60 @@
---
title: "GitHub Copilot"
description: "Use your GitHub Copilot subscription to power BrowserOS"
---
Connect your GitHub Copilot subscription to BrowserOS and access 19+ models — including Claude, GPT-5, and Gemini — through a single GitHub sign-in. No API keys needed.
<Info>
**Free tier** includes GPT-5 Mini, Claude Haiku 4.5, GPT-4o, and GPT-4.1. **Copilot Pro** ($10/month) unlocks Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Pro, GPT-5.4, and more.
</Info>
## Setup
**1.** Open BrowserOS and go to **Settings** (`chrome://browseros/settings`). You'll see the AI Providers section.
![AI Settings screen](/images/setting-up-copilot/llm-screen.png)
**2.** Click **USE** on the **GitHub Copilot** card. A device code will appear — copy it, then click the link to open GitHub's device authorization page.
![Device code displayed](/images/setting-up-copilot/device-code.png)
**3.** Select your GitHub account to authorize.
![Select GitHub account](/images/setting-up-copilot/select-account.png)
**4.** Paste the device code and authorize BrowserOS to access your Copilot subscription.
![Authorize device](/images/setting-up-copilot/authorize-device.png)
**5.** Once authorized, GitHub Copilot will appear as a provider in your settings. Select a model and start using it.
## Available Models
### Free Tier
| Model | Context Window |
|-------|---------------|
| `gpt-5-mini` | 128K |
| `claude-haiku-4.5` | 128K |
| `gpt-4o` | 64K |
| `gpt-4.1` | 64K |
### Copilot Pro / Pro+
| Model | Context Window |
|-------|---------------|
| `claude-sonnet-4.6` | 200K |
| `claude-opus-4.6` | 200K |
| `gemini-2.5-pro` | 1M |
| `gemini-3-pro-preview` | 1M |
| `gpt-5.4` | 400K |
| `gpt-5.3-codex` | 400K |
| `gpt-5.2-codex` | 400K |
| `grok-code-fast-1` | 128K |
<Tip>
GitHub Copilot is the most versatile provider — one subscription gives you access to models from OpenAI, Anthropic, Google, and xAI. Great if you want to switch between models for different tasks.
</Tip>
## Disconnecting
To disconnect your GitHub account, go to **Settings**, find the GitHub Copilot provider, and click **Disconnect**. Your OAuth tokens will be immediately deleted from your machine.

View File

@@ -0,0 +1,39 @@
---
title: "Qwen Code"
description: "Use your Qwen Code account to power BrowserOS"
---
Connect your Qwen Code account to BrowserOS and access Alibaba's coding models with up to a **1 million token context window** — the largest of any provider we support. No API keys needed.
## Setup
**1.** Open BrowserOS and go to **Settings** (`chrome://browseros/settings`). You'll see the AI Providers section.
![AI Settings screen](/images/setting-up-qwen/llm-screen.png)
**2.** Click **USE** on the **Qwen Code** card. You'll be prompted to sign in with your Qwen account.
![Select Qwen Code](/images/setting-up-qwen/select-qwen.png)
**3.** Sign in with your Alibaba Cloud / Qwen account to authorize BrowserOS.
![Qwen sign in](/images/setting-up-qwen/qwen-signin.png)
**4.** Once authorized, Qwen Code will appear as a provider in your settings. Select a model and start using it.
## Available Models
| Model | Context Window |
|-------|---------------|
| `coder-model` | 1M |
| `qwen3-coder-plus` | 1M |
| `qwen3-coder-flash` | 1M |
| `qwen3.5-plus` | 1M |
<Tip>
Qwen Code's 1 million token context window is ideal for tasks that involve long documents, entire documentation sites, or working across many browser tabs simultaneously — the agent can hold everything in context at once.
</Tip>
## Disconnecting
To disconnect your Qwen account, go to **Settings**, find the Qwen Code provider, and click **Disconnect**. Your OAuth tokens will be immediately deleted from your machine.

View File

@@ -0,0 +1 @@
<svg fill="currentColor" fill-rule="evenodd" height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>GithubCopilot</title><path d="M19.245 5.364c1.322 1.36 1.877 3.216 2.11 5.817.622 0 1.2.135 1.592.654l.73.964c.21.278.323.61.323.955v2.62c0 .339-.173.669-.453.868C20.239 19.602 16.157 21.5 12 21.5c-4.6 0-9.205-2.583-11.547-4.258-.28-.2-.452-.53-.453-.868v-2.62c0-.345.113-.679.321-.956l.73-.963c.392-.517.974-.654 1.593-.654l.029-.297c.25-2.446.81-4.213 2.082-5.52 2.461-2.54 5.71-2.851 7.146-2.864h.198c1.436.013 4.685.323 7.146 2.864zm-7.244 4.328c-.284 0-.613.016-.962.05-.123.447-.305.85-.57 1.108-1.05 1.023-2.316 1.18-2.994 1.18-.638 0-1.306-.13-1.851-.464-.516.165-1.012.403-1.044.996a65.882 65.882 0 00-.063 2.884l-.002.48c-.002.563-.005 1.126-.013 1.69.002.326.204.63.51.765 2.482 1.102 4.83 1.657 6.99 1.657 2.156 0 4.504-.555 6.985-1.657a.854.854 0 00.51-.766c.03-1.682.006-3.372-.076-5.053-.031-.596-.528-.83-1.046-.996-.546.333-1.212.464-1.85.464-.677 0-1.942-.157-2.993-1.18-.266-.258-.447-.661-.57-1.108-.32-.032-.64-.049-.96-.05zm-2.525 4.013c.539 0 .976.426.976.95v1.753c0 .525-.437.95-.976.95a.964.964 0 01-.976-.95v-1.752c0-.525.437-.951.976-.951zm5 0c.539 0 .976.426.976.95v1.753c0 .525-.437.95-.976.95a.964.964 0 01-.976-.95v-1.752c0-.525.437-.951.976-.951zM7.635 5.087c-1.05.102-1.935.438-2.385.906-.975 1.037-.765 3.668-.21 4.224.405.394 1.17.657 1.995.657h.09c.649-.013 1.785-.176 2.73-1.11.435-.41.705-1.433.675-2.47-.03-.834-.27-1.52-.63-1.813-.39-.336-1.275-.482-2.265-.394zm6.465.394c-.36.292-.6.98-.63 1.813-.03 1.037.24 2.06.675 2.47.968.957 2.136 1.104 2.776 1.11h.044c.825 0 1.59-.263 1.995-.657.555-.556.765-3.187-.21-4.224-.45-.468-1.335-.804-2.385-.906-.99-.088-1.875.058-2.265.394zM12 7.615c-.24 0-.525.015-.84.044.03.16.045.336.06.526l-.001.159a2.94 2.94 0 01-.014.25c.225-.022.425-.027.612-.028h.366c.187 0 .387.006.612.028-.015-.146-.015-.277-.015-.409.015-.19.03-.365.06-.526a9.29 9.29 0 00-.84-.044z"></path></svg>

After

Width:  |  Height:  |  Size: 2.0 KiB

View File

@@ -0,0 +1 @@
<svg fill="currentColor" fill-rule="evenodd" height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>OpenAI</title><path d="M9.205 8.658v-2.26c0-.19.072-.333.238-.428l4.543-2.616c.619-.357 1.356-.523 2.117-.523 2.854 0 4.662 2.212 4.662 4.566 0 .167 0 .357-.024.547l-4.71-2.759a.797.797 0 00-.856 0l-5.97 3.473zm10.609 8.8V12.06c0-.333-.143-.57-.429-.737l-5.97-3.473 1.95-1.118a.433.433 0 01.476 0l4.543 2.617c1.309.76 2.189 2.378 2.189 3.948 0 1.808-1.07 3.473-2.76 4.163zM7.802 12.703l-1.95-1.142c-.167-.095-.239-.238-.239-.428V5.899c0-2.545 1.95-4.472 4.591-4.472 1 0 1.927.333 2.712.928L8.23 5.067c-.285.166-.428.404-.428.737v6.898zM12 15.128l-2.795-1.57v-3.33L12 8.658l2.795 1.57v3.33L12 15.128zm1.796 7.23c-1 0-1.927-.332-2.712-.927l4.686-2.712c.285-.166.428-.404.428-.737v-6.898l1.974 1.142c.167.095.238.238.238.428v5.233c0 2.545-1.974 4.472-4.614 4.472zm-5.637-5.303l-4.544-2.617c-1.308-.761-2.188-2.378-2.188-3.948A4.482 4.482 0 014.21 6.327v5.423c0 .333.143.571.428.738l5.947 3.449-1.95 1.118a.432.432 0 01-.476 0zm-.262 3.9c-2.688 0-4.662-2.021-4.662-4.519 0-.19.024-.38.047-.57l4.686 2.71c.286.167.571.167.856 0l5.97-3.448v2.26c0 .19-.07.333-.237.428l-4.543 2.616c-.619.357-1.356.523-2.117.523zm5.899 2.83a5.947 5.947 0 005.827-4.756C22.287 18.339 24 15.84 24 13.296c0-1.665-.713-3.282-1.998-4.448.119-.5.19-.999.19-1.498 0-3.401-2.759-5.947-5.946-5.947-.642 0-1.26.095-1.88.31A5.962 5.962 0 0010.205 0a5.947 5.947 0 00-5.827 4.757C1.713 5.447 0 7.945 0 10.49c0 1.666.713 3.283 1.998 4.448-.119.5-.19 1-.19 1.499 0 3.401 2.759 5.946 5.946 5.946.642 0 1.26-.095 1.88-.309a5.96 5.96 0 004.162 1.713z"></path></svg>

After

Width:  |  Height:  |  Size: 1.6 KiB

View File

@@ -0,0 +1 @@
<svg fill="currentColor" fill-rule="evenodd" height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>Qwen</title><path d="M12.604 1.34c.393.69.784 1.382 1.174 2.075a.18.18 0 00.157.091h5.552c.174 0 .322.11.446.327l1.454 2.57c.19.337.24.478.024.837-.26.43-.513.864-.76 1.3l-.367.658c-.106.196-.223.28-.04.512l2.652 4.637c.172.301.111.494-.043.77-.437.785-.882 1.564-1.335 2.34-.159.272-.352.375-.68.37-.777-.016-1.552-.01-2.327.016a.099.099 0 00-.081.05 575.097 575.097 0 01-2.705 4.74c-.169.293-.38.363-.725.364-.997.003-2.002.004-3.017.002a.537.537 0 01-.465-.271l-1.335-2.323a.09.09 0 00-.083-.049H4.982c-.285.03-.553-.001-.805-.092l-1.603-2.77a.543.543 0 01-.002-.54l1.207-2.12a.198.198 0 000-.197 550.951 550.951 0 01-1.875-3.272l-.79-1.395c-.16-.31-.173-.496.095-.965.465-.813.927-1.625 1.387-2.436.132-.234.304-.334.584-.335a338.3 338.3 0 012.589-.001.124.124 0 00.107-.063l2.806-4.895a.488.488 0 01.422-.246c.524-.001 1.053 0 1.583-.006L11.704 1c.341-.003.724.032.9.34zm-3.432.403a.06.06 0 00-.052.03L6.254 6.788a.157.157 0 01-.135.078H3.253c-.056 0-.07.025-.041.074l5.81 10.156c.025.042.013.062-.034.063l-2.795.015a.218.218 0 00-.2.116l-1.32 2.31c-.044.078-.021.118.068.118l5.716.008c.046 0 .08.02.104.061l1.403 2.454c.046.081.092.082.139 0l5.006-8.76.783-1.382a.055.055 0 01.096 0l1.424 2.53a.122.122 0 00.107.062l2.763-.02a.04.04 0 00.035-.02.041.041 0 000-.04l-2.9-5.086a.108.108 0 010-.113l.293-.507 1.12-1.977c.024-.041.012-.062-.035-.062H9.2c-.059 0-.073-.026-.043-.077l1.434-2.505a.107.107 0 000-.114L9.225 1.774a.06.06 0 00-.053-.031zm6.29 8.02c.046 0 .058.02.034.06l-.832 1.465-2.613 4.585a.056.056 0 01-.05.029.058.058 0 01-.05-.029L8.498 9.841c-.02-.034-.01-.052.028-.054l.216-.012 6.722-.012z"></path></svg>

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 717 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 815 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 637 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 687 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 825 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 634 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 837 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 712 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 843 KiB

View File

@@ -1,4 +1,4 @@
import { Check } from 'lucide-react'
import { Check, Plus } from 'lucide-react'
import type { FC, PropsWithChildren } from 'react'
import { useState } from 'react'
import {
@@ -77,6 +77,19 @@ export const ChatProviderSelector: FC<
)
})}
</CommandGroup>
<div className="border-border border-t p-1">
<button
type="button"
className="flex w-full items-center gap-3 rounded-md p-2 text-muted-foreground text-sm transition-colors hover:bg-accent hover:text-foreground"
onClick={() => {
window.open('/app.html#/settings/ai', '_blank')
setOpen(false)
}}
>
<Plus className="h-4 w-4" />
Add Provider
</button>
</div>
</CommandList>
</Command>
</PopoverContent>

View File

@@ -38,6 +38,7 @@ import {
} from '@/lib/llm-providers/useOAuthProviderFlow'
import { track } from '@/lib/metrics/track'
import { ConfiguredProvidersList } from './ConfiguredProvidersList'
import { DeviceCodeDialog } from './DeviceCodeDialog'
import {
DeleteRemoteLlmProviderDocument,
GetRemoteLlmProvidersDocument,
@@ -45,6 +46,7 @@ import {
import type { IncompleteProvider } from './IncompleteProviderCard'
import { IncompleteProvidersList } from './IncompleteProvidersList'
import { LlmProvidersHeader } from './LlmProvidersHeader'
import { McpPromoBanner } from './McpPromoBanner'
import { NewProviderDialog } from './NewProviderDialog'
import { ProviderTemplatesSection } from './ProviderTemplatesSection'
@@ -173,6 +175,16 @@ export const AISettingsPage: FC = () => {
saveProvider,
)
const activeDeviceCode =
chatgptPro.pendingDeviceCode ??
copilot.pendingDeviceCode ??
qwenCode.pendingDeviceCode
const clearActiveDeviceCode = () => {
chatgptPro.clearDeviceCode()
copilot.clearDeviceCode()
qwenCode.clearDeviceCode()
}
const oauthFlows: Record<
string,
{
@@ -347,6 +359,8 @@ export const AISettingsPage: FC = () => {
onAddProvider={handleAddProvider}
/>
<McpPromoBanner />
<ProviderTemplatesSection onUseTemplate={handleUseTemplate} />
<ConfiguredProvidersList
@@ -421,6 +435,11 @@ export const AISettingsPage: FC = () => {
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
<DeviceCodeDialog
deviceCode={activeDeviceCode}
onClose={clearActiveDeviceCode}
/>
</div>
)
}

View File

@@ -0,0 +1,81 @@
import { Check, Copy, ExternalLink } from 'lucide-react'
import { type FC, useState } from 'react'
import { Button } from '@/components/ui/button'
import {
Dialog,
DialogContent,
DialogDescription,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog'
import type { PendingDeviceCode } from '@/lib/llm-providers/useOAuthProviderFlow'
interface DeviceCodeDialogProps {
deviceCode: PendingDeviceCode | null
onClose: () => void
}
export const DeviceCodeDialog: FC<DeviceCodeDialogProps> = ({
deviceCode,
onClose,
}) => {
const [copied, setCopied] = useState(false)
const handleCopy = async () => {
if (!deviceCode) return
try {
await navigator.clipboard.writeText(deviceCode.userCode)
setCopied(true)
setTimeout(() => setCopied(false), 2000)
} catch {
// Clipboard API failed
}
}
return (
<Dialog open={!!deviceCode} onOpenChange={(open) => !open && onClose()}>
<DialogContent className="sm:max-w-md">
<DialogHeader>
<DialogTitle>Connect to {deviceCode?.providerName}</DialogTitle>
<DialogDescription>
Paste this code on the {deviceCode?.providerName} page that just
opened in your browser.
</DialogDescription>
</DialogHeader>
<div className="flex flex-col items-center gap-4 py-4">
<div className="flex items-center gap-3 rounded-xl border-2 border-[var(--accent-orange)]/40 border-dashed bg-[var(--accent-orange)]/5 px-6 py-4">
<code className="font-bold font-mono text-2xl text-foreground tracking-widest">
{deviceCode?.userCode}
</code>
<Button
variant="ghost"
size="icon"
onClick={handleCopy}
className="shrink-0"
>
{copied ? (
<Check className="h-4 w-4 text-green-600" />
) : (
<Copy className="h-4 w-4" />
)}
</Button>
</div>
<p className="text-center text-muted-foreground text-xs">
This dialog will close automatically once authentication completes.
</p>
{deviceCode?.verificationUri && (
<a
href={deviceCode.verificationUri}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-1 text-[var(--accent-orange)] text-xs transition-colors hover:underline"
>
Open verification page
<ExternalLink className="h-3 w-3" />
</a>
)}
</div>
</DialogContent>
</Dialog>
)
}

View File

@@ -0,0 +1,57 @@
import { ArrowRight, Server, X } from 'lucide-react'
import { type FC, useState } from 'react'
import { useNavigate } from 'react-router'
import { Button } from '@/components/ui/button'
import { MCP_PROMO_BANNER_CLICKED_EVENT } from '@/lib/constants/analyticsEvents'
import { track } from '@/lib/metrics/track'
export const McpPromoBanner: FC = () => {
const [dismissed, setDismissed] = useState(false)
const navigate = useNavigate()
if (dismissed) return null
const handleClick = () => {
track(MCP_PROMO_BANNER_CLICKED_EVENT)
navigate('/settings/mcp')
}
return (
<div className="relative flex items-center gap-4 rounded-xl border border-border bg-card p-4 shadow-sm transition-all hover:shadow-md">
<div className="flex h-10 w-10 shrink-0 items-center justify-center rounded-lg bg-[var(--accent-orange)]/10">
<Server className="h-5 w-5 text-[var(--accent-orange)]" />
</div>
<div className="min-w-0 flex-1">
<p className="flex items-center gap-2 font-semibold text-sm">
Use BrowserOS with Claude Code, Cursor & more
<span className="text-[var(--accent-orange)] text-xs">
(66+ tools)
</span>
<span className="inline-flex items-center gap-1 rounded-full bg-[var(--accent-orange)]/10 px-2.5 py-1 font-semibold text-[var(--accent-orange)] text-xs">
<span className="h-1.5 w-1.5 rounded-full bg-[var(--accent-orange)]" />
New
</span>
</p>
<p className="text-muted-foreground text-xs">
Connect your favorite coding tools to BrowserOS as an MCP server
</p>
</div>
<Button
variant="outline"
size="sm"
onClick={handleClick}
className="shrink-0 border-[var(--accent-orange)] bg-[var(--accent-orange)]/10 text-[var(--accent-orange)] hover:bg-[var(--accent-orange)]/20 hover:text-[var(--accent-orange)]"
>
Set up
<ArrowRight className="ml-1 h-3 w-3" />
</Button>
<button
type="button"
onClick={() => setDismissed(true)}
className="absolute top-2 right-2 rounded-sm p-1 text-muted-foreground opacity-50 transition-opacity hover:opacity-100"
>
<X className="h-3.5 w-3.5" />
</button>
</div>
)
}

View File

@@ -7,12 +7,14 @@ import { cn } from '@/lib/utils'
interface ProviderTemplateCardProps {
template: ProviderTemplate
highlighted?: boolean
isNew?: boolean
onUseTemplate: (template: ProviderTemplate) => void
}
export const ProviderTemplateCard: FC<ProviderTemplateCardProps> = ({
template,
highlighted = false,
isNew = false,
onUseTemplate,
}) => {
return (
@@ -20,12 +22,19 @@ export const ProviderTemplateCard: FC<ProviderTemplateCardProps> = ({
type="button"
onClick={() => onUseTemplate(template)}
className={cn(
'group flex w-full items-center gap-3 rounded-lg border bg-background p-4 text-left transition-all hover:border-[var(--accent-orange)] hover:shadow-md',
'group relative flex w-full items-center gap-3 rounded-lg border bg-background p-4 text-left transition-all hover:border-[var(--accent-orange)] hover:shadow-md',
highlighted
? 'border-orange-300/80 bg-orange-50/30 shadow-sm ring-1 ring-orange-300/45 dark:bg-orange-500/5'
: 'border-border',
: isNew
? 'border-2 border-[var(--accent-orange)]/50'
: 'border-border',
)}
>
{isNew && (
<span className="absolute -top-2 left-3 rounded-full bg-[var(--accent-orange)] px-2 py-0.5 font-semibold text-[9px] text-white uppercase tracking-wider">
New
</span>
)}
<div className="flex min-w-0 flex-1 items-center gap-3">
<ProviderIcon
type={template.id}

View File

@@ -58,14 +58,21 @@ export const ProviderTemplatesSection: FC<ProviderTemplatesSectionProps> = ({
<CollapsibleContent>
<div className="mt-4 grid gap-3 sm:grid-cols-2 lg:grid-cols-3">
{filteredTemplates.map((template) => (
<ProviderTemplateCard
key={template.id}
template={template}
highlighted={template.id === 'moonshot'}
onUseTemplate={onUseTemplate}
/>
))}
{filteredTemplates.map((template) => {
const isNew =
template.id === 'chatgpt-pro' ||
template.id === 'github-copilot' ||
template.id === 'qwen-code'
return (
<ProviderTemplateCard
key={template.id}
template={template}
highlighted={template.id === 'moonshot'}
isNew={isNew}
onUseTemplate={onUseTemplate}
/>
)
})}
</div>
</CollapsibleContent>
</div>

View File

@@ -1,31 +1,40 @@
import { Check, Copy, ExternalLink, Globe, Server } from 'lucide-react'
import { type FC, useState } from 'react'
import {
Check,
Copy,
ExternalLink,
Loader2,
RefreshCw,
Server,
} from 'lucide-react'
import { type FC, useCallback, useState } from 'react'
import { toast } from 'sonner'
import { Button } from '@/components/ui/button'
import { MCP_SERVER_RESTARTED_EVENT } from '@/lib/constants/analyticsEvents'
import { sendServerMessage } from '@/lib/messaging/server/serverMessages'
import { track } from '@/lib/metrics/track'
interface MCPServerHeaderProps {
serverUrl: string | null
isLoading: boolean
error: string | null
title?: string
description?: string
remoteAccessEnabled?: boolean
onServerRestart?: () => void
}
const DOCS_URL = 'https://docs.browseros.com/features/use-with-claude-code'
const HEALTH_CHECK_TIMEOUT_MS = 60_000
const HEALTH_CHECK_INTERVAL_MS = 2_000
export const MCPServerHeader: FC<MCPServerHeaderProps> = ({
serverUrl,
isLoading,
error,
title = 'BrowserOS MCP Server',
description = 'Connect BrowserOS to MCP clients like claude code, gemini and others.',
remoteAccessEnabled = false,
onServerRestart,
}) => {
const [isCopied, setIsCopied] = useState(false)
const [isRestarting, setIsRestarting] = useState(false)
const handleCopy = async () => {
if (!serverUrl) return
try {
await navigator.clipboard.writeText(serverUrl)
setIsCopied(true)
@@ -35,6 +44,57 @@ export const MCPServerHeader: FC<MCPServerHeaderProps> = ({
}
}
const checkServerHealth = useCallback(async (): Promise<boolean> => {
try {
const result = await sendServerMessage('checkHealth', undefined)
return result.healthy
} catch {
return false
}
}, [])
const handleRestart = async () => {
setIsRestarting(true)
try {
const { getBrowserOSAdapter } = await import('@/lib/browseros/adapter')
const { BROWSEROS_PREFS } = await import('@/lib/browseros/prefs')
const adapter = getBrowserOSAdapter()
await adapter.setPref(BROWSEROS_PREFS.RESTART_SERVER, true)
const startTime = Date.now()
const waitForHealth = (): Promise<boolean> =>
new Promise((resolve) => {
const check = async () => {
if (Date.now() - startTime >= HEALTH_CHECK_TIMEOUT_MS) {
resolve(false)
return
}
if (await checkServerHealth()) {
resolve(true)
return
}
setTimeout(check, HEALTH_CHECK_INTERVAL_MS)
}
setTimeout(check, HEALTH_CHECK_INTERVAL_MS)
})
const healthy = await waitForHealth()
if (healthy) {
track(MCP_SERVER_RESTARTED_EVENT)
toast.success('Server restarted successfully')
onServerRestart?.()
} else {
toast.error('Server did not respond. Try restarting the browser.')
}
} catch (err) {
toast.error(
err instanceof Error ? err.message : 'Failed to restart server',
)
} finally {
setIsRestarting(false)
}
}
return (
<div className="rounded-xl border border-border bg-card p-6 shadow-sm transition-all hover:shadow-md">
<div className="flex items-start gap-4">
@@ -43,18 +103,21 @@ export const MCPServerHeader: FC<MCPServerHeaderProps> = ({
</div>
<div className="flex-1">
<div className="mb-1 flex items-center justify-between">
<h2 className="font-semibold text-xl">{title}</h2>
<h2 className="font-semibold text-xl">BrowserOS MCP Server</h2>
<a
href={DOCS_URL}
target="_blank"
rel="noopener noreferrer"
className="flex items-center gap-1 text-muted-foreground text-sm transition-colors hover:text-[var(--accent-orange)]"
>
Setup a client
Docs
<ExternalLink className="h-3.5 w-3.5" />
</a>
</div>
<p className="mb-6 text-muted-foreground text-sm">{description}</p>
<p className="mb-6 text-muted-foreground text-sm">
Connect BrowserOS to MCP clients like Claude Code, Gemini CLI and
others.
</p>
<div className="flex flex-col gap-2 sm:flex-row sm:items-center">
<span className="whitespace-nowrap font-medium text-sm">
@@ -76,6 +139,7 @@ export const MCPServerHeader: FC<MCPServerHeaderProps> = ({
onClick={handleCopy}
disabled={!serverUrl || isLoading}
className="shrink-0"
title="Copy URL"
>
{isCopied ? (
<Check className="h-4 w-4 text-green-600" />
@@ -83,19 +147,22 @@ export const MCPServerHeader: FC<MCPServerHeaderProps> = ({
<Copy className="h-4 w-4" />
)}
</Button>
<Button
variant="outline"
size="icon"
onClick={handleRestart}
disabled={isLoading || isRestarting}
className="shrink-0"
title="Restart server"
>
{isRestarting ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<RefreshCw className="h-4 w-4" />
)}
</Button>
</div>
</div>
{remoteAccessEnabled && serverUrl && !isLoading && (
<div className="mt-3 flex items-start gap-2 rounded-lg bg-muted/50 px-3 py-2">
<Globe className="mt-0.5 h-3.5 w-3.5 shrink-0 text-muted-foreground" />
<p className="text-muted-foreground text-xs">
External access is enabled. To connect from another device,
replace <span className="font-mono">127.0.0.1</span> with this
machine's IP address.
</p>
</div>
)}
</div>
</div>
</div>

View File

@@ -4,7 +4,7 @@ import type { McpTool } from '@/lib/mcp/client'
import { sendServerMessage } from '@/lib/messaging/server/serverMessages'
import { MCPServerHeader } from './MCPServerHeader'
import { MCPToolsSection } from './MCPToolsSection'
import { ServerSettingsCard } from './ServerSettingsCard'
import { QuickSetupSection } from './QuickSetupSection'
/** @public */
export const MCPSettingsPage: FC = () => {
@@ -12,8 +12,6 @@ export const MCPSettingsPage: FC = () => {
const [urlLoading, setUrlLoading] = useState(true)
const [urlError, setUrlError] = useState<string | null>(null)
const [remoteAccessEnabled, setRemoteAccessEnabled] = useState(false)
const [tools, setTools] = useState<McpTool[]>([])
const [toolsLoading, setToolsLoading] = useState(false)
const [toolsError, setToolsError] = useState<string | null>(null)
@@ -82,13 +80,10 @@ export const MCPSettingsPage: FC = () => {
serverUrl={serverUrl}
isLoading={urlLoading}
error={urlError}
remoteAccessEnabled={remoteAccessEnabled}
onServerRestart={loadServerUrlAndTools}
/>
<ServerSettingsCard
onServerRestart={loadServerUrlAndTools}
onRemoteAccessChange={setRemoteAccessEnabled}
/>
<QuickSetupSection serverUrl={serverUrl} />
<MCPToolsSection
tools={tools}

View File

@@ -0,0 +1,162 @@
import { Check, Copy, Terminal } from 'lucide-react'
import { type FC, useState } from 'react'
import { Button } from '@/components/ui/button'
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs'
interface QuickSetupSectionProps {
serverUrl: string | null
}
interface ClientConfig {
id: string
name: string
type: 'command' | 'json'
getSnippet: (url: string) => string
fileName?: string
}
const clients: ClientConfig[] = [
{
id: 'claude-code',
name: 'Claude Code',
type: 'command',
getSnippet: (url) =>
`claude mcp add --transport http browseros ${url} --scope user`,
},
{
id: 'gemini-cli',
name: 'Gemini CLI',
type: 'command',
getSnippet: (url) =>
`gemini mcp add local-server ${url} --transport http --scope user`,
},
{
id: 'codex',
name: 'Codex',
type: 'command',
getSnippet: (url) => `codex mcp add browseros ${url}`,
},
{
id: 'claude-desktop',
name: 'Claude Desktop',
type: 'json',
fileName: 'claude_desktop_config.json',
getSnippet: (url) =>
JSON.stringify(
{
mcpServers: {
browserOS: {
command: 'npx',
args: ['mcp-remote', url],
},
},
},
null,
2,
),
},
{
id: 'openclaw',
name: 'OpenClaw',
type: 'json',
fileName: 'openclaw.json',
getSnippet: (url) =>
JSON.stringify(
{
mcpServers: {
browseros: { url },
},
},
null,
2,
),
},
]
const CopyButton: FC<{ text: string }> = ({ text }) => {
const [copied, setCopied] = useState(false)
const handleCopy = async () => {
try {
await navigator.clipboard.writeText(text)
setCopied(true)
setTimeout(() => setCopied(false), 2000)
} catch {
// Clipboard API failed
}
}
return (
<Button
variant="ghost"
size="icon-sm"
onClick={handleCopy}
className="shrink-0 text-muted-foreground hover:text-foreground"
>
{copied ? (
<Check className="h-3.5 w-3.5 text-green-600" />
) : (
<Copy className="h-3.5 w-3.5" />
)}
</Button>
)
}
export const QuickSetupSection: FC<QuickSetupSectionProps> = ({
serverUrl,
}) => {
if (!serverUrl) return null
return (
<div className="rounded-xl border border-border bg-card p-6 shadow-sm transition-all hover:shadow-md">
<div className="flex items-start gap-4">
<div className="flex h-12 w-12 shrink-0 items-center justify-center rounded-xl bg-[var(--accent-orange)]/10">
<Terminal className="h-6 w-6 text-[var(--accent-orange)]" />
</div>
<div className="flex-1">
<h2 className="mb-1 font-semibold text-xl">Quick Setup</h2>
<p className="mb-4 text-muted-foreground text-sm">
Copy and run the command for your tool
</p>
<Tabs defaultValue="claude-code">
<TabsList className="mb-3 flex-wrap">
{clients.map((client) => (
<TabsTrigger key={client.id} value={client.id}>
{client.name}
</TabsTrigger>
))}
</TabsList>
{clients.map((client) => {
const snippet = client.getSnippet(serverUrl)
return (
<TabsContent key={client.id} value={client.id}>
<div className="space-y-3">
{client.fileName && (
<p className="text-muted-foreground text-xs">
Add to{' '}
<code className="rounded bg-muted px-1 py-0.5 font-mono text-xs">
{client.fileName}
</code>
</p>
)}
<div className="flex items-start gap-2 rounded-lg border border-border bg-background px-3 py-2.5">
<pre className="flex-1 overflow-x-auto whitespace-pre-wrap break-all font-mono text-xs">
{client.type === 'command' && (
<span className="mr-1 text-muted-foreground">$</span>
)}
{snippet}
</pre>
<CopyButton text={snippet} />
</div>
</div>
</TabsContent>
)
})}
</Tabs>
</div>
</div>
</div>
)
}

View File

@@ -105,18 +105,40 @@ export const UsagePage: FC = () => {
</div>
<div className="rounded-xl border p-5">
<div className="flex items-center gap-3">
<CreditCard className="h-5 w-5 text-muted-foreground" />
<div>
<p className="flex items-center gap-2 font-semibold text-sm">
Need more credits?
<span className="rounded-full bg-muted px-2 py-0.5 font-medium text-[10px] text-muted-foreground uppercase tracking-wide">
Coming soon
</span>
</p>
<p className="text-muted-foreground text-xs">
Additional credit packages will be available soon
</p>
</div>
</div>
</div>
<div className="rounded-xl border border-[var(--accent-orange)]/30 bg-[var(--accent-orange)]/5 p-5">
<div className="flex items-center justify-between">
<div className="flex items-center gap-3">
<CreditCard className="h-5 w-5 text-muted-foreground" />
<Zap className="h-5 w-5 text-[var(--accent-orange)]" />
<div>
<p className="font-semibold text-sm">Need more credits?</p>
<p className="font-semibold text-sm">Want unlimited usage?</p>
<p className="text-muted-foreground text-xs">
Additional credit packages coming soon
Add your own LLM provider no credit limits
</p>
</div>
</div>
<Button variant="outline" size="sm" disabled className="opacity-50">
Add Credits
<Button
variant="outline"
size="sm"
className="border-[var(--accent-orange)] bg-[var(--accent-orange)]/10 text-[var(--accent-orange)] hover:bg-[var(--accent-orange)]/20"
asChild
>
<a href="/app.html#/settings/ai">Add Provider</a>
</Button>
</div>
</div>

View File

@@ -36,6 +36,7 @@ export const Chat = () => {
stop,
agentUrlError,
chatError,
selectedProvider,
getActionForMessage,
liked,
onClickLike,
@@ -224,7 +225,9 @@ export const Chat = () => {
/>
)}
{agentUrlError && <ChatError error={agentUrlError} />}
{chatError && <ChatError error={chatError} />}
{chatError && (
<ChatError error={chatError} providerType={selectedProvider?.type} />
)}
</main>
<ChatFooter

View File

@@ -2,11 +2,6 @@ import { AlertCircle, RefreshCw } from 'lucide-react'
import type { FC } from 'react'
// import { useMemo } from 'react'
import { Button } from '@/components/ui/button'
import {
KIMI_RATE_LIMIT_DOCS_CLICKED_EVENT,
KIMI_RATE_LIMIT_PLATFORM_CLICKED_EVENT,
} from '@/lib/constants/analyticsEvents'
import { track } from '@/lib/metrics/track'
// --- Commented out for Kimi partnership launch (restore after) ---
// const SURVEY_DIRECTIONS = [
@@ -24,16 +19,22 @@ import { track } from '@/lib/metrics/track'
interface ChatErrorProps {
error: Error
onRetry?: () => void
providerType?: string
}
function parseErrorMessage(message: string): {
function parseErrorMessage(
message: string,
providerType?: string,
): {
text: string
url?: string
isRateLimit?: boolean
isCreditsExhausted?: boolean
isConnectionError?: boolean
} {
// Detect MCP server connection failures
const isBrowserosProvider = providerType === 'browseros'
// Detect MCP server connection failures (universal — affects all providers)
if (
(message.includes('Failed to fetch') || message.includes('fetch failed')) &&
message.includes('127.0.0.1')
@@ -45,10 +46,12 @@ function parseErrorMessage(message: string): {
}
}
// Detect credit exhaustion from gateway
// Detect credit exhaustion from gateway (BrowserOS provider only)
if (
message.includes('CREDITS_EXHAUSTED') ||
message.includes('Daily credits exhausted')
isBrowserosProvider &&
(message.includes('CREDITS_EXHAUSTED') ||
message.includes('Credits exhausted') ||
message.includes('Daily credits exhausted'))
) {
return {
text: 'Daily credits exhausted. Credits reset at midnight UTC.',
@@ -58,8 +61,11 @@ function parseErrorMessage(message: string): {
}
}
// Detect BrowserOS rate limit (unique pattern, no provider uses this)
if (message.includes('BrowserOS LLM daily limit reached')) {
// Detect BrowserOS rate limit (BrowserOS provider only)
if (
isBrowserosProvider &&
message.includes('BrowserOS LLM daily limit reached')
) {
return {
text: 'Add your own API key for unlimited usage.',
url: 'https://dub.sh/browseros-usage-limit',
@@ -83,9 +89,13 @@ function parseErrorMessage(message: string): {
return { text: text || 'An unexpected error occurred', url }
}
export const ChatError: FC<ChatErrorProps> = ({ error, onRetry }) => {
export const ChatError: FC<ChatErrorProps> = ({
error,
onRetry,
providerType,
}) => {
const { text, url, isRateLimit, isCreditsExhausted, isConnectionError } =
parseErrorMessage(error.message)
parseErrorMessage(error.message, providerType)
// --- Commented out for Kimi partnership launch (restore after) ---
// const surveyUrl = useMemo(
@@ -151,31 +161,15 @@ export const ChatError: FC<ChatErrorProps> = ({ error, onRetry }) => {
View Usage & Billing
</a>
)}
{isRateLimit && !isCreditsExhausted && (
<div className="flex flex-col items-center gap-1">
<p className="text-muted-foreground text-xs">
{/* biome-ignore lint/a11y/useValidAnchor: link with click tracking */}
<a
href="https://docs.browseros.com/features/bring-your-own-llm#kimi-k2-5-%E2%80%94-in-partnership-with-moonshot-ai"
target="_blank"
rel="noopener noreferrer"
className="underline hover:text-foreground"
onClick={() => track(KIMI_RATE_LIMIT_DOCS_CLICKED_EVENT)}
>
Learn how to get a Kimi API key
</a>
{' or '}
<a
href="https://platform.moonshot.ai"
target="_blank"
rel="noopener noreferrer"
className="underline hover:text-foreground"
onClick={() => track(KIMI_RATE_LIMIT_PLATFORM_CLICKED_EVENT)}
>
get your API key
</a>
</p>
</div>
{isRateLimit && providerType === 'browseros' && (
<a
href="/app.html#/settings/ai"
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1.5 rounded-md border border-[var(--accent-orange)] bg-[var(--accent-orange)]/10 px-3 py-1.5 font-medium text-[var(--accent-orange)] text-xs transition-colors hover:bg-[var(--accent-orange)]/20"
>
Add your own provider for unlimited usage
</a>
)}
{onRetry && (
<Button

View File

@@ -67,6 +67,10 @@ export const QWEN_CODE_OAUTH_DISCONNECTED_EVENT =
/** @public */
export const HUB_PROVIDER_ADDED_EVENT = 'settings.hub_provider.added'
/** @public */
export const MCP_PROMO_BANNER_CLICKED_EVENT =
'settings.mcp_promo_banner.clicked'
/** @public */
export const MCP_EXTERNAL_ACCESS_ENABLED_EVENT =
'settings.mcp_external_access.enabled'

View File

@@ -1,4 +1,4 @@
import { useEffect, useRef } from 'react'
import { useEffect, useRef, useState } from 'react'
import { toast } from 'sonner'
import { track } from '@/lib/metrics/track'
import {
@@ -20,10 +20,18 @@ export interface OAuthProviderFlowConfig {
clientAuth?: ClientAuthConfig
}
export interface PendingDeviceCode {
userCode: string
providerName: string
verificationUri: string
}
interface OAuthProviderFlowReturn {
status: { authenticated: boolean; email?: string } | null
disconnect: () => Promise<void>
startOAuthFlow: (agentServerUrl: string | undefined) => Promise<void>
pendingDeviceCode: PendingDeviceCode | null
clearDeviceCode: () => void
}
export function useOAuthProviderFlow(
@@ -35,6 +43,8 @@ export function useOAuthProviderFlow(
config.providerType,
)
const flowStartedRef = useRef(false)
const [pendingDeviceCode, setPendingDeviceCode] =
useState<PendingDeviceCode | null>(null)
// Auto-create provider when OAuth completes
// biome-ignore lint/correctness/useExhaustiveDependencies: intentional — only trigger on auth status change
@@ -57,6 +67,7 @@ export function useOAuthProviderFlow(
createdAt: now,
updatedAt: now,
})
setPendingDeviceCode(null)
track(config.completedEvent, { email: status.email })
toast.success(`${config.displayName} Connected`, {
description: status.email
@@ -104,9 +115,10 @@ export function useOAuthProviderFlow(
deviceData.verification_uri_complete ?? deviceData.verification_uri
window.open(verificationUri, '_blank')
track(config.startedEvent)
toast.info(`Enter code: ${deviceData.user_code}`, {
description: `Paste this code on the ${config.displayName} page that just opened.`,
duration: 60_000,
setPendingDeviceCode({
userCode: deviceData.user_code,
providerName: config.displayName,
verificationUri,
})
startTokenPolling(auth, deviceData, codeVerifier, async (token) => {
@@ -142,9 +154,10 @@ export function useOAuthProviderFlow(
window.open(data.verificationUri, '_blank')
startPolling()
track(config.startedEvent)
toast.info(`Enter code: ${data.userCode}`, {
description: `Paste this code on the ${config.displayName} page that just opened.`,
duration: 60_000,
setPendingDeviceCode({
userCode: data.userCode,
providerName: config.displayName,
verificationUri: data.verificationUri,
})
return
}
@@ -163,5 +176,7 @@ export function useOAuthProviderFlow(
status,
disconnect,
startOAuthFlow,
pendingDeviceCode,
clearDeviceCode: () => setPendingDeviceCode(null),
}
}

View File

@@ -54,13 +54,18 @@ export default defineConfig({
},
permissions: [
'topSites',
'tabs',
'tabGroups',
'storage',
'unlimitedStorage',
'scripting',
'tabs',
'tabGroups',
'sidePanel',
'bookmarks',
'history',
'browserOS',
'alarms',
'webNavigation',
'downloads',
],
host_permissions: [
'http://127.0.0.1/*',

View File

@@ -1 +1,2 @@
browseros-cli
dist

View File

@@ -0,0 +1,47 @@
version: 2
project_name: browseros-cli
builds:
- main: .
binary: browseros-cli
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -s -w -X main.version={{ .Version }}
targets:
- darwin_amd64
- darwin_arm64
- linux_amd64
- linux_arm64
- windows_amd64
- windows_arm64
archives:
- format: tar.gz
format_overrides:
- goos: windows
format: zip
name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
files:
- "none*"
checksum:
name_template: checksums.txt
changelog:
sort: asc
filters:
exclude:
- "^docs:"
- "^test:"
- "^ci:"
release:
github:
owner: browseros-ai
name: BrowserOS
prerelease: auto
name_template: "browseros-cli v{{ .Version }}"

View File

@@ -18,3 +18,9 @@ vet:
test:
go test -tags integration -v -timeout 120s ./...
release-dry:
goreleaser release --snapshot --clean
release:
goreleaser release --clean

View File

@@ -17,8 +17,10 @@ import (
)
func init() {
var autoDiscover bool
cmd := &cobra.Command{
Use: "init",
Use: "init [url]",
Short: "Configure the BrowserOS server connection",
Long: `Set up the CLI by providing the MCP server URL from BrowserOS.
@@ -26,33 +28,59 @@ Open BrowserOS → Settings → BrowserOS MCP to find your Server URL.
The URL looks like: http://127.0.0.1:9004/mcp
The port varies per installation, so this step is required on first use.
Run again if your port changes.`,
Run again if your port changes.
Three modes:
browseros-cli init <url> Non-interactive, use the provided URL
browseros-cli init --auto Auto-discover from ~/.browseros/server.json
browseros-cli init Interactive prompt`,
Annotations: map[string]string{"group": "Setup:"},
Args: cobra.NoArgs,
Args: cobra.MaximumNArgs(1),
Run: func(cmd *cobra.Command, args []string) {
bold := color.New(color.Bold)
green := color.New(color.FgGreen)
dim := color.New(color.Faint)
fmt.Println()
bold.Println("BrowserOS CLI Setup")
fmt.Println()
fmt.Println("Open BrowserOS → Settings → BrowserOS MCP")
fmt.Println("Copy the Server URL shown there.")
fmt.Println()
dim.Println("It looks like: http://127.0.0.1:9004/mcp")
fmt.Println()
var input string
reader := bufio.NewReader(os.Stdin)
fmt.Print("Server URL: ")
input, err := reader.ReadString('\n')
if err != nil {
output.Error("failed to read input", 1)
}
input = strings.TrimSpace(input)
switch {
case len(args) == 1:
// Non-interactive: URL provided as argument
input = args[0]
if input == "" {
output.Error("no URL provided", 1)
case autoDiscover:
// Auto-discover: server.json → config → probe common ports
discovered := probeRunningServer()
if discovered == "" {
output.Error("auto-discovery failed: no running BrowserOS found.\n\n"+
" If not running: browseros-cli launch\n"+
" If not installed: browseros-cli install", 1)
}
input = discovered
fmt.Printf("Auto-discovered server at %s\n", input)
default:
// Interactive prompt (original behavior)
fmt.Println()
bold.Println("BrowserOS CLI Setup")
fmt.Println()
fmt.Println("Open BrowserOS → Settings → BrowserOS MCP")
fmt.Println("Copy the Server URL shown there.")
fmt.Println()
dim.Println("It looks like: http://127.0.0.1:9004/mcp")
fmt.Println()
reader := bufio.NewReader(os.Stdin)
fmt.Print("Server URL: ")
line, err := reader.ReadString('\n')
if err != nil {
output.Error("failed to read input", 1)
}
input = strings.TrimSpace(line)
if input == "" {
output.Error("no URL provided", 1)
}
}
baseURL := normalizeServerURL(input)
@@ -88,5 +116,6 @@ Run again if your port changes.`,
},
}
cmd.Flags().BoolVar(&autoDiscover, "auto", false, "Auto-discover server URL from ~/.browseros/server.json")
rootCmd.AddCommand(cmd)
}

View File

@@ -0,0 +1,247 @@
package cmd
import (
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"runtime"
"time"
"browseros-cli/output"
"github.com/fatih/color"
"github.com/spf13/cobra"
)
func init() {
cmd := &cobra.Command{
Use: "install",
Short: "Download and install BrowserOS for the current platform",
Long: `Download BrowserOS for your platform and start the installation.
macOS: Downloads .dmg, mounts it, and copies BrowserOS to /Applications
Windows: Downloads installer .exe and launches it
Linux: Downloads AppImage (or .deb with --deb flag)
After installation:
browseros-cli launch # start BrowserOS
browseros-cli init --auto # configure the CLI`,
Annotations: map[string]string{"group": "Setup:"},
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
dir, _ := cmd.Flags().GetString("dir")
deb, _ := cmd.Flags().GetBool("deb")
if deb && runtime.GOOS != "linux" {
output.Error("--deb is only available on Linux", 1)
}
downloadURL, filename := resolveDownload(deb)
destPath := filepath.Join(dir, filename)
bold := color.New(color.Bold)
green := color.New(color.FgGreen)
dim := color.New(color.Faint)
bold.Printf("Downloading BrowserOS for %s...\n", platformDisplayName())
dim.Printf(" %s\n", downloadURL)
fmt.Println()
client := &http.Client{Timeout: 10 * time.Minute}
resp, err := client.Get(downloadURL)
if err != nil {
output.Errorf(1, "download failed: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
output.Errorf(1, "download failed: HTTP %d", resp.StatusCode)
}
file, err := os.Create(destPath)
if err != nil {
output.Errorf(1, "create file: %v", err)
}
written, err := io.Copy(file, resp.Body)
file.Close()
if err != nil {
os.Remove(destPath)
output.Errorf(1, "download interrupted: %v", err)
}
green.Printf("Downloaded %s (%.1f MB)\n", filename, float64(written)/(1024*1024))
fmt.Println()
runPostInstall(destPath, deb, dim)
fmt.Println()
bold.Println("Next steps:")
dim.Println(" browseros-cli launch # start BrowserOS")
dim.Println(" browseros-cli init --auto # configure the CLI")
},
}
cmd.Flags().String("dir", ".", "Directory to download the installer to")
cmd.Flags().Bool("deb", false, "Download .deb package instead of AppImage (Linux only)")
rootCmd.AddCommand(cmd)
}
func resolveDownload(deb bool) (url, filename string) {
switch runtime.GOOS {
case "darwin":
return "https://files.browseros.com/download/BrowserOS.dmg", "BrowserOS.dmg"
case "windows":
return "https://files.browseros.com/download/BrowserOS_installer.exe", "BrowserOS_installer.exe"
case "linux":
if deb {
return "https://cdn.browseros.com/download/BrowserOS.deb", "BrowserOS.deb"
}
return "https://files.browseros.com/download/BrowserOS.AppImage", "BrowserOS.AppImage"
default:
output.Errorf(1, "unsupported platform: %s/%s\n Download manually from https://browseros.com", runtime.GOOS, runtime.GOARCH)
return "", ""
}
}
func platformDisplayName() string {
switch runtime.GOOS {
case "darwin":
return "macOS"
case "windows":
return "Windows"
case "linux":
return "Linux"
default:
return runtime.GOOS
}
}
func runPostInstall(path string, deb bool, dim *color.Color) {
switch runtime.GOOS {
case "darwin":
installMacOS(path, dim)
case "linux":
if deb {
dim.Println("Install the .deb package:")
fmt.Printf(" sudo dpkg -i %s\n", path)
} else {
os.Chmod(path, 0755)
dim.Printf("AppImage is ready to run: ./%s\n", filepath.Base(path))
}
case "windows":
fmt.Println("Launching installer...")
if err := exec.Command("cmd", "/c", "start", "", path).Run(); err != nil {
dim.Printf("Could not launch installer automatically. Run: %s\n", path)
} else {
dim.Println("Follow the installer prompts to complete setup.")
}
}
}
// installMacOS mounts the DMG and copies BrowserOS.app to /Applications.
func installMacOS(dmgPath string, dim *color.Color) {
fmt.Println("Mounting disk image...")
mountOut, err := exec.Command("hdiutil", "attach", dmgPath, "-nobrowse", "-quiet").Output()
if err != nil {
dim.Println("Could not mount DMG automatically.")
dim.Printf(" Open it manually: open %s\n", dmgPath)
return
}
// Find the mount point (last field of last line of hdiutil output)
mountPoint := ""
for _, line := range splitLines(string(mountOut)) {
fields := splitTabs(line)
if len(fields) > 0 {
mountPoint = fields[len(fields)-1]
}
}
if mountPoint == "" {
dim.Println("DMG mounted but could not determine mount point.")
dim.Printf(" Open it manually: open %s\n", dmgPath)
return
}
// Look for BrowserOS.app in the mounted volume
appSrc := filepath.Join(mountPoint, "BrowserOS.app")
if _, err := os.Stat(appSrc); err != nil {
dim.Printf("DMG mounted at %s but BrowserOS.app not found inside.\n", mountPoint)
dim.Printf(" Check the volume manually: open %s\n", mountPoint)
exec.Command("hdiutil", "detach", mountPoint, "-quiet").Run()
return
}
appDest := "/Applications/BrowserOS.app"
fmt.Printf("Installing to %s...\n", appDest)
// Remove existing installation if present
os.RemoveAll(appDest)
// Copy using cp -R (preserves code signatures, symlinks, etc.)
if err := exec.Command("cp", "-R", appSrc, appDest).Run(); err != nil {
dim.Printf("Could not copy to /Applications (may need sudo).\n")
dim.Printf(" Try: sudo cp -R \"%s\" /Applications/\n", appSrc)
exec.Command("hdiutil", "detach", mountPoint, "-quiet").Run()
return
}
// Unmount
exec.Command("hdiutil", "detach", mountPoint, "-quiet").Run()
// Clean up DMG
os.Remove(dmgPath)
fmt.Println("BrowserOS installed to /Applications/BrowserOS.app")
}
func splitLines(s string) []string {
var lines []string
for _, line := range filepath.SplitList(s) {
lines = append(lines, line)
}
// filepath.SplitList uses : on unix, not newlines — use manual split
result := []string{}
start := 0
for i := 0; i < len(s); i++ {
if s[i] == '\n' {
line := s[start:i]
if len(line) > 0 {
result = append(result, line)
}
start = i + 1
}
}
if start < len(s) {
result = append(result, s[start:])
}
return result
}
func splitTabs(s string) []string {
result := []string{}
start := 0
for i := 0; i < len(s); i++ {
if s[i] == '\t' {
field := s[start:i]
if len(field) > 0 {
result = append(result, field)
}
start = i + 1
}
}
if start < len(s) {
field := s[start:]
if len(field) > 0 {
result = append(result, field)
}
}
return result
}

View File

@@ -0,0 +1,287 @@
package cmd
import (
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"time"
"browseros-cli/output"
"github.com/fatih/color"
"github.com/spf13/cobra"
)
// macOS bundle identifier — verified from BrowserOS.app/Contents/Info.plist
const browserOSBundleID = "com.browseros.BrowserOS"
func init() {
cmd := &cobra.Command{
Use: "launch",
Short: "Launch the BrowserOS application",
Long: `Find and launch the BrowserOS application.
Uses platform-native detection to find BrowserOS, launches it,
and waits for the server to become ready.
If BrowserOS is already running, reports the server URL.`,
Annotations: map[string]string{"group": "Setup:"},
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
green := color.New(color.FgGreen)
dim := color.New(color.Faint)
waitSecs, _ := cmd.Flags().GetInt("wait")
if url := probeRunningServer(); url != "" {
green.Printf("BrowserOS is already running at %s\n", url)
return
}
if !isBrowserOSInstalled() {
output.Error("BrowserOS is not installed.\n\n"+
" To install: browseros-cli install", 1)
}
fmt.Println("Launching BrowserOS...")
if err := startBrowserOS(); err != nil {
output.Errorf(1, "failed to launch: %v", err)
}
fmt.Print("Waiting for server")
url, ok := waitForServer(time.Duration(waitSecs) * time.Second)
fmt.Println()
if !ok {
output.Error("BrowserOS launched but server didn't respond within "+
fmt.Sprintf("%d seconds.\n", waitSecs)+
" Check if BrowserOS is fully loaded, then retry.", 1)
}
green.Printf("BrowserOS is ready at %s\n", url)
fmt.Println()
dim.Println("Next: browseros-cli init --auto")
},
}
cmd.Flags().Int("wait", 30, "Seconds to wait for server to start")
rootCmd.AddCommand(cmd)
}
// ---------------------------------------------------------------------------
// Server probing
// ---------------------------------------------------------------------------
// probeRunningServer checks server.json, config, and common ports for a running server.
func probeRunningServer() string {
check := func(baseURL string) bool {
client := &http.Client{Timeout: 2 * time.Second}
resp, err := client.Get(baseURL + "/health")
if err != nil {
return false
}
resp.Body.Close()
return resp.StatusCode == 200
}
// 1. server.json — written by BrowserOS on startup with the actual port
if url := loadBrowserosServerURL(); url != "" && check(url) {
return url
}
// 2. Saved config / env var
if url := defaultServerURL(); url != "" && check(url) {
return url
}
// 3. Probe common BrowserOS ports as last resort
for _, port := range []int{9100, 9200, 9300} {
url := fmt.Sprintf("http://127.0.0.1:%d", port)
if check(url) {
return url
}
}
return ""
}
// ---------------------------------------------------------------------------
// Platform-native installation detection
// ---------------------------------------------------------------------------
// isBrowserOSInstalled checks if BrowserOS is installed using platform-native methods.
//
// macOS: `open -Ra "BrowserOS"` — queries Launch Services (finds apps anywhere)
// Linux: checks /usr/bin/browseros (.deb), browseros.desktop, or AppImage files
// Windows: checks executable at %LOCALAPPDATA%\BrowserOS\Application\BrowserOS.exe
// and registry uninstall key (per-user Chromium install pattern)
func isBrowserOSInstalled() bool {
switch runtime.GOOS {
case "darwin":
// open -Ra checks if Launch Services knows about the app without launching it.
// Works regardless of where the app is installed.
return exec.Command("open", "-Ra", "BrowserOS").Run() == nil
case "linux":
// .deb install puts `browseros` in /usr/bin/
if _, err := exec.LookPath("browseros"); err == nil {
return true
}
// .deb also creates browseros.desktop
for _, dir := range []string{
"/usr/share/applications",
filepath.Join(userHomeDir(), ".local/share/applications"),
} {
if _, err := os.Stat(filepath.Join(dir, "browseros.desktop")); err == nil {
return true
}
}
// AppImage — user may have it in ~/Downloads, ~/Applications, etc.
return findLinuxAppImage() != ""
case "windows":
// Chromium per-user install: %LOCALAPPDATA%\BrowserOS\Application\BrowserOS.exe
if exePath := windowsBrowserOSExe(); exePath != "" {
if _, err := os.Stat(exePath); err == nil {
return true
}
}
// Fallback: check uninstall registry (per-user install uses HKCU)
for _, root := range []string{"HKCU", "HKLM"} {
key := root + `\Software\Microsoft\Windows\CurrentVersion\Uninstall\BrowserOS`
if exec.Command("reg", "query", key, "/v", "DisplayName").Run() == nil {
return true
}
}
return false
}
return false
}
// ---------------------------------------------------------------------------
// Platform-native launch
// ---------------------------------------------------------------------------
// startBrowserOS launches BrowserOS using platform-native methods.
//
// macOS: `open -b com.browseros.BrowserOS` — launches by bundle ID
// Linux: runs `browseros` binary or AppImage directly
// Windows: runs BrowserOS.exe from the known install path
func startBrowserOS() error {
switch runtime.GOOS {
case "darwin":
// Launch by bundle ID via Launch Services — no hardcoded paths needed.
return exec.Command("open", "-b", browserOSBundleID).Run()
case "linux":
// .deb install: browseros is in PATH
if p, err := exec.LookPath("browseros"); err == nil {
return startDetached(p)
}
// AppImage: run it directly
if appImage := findLinuxAppImage(); appImage != "" {
return startDetached(appImage)
}
// .desktop file: use gtk-launch (not xdg-open, which opens by MIME type)
if _, err := exec.LookPath("gtk-launch"); err == nil {
return exec.Command("gtk-launch", "browseros").Run()
}
return fmt.Errorf("BrowserOS found but could not determine how to launch it")
case "windows":
if exePath := windowsBrowserOSExe(); exePath != "" {
if _, err := os.Stat(exePath); err == nil {
return startDetached(exePath)
}
}
return fmt.Errorf("BrowserOS.exe not found at expected location")
default:
return fmt.Errorf("unsupported platform: %s", runtime.GOOS)
}
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
// startDetached starts a process in the background without inheriting stdio.
func startDetached(path string, args ...string) error {
cmd := exec.Command(path, args...)
cmd.Stdout = nil
cmd.Stderr = nil
cmd.Stdin = nil
return cmd.Start()
}
// windowsBrowserOSExe returns the expected BrowserOS.exe path on Windows.
// Chromium per-user installs go to %LOCALAPPDATA%\<base_app_name>\Application\<binary>.
// base_app_name = "BrowserOS" (from chromium_install_modes.h)
func windowsBrowserOSExe() string {
localAppData := os.Getenv("LOCALAPPDATA")
if localAppData == "" {
return ""
}
return filepath.Join(localAppData, "BrowserOS", "Application", "BrowserOS.exe")
}
// findLinuxAppImage searches common locations for a BrowserOS AppImage.
func findLinuxAppImage() string {
home := userHomeDir()
if home == "" {
return ""
}
for _, dir := range []string{
home,
filepath.Join(home, "Applications"),
filepath.Join(home, "Downloads"),
"/opt",
} {
entries, err := os.ReadDir(dir)
if err != nil {
continue
}
for _, e := range entries {
name := e.Name()
if strings.HasPrefix(name, "BrowserOS") && strings.HasSuffix(name, ".AppImage") {
return filepath.Join(dir, name)
}
}
}
return ""
}
// userHomeDir returns the home directory or empty string.
func userHomeDir() string {
home, err := os.UserHomeDir()
if err != nil {
return ""
}
return home
}
// waitForServer polls until a BrowserOS server responds or timeout.
func waitForServer(maxWait time.Duration) (string, bool) {
client := &http.Client{Timeout: 2 * time.Second}
deadline := time.Now().Add(maxWait)
for time.Now().Before(deadline) {
// server.json is written by BrowserOS on startup with the actual port
if url := loadBrowserosServerURL(); url != "" {
resp, err := client.Get(url + "/health")
if err == nil {
resp.Body.Close()
if resp.StatusCode == 200 {
return url, true
}
}
}
fmt.Print(".")
time.Sleep(1 * time.Second)
}
return "", false
}

View File

@@ -167,10 +167,17 @@ func envBool(key string) bool {
}
func defaultServerURL() string {
// 1. Explicit env var always wins
if env := normalizeServerURL(os.Getenv("BROWSEROS_URL")); env != "" {
return env
}
// 2. Live discovery file from running BrowserOS (most current)
if url := loadBrowserosServerURL(); url != "" {
return url
}
// 3. Saved config (may be stale if port changed)
cfg, err := config.Load()
if err == nil {
if url := normalizeServerURL(cfg.ServerURL); url != "" {
@@ -178,10 +185,6 @@ func defaultServerURL() string {
}
}
if url := loadBrowserosServerURL(); url != "" {
return url
}
return ""
}
@@ -225,6 +228,9 @@ func validateServerURL(raw string) (string, error) {
}
return "", fmt.Errorf(
"BrowserOS server URL is not configured.\n Open BrowserOS -> Settings -> BrowserOS MCP and copy the Server URL.\n Then run: browseros-cli init",
"BrowserOS server URL is not configured.\n\n" +
" If BrowserOS is running: browseros-cli init --auto\n" +
" If BrowserOS is closed: browseros-cli launch\n" +
" If not installed: browseros-cli install",
)
}

View File

@@ -44,7 +44,10 @@ func (c *Client) connect(ctx context.Context) (*sdkmcp.ClientSession, error) {
session, err := sdkClient.Connect(ctx, transport, nil)
if err != nil {
return nil, fmt.Errorf("cannot connect to BrowserOS at %s: %w\n Is the server running? Try: browseros-cli init", c.BaseURL, err)
return nil, fmt.Errorf("cannot connect to BrowserOS at %s: %w\n\n"+
" If BrowserOS is running on a different port: browseros-cli init --auto\n"+
" If BrowserOS is not running: browseros-cli launch\n"+
" If not installed: browseros-cli install", c.BaseURL, err)
}
return session, nil
}
@@ -184,7 +187,10 @@ func (c *Client) Status() (map[string]any, error) {
func (c *Client) restGET(path string) (map[string]any, error) {
resp, err := c.HTTPClient.Get(c.BaseURL + path)
if err != nil {
return nil, fmt.Errorf("cannot connect to BrowserOS at %s: %w\n Try: browseros-cli init", c.BaseURL, err)
return nil, fmt.Errorf("cannot connect to BrowserOS at %s: %w\n\n"+
" If BrowserOS is running on a different port: browseros-cli init --auto\n"+
" If BrowserOS is not running: browseros-cli launch\n"+
" If not installed: browseros-cli install", c.BaseURL, err)
}
defer resp.Body.Close()

View File

@@ -16,7 +16,10 @@
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",

View File

@@ -2,24 +2,21 @@
"agent": {
"type": "orchestrator-executor",
"orchestrator": {
"type": "single",
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
"baseUrl": "https://api.fireworks.ai/inference/v1"
},
"executor": {
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
"baseUrl": "https://api.fireworks.ai/inference/v1"
}
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/orchestrator-executor-webvoyager-test",
"num_workers": 3,
"dataset": "../data/webbench-2of4-50.jsonl",
"num_workers": 10,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
@@ -28,8 +25,12 @@
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
"timeout_ms": 1800000
}

View File

@@ -23,7 +23,10 @@
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",

View File

@@ -1,23 +0,0 @@
{
"agent": {
"type": "orchestrator-executor",
"orchestrator": {
"provider": "openrouter",
"model": "openai/gpt-4o",
"apiKey": "OPENROUTER_API_KEY",
"maxTurns": 3
},
"executor": {
"provider": "openrouter",
"model": "openai/gpt-4o",
"apiKey": "OPENROUTER_API_KEY"
}
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/debug-test",
"num_workers": 1,
"browseros": {
"server_url": "http://127.0.0.1:9110"
},
"timeout_ms": 90000
}

View File

@@ -1,21 +0,0 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
},
"dataset": "../data/test-set.jsonl",
"output_dir": "../results/fireworks-minimax-k2p5-test-set",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110"
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/o4-mini-high",
"timeout_ms": 3600000
}

View File

@@ -1,18 +0,0 @@
{
"agent": {
"type": "single",
"provider": "openrouter",
"model": "openai/gpt-4.1",
"apiKey": "OPENROUTER_API_KEY"
},
"dataset": "../data/mind2web_e2e_test.jsonl",
"output_dir": "../results/mind2web-test",
"num_workers": 5,
"browseros": {
"server_url": "http://127.0.0.1:9110"
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 300000
}

View File

@@ -1,32 +0,0 @@
{
"agent": {
"type": "orchestrator-executor",
"orchestrator": {
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1"
},
"executor": {
"provider": "clado-action",
"model": "qwen3-vl-30b-a3b-instruct",
"apiKey": "",
"baseUrl": "https://clado-ai--clado-browseros-action-actionmodel-generate.modal.run"
}
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/orchestrator-executor-clado-webvoyager-test",
"num_workers": 3,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -9,12 +9,20 @@
"turnLimit": 100
},
"dataset": "../data/test-set.jsonl",
"output_dir": "../results/gemini-computer-use-test-set2",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110"
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",

View File

@@ -6,11 +6,20 @@
"apiKey": "OPENROUTER_API_KEY"
},
"dataset": "../data/mind2web.jsonl",
"output_dir": "../results/mind2web-full",
"num_workers": 5,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110"
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",

View File

@@ -8,16 +8,20 @@
"supportsImages": true
},
"dataset": "../data/webvoyager.jsonl",
"output_dir": "../results/webvoyager-cdp-server",
"num_workers": 3,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
"headless": false
},
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",

View File

@@ -9,14 +9,22 @@
"turnLimit": 100
},
"dataset": "../data/test-set.jsonl",
"output_dir": "../results/yutori-navigator",
"num_workers": 1,
"restart_server_per_task": true,
"browseros": {
"server_url": "http://127.0.0.1:9110"
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": false
},
"timeout_ms": 1200000,
"captcha": {
"api_key_env": "NOPECHA_API_KEY"
},
"graders": ["performance_grader"],
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1"
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -1,25 +0,0 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/tool-loop-webvoyager-test",
"num_workers": 3,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -1,25 +0,0 @@
{
"agent": {
"type": "single",
"provider": "openai-compatible",
"model": "accounts/fireworks/models/kimi-k2p5",
"apiKey": "FIREWORKS_API_KEY",
"baseUrl": "https://api.fireworks.ai/inference/v1",
"supportsImages": true
},
"dataset": "../data/webvoyager_e2e_test.jsonl",
"output_dir": "../results/webvoyager-test",
"num_workers": 3,
"browseros": {
"server_url": "http://127.0.0.1:9110",
"base_cdp_port": 9010,
"base_server_port": 9110,
"base_extension_port": 9310,
"load_extensions": false,
"headless": true
},
"grader_api_key_env": "OPENROUTER_API_KEY",
"grader_base_url": "https://openrouter.ai/api/v1",
"grader_model": "openai/gpt-4.1",
"timeout_ms": 1200000
}

View File

@@ -0,0 +1,220 @@
/**
* Test script for Clado API endpoints (grounding + action models)
*
* Usage:
* bun apps/eval/scripts/test-clado-api.ts [screenshot-path]
*
* If no screenshot provided, captures one from a running BrowserOS server.
*/
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'
const ACTION_URL =
'https://clado-ai--clado-browseros-action-actionmodel-generate.modal.run'
const ACTION_HEALTH_URL =
'https://clado-ai--clado-browseros-action-actionmodel-health.modal.run'
const GROUNDING_URL =
'https://clado-ai--clado-browseros-grounding-groundingmodel-generate.modal.run'
const GROUNDING_HEALTH_URL =
'https://clado-ai--clado-browseros-grounding-groundingmodel-health.modal.run'
async function checkHealth(name: string, url: string): Promise<boolean> {
console.log(`\n--- ${name} health check ---`)
console.log(` URL: ${url}`)
const start = performance.now()
try {
const resp = await fetch(url, { signal: AbortSignal.timeout(30_000) })
const elapsed = ((performance.now() - start) / 1000).toFixed(2)
const body = await resp.text()
console.log(` Status: ${resp.status} (${elapsed}s)`)
console.log(` Body: ${body.slice(0, 200)}`)
return resp.ok
} catch (err) {
const elapsed = ((performance.now() - start) / 1000).toFixed(2)
console.log(
` FAILED (${elapsed}s): ${err instanceof Error ? err.message : err}`,
)
return false
}
}
async function testGenerate(
name: string,
url: string,
payload: Record<string, unknown>,
): Promise<Record<string, unknown> | null> {
console.log(`\n--- ${name} generate ---`)
console.log(` URL: ${url}`)
console.log(` Instruction: ${payload.instruction}`)
console.log(
` Image size: ${((payload.image_base64 as string).length / 1024).toFixed(0)} KB (base64)`,
)
if (payload.history) console.log(` History: ${payload.history}`)
const start = performance.now()
try {
const resp = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
signal: AbortSignal.timeout(120_000),
})
const elapsed = ((performance.now() - start) / 1000).toFixed(2)
if (!resp.ok) {
const body = await resp.text()
console.log(` FAILED: HTTP ${resp.status} (${elapsed}s)`)
console.log(` Body: ${body.slice(0, 400)}`)
return null
}
const result = (await resp.json()) as Record<string, unknown>
console.log(` Status: ${resp.status} (${elapsed}s)`)
console.log(` Action: ${result.action}`)
if (result.x !== null && result.x !== undefined)
console.log(` Coordinates: (${result.x}, ${result.y})`)
if (result.text)
console.log(` Text: ${(result.text as string).slice(0, 100)}`)
if (result.key) console.log(` Key: ${result.key}`)
if (result.inference_time_seconds)
console.log(` Inference: ${result.inference_time_seconds}s`)
// Show thinking if present
const raw = result.raw_response as string | undefined
if (raw) {
const thinkMatch = raw.match(/<thinking>([\s\S]*?)<\/thinking>/)
if (thinkMatch) {
const thinking = thinkMatch[1].trim()
console.log(
` Thinking: ${thinking.slice(0, 200)}${thinking.length > 200 ? '...' : ''}`,
)
}
}
return result
} catch (err) {
const elapsed = ((performance.now() - start) / 1000).toFixed(2)
console.log(
` FAILED (${elapsed}s): ${err instanceof Error ? err.message : err}`,
)
return null
}
}
async function loadScreenshot(path?: string): Promise<string> {
if (path) {
const resolved = resolve(path)
console.log(`Loading screenshot: ${resolved}`)
const data = await readFile(resolved)
return data.toString('base64')
}
// Try to capture from a running BrowserOS server
const serverUrl = process.env.BROWSEROS_URL || 'http://127.0.0.1:9110'
console.log(
`No screenshot path provided. Trying to capture from ${serverUrl}...`,
)
const { Client } = await import('@modelcontextprotocol/sdk/client/index.js')
const { StreamableHTTPClientTransport } = await import(
'@modelcontextprotocol/sdk/client/streamableHttp.js'
)
const client = new Client({ name: 'clado-test', version: '1.0.0' })
const transport = new StreamableHTTPClientTransport(
new URL(`${serverUrl}/mcp`),
{ requestInit: { headers: { 'X-BrowserOS-Source': 'sdk-internal' } } },
)
try {
await client.connect(transport)
const result = (await client.callTool({
name: 'take_screenshot',
arguments: { format: 'png', page: 1 },
})) as { content: Array<{ type: string; data?: string }> }
const imageContent = result.content?.find((c) => c.type === 'image')
if (!imageContent?.data)
throw new Error('No image data in screenshot response')
console.log(
`Captured screenshot (${(imageContent.data.length / 1024).toFixed(0)} KB base64)`,
)
return imageContent.data
} finally {
try {
await transport.close()
} catch {}
}
}
async function main() {
const screenshotPath = process.argv[2]
console.log('=== Clado API Test ===\n')
// Health checks (parallel)
const [actionHealthy, groundingHealthy] = await Promise.all([
checkHealth('Action Model', ACTION_HEALTH_URL),
checkHealth('Grounding Model', GROUNDING_HEALTH_URL),
])
if (!actionHealthy && !groundingHealthy) {
console.log('\nBoth endpoints are down. Exiting.')
process.exit(1)
}
// Load screenshot
let imageBase64: string
try {
imageBase64 = await loadScreenshot(screenshotPath)
} catch (err) {
console.log(
`\nFailed to load screenshot: ${err instanceof Error ? err.message : err}`,
)
console.log(
'Provide a screenshot path: bun apps/eval/scripts/test-clado-api.ts path/to/screenshot.png',
)
process.exit(1)
}
const instruction = 'Click on the search button or search bar'
// Test grounding model
if (groundingHealthy) {
await testGenerate('Grounding Model', GROUNDING_URL, {
instruction,
image_base64: imageBase64,
})
} else {
console.log('\nSkipping grounding model (unhealthy)')
}
// Test action model (no history)
if (actionHealthy) {
const result = await testGenerate('Action Model (step 1)', ACTION_URL, {
instruction,
image_base64: imageBase64,
history: 'None',
})
// Test action model with history (simulate multi-turn)
if (result && result.action === 'click') {
await testGenerate('Action Model (step 2, with history)', ACTION_URL, {
instruction: 'Type "hello world" in the search bar',
image_base64: imageBase64,
history: `click(${result.x}, ${result.y})`,
})
}
} else {
console.log('\nSkipping action model (unhealthy)')
}
console.log('\n=== Done ===')
}
main().catch((err) => {
console.error('Fatal:', err)
process.exit(1)
})

View File

@@ -47,7 +47,7 @@ interface RunSummary {
runId: string
configName: string
date: string
passRate: number
avgScore: number
total: number
completed: number
failed: number
@@ -135,20 +135,20 @@ const runs: RunSummary[] = manifests
const failed = m.tasks.filter((t) => t.status === 'failed').length
const timeout = m.tasks.filter((t) => t.status === 'timeout').length
let graded = 0
let passed = 0
let scoredCount = 0
let scoreSum = 0
for (const task of m.tasks) {
if (!task.graderResults) continue
for (const name of PASS_FAIL_GRADER_ORDER) {
if (task.graderResults[name]) {
graded++
if (task.graderResults[name].pass) passed++
scoredCount++
scoreSum += task.graderResults[name].score ?? 0
break
}
}
}
const passRate = graded > 0 ? passed / graded : 0
const avgScore = scoredCount > 0 ? (scoreSum / scoredCount) * 100 : 0
const durations = m.tasks
.filter((t) => t.durationMs > 0)
.map((t) => t.durationMs)
@@ -170,7 +170,7 @@ const runs: RunSummary[] = manifests
runId: m.runId,
configName,
date,
passRate,
avgScore,
total,
completed,
failed,
@@ -242,7 +242,7 @@ const html = `<!DOCTYPE html>
.stat-value.big { font-size: 2.5rem; font-weight: 700; }
.pass { color: #3fb950; }
.fail { color: #f85149; }
.neutral { color: #8b949e; }
.neutral { color: #f0883e; }
.trend-up { color: #3fb950; }
.trend-down { color: #f85149; }
.trend-flat { color: #8b949e; }
@@ -314,7 +314,7 @@ const html = `<!DOCTYPE html>
<th>Model</th>
<th>Dataset</th>
<th>Architecture</th>
<th>Pass Rate</th>
<th>Score</th>
<th>Tasks</th>
<th>Timeout</th>
<th>Avg Duration</th>
@@ -327,7 +327,6 @@ const html = `<!DOCTYPE html>
.reverse()
.map((r) => {
const viewerUrl = `viewer.html?run=${encodeURIComponent(r.runId)}`
const passed = Math.round(r.passRate * r.total)
const archLabel =
r.agentType === 'orchestrator-executor'
? 'Orch-Exec'
@@ -342,7 +341,7 @@ const html = `<!DOCTYPE html>
<td class="mono" style="max-width:200px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;" title="${escHtml(r.model)}">${escHtml(r.model)}</td>
<td>${escHtml(r.dataset)}</td>
<td>${escHtml(archLabel)}</td>
<td class="${r.passRate >= 0.7 ? 'pass' : r.passRate >= 0.4 ? 'neutral' : 'fail'}">${(r.passRate * 100).toFixed(1)}% <span style="color:#6e7681;font-size:11px;">(${passed}/${r.total})</span></td>
<td class="${r.avgScore >= 75 ? 'pass' : r.avgScore >= 40 ? 'neutral' : 'fail'}">${r.avgScore.toFixed(1)}%</td>
<td>${r.total}</td>
<td class="${r.timeout > 0 ? 'neutral' : ''}">${r.timeout}</td>
<td>${(r.avgDurationMs / 1000).toFixed(0)}s</td>
@@ -386,10 +385,12 @@ const html = `<!DOCTYPE html>
: latest.agentType === 'single' ? 'Single Agent (Tool Loop)'
: latest.agentType === 'gemini-computer-use' ? 'Gemini Computer Use'
: latest.agentType || 'Unknown';
var scoreColor = latest.avgScore >= 75 ? '#3fb950' : latest.avgScore >= 40 ? '#f0883e' : '#f85149';
el.innerHTML =
'<div class="config-detail"><span class="cd-label">Architecture</span><span class="cd-value">' + archLabel + '</span></div>' +
'<div class="config-detail"><span class="cd-label">Model</span><span class="cd-value">' + (latest.model || 'unknown') + '</span></div>' +
'<div class="config-detail"><span class="cd-label">Dataset</span><span class="cd-value">' + (latest.dataset || 'unknown') + '</span></div>' +
'<div class="config-detail"><span class="cd-label">Latest Score</span><span class="cd-value" style="color:' + scoreColor + ';">' + latest.avgScore.toFixed(1) + '%</span></div>' +
'<div class="config-detail"><span class="cd-label">Tasks</span><span class="cd-value">' + latest.total + '</span></div>' +
'<div class="config-detail"><span class="cd-label">Runs</span><span class="cd-value">' + runs.length + '</span></div>';
}
@@ -400,15 +401,16 @@ const html = `<!DOCTYPE html>
if (runs.length === 0) { el.innerHTML = ''; return; }
var latest = runs[runs.length - 1];
var prev = runs.length >= 2 ? runs[runs.length - 2] : null;
var best = Math.max.apply(null, runs.map(function(r) { return r.passRate; }));
var delta = prev ? latest.passRate - prev.passRate : 0;
var best = Math.max.apply(null, runs.map(function(r) { return r.avgScore; }));
var delta = prev ? latest.avgScore - prev.avgScore : 0;
var sign = delta > 0 ? '+' : '';
var trendCls = delta > 0 ? 'trend-up' : delta < 0 ? 'trend-down' : 'trend-flat';
var latestColor = latest.avgScore >= 75 ? 'pass' : latest.avgScore >= 40 ? 'neutral' : 'fail';
el.innerHTML =
'<div class="stat-card"><div class="stat-label">Latest Pass Rate</div><div class="stat-value big ' + (latest.passRate >= 0.7 ? 'pass' : 'fail') + '">' + (latest.passRate * 100).toFixed(1) + '%</div></div>' +
'<div class="stat-card"><div class="stat-label">Trend</div><div class="stat-value ' + trendCls + '">' + (prev ? sign + (delta * 100).toFixed(1) + ' pp' : 'N/A') + '</div></div>' +
'<div class="stat-card"><div class="stat-label">Best Score</div><div class="stat-value pass">' + (best * 100).toFixed(1) + '%</div></div>' +
'<div class="stat-card"><div class="stat-label">Latest Score</div><div class="stat-value big ' + latestColor + '">' + latest.avgScore.toFixed(1) + '%</div></div>' +
'<div class="stat-card"><div class="stat-label">Trend</div><div class="stat-value ' + trendCls + '">' + (prev ? sign + delta.toFixed(1) + ' pp' : 'N/A') + '</div></div>' +
'<div class="stat-card"><div class="stat-label">Best Score</div><div class="stat-value pass">' + best.toFixed(1) + '%</div></div>' +
'<div class="stat-card"><div class="stat-label">Avg Duration</div><div class="stat-value">' + (latest.avgDurationMs / 1000).toFixed(0) + 's</div></div>' +
'<div class="stat-card"><div class="stat-label">Runs</div><div class="stat-value">' + runs.length + '</div></div>';
}
@@ -436,7 +438,7 @@ const html = `<!DOCTYPE html>
return;
}
var scores = runs.map(function(r) { return r.passRate * 100; });
var scores = runs.map(function(r) { return r.avgScore; });
var minY = Math.max(0, Math.floor(Math.min.apply(null, scores) / 10) * 10 - 10);
var maxY = Math.min(100, Math.ceil(Math.max.apply(null, scores) / 10) * 10 + 10);
if (minY === maxY) { minY = Math.max(0, minY - 10); maxY = Math.min(100, maxY + 10); }
@@ -463,7 +465,7 @@ const html = `<!DOCTYPE html>
ctx.strokeStyle = '#58a6ff'; ctx.lineWidth = 2; ctx.beginPath();
runs.forEach(function(r, i) {
var px = pad.left + (runs.length === 1 ? plotW / 2 : (i / (runs.length - 1)) * plotW);
var py2 = pad.top + plotH - ((r.passRate * 100 - minY) / (maxY - minY)) * plotH;
var py2 = pad.top + plotH - ((r.avgScore - minY) / (maxY - minY)) * plotH;
if (i === 0) ctx.moveTo(px, py2); else ctx.lineTo(px, py2);
});
ctx.stroke();
@@ -471,10 +473,10 @@ const html = `<!DOCTYPE html>
// Dots
runs.forEach(function(r, i) {
var px = pad.left + (runs.length === 1 ? plotW / 2 : (i / (runs.length - 1)) * plotW);
var py2 = pad.top + plotH - ((r.passRate * 100 - minY) / (maxY - minY)) * plotH;
var py2 = pad.top + plotH - ((r.avgScore - minY) / (maxY - minY)) * plotH;
dotPositions.push({ x: px, y: py2, run: r });
ctx.beginPath(); ctx.arc(px, py2, 4, 0, Math.PI * 2);
ctx.fillStyle = r.passRate >= 0.7 ? '#3fb950' : '#f85149';
ctx.fillStyle = r.avgScore >= 75 ? '#3fb950' : r.avgScore >= 40 ? '#f0883e' : '#f85149';
ctx.fill(); ctx.strokeStyle = '#0d1117'; ctx.lineWidth = 2; ctx.stroke();
});
}
@@ -491,11 +493,10 @@ const html = `<!DOCTYPE html>
if (closest && closestDist < 40) {
var r = closest.run;
var passed = Math.round(r.passRate * r.total);
document.getElementById('tt-date').textContent = r.date;
document.getElementById('tt-score').textContent = (r.passRate * 100).toFixed(1) + '%';
document.getElementById('tt-score').style.color = r.passRate >= 0.7 ? '#3fb950' : '#f85149';
document.getElementById('tt-detail').textContent = passed + '/' + r.total + ' pass \\u00B7 ' + (r.avgDurationMs / 1000).toFixed(0) + 's avg \\u00B7 ' + r.model;
document.getElementById('tt-score').textContent = r.avgScore.toFixed(1) + '%';
document.getElementById('tt-score').style.color = r.avgScore >= 75 ? '#3fb950' : r.avgScore >= 40 ? '#f0883e' : '#f85149';
document.getElementById('tt-detail').textContent = 'score ' + r.avgScore.toFixed(1) + '% \\u00B7 ' + r.total + ' tasks \\u00B7 ' + (r.avgDurationMs / 1000).toFixed(0) + 's avg \\u00B7 ' + r.model;
tooltip.style.display = 'block';
var tx = closest.x + 12, ty = closest.y - 50;
@@ -508,7 +509,7 @@ const html = `<!DOCTYPE html>
ctx.beginPath(); ctx.arc(closest.x, closest.y, 7, 0, Math.PI * 2);
ctx.fillStyle = 'rgba(88, 166, 255, 0.3)'; ctx.fill();
ctx.beginPath(); ctx.arc(closest.x, closest.y, 5, 0, Math.PI * 2);
ctx.fillStyle = r.passRate >= 0.7 ? '#3fb950' : '#f85149'; ctx.fill();
ctx.fillStyle = r.avgScore >= 75 ? '#3fb950' : r.avgScore >= 40 ? '#f0883e' : '#f85149'; ctx.fill();
ctx.strokeStyle = '#e6edf3'; ctx.lineWidth = 2; ctx.stroke();
canvas.style.cursor = 'pointer';
} else {
@@ -584,7 +585,7 @@ console.log(` View at: ${cdnBaseUrl}/report.html`)
// Print summary
console.log('\nScore trend:')
for (const run of runs.slice(-10)) {
const bar = '\u2588'.repeat(Math.round(run.passRate * 20))
const pct = (run.passRate * 100).toFixed(0).padStart(3)
const bar = '\u2588'.repeat(Math.round(run.avgScore / 5))
const pct = run.avgScore.toFixed(0).padStart(3)
console.log(` ${run.date} ${pct}% ${bar}`)
}

View File

@@ -11,6 +11,7 @@
import type { ResolvedAgentConfig } from '@browseros/server/agent/types'
import { Browser } from '@browseros/server/browser'
import { CdpBackend } from '@browseros/server/browser/backends/cdp'
import { CaptchaWaiter } from '../../capture/captcha-waiter'
import { DEFAULT_TIMEOUT_MS } from '../../constants'
import type {
EvalConfig,
@@ -161,6 +162,13 @@ export class OrchestratorExecutorEvaluator implements AgentEvaluator {
const browser = new Browser(cdp, CONTROLLER_STUB)
capture.screenshot.setBrowser(browser)
const captchaWaiter = config.captcha
? new CaptchaWaiter({
waitTimeoutMs: config.captcha.wait_timeout_ms,
pollIntervalMs: config.captcha.poll_interval_ms,
})
: null
try {
// Build capture callbacks (same pattern as single-agent.ts)
const callbacks: ExecutorCallbacks = {
@@ -172,6 +180,12 @@ export class OrchestratorExecutorEvaluator implements AgentEvaluator {
},
onToolCallFinish: async () => {
try {
if (captchaWaiter) {
await captchaWaiter.waitIfCaptchaPresent(
browser,
capture.getActivePageId(),
)
}
const screenshotNum = await capture.screenshot.capture(
capture.getActivePageId(),
)

View File

@@ -7,6 +7,7 @@ import type { ResolvedAgentConfig } from '@browseros/server/agent/types'
import { Browser } from '@browseros/server/browser'
import { CdpBackend } from '@browseros/server/browser/backends/cdp'
import { registry } from '@browseros/server/tools/registry'
import { CaptchaWaiter } from '../capture/captcha-waiter'
import { DEFAULT_TIMEOUT_MS } from '../constants'
import type { EvalConfig, TaskMetadata } from '../types'
import { resolveProviderConfig } from '../utils/resolve-provider-config'
@@ -78,6 +79,13 @@ export class SingleAgentEvaluator implements AgentEvaluator {
}
: undefined
const captchaWaiter = config.captcha
? new CaptchaWaiter({
waitTimeoutMs: config.captcha.wait_timeout_ms,
pollIntervalMs: config.captcha.poll_interval_ms,
})
: null
let agent: AiSdkAgent | null = null
try {
@@ -112,6 +120,12 @@ export class SingleAgentEvaluator implements AgentEvaluator {
experimental_onToolCallFinish: async () => {
try {
if (captchaWaiter) {
await captchaWaiter.waitIfCaptchaPresent(
browser,
capture.getActivePageId(),
)
}
const screenshotNum = await capture.screenshot.capture(
capture.getActivePageId(),
)

View File

@@ -0,0 +1,115 @@
import type { Browser } from '@browseros/server/browser'
export interface CaptchaWaitResult {
detected: boolean
type: 'recaptcha' | 'hcaptcha' | 'turnstile' | 'none'
solved: boolean
waitDurationMs: number
}
interface CaptchaWaiterConfig {
waitTimeoutMs: number
pollIntervalMs: number
}
const DETECTION_SCRIPT = `(() => {
const recaptcha = document.querySelector('iframe[src*="recaptcha"]')
if (recaptcha) {
const response = document.getElementById('g-recaptcha-response')
return { type: 'recaptcha', solved: !!(response && response.value) }
}
const hcaptcha = document.querySelector('iframe[src*="hcaptcha"]')
if (hcaptcha) {
const response = document.querySelector('[name="h-captcha-response"]')
return { type: 'hcaptcha', solved: !!(response && response.value) }
}
const turnstile = document.querySelector('iframe[src*="challenges.cloudflare.com"]')
if (turnstile) {
const response = document.querySelector('[name="cf-turnstile-response"]')
return { type: 'turnstile', solved: !!(response && response.value) }
}
return { type: 'none', solved: false }
})()`
export class CaptchaWaiter {
private readonly config: CaptchaWaiterConfig
constructor(config: CaptchaWaiterConfig) {
this.config = config
}
async waitIfCaptchaPresent(
browser: Browser,
pageId: number,
): Promise<CaptchaWaitResult> {
const start = Date.now()
try {
const initial = await this.detect(browser, pageId)
if (initial.type === 'none') {
return {
detected: false,
type: 'none',
solved: false,
waitDurationMs: Date.now() - start,
}
}
if (initial.solved) {
return {
detected: true,
type: initial.type,
solved: true,
waitDurationMs: Date.now() - start,
}
}
// Poll until solved or timeout
while (Date.now() - start < this.config.waitTimeoutMs) {
await sleep(this.config.pollIntervalMs)
const check = await this.detect(browser, pageId)
if (check.solved || check.type === 'none') {
return {
detected: true,
type: initial.type,
solved: check.solved,
waitDurationMs: Date.now() - start,
}
}
}
return {
detected: true,
type: initial.type,
solved: false,
waitDurationMs: Date.now() - start,
}
} catch {
return {
detected: false,
type: 'none',
solved: false,
waitDurationMs: Date.now() - start,
}
}
}
private async detect(
browser: Browser,
pageId: number,
): Promise<{ type: CaptchaWaitResult['type']; solved: boolean }> {
const result = await browser.evaluate(pageId, DETECTION_SCRIPT)
if (result.error || !result.value) {
return { type: 'none', solved: false }
}
const val = result.value as { type: string; solved: boolean }
return {
type: (val.type as CaptchaWaitResult['type']) ?? 'none',
solved: val.solved ?? false,
}
}
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms))
}

View File

@@ -1,4 +1,5 @@
export { callMcpTool } from '../utils/mcp-client'
export { CaptchaWaiter } from './captcha-waiter'
export { CaptureContext } from './context'
export { MessageLogger } from './message-logger'
export { ScreenshotCapture } from './screenshot'

View File

@@ -1221,6 +1221,12 @@
const graders = task.graderResults || {};
const keys = Object.keys(graders);
if (keys.length === 0) return { label: '', cls: '' };
const firstKey = keys[0];
const score = graders[firstKey].score;
if (typeof score === 'number') {
const pct = Math.round(score * 100);
return { label: pct + '%', cls: pct >= 75 ? 'pass' : 'fail' };
}
const anyPass = keys.some((k) => graders[k].pass);
return { label: anyPass ? 'PASS' : 'FAIL', cls: anyPass ? 'pass' : 'fail' };
}

View File

@@ -14,7 +14,13 @@
* Each worker gets isolated ports: base + workerIndex offset.
*/
import { existsSync, mkdtempSync, rmSync } from 'node:fs'
import {
existsSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { type Subprocess, spawn, spawnSync } from 'bun'
@@ -37,7 +43,7 @@ const BROWSEROS_BINARY =
const CONTROLLER_EXT_DIR = join(MONOREPO_ROOT, 'apps/controller-ext/dist')
const CAPTCHA_EXT_DIR = join(
dirname(fileURLToPath(import.meta.url)),
'../../../extensions/nopecha',
'../../extensions/nopecha',
)
export class BrowserOSAppManager {
@@ -149,7 +155,6 @@ export class BrowserOSAppManager {
'--use-mock-keychain',
'--disable-browseros-server',
'--disable-browseros-extensions',
'--incognito',
...(this.headless ? ['--headless=new'] : []),
'--window-size=1440,900',
`--remote-debugging-port=${cdp}`,
@@ -319,4 +324,22 @@ export class BrowserOSAppManager {
})
return (result.stdout?.toString().trim() ?? '').length > 0
}
/**
* Patch NopeCHA extension manifest with API key.
* Call once before launching any workers — the extension directory is shared.
*/
static patchNopechaApiKey(apiKey: string): void {
const manifestPath = join(CAPTCHA_EXT_DIR, 'manifest.json')
if (!existsSync(manifestPath)) {
console.log(
'[BROWSEROS] NopeCHA extension not found, skipping API key patch',
)
return
}
const manifest = JSON.parse(readFileSync(manifestPath, 'utf-8'))
manifest.nopecha = { ...manifest.nopecha, key: apiKey }
writeFileSync(manifestPath, JSON.stringify(manifest, null, 2))
console.log('[BROWSEROS] NopeCHA API key patched')
}
}

View File

@@ -93,6 +93,15 @@ export class ParallelExecutor {
BrowserOSAppManager.buildExtensions()
}
// Patch NopeCHA API key before launching any workers
const captchaConfig = this.config.config.captcha
if (captchaConfig) {
const apiKey = process.env[captchaConfig.api_key_env]
if (apiKey) {
BrowserOSAppManager.patchNopechaApiKey(apiKey)
}
}
this.queue = new TaskQueue(tasks)
const totalTasks = tasks.length

View File

@@ -71,6 +71,13 @@ export const EvalConfigSchema = z.object({
grader_api_key_env: z.string().optional(),
grader_base_url: z.string().url().optional(),
timeout_ms: z.number().int().min(30000).max(3600000).optional(),
captcha: z
.object({
api_key_env: z.string().default('NOPECHA_API_KEY'),
wait_timeout_ms: z.number().int().min(1000).max(120000).default(30000),
poll_interval_ms: z.number().int().min(200).max(5000).default(1000),
})
.optional(),
})
export type SingleAgentConfig = z.infer<typeof SingleAgentConfigSchema>

View File

@@ -0,0 +1,136 @@
import { beforeEach, describe, expect, it, mock } from 'bun:test'
import { CaptchaWaiter } from '../../src/capture/captcha-waiter'
function createMockBrowser(
evaluateResults: Array<{ value?: unknown; error?: string }>,
) {
let callIndex = 0
return {
evaluate: mock(async (_page: number, _expr: string) => {
const result = evaluateResults[callIndex] ?? evaluateResults.at(-1)!
callIndex++
return result
}),
} as any
}
describe('CaptchaWaiter', () => {
let waiter: CaptchaWaiter
beforeEach(() => {
waiter = new CaptchaWaiter({
waitTimeoutMs: 5000,
pollIntervalMs: 100,
})
})
it('returns immediately when no CAPTCHA detected', async () => {
const browser = createMockBrowser([
{ value: { type: 'none', solved: false } },
])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(false)
expect(result.type).toBe('none')
expect(result.solved).toBe(false)
expect(browser.evaluate).toHaveBeenCalledTimes(1)
})
it('returns immediately when CAPTCHA already solved', async () => {
const browser = createMockBrowser([
{ value: { type: 'recaptcha', solved: true } },
])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(true)
expect(result.type).toBe('recaptcha')
expect(result.solved).toBe(true)
expect(browser.evaluate).toHaveBeenCalledTimes(1)
})
it('polls until CAPTCHA is solved', async () => {
const browser = createMockBrowser([
{ value: { type: 'hcaptcha', solved: false } },
{ value: { type: 'hcaptcha', solved: false } },
{ value: { type: 'hcaptcha', solved: true } },
])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(true)
expect(result.type).toBe('hcaptcha')
expect(result.solved).toBe(true)
expect(browser.evaluate).toHaveBeenCalledTimes(3)
})
it('polls until CAPTCHA disappears', async () => {
const browser = createMockBrowser([
{ value: { type: 'turnstile', solved: false } },
{ value: { type: 'turnstile', solved: false } },
{ value: { type: 'none', solved: false } },
])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(true)
expect(result.type).toBe('turnstile')
expect(result.solved).toBe(false)
expect(browser.evaluate).toHaveBeenCalledTimes(3)
})
it('times out if CAPTCHA never solves', async () => {
const shortWaiter = new CaptchaWaiter({
waitTimeoutMs: 300,
pollIntervalMs: 100,
})
const browser = createMockBrowser([
{ value: { type: 'recaptcha', solved: false } },
])
const result = await shortWaiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(true)
expect(result.type).toBe('recaptcha')
expect(result.solved).toBe(false)
expect(result.waitDurationMs).toBeGreaterThanOrEqual(250)
})
it('handles browser.evaluate errors gracefully', async () => {
const browser = createMockBrowser([{ error: 'Page crashed' }])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(false)
expect(result.type).toBe('none')
expect(result.solved).toBe(false)
})
it('handles browser.evaluate throwing', async () => {
const browser = {
evaluate: mock(async () => {
throw new Error('Connection lost')
}),
} as any
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.detected).toBe(false)
expect(result.type).toBe('none')
expect(result.solved).toBe(false)
})
it('tracks wait duration', async () => {
const browser = createMockBrowser([
{ value: { type: 'recaptcha', solved: false } },
{ value: { type: 'recaptcha', solved: false } },
{ value: { type: 'recaptcha', solved: true } },
])
const result = await waiter.waitIfCaptchaPresent(browser, 1)
expect(result.waitDurationMs).toBeGreaterThanOrEqual(150)
})
})

View File

@@ -0,0 +1,172 @@
/**
* End-to-end test for CAPTCHA solver integration.
*
* Runs a single eval task against Google's reCAPTCHA demo page:
* 1. Launches BrowserOS (headed) with NopeCHA extension loaded
* 2. Agent navigates to reCAPTCHA demo, fills form
* 3. CaptchaWaiter polls until NopeCHA solves the CAPTCHA
* 4. Screenshot is captured AFTER solve
* 5. Verifies: task completed, screenshots exist, metadata saved
*
* Prerequisites:
* - NOPECHA_API_KEY env var set
* - FIREWORKS_API_KEY env var set (or swap agent config)
* - NopeCHA extension at extensions/nopecha/ (run the install step from CI)
* - BrowserOS binary available
*
* Run:
* bun --env-file=apps/eval/.env.development apps/eval/tests/e2e/captcha-e2e.ts
*/
import { existsSync, readdirSync, readFileSync, rmSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { BrowserOSAppManager } from '../../src/runner/browseros-app-manager'
import { createTaskExecutor } from '../../src/runner/task-executor'
import { EvalConfigSchema } from '../../src/types/config'
import { TaskSchema } from '../../src/types/task'
const HERE = dirname(fileURLToPath(import.meta.url))
const OUTPUT_DIR = join(HERE, 'results')
const EVAL_CONFIG = {
agent: {
type: 'single' as const,
provider: 'openai-compatible' as const,
model: 'accounts/fireworks/models/kimi-k2p5',
apiKey: 'FIREWORKS_API_KEY',
baseUrl: 'https://api.fireworks.ai/inference/v1',
supportsImages: true,
},
dataset: 'inline',
num_workers: 1,
restart_server_per_task: true,
browseros: {
server_url: 'http://127.0.0.1:9110',
base_cdp_port: 9010,
base_server_port: 9110,
base_extension_port: 9310,
load_extensions: false,
headless: false,
},
captcha: { api_key_env: 'NOPECHA_API_KEY' },
timeout_ms: 120000,
}
const TASK = {
query_id: 'captcha-e2e-1',
dataset: 'captcha-test',
query:
"Go to the Google reCAPTCHA demo page. Wait for the CAPTCHA to appear. Click the 'I'm not a robot' checkbox. Once the CAPTCHA is solved, fill in the 'Name' field with 'Test User' and the 'Email' field with 'test@example.com'. Then click the Submit button.",
start_url: 'https://www.google.com/recaptcha/api2/demo',
metadata: { original_task_id: 'captcha-e2e-1' },
}
// ── Helpers ────────────────────────────────────────────────────────────
function log(msg: string) {
console.log(`[captcha-e2e] ${msg}`)
}
function fail(msg: string): never {
console.error(`\n[FAIL] ${msg}`)
process.exit(1)
}
function pass(msg: string) {
console.log(`\n[PASS] ${msg}`)
}
function preflight() {
if (!process.env.NOPECHA_API_KEY) {
fail('NOPECHA_API_KEY env var not set')
}
if (!process.env.FIREWORKS_API_KEY) {
fail('FIREWORKS_API_KEY env var not set — needed for the agent LLM')
}
const extDir = join(HERE, '../../extensions/nopecha')
if (!existsSync(join(extDir, 'manifest.json'))) {
fail(`NopeCHA extension not found at ${extDir}`)
}
}
// ── Main ──────────────────────────────────────────────────────────────
async function main() {
preflight()
const config = EvalConfigSchema.parse(EVAL_CONFIG)
const task = TaskSchema.parse(TASK)
const taskDir = join(OUTPUT_DIR, task.query_id)
if (existsSync(taskDir)) {
rmSync(taskDir, { recursive: true, force: true })
}
const captcha = config.captcha
if (!captcha) fail('captcha config block missing')
const apiKey = process.env[captcha.api_key_env]
if (!apiKey) fail(`${captcha.api_key_env} env var is empty`)
BrowserOSAppManager.patchNopechaApiKey(apiKey)
const app = new BrowserOSAppManager(
0,
{
cdp: config.browseros.base_cdp_port,
server: config.browseros.base_server_port,
extension: config.browseros.base_extension_port,
},
config.browseros.load_extensions,
config.browseros.headless,
)
try {
log('Starting BrowserOS stack (headed + NopeCHA extension)...')
await app.restart()
log(`BrowserOS ready at ${app.getServerUrl()}`)
const runConfig = {
...config,
browseros: { ...config.browseros, server_url: app.getServerUrl() },
}
const executor = createTaskExecutor(runConfig, OUTPUT_DIR, null)
log(`Running task: ${task.query_id}`)
log(` start_url: ${task.start_url}`)
const result = await executor.execute(task)
log(`\nTask status: ${result.status}`)
if (result.status === 'failed') {
const err = 'error' in result ? result.error : null
fail(`Task failed: ${err?.message ?? 'unknown error'}`)
}
const metadataPath = join(taskDir, 'metadata.json')
if (!existsSync(metadataPath)) fail('metadata.json not found')
const metadata = JSON.parse(readFileSync(metadataPath, 'utf-8'))
log(` Duration: ${metadata.total_duration_ms}ms`)
log(` Steps: ${metadata.total_steps}`)
log(` Termination: ${metadata.termination_reason}`)
const screenshotDir = join(taskDir, 'screenshots')
const screenshots = existsSync(screenshotDir)
? readdirSync(screenshotDir).filter((f) => f.endsWith('.png'))
: []
log(` Screenshots: ${screenshots.length}`)
if (screenshots.length === 0) fail('No screenshots captured')
pass(
`${screenshots.length} screenshots, ${metadata.total_steps} steps, ${metadata.total_duration_ms}ms`,
)
} finally {
log('Shutting down BrowserOS...')
await app.killApp()
}
}
main().catch((err) => {
console.error(err)
process.exit(1)
})

View File

@@ -11,12 +11,14 @@ import {
type ModelMessage,
stepCountIs,
ToolLoopAgent,
type ToolSet,
type UIMessage,
wrapLanguageModel,
} from 'ai'
import type { Browser } from '../browser/browser'
import type { KlavisClient } from '../lib/clients/klavis/klavis-client'
import { logger } from '../lib/logger'
import { metrics } from '../lib/metrics'
import { isSoulBootstrap, readSoul } from '../lib/soul'
import { buildSkillsCatalog } from '../skills/catalog'
import { loadSkills } from '../skills/loader'
@@ -114,7 +116,44 @@ export class AiSdkAgent {
klavisClient: config.klavisClient,
browserosId: config.browserosId,
})
const { clients, tools: externalMcpTools } = await createMcpClients(specs)
const { clients, tools: rawExternalMcpTools } =
await createMcpClients(specs)
// Wrap external MCP tools (Klavis, custom) with metrics
const externalMcpTools: ToolSet = {}
for (const [name, t] of Object.entries(rawExternalMcpTools)) {
const originalExecute = t.execute
externalMcpTools[name] = {
...t,
execute: originalExecute
? async (
...args: Parameters<NonNullable<typeof originalExecute>>
) => {
const startTime = performance.now()
try {
const result = await originalExecute(...args)
metrics.log('tool_executed', {
tool_name: name,
duration_ms: Math.round(performance.now() - startTime),
success: true,
source: 'chat',
})
return result
} catch (error) {
metrics.log('tool_executed', {
tool_name: name,
duration_ms: Math.round(performance.now() - startTime),
success: false,
error_message:
error instanceof Error ? error.message : String(error),
source: 'chat',
})
throw error
}
}
: undefined,
}
}
// Add filesystem tools (Pi coding agent) — skip in chat mode (read-only)
const filesystemTools = config.resolvedConfig.chatMode

View File

@@ -64,6 +64,7 @@ export function buildBrowserToolSet(
tool_name: def.name,
duration_ms: Math.round(performance.now() - startTime),
success: !result.isError,
source: 'chat',
})
return {
@@ -85,6 +86,7 @@ export function buildBrowserToolSet(
success: false,
error_message:
error instanceof Error ? error.message : 'Unknown error',
source: 'chat',
})
return {

View File

@@ -1,45 +0,0 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*/
import { LLM_PROVIDERS } from '@browseros/shared/schemas/llm'
import { createMiddleware } from 'hono/factory'
import type { RateLimiter } from '../../lib/rate-limiter/rate-limiter'
import type { ChatRequest } from '../types'
interface RateLimitMiddlewareDeps {
rateLimiter?: RateLimiter
browserosId?: string
}
type ChatValidationInput = {
in: { json: ChatRequest }
out: { json: ChatRequest }
}
export function createBrowserosRateLimitMiddleware(
deps: RateLimitMiddlewareDeps,
) {
return createMiddleware<object, '*', ChatValidationInput>(async (c, next) => {
const { rateLimiter, browserosId } = deps
if (!rateLimiter || !browserosId) {
return next()
}
const request = c.req.valid('json')
if (request.provider === LLM_PROVIDERS.BROWSEROS) {
rateLimiter.check(browserosId)
rateLimiter.record({
conversationId: request.conversationId,
browserosId,
provider: request.provider,
})
}
return next()
})
}

View File

@@ -5,10 +5,8 @@ import type { Browser } from '../../browser/browser'
import { KlavisClient } from '../../lib/clients/klavis/klavis-client'
import { logger } from '../../lib/logger'
import { metrics } from '../../lib/metrics'
import type { RateLimiter } from '../../lib/rate-limiter/rate-limiter'
import { Sentry } from '../../lib/sentry'
import type { ToolRegistry } from '../../tools/tool-registry'
import { createBrowserosRateLimitMiddleware } from '../middleware/rate-limit'
import { ChatService } from '../services/chat-service'
import { ChatRequestSchema } from '../types'
import { ConversationIdParamSchema } from '../utils/validation'
@@ -17,12 +15,11 @@ interface ChatRouteDeps {
browser: Browser
registry: ToolRegistry
browserosId?: string
rateLimiter?: RateLimiter
aiSdkDevtoolsEnabled?: boolean
}
export function createChatRoutes(deps: ChatRouteDeps) {
const { browserosId, rateLimiter } = deps
const { browserosId } = deps
const sessionStore = new SessionStore()
const klavisClient = new KlavisClient()
@@ -36,46 +33,41 @@ export function createChatRoutes(deps: ChatRouteDeps) {
})
return new Hono()
.post(
'/',
zValidator('json', ChatRequestSchema),
createBrowserosRateLimitMiddleware({ rateLimiter, browserosId }),
async (c) => {
const request = c.req.valid('json')
.post('/', zValidator('json', ChatRequestSchema), async (c) => {
const request = c.req.valid('json')
// Sentry + metrics (HTTP concerns only)
Sentry.getCurrentScope().setTag(
'request-type',
request.isScheduledTask ? 'schedule' : 'chat',
)
Sentry.setContext('request', {
provider: request.provider,
model: request.model,
baseUrl: request.baseUrl
? (() => {
try {
return new URL(request.baseUrl).origin
} catch {
return undefined
}
})()
: undefined,
})
// Sentry + metrics (HTTP concerns only)
Sentry.getCurrentScope().setTag(
'request-type',
request.isScheduledTask ? 'schedule' : 'chat',
)
Sentry.setContext('request', {
provider: request.provider,
model: request.model,
baseUrl: request.baseUrl
? (() => {
try {
return new URL(request.baseUrl).origin
} catch {
return undefined
}
})()
: undefined,
})
metrics.log('chat.request', {
provider: request.provider,
model: request.model,
})
metrics.log('chat.request', {
provider: request.provider,
model: request.model,
})
logger.info('Chat request received', {
conversationId: request.conversationId,
provider: request.provider,
model: request.model,
})
logger.info('Chat request received', {
conversationId: request.conversationId,
provider: request.provider,
model: request.model,
})
return service.processMessage(request, c.req.raw.signal)
},
)
return service.processMessage(request, c.req.raw.signal)
})
.delete(
'/:conversationId',
zValidator('param', ConversationIdParamSchema),

View File

@@ -50,7 +50,11 @@ export function createMcpRoutes(deps: McpRouteDeps) {
await mcpServer.connect(transport)
return transport.handleRequest(c)
} catch (error) {
Sentry.captureException(error)
Sentry.withScope((scope) => {
scope.setTag('route', 'mcp')
scope.setTag('scopeId', scopeId)
Sentry.captureException(error)
})
logger.error('Error handling MCP request', {
error: error instanceof Error ? error.message : String(error),
})

View File

@@ -19,6 +19,7 @@ import { KlavisClient } from '../lib/clients/klavis/klavis-client'
import { initializeOAuth } from '../lib/clients/oauth'
import { getDb } from '../lib/db'
import { logger } from '../lib/logger'
import { Sentry } from '../lib/sentry'
import { createChatRoutes } from './routes/chat'
import { createCreditsRoutes } from './routes/credits'
import { createGraphRoutes } from './routes/graph'
@@ -71,7 +72,6 @@ export async function createHttpServer(config: HttpServerConfig) {
browserosId,
executionDir,
resourcesDir,
rateLimiter,
version,
browser,
controller,
@@ -161,7 +161,6 @@ export async function createHttpServer(config: HttpServerConfig) {
browser,
registry,
browserosId,
rateLimiter,
aiSdkDevtoolsEnabled: config.aiSdkDevtoolsEnabled,
}),
)
@@ -196,6 +195,12 @@ export async function createHttpServer(config: HttpServerConfig) {
return c.json(error.toJSON(), error.statusCode as ContentfulStatusCode)
}
Sentry.withScope((scope) => {
scope.setTag('route', c.req.path)
scope.setTag('method', c.req.method)
Sentry.captureException(error)
})
logger.error('Unhandled Error', {
message: error.message,
stack: error.stack,

View File

@@ -111,7 +111,7 @@ export function registerKlavisTools(
metrics.log('tool_executed', {
tool_name: tool.name,
source: 'klavis',
source: 'mcp',
duration_ms: Math.round(performance.now() - startTime),
success: !result.isError,
})
@@ -123,7 +123,7 @@ export function registerKlavisTools(
metrics.log('tool_executed', {
tool_name: tool.name,
source: 'klavis',
source: 'mcp',
duration_ms: Math.round(performance.now() - startTime),
success: false,
error_message: errorText,

View File

@@ -25,6 +25,7 @@ export function registerTools(
tool_name: tool.name,
duration_ms: Math.round(performance.now() - startTime),
success: !result.isError,
source: 'mcp',
})
return {
@@ -40,6 +41,7 @@ export function registerTools(
duration_ms: Math.round(performance.now() - startTime),
success: false,
error_message: errorText,
source: 'mcp',
})
return {

View File

@@ -16,7 +16,6 @@ import { LLMConfigSchema } from '@browseros/shared/schemas/llm'
import { z } from 'zod'
import type { ControllerBackend } from '../browser/backends/controller'
import type { Browser } from '../browser/browser'
import type { RateLimiter } from '../lib/rate-limiter/rate-limiter'
import type { ToolRegistry } from '../tools/tool-registry'
// Re-export browser context types for consumers
@@ -99,8 +98,6 @@ export interface HttpServerConfig {
browserosId?: string
executionDir: string
resourcesDir: string
rateLimiter?: RateLimiter
codegenServiceUrl?: string
aiSdkDevtoolsEnabled?: boolean

View File

@@ -5,15 +5,6 @@
*/
import type { Database } from 'bun:sqlite'
// id is the conversation_id - using it as PK ensures same conversation is only counted once
const RATE_LIMITER_TABLE = `
CREATE TABLE IF NOT EXISTS rate_limiter (
id TEXT PRIMARY KEY,
browseros_id TEXT NOT NULL,
provider TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
)`
const IDENTITY_TABLE = `
CREATE TABLE IF NOT EXISTS identity (
id INTEGER PRIMARY KEY CHECK (id = 1),
@@ -36,7 +27,6 @@ CREATE TABLE IF NOT EXISTS oauth_tokens (
)`
export function initSchema(db: Database): void {
db.exec(RATE_LIMITER_TABLE)
db.exec(IDENTITY_TABLE)
db.exec(OAUTH_TOKENS_TABLE)
}

View File

@@ -1,32 +0,0 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*/
import { HttpAgentError } from '../../agent/errors'
export class RateLimitError extends HttpAgentError {
constructor(
public used: number,
public limit: number,
) {
super(
`Daily limit reached (${used}/${limit}). Add your own API key for unlimited usage. https://dub.sh/browseros-usage-limit`,
429,
'RATE_LIMIT_EXCEEDED',
)
}
override toJSON() {
return {
error: {
name: this.name,
message: this.message,
code: this.code,
statusCode: this.statusCode,
used: this.used,
limit: this.limit,
},
}
}
}

View File

@@ -1,55 +0,0 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*
* Rate limit configuration fetching from remote config service.
*/
import { RATE_LIMITS } from '@browseros/shared/constants/limits'
import { INLINED_ENV } from '../../env'
import { fetchBrowserOSConfig } from '../clients/gateway'
import { logger } from '../logger'
export async function fetchDailyRateLimit(
browserosId: string,
): Promise<number> {
if (process.env.NODE_ENV === 'test') {
logger.info('Test mode: rate limiting disabled')
return RATE_LIMITS.TEST_DAILY
}
if (process.env.NODE_ENV === 'development') {
logger.info('Dev mode: using dev rate limit', {
dailyRateLimit: RATE_LIMITS.DEV_DAILY,
})
return RATE_LIMITS.DEV_DAILY
}
const configUrl = INLINED_ENV.BROWSEROS_CONFIG_URL
if (!configUrl) {
logger.info('No BROWSEROS_CONFIG_URL, using default rate limit', {
dailyRateLimit: RATE_LIMITS.DEFAULT_DAILY,
})
return RATE_LIMITS.DEFAULT_DAILY
}
try {
const browserosConfig = await fetchBrowserOSConfig(configUrl, browserosId)
const defaultProvider = browserosConfig.providers.find(
(p) => p.name === 'default',
)
const dailyRateLimit =
defaultProvider?.dailyRateLimit ?? RATE_LIMITS.DEFAULT_DAILY
logger.info('Rate limit config fetched', { dailyRateLimit })
return dailyRateLimit
} catch (error) {
logger.warn('Failed to fetch rate limit config, using default', {
error: error instanceof Error ? error.message : String(error),
dailyRateLimit: RATE_LIMITS.DEFAULT_DAILY,
})
return RATE_LIMITS.DEFAULT_DAILY
}
}

View File

@@ -1,74 +0,0 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*/
import type { Database } from 'bun:sqlite'
import { RATE_LIMITS } from '@browseros/shared/constants/limits'
import { logger } from '../logger'
import { metrics } from '../metrics'
import { RateLimitError } from './errors'
export interface RecordParams {
conversationId: string
browserosId: string
provider: string
}
export class RateLimiter {
private countStmt: ReturnType<Database['prepare']>
private insertStmt: ReturnType<Database['prepare']>
private dailyRateLimit: number
constructor(
db: Database,
dailyRateLimit: number = RATE_LIMITS.DEFAULT_DAILY,
) {
this.dailyRateLimit = dailyRateLimit
this.countStmt = db.prepare(`
SELECT COUNT(*) as count
FROM rate_limiter
WHERE browseros_id = ?
AND date(created_at) = date('now')
`)
// INSERT OR IGNORE: duplicate conversation_ids are silently ignored
// This ensures the same conversation is only counted once for rate limiting
this.insertStmt = db.prepare(`
INSERT OR IGNORE INTO rate_limiter
(id, browseros_id, provider)
VALUES (?, ?, ?)
`)
}
check(browserosId: string): void {
const count = this.getTodayCount(browserosId)
if (count >= this.dailyRateLimit) {
logger.warn('Rate limit exceeded', {
browserosId,
count,
dailyRateLimit: this.dailyRateLimit,
})
metrics.log('rate_limit.triggered', {
count,
daily_limit: this.dailyRateLimit,
})
throw new RateLimitError(count, this.dailyRateLimit)
}
}
record(params: RecordParams): void {
const { conversationId, browserosId, provider } = params
this.insertStmt.run(conversationId, browserosId, provider)
}
private getTodayCount(browserosId: string): number {
const row = this.countStmt.get(browserosId) as { count: number } | null
return row?.count ?? 0
}
}
export { RateLimitError } from './errors'

View File

@@ -21,6 +21,15 @@ Sentry.init({
release: VERSION,
beforeSend(event) {
// Group tool execution errors by tool name instead of generic "execute"
const message = event.exception?.values?.[0]?.value ?? ''
if (message.startsWith('Internal error in ')) {
const toolName = message.match(/Internal error in (\S+):/)?.[1]
if (toolName) {
event.fingerprint = ['tool-execution', toolName]
}
}
return sanitizeEvent(event)
},
})

View File

@@ -29,8 +29,6 @@ import { identity } from './lib/identity'
import { logger } from './lib/logger'
import { metrics } from './lib/metrics'
import { isPortInUseError } from './lib/port-binding'
import { fetchDailyRateLimit } from './lib/rate-limiter/fetch-config'
import { RateLimiter } from './lib/rate-limiter/rate-limiter'
import { Sentry } from './lib/sentry'
import { seedSoulTemplate } from './lib/soul'
import { migrateBuiltinSkills } from './skills/migrate'
@@ -59,8 +57,6 @@ export class Application {
await this.initCoreServices()
const dailyRateLimit = await fetchDailyRateLimit(identity.getBrowserOSId())
const controller = new ControllerBackend({
port: this.config.extensionPort,
})
@@ -104,7 +100,6 @@ export class Application {
browserosId: identity.getBrowserOSId(),
executionDir: this.config.executionDir,
resourcesDir: this.config.resourcesDir,
rateLimiter: new RateLimiter(this.getDb(), dailyRateLimit),
codegenServiceUrl: this.config.codegenServiceUrl,
aiSdkDevtoolsEnabled: this.config.aiSdkDevtoolsEnabled,
@@ -274,13 +269,4 @@ export class Application {
logger.info(` HTTP Server: http://127.0.0.1:${this.config.serverPort}`)
logger.info('')
}
private getDb(): Database {
if (!this.db) {
throw new Error(
'Database not initialized. Call initCoreServices() first.',
)
}
return this.db
}
}

View File

@@ -264,6 +264,7 @@ export function executeWithMetrics(
tool_name: toolName,
duration_ms: Math.round(performance.now() - startTime),
success: !result.isError,
source: 'chat',
})
return result
},
@@ -278,6 +279,7 @@ export function executeWithMetrics(
duration_ms: Math.round(performance.now() - startTime),
success: false,
error_message: errorText,
source: 'chat',
})
return { text: errorText, isError: true }
},

View File

@@ -1,149 +0,0 @@
/**
* @license
* Copyright 2025 BrowserOS
* SPDX-License-Identifier: AGPL-3.0-or-later
*
* Integration tests for RateLimiter
* Uses in-memory SQLite to test actual database behavior
*/
import { Database } from 'bun:sqlite'
import { beforeEach, describe, expect, it } from 'bun:test'
import {
RateLimitError,
RateLimiter,
} from '../../src/agent/rate-limiter/rate-limiter'
const DAILY_RATE_LIMIT_TEST = 3
function createTestDb(): Database {
const db = new Database(':memory:')
db.exec('PRAGMA journal_mode = WAL')
db.exec(`
CREATE TABLE IF NOT EXISTS rate_limiter (
id TEXT PRIMARY KEY,
browseros_id TEXT NOT NULL,
provider TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
)
`)
return db
}
describe('RateLimiter', () => {
let db: Database
let rateLimiter: RateLimiter
beforeEach(() => {
db = createTestDb()
rateLimiter = new RateLimiter(db, DAILY_RATE_LIMIT_TEST)
})
describe('check()', () => {
it('allows first 3 conversations (check before record)', () => {
const browserosId = 'test-browseros-id'
// Simulates real flow: check() then record() for each conversation
for (let i = 1; i <= 3; i++) {
expect(() => rateLimiter.check(browserosId)).not.toThrow()
rateLimiter.record({
conversationId: `conv-${i}`,
browserosId,
provider: 'browseros',
})
}
})
it('blocks 4th conversation with RateLimitError', () => {
const browserosId = 'test-browseros-id'
// Use up all 3 slots
for (let i = 1; i <= 3; i++) {
rateLimiter.check(browserosId)
rateLimiter.record({
conversationId: `conv-${i}`,
browserosId,
provider: 'browseros',
})
}
// 4th should be blocked
expect(() => rateLimiter.check(browserosId)).toThrow(RateLimitError)
try {
rateLimiter.check(browserosId)
} catch (error) {
expect(error).toBeInstanceOf(RateLimitError)
const rateLimitError = error as RateLimitError
expect(rateLimitError.used).toBe(3)
expect(rateLimitError.limit).toBe(3)
expect(rateLimitError.statusCode).toBe(429)
}
})
})
describe('record() with duplicate conversation IDs', () => {
it('ignores duplicate conversation IDs (same conversation counted once)', () => {
const browserosId = 'test-browseros-id'
const sameConversationId = 'duplicate-conv-id'
// Record the same conversation 5 times
for (let i = 0; i < 5; i++) {
rateLimiter.record({
conversationId: sameConversationId,
browserosId,
provider: 'browseros',
})
}
// Should still pass - only counts as 1 conversation
expect(() => rateLimiter.check(browserosId)).not.toThrow()
// Add 2 more unique conversations (total 3)
rateLimiter.record({
conversationId: 'unique-conv-1',
browserosId,
provider: 'browseros',
})
rateLimiter.record({
conversationId: 'unique-conv-2',
browserosId,
provider: 'browseros',
})
// Now at limit (3 unique conversations)
expect(() => rateLimiter.check(browserosId)).toThrow(RateLimitError)
})
})
describe('separate limits per browserosId', () => {
it('tracks limits independently for different users', () => {
const user1 = 'browseros-user-1'
const user2 = 'browseros-user-2'
// User 1 uses all 3 conversations
for (let i = 1; i <= 3; i++) {
rateLimiter.record({
conversationId: `user1-conv-${i}`,
browserosId: user1,
provider: 'browseros',
})
}
// User 1 is blocked
expect(() => rateLimiter.check(user1)).toThrow(RateLimitError)
// User 2 should still have full quota
expect(() => rateLimiter.check(user2)).not.toThrow()
// User 2 can use their quota
rateLimiter.record({
conversationId: 'user2-conv-1',
browserosId: user2,
provider: 'browseros',
})
expect(() => rateLimiter.check(user2)).not.toThrow()
})
})
})

View File

@@ -99,12 +99,6 @@ describe('Application.start', () => {
log: mock(() => {}),
},
}))
mock.module('../src/lib/rate-limiter/fetch-config', () => ({
fetchDailyRateLimit: mock(async () => 100),
}))
mock.module('../src/lib/rate-limiter/rate-limiter', () => ({
RateLimiter: class {},
}))
mock.module('../src/lib/sentry', () => ({
Sentry: {
setContext: mock(() => {}),

View File

@@ -0,0 +1,227 @@
# BrowserOS Analytics Events
All tracked events across the BrowserOS platform. Events are sent to PostHog.
## Event Naming Convention
- **Server events**: `browseros.server.<event>` — sent via `metrics.log()` from the MCP/HTTP server
- **Extension events**: `browseros.native.extension.<event>` — sent via `track()` from the Chrome extension
- **Native events**: `browseros.native.<event>` — sent from the Chromium browser process
## Server Events
Prefix: `browseros.server.`
| Event | Properties | Description |
|-------|-----------|-------------|
| `http_server.started` | `version` | Server boot completed |
| `mcp.request` | `scopeId` | Every `POST /mcp` from external MCP clients (Claude Code, Cursor, etc.) |
| `mcp.rejected` | — | MCP request rejected (e.g. auth failure) |
| `chat.request` | `provider`, `model` | Every `POST /chat` from the built-in BrowserOS agent |
| `chat.aborted` | — | Chat request was aborted by the user |
| `chat-v2.request` | — | Chat v2 endpoint request (deprecated) |
| `tool_executed` | `tool_name`, `duration_ms`, `success`, `error_message?`, `source` | A tool was executed. `source` = `mcp` (external client) or `chat` (built-in agent) |
| `rate_limit.triggered` | — | Rate limit was hit |
### Global Properties (attached to all server events)
| Property | Description |
|----------|-------------|
| `client_id` | Client identifier from BrowserOS config |
| `install_id` | Installation identifier |
| `browseros_version` | BrowserOS browser version |
| `chromium_version` | Underlying Chromium version |
| `server_version` | MCP server version |
## Extension Events — UI Interactions
Prefix: `browseros.native.extension.`
### Chat & Sidepanel
| Event | Properties | Description |
|-------|-----------|-------------|
| `ui.message.sent` | — | User sent a message |
| `ui.message.like` | — | User liked a message |
| `ui.message.dislike` | — | User disliked a message |
| `ui.provider.selected` | — | User selected an LLM provider |
| `ui.conversation.reset` | — | User reset conversation |
| `sidepanel.ai.triggered` | — | AI triggered from sidepanel |
| `sidepanel.mode.changed` | — | Chat/agent mode changed in sidepanel |
| `sidepanel.generation.stopped` | — | User stopped generation in sidepanel |
| `sidepanel.message.copied` | — | User copied a message in sidepanel |
| `sidepanel.suggestion.clicked` | — | User clicked a suggestion in sidepanel |
| `sidepanel.tab.toggled` | — | Tab toggled in sidepanel |
| `sidepanel.tab.removed` | — | Tab removed in sidepanel |
| `sidepanel.voice.recording_started` | — | Voice recording started in sidepanel |
| `sidepanel.voice.recording_stopped` | — | Voice recording stopped in sidepanel |
| `sidepanel.voice.transcription_completed` | — | Voice transcription completed in sidepanel |
| `sidepanel.voice.error` | — | Voice error in sidepanel |
| `glow.generation.stopped` | — | User stopped generation in glow mode |
### New Tab Page
| Event | Properties | Description |
|-------|-----------|-------------|
| `newtab.opened` | — | New tab page loaded |
| `newtab.ai.triggered` | `mode`, `tabs_count` | User triggered AI from new tab |
| `newtab.search.executed` | `search_engine` | User executed a search |
| `newtab.chat.started` | `mode`, `tabs_count` | Inline chat started on new tab |
| `newtab.chat.stopped` | — | Inline chat stopped |
| `newtab.chat.reset` | — | Inline chat reset |
| `newtab.chat.suggestion_clicked` | — | User clicked a chat suggestion |
| `newtab.chat.mode_changed` | — | Chat mode changed on new tab |
| `newtab.workspace.opened` | — | Workspace selector opened |
| `newtab.tabs.opened` | — | Tab picker opened |
| `newtab.tab.toggled` | `action` | Tab selected/deselected |
| `newtab.tab.removed` | — | Tab removed from context |
| `newtab.apps.opened` | `has_connected_apps`, `connected_count?` | Apps selector opened |
| `newtab.tip.dismissed` | — | Tip card dismissed |
| `newtab.voice.recording_started` | — | Voice recording started on new tab |
| `newtab.voice.recording_stopped` | — | Voice recording stopped on new tab |
| `newtab.voice.transcription_completed` | — | Voice transcription completed on new tab |
| `newtab.voice.error` | `error` | Voice error on new tab |
| `newtab.scheduled_task.viewed_results` | — | User viewed scheduled task results on new tab |
| `newtab.scheduled_task.view_more` | — | User clicked "view more" for scheduled tasks |
### Settings — AI Providers
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.page.viewed` | `page` | Settings page loaded (tracks which page) |
| `settings.ai_provider.added` | — | Custom AI provider added |
| `settings.hub_provider.added` | — | Hub provider added |
| `settings.search_provider.changed` | — | Search provider changed |
| `settings.mcp_promo_banner.clicked` | — | MCP promo banner clicked on providers page |
### Settings — OAuth Providers
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.chatgpt_pro.oauth_started` | — | ChatGPT Pro OAuth flow started |
| `settings.chatgpt_pro.oauth_completed` | — | ChatGPT Pro OAuth flow completed |
| `settings.chatgpt_pro.oauth_disconnected` | — | ChatGPT Pro disconnected |
| `settings.github_copilot.oauth_started` | — | GitHub Copilot OAuth flow started |
| `settings.github_copilot.oauth_completed` | — | GitHub Copilot OAuth flow completed |
| `settings.github_copilot.oauth_disconnected` | — | GitHub Copilot disconnected |
| `settings.qwen_code.oauth_started` | — | Qwen Code OAuth flow started |
| `settings.qwen_code.oauth_completed` | — | Qwen Code OAuth flow completed |
| `settings.qwen_code.oauth_disconnected` | — | Qwen Code disconnected |
### Settings — Kimi / Moonshot
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.kimi.api_key_configured` | — | Kimi API key was configured |
| `settings.kimi.api_key_guide_clicked` | — | User clicked Kimi API key guide |
| `ui.rate_limit.kimi_docs_clicked` | — | User clicked Kimi docs from rate limit notice |
| `ui.rate_limit.moonshot_platform_clicked` | — | User clicked Moonshot platform link from rate limit |
### Settings — MCP Server
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.mcp_external_access.enabled` | — | External MCP access enabled |
| `settings.mcp_external_access.disabled` | — | External MCP access disabled |
| `settings.mcp_server.restarted` | — | MCP server manually restarted |
| `settings.managed_mcp.added` | — | Managed MCP server connected (e.g. Gmail, Slack) |
| `settings.custom_mcp.added` | — | Custom MCP server added |
### Settings — Scheduled Tasks
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.scheduled_task.created` | — | New scheduled task created |
| `settings.scheduled_task.edited` | — | Scheduled task edited |
| `settings.scheduled_task.deleted` | — | Scheduled task deleted |
| `settings.scheduled_task.toggled` | — | Scheduled task enabled/disabled |
| `settings.scheduled_task.prompt_refined` | — | Task prompt was refined |
| `settings.scheduled_task.tested` | — | Scheduled task was tested |
| `settings.scheduled_task.viewed_results` | — | Task results viewed in settings |
| `settings.scheduled_task.cancelled` | — | Running task was cancelled |
| `settings.scheduled_task.retried` | — | Task run was retried |
### Settings — Workflows
| Event | Properties | Description |
|-------|-----------|-------------|
| `settings.graph.created` | — | New workflow graph created |
| `settings.graph.saved` | — | Workflow graph saved |
| `settings.graph.updated` | — | Workflow graph updated |
| `settings.graph.message.like` | — | Workflow message liked |
| `settings.graph.message.dislike` | — | Workflow message disliked |
| `settings.workflow.deleted` | — | Workflow deleted |
| `settings.workflow.run_started` | — | Workflow run started |
| `settings.workflow.run_stopped` | — | Workflow run stopped |
| `settings.workflow.run_retried` | — | Workflow run retried |
| `settings.workflow.run_completed` | — | Workflow run completed |
### Onboarding
| Event | Properties | Description |
|-------|-----------|-------------|
| `onboarding.started` | — | Onboarding flow started |
| `onboarding.step.viewed` | — | Onboarding step viewed |
| `onboarding.step.completed` | — | Onboarding step completed |
| `onboarding.about.submitted` | — | User submitted "about me" info |
| `onboarding.soul.selected` | — | User selected a soul/persona |
| `onboarding.connect_apps.viewed` | — | Connect apps step viewed |
| `onboarding.app.connected` | — | App connected during onboarding |
| `onboarding.connect_apps.skipped` | — | Connect apps step skipped |
| `onboarding.signin.completed` | — | Sign-in completed during onboarding |
| `onboarding.signin.skipped` | — | Sign-in skipped during onboarding |
| `onboarding.demo.triggered` | — | Demo triggered during onboarding |
| `onboarding.feature.clicked` | — | Feature card clicked during onboarding |
| `onboarding.completed` | — | Onboarding flow completed |
### Breadcrumb Nudges
| Event | Properties | Description |
|-------|-----------|-------------|
| `breadcrumb.schedule.clicked` | — | Schedule nudge clicked |
| `breadcrumb.connect.clicked` | — | Connect app nudge clicked |
| `breadcrumb.connect.manual` | — | Manual connect triggered from nudge |
| `breadcrumb.connect.completed` | — | App connection completed from nudge |
| `breadcrumb.schedule.dismissed` | — | Schedule nudge dismissed |
### JTBD Popup
| Event | Properties | Description |
|-------|-----------|-------------|
| `ui.jtbd_popup.shown` | — | Jobs-to-be-done popup shown |
| `ui.jtbd_popup.clicked` | — | JTBD popup clicked |
| `ui.jtbd_popup.dismissed` | — | JTBD popup dismissed |
## Native Events (Chromium Browser Process)
Prefix: `browseros.native.`
These events come from the BrowserOS Chromium browser, not the extension or server.
| Event | Description |
|-------|-------------|
| `alive` | Heartbeat — BrowserOS instance is running |
| `llmhub.shown` | LLM Hub UI shown |
| `llmhub.panecount.changed` | Number of panes changed in LLM Hub |
| `llmhub.provider.switched` | Provider switched in LLM Hub |
| `llmchat.created` | New LLM chat session created |
| `llmchat.content.copied` | Content copied from LLM chat |
| `llmchat.provider.changed` | Provider changed in LLM chat |
| `llmchat.menu.hub` | Hub menu clicked |
| `llmchat.menu.help` | Help menu clicked |
| `llmchat.menu.newtab` | New tab menu clicked |
| `llmchat.menu.refresh` | Refresh menu clicked |
| `settings.provider.added` | Provider added via native settings |
| `settings.default_provider.changed` | Default provider changed via native settings |
| `server.ota.success` | Server OTA update succeeded |
| `server.ota.error` | Server OTA update failed |
| `server.ota.cleanup` | Server OTA cleanup ran |
| `server.ota.busy` | Server OTA busy (update in progress) |
## Other Events
| Event | Description |
|-------|-------------|
| `browseros.cdn.downloads` | CDN download tracking |
| `browseros.agent.feedback` | Agent feedback submitted |
| `browseros:update_ping` | Update ping (legacy) |

View File

@@ -6,12 +6,6 @@
* Centralized limits and thresholds.
*/
export const RATE_LIMITS = {
DEFAULT_DAILY: 5,
DEV_DAILY: 100,
TEST_DAILY: Infinity,
} as const
export const AGENT_LIMITS = {
MAX_TURNS: 100,
DEFAULT_CONTEXT_WINDOW: 200_000,