feat: enhance onboarding and replay functionality in browser extension

- Added a feature to open the onboarding page for new installs in production.
- Implemented a user manual replay request handler, validating input data and managing replay steps.
- Introduced a side panel for replay setup, allowing users to navigate and initialize replay sessions.
- Enhanced the chat header with options to share conversations and save them as skills, improving user interaction.
- Added error handling and validation for user-provided host URLs to mitigate security risks.
This commit is contained in:
ropzislaw
2026-02-15 21:23:03 +08:00
parent 66bbbc1777
commit 29d6739ccc
17 changed files with 2093 additions and 11 deletions

View File

@@ -0,0 +1,267 @@
# Sidepanel 入口链路new-aipex vs aipex 缺口分析
> **目的**:从 sidepanel 入口出发,系统梳理 `new-aipex` 相比 `aipex` 在端到端链路上的不足之处,为后续补齐工作提供可执行清单。
>
> **生成日期**2026-02-15
---
## 1. 对比基线与评估维度
| 项目 | 根路径 | 入口文件 |
|------|--------|----------|
| aipex旧版全功能 | `aipex/` | `src/sidepanel.tsx``ChatBot`2800+ 行单体组件) |
| new-aipex新版模块化 | `new-aipex/packages/` | `browser-ext/src/pages/sidepanel/index.tsx``app-root.tsx``ChatBot`(组件化 + hooks |
**评估维度**:触发与打开 / 初始化与配置加载 / 消息桥接 / Agent 与工具 / 会话持久化 / 会话历史 / 回放链路 / UI 功能 / 服务依赖 / 错误处理 / 可观测性
---
## 2. Sidepanel 端到端链路对比
### 2.1 链路流程图
```mermaid
flowchart TD
subgraph trigger [触发阶段]
iconClick[扩展图标点击]
shortcut[Ctrl/Cmd+M 快捷键]
omni[Omni 选择 ai-chat]
external[外部网站 openWithPrompt]
replay[外部网站 REPLAY_USER_MANUAL]
end
subgraph bg [Background Service Worker]
bgOpen[chrome.sidePanel.open]
bgRelay[消息中转与状态管理]
end
subgraph sp [Sidepanel 初始化]
vmInit[VM 初始化 - QuickJS/ZenFS]
reactMount[React 挂载]
providerTree[Provider 树]
agentCreate[Agent/MessageHandler 创建]
toolLoad[工具加载与过滤]
skillInit[Skill 系统初始化]
interventionInit[Intervention 初始化]
pendingPrompt[Pending Prompt 消费]
end
subgraph chat [首次交互]
userInput[用户输入]
stream[流式响应]
persist[会话持久化]
end
iconClick --> bgOpen
shortcut --> bgRelay --> bgOpen
omni --> bgRelay
external --> bgOpen
replay --> bgOpen
bgOpen --> vmInit
vmInit --> reactMount
reactMount --> providerTree
providerTree --> agentCreate
agentCreate --> toolLoad
agentCreate --> skillInit
agentCreate --> interventionInit
agentCreate --> pendingPrompt
pendingPrompt --> userInput
toolLoad --> userInput
userInput --> stream
stream --> persist
```
### 2.2 新旧链路步骤对照
| 阶段 | aipex (旧版) | new-aipex (新版) | 差异 |
|------|-------------|-----------------|------|
| **触发 → 打开** | `chrome.action.onClicked` / `open-aipex` 快捷键 / content script `open-sidepanel` / `openWithPrompt` / `REPLAY_USER_MANUAL` | `chrome.action.onClicked` / `open-aipex` 快捷键 / content script 无 `open-sidepanel` / `openWithPrompt` | 缺少 `REPLAY_USER_MANUAL` 外部触发 |
| **选中文本传递** | `open-sidepanel` 可携带 `selectedText``get-selected-text` 取回 | 无此机制 | **缺失** |
| **VM 初始化** | `initializeVM()` — QuickJS + ZenFS | 同 | 一致 |
| **React 挂载** | `ReactDOM.createRoot``SidepanelApp` | `renderChatApp()``App` | 一致 |
| **Provider 树** | `ThemeProvider → I18nProvider → ChatBot` | `I18nProvider → ThemeProvider → AuthProvider → ChatApp` | 新版多了 `AuthProvider` |
| **配置加载** | `useStorage` 逐个读 `aiHost/aiToken/aiModel` + 等待 `isLoading*` | `useChatConfig({ storageAdapter })` 统一读取 | 新版更简洁 |
| **Agent 创建** | `new MessageHandler(config)` — 含 `DefaultBackgroundContextManager` + `getAllTools()` + `filterToolsByMode` | `useAgent({ settings, modelFactory, storage, contextProviders, tools })``AIPex.create()` | 架构不同但功能对等 |
| **工具加载** | `getAllTools()` — 约 70+ MCP 工具 | `allBrowserTools` — 32 个注册工具 | **缺口:~50 个工具未暴露** |
| **Skill 初始化** | `skillManager.initialize()` + 订阅 `skill_enabled/skill_disabled` | VM 层初始化 skill无显式事件订阅 | 事件订阅缺失 |
| **Intervention 初始化** | `interventionManager.initialize()` + 5 个事件监听 (start/complete/cancel/timeout/error) | `InterventionModeProvider` + `InterventionUI` 组件化 | 功能对等但实现方式不同 |
| **Pending Prompt** | `useEffect` 检查 `aipex-pending-prompt``setInput()` | `usePendingPrompt()` hook → `initialInput` prop | 一致 |
| **Port 长连接** | `chrome.runtime.connect({ name: 'sidepanel' })` | 同 | 一致 |
| **Visibility 清理** | `visibilitychange``stop-recording` + `stop-capture` | 同 | 一致 |
| **模型列表** | `fetchModels()` → 手动管理 `models` state + 自定义模型追加 | `fetchModelsForSelector()` 在 InputArea 内自动加载 + BYOK 自定义模型 | 新版更灵活 |
| **Conversation 呼吸边框** | heartbeat → `aipex-conversation-active` → content script 边框 | 同(`useConversationHeartbeat` | 一致 |
| **流式输出** | `MessageHandler._startStream` — 手动 fetch + `StreamingParser` | `AIPex.chat()``AsyncGenerator<AgentEvent>``ChatAdapter` | 新版架构更清晰 |
| **会话持久化** | `ConversationStorage` (IndexedDB, LRU 5 条) | `conversationStorage` (IndexedDB, LRU 5 条) + debounced auto-save | 一致 |
| **会话历史 UI** | `ConversationHistory` 组件 (header 内) | `ConversationHistory` 组件 (BrowserChatHeader 内) | 一致 |
| **Replay 链路** | `NAVIGATE_AND_SETUP_REPLAY` 监听 → `UseCaseDetail` 页面 | 不存在 | **完全缺失** |
| **Chat 图片导出** | sidepanel 内 `provide-current-chat-images` listener | `ChatImagesListener` 组件 | 一致 |
---
## 3. 缺口清单
### GAP-01: Replay 外部触发链路缺失
| 属性 | 值 |
|------|-----|
| **现象** | 旧版支持外部网站发送 `REPLAY_USER_MANUAL` → background 转发 `NAVIGATE_AND_SETUP_REPLAY` → sidepanel 接收并跳转到 `UseCaseDetail`。新版 `background.ts` 和 sidepanel 中均不存在这两个消息类型。 |
| **证据** | 旧版 `aipex/src/background.ts` L3151-3199`REPLAY_USER_MANUAL``aipex/src/lib/components/chatbot/index.tsx` L1373-1424`NAVIGATE_AND_SETUP_REPLAY` listener。新版 `rg "REPLAY_USER_MANUAL\|NAVIGATE_AND_SETUP_REPLAY" new-aipex/` → 零匹配。 |
| **影响** | claudechrome.com 网站上的 "Replay Manual" 功能无法调起新版扩展。 |
| **优先级** | **P1** |
| **建议** | 在 `background.ts` `onMessageExternal` 和 sidepanel ChatApp 中补齐消息协议。 |
---
### GAP-02: 未迁移的后台服务
| 属性 | 值 |
|------|-----|
| **现象** | 旧版多个与 sidepanel 功能关联的后台服务在新版中不存在。 |
| **证据** | `new-aipex/migration/LOGIC_INCONSISTENCIES_BY_PACKAGE.md` 第 4.2 节。 |
| **影响** | |
| 缺失服务 | 旧版路径 | 影响范围 |
|----------|---------|---------|
| `version-checker.ts` | `aipex/src/lib/services/version-checker.ts` | 无法检测扩展新版本并通知用户 |
| `web-auth.ts` | `aipex/src/lib/services/web-auth.ts` | 旧版 Cookie-based auth 辅助函数缺失(新版有 `AuthProvider` 替代,但功能不完全对等) |
| `recording-upload.ts` | `aipex/src/lib/services/recording-upload.ts` | 无法将录屏上传到托管服务 |
| `screenshot-upload.ts` | `aipex/src/lib/services/screenshot-upload.ts` | 无法将截图上传到托管服务 |
| `user-manuals-api.ts` | `aipex/src/lib/services/user-manuals-api.ts` | 无法从服务端获取用户手册数据 |
| `replay-controller.ts` | `aipex/src/lib/services/replay-controller.ts` | 无法执行步骤回放控制 |
| **优先级** | **P1**auth 相关)/ **P2**(其余) |
| **建议** | 按依赖顺序逐步迁移到 `packages/browser-ext/src/services/` 或新建 `packages/services`。 |
---
### GAP-03: Voice 输入仅支持 Web Speech API
| 属性 | 值 |
|------|-----|
| **现象** | 旧版支持三层 STT 降级Server STT → ElevenLabs → Web Speech API。新版仅保留 Web Speech API。 |
| **证据** | `new-aipex/migration/LOGIC_INCONSISTENCIES_BY_PACKAGE.md` 第 2.7 节。新版 `packages/aipex-react/src/components/voice/VoiceInput.tsx``packages/browser-ext/src/lib/browser-chat-input-area.tsx` 仅使用 Web Speech。 |
| **影响** | 非英语语言识别准确率下降BYOK 用户无法使用 ElevenLabs 高精度识别。 |
| **优先级** | **P1** |
| **建议** | 在 `packages/aipex-react/src/lib/voice/` 中恢复 ElevenLabs + Server STT 路径。 |
---
### GAP-04: Install 时自动打开引导页缺失
| 属性 | 值 |
|------|-----|
| **现象** | 旧版 `onInstalled` 在 production 环境下会自动打开 `https://www.claudechrome.com` 引导用户。新版仅打日志。 |
| **证据** | 旧版 `aipex/src/background.ts` L1082-1089新版 `new-aipex/packages/browser-ext/src/background.ts` L31-39。 |
| **影响** | 新用户安装后无引导体验。 |
| **优先级** | **P2** |
| **建议** | 在 `onInstalled` install 分支中恢复引导页打开逻辑。 |
---
### GAP-05: React ErrorBoundary 缺失
| 属性 | 值 |
|------|-----|
| **现象** | 新版 sidepanel 的 React 组件树中没有包裹 `ErrorBoundary`。任何渲染错误会导致整个 sidepanel 白屏。 |
| **证据** | `rg "ErrorBoundary" new-aipex/` → 零匹配。旧版同样缺少,但旧版组件更少、更稳定。 |
| **影响** | Agent 工具回调中的渲染错误、异步组件加载失败等场景会导致不可恢复的白屏。 |
| **优先级** | **P1** |
| **建议** | 在 `app-root.tsx``<App />` 外层包裹 `ErrorBoundary`,提供友好的错误恢复 UI。 |
---
### GAP-06: Skill 系统事件订阅缺失
| 属性 | 值 |
|------|-----|
| **现象** | 旧版在 ChatBot 初始化时显式订阅 `skillManager``skill_enabled` / `skill_disabled` 事件。新版 VM 初始化了 skill 系统,但 sidepanel UI 层未订阅 skill 变更事件。 |
| **证据** | 旧版 `aipex/src/lib/components/chatbot/index.tsx` L1313-1325`skillManager.subscribe`)。新版 `app-root.tsx``chatbot.tsx` 中无 skill 事件订阅。 |
| **影响** | Skill 启用/禁用后 UI 可能无法实时反映变化(例如 skill tag 不更新)。 |
| **优先级** | **P2** |
| **建议** | 在 ChatApp 或 ChatBot 内添加 `skillManager` 事件订阅并传播到 UI。 |
---
### GAP-07: Share分享对话功能缺失
| 属性 | 值 |
|------|-----|
| **现象** | 旧版 sidepanel header 包含"分享"和"保存为 Skill"按钮。新版 `BrowserChatHeader` 中无此功能。 |
| **证据** | 旧版 `aipex/src/lib/components/chatbot/index.tsx` 中存在 `handleShareConversation`L1744-1793`handleSaveAsSkill`L1796-1798。新版 `browser-chat-header.tsx` 中无对应功能。 |
| **影响** | 用户无法分享当前对话或将对话保存为可复用的 Skill。 |
| **优先级** | **P2** |
| **建议** | 在 `BrowserChatHeader` 中添加分享与 Skill 保存入口。 |
---
## 4. 缺口优先级汇总
| 优先级 | 缺口编号 | 简述 |
|--------|---------|------|
| **P1** | GAP-01 | Replay 外部触发链路缺失 |
| **P1** | GAP-03 | Voice 输入仅 Web Speech API |
| **P1** | GAP-05 | React ErrorBoundary 缺失 |
| **P2** | GAP-02 | 未迁移的后台服务 |
| **P2** | GAP-04 | Install 引导页缺失 |
| **P2** | GAP-06 | Skill 事件订阅缺失 |
| **P2** | GAP-07 | 分享 / 保存为 Skill 功能缺失 |
---
## 5. 补齐路线建议
### Phase 1P1 — 功能完整性)
1. **补齐 Replay 外部触发** — background 中处理 `REPLAY_USER_MANUAL`sidepanel 中监听 `NAVIGATE_AND_SETUP_REPLAY`
2. **恢复 Voice 多层 STT** — 在 voice engine 中添加 ElevenLabs 和 Server STT 路径。
3. **添加 ErrorBoundary**`app-root.tsx` 顶层包裹。
### Phase 2P2 — 体验优化)
4. Install 引导页、Skill 事件订阅、分享/Skill 保存、后台服务迁移。
---
## 6. 安全评审卡
### 6.1 入口与信任边界
| 入口 | 信任级别 | 校验机制 |
|------|---------|---------|
| `chrome.action.onClicked` | 可信(用户主动触发) | 无需额外校验 |
| `chrome.commands` 快捷键 | 可信(用户主动触发) | 无需额外校验 |
| Content script → `open-sidepanel` | 半可信content script 上下文) | Background 仅执行 `sidePanel.open`,不传递数据 |
| `onMessageExternal` (`openWithPrompt`) | 外部来源 | `manifest.json` `externally_connectable` 限制来源域名prompt 类型校验 (`typeof === "string"`)timestamp 过期丢弃5s TTL|
| `onMessageExternal` (`REPLAY_USER_MANUAL`) | 外部来源 | **旧版已有**`externally_connectable` 域名白名单;**新版缺失** — 需补齐时同步实现。 |
### 6.2 外部消息校验路径
- `openWithPrompt`prompt 必须为 `string` 类型 + 非空;通过 `chrome.storage.local` 中转(非直接注入 UIsidepanel 消费时检查 timestamp<5s)。**状态已实现安全。**
- `REPLAY_USER_MANUAL`**新版缺失**补齐时需确保 `steps` 数组做 schema 校验防止注入恶意步骤数据
### 6.3 敏感数据与日志
- `aiToken` 存储在 `chrome.storage.local`UI 中未展示明文新版 `AIPex.sanitizeErrorMessage()` 内建 Bearer token / API key 脱敏。**状态安全。**
- 旧版大量 `console.log` 包含 emoji debug 信息可能泄露会话内容新版日志更精简。**建议**production build 时移除或降级 debug 日志
### 6.4 高风险工具默认策略
- 旧版所有 MCP 工具默认暴露新版通过 `allBrowserTools` 白名单控制 `filterToolsByMode` background 模式下过滤 computer/screenshot 工具。**状态新版更安全。**
### 6.5 待验证测试点
- [ ] `ErrorBoundary` 添加后的错误恢复行为
- [ ] `REPLAY_USER_MANUAL` 补齐后的 steps 数据校验
- [ ] Production build 日志级别控制
- [ ] BYOK 自定义模型的 `aiHost` URL 校验 SSRF
- [ ] `externally_connectable` 域名列表与实际部署域名一致性
---
> **交叉引用**:本文档与以下已有迁移文档互补
> - `migration/TOOL_SURFACE_AUDIT.md` — 工具级对比详表
> - `migration/LOGIC_INCONSISTENCIES_BY_PACKAGE.md` — 按包的逻辑差异
> - `migration/MIGRATION_STRATEGY.md` — 整体迁移规划

View File

@@ -0,0 +1,102 @@
import { fireEvent, render, screen } from "@testing-library/react";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { ErrorBoundary } from "./ErrorBoundary";
// Component that throws on render
function ThrowingComponent({ shouldThrow }: { shouldThrow: boolean }) {
if (shouldThrow) {
throw new Error("Test rendering error");
}
return <div>Normal content</div>;
}
describe("ErrorBoundary", () => {
// Suppress React error boundary console.error during tests
const originalConsoleError = console.error;
beforeEach(() => {
console.error = vi.fn();
});
afterEach(() => {
console.error = originalConsoleError;
});
it("renders children when no error occurs", () => {
render(
<ErrorBoundary>
<div>Hello World</div>
</ErrorBoundary>,
);
expect(screen.getByText("Hello World")).toBeTruthy();
});
it("renders fallback UI when a child throws", () => {
render(
<ErrorBoundary>
<ThrowingComponent shouldThrow={true} />
</ErrorBoundary>,
);
expect(screen.getByText("Something went wrong")).toBeTruthy();
expect(screen.getByText("Try Again")).toBeTruthy();
});
it("shows error details in expandable section", () => {
render(
<ErrorBoundary>
<ThrowingComponent shouldThrow={true} />
</ErrorBoundary>,
);
expect(screen.getByText("Error details")).toBeTruthy();
expect(screen.getByText("Test rendering error")).toBeTruthy();
});
it("resets error state when Try Again is clicked", () => {
// Use a stateful wrapper so we can control the throw behavior
let shouldThrow = true;
function ConditionalThrower() {
if (shouldThrow) throw new Error("Boom");
return <div>Recovered</div>;
}
render(
<ErrorBoundary>
<ConditionalThrower />
</ErrorBoundary>,
);
expect(screen.getByText("Something went wrong")).toBeTruthy();
// Stop throwing before clicking retry
shouldThrow = false;
fireEvent.click(screen.getByText("Try Again"));
expect(screen.getByText("Recovered")).toBeTruthy();
});
it("renders custom fallback when provided", () => {
render(
<ErrorBoundary fallback={<div>Custom error UI</div>}>
<ThrowingComponent shouldThrow={true} />
</ErrorBoundary>,
);
expect(screen.getByText("Custom error UI")).toBeTruthy();
});
it("calls onError callback when an error is caught", () => {
const onError = vi.fn();
render(
<ErrorBoundary onError={onError}>
<ThrowingComponent shouldThrow={true} />
</ErrorBoundary>,
);
expect(onError).toHaveBeenCalledTimes(1);
expect(onError).toHaveBeenCalledWith(
expect.objectContaining({ message: "Test rendering error" }),
expect.objectContaining({ componentStack: expect.any(String) }),
);
});
});

View File

@@ -0,0 +1,150 @@
/**
* ErrorBoundary
* Catches React rendering errors and displays a recovery UI
* instead of an unrecoverable white screen.
*/
import React from "react";
export interface ErrorBoundaryProps {
children: React.ReactNode;
/** Optional fallback UI to render when an error is caught. */
fallback?: React.ReactNode;
/** Optional callback invoked when an error is caught. */
onError?: (error: Error, errorInfo: React.ErrorInfo) => void;
}
interface ErrorBoundaryState {
hasError: boolean;
error: Error | null;
}
export class ErrorBoundary extends React.Component<
ErrorBoundaryProps,
ErrorBoundaryState
> {
constructor(props: ErrorBoundaryProps) {
super(props);
this.state = { hasError: false, error: null };
}
static getDerivedStateFromError(error: Error): ErrorBoundaryState {
return { hasError: true, error };
}
componentDidCatch(error: Error, errorInfo: React.ErrorInfo): void {
// Log error without exposing sensitive data
console.error(
"[ErrorBoundary] Uncaught rendering error:",
error.message,
);
this.props.onError?.(error, errorInfo);
}
private handleRetry = (): void => {
this.setState({ hasError: false, error: null });
};
render(): React.ReactNode {
if (this.state.hasError) {
if (this.props.fallback) {
return this.props.fallback;
}
return (
<div
style={{
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
height: "100%",
padding: "24px",
fontFamily:
'-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif',
textAlign: "center",
color: "var(--foreground, #333)",
backgroundColor: "var(--background, #fff)",
}}
>
<div
style={{
fontSize: "48px",
marginBottom: "16px",
lineHeight: 1,
}}
aria-hidden="true"
>
!
</div>
<h2
style={{
fontSize: "18px",
fontWeight: 600,
margin: "0 0 8px",
}}
>
Something went wrong
</h2>
<p
style={{
fontSize: "14px",
color: "var(--muted-foreground, #666)",
margin: "0 0 20px",
maxWidth: "320px",
}}
>
An unexpected error occurred. Click the button below to try
recovering.
</p>
<button
type="button"
onClick={this.handleRetry}
style={{
padding: "8px 20px",
fontSize: "14px",
fontWeight: 500,
borderRadius: "6px",
border: "1px solid var(--border, #ddd)",
backgroundColor: "var(--primary, #333)",
color: "var(--primary-foreground, #fff)",
cursor: "pointer",
}}
>
Try Again
</button>
{this.state.error && (
<details
style={{
marginTop: "16px",
fontSize: "12px",
color: "var(--muted-foreground, #888)",
maxWidth: "400px",
textAlign: "left",
}}
>
<summary style={{ cursor: "pointer" }}>Error details</summary>
<pre
style={{
marginTop: "8px",
padding: "8px",
borderRadius: "4px",
backgroundColor: "var(--muted, #f5f5f5)",
overflow: "auto",
whiteSpace: "pre-wrap",
wordBreak: "break-word",
maxHeight: "120px",
}}
>
{this.state.error.message}
</pre>
</details>
)}
</div>
);
}
return this.props.children;
}
}

View File

@@ -0,0 +1 @@
export { ErrorBoundary, type ErrorBoundaryProps } from "./ErrorBoundary";

View File

@@ -31,6 +31,11 @@ chrome.commands.onCommand.addListener((command) => {
chrome.runtime.onInstalled.addListener((details) => {
if (details.reason === "install") {
console.log("AIPex extension installed");
// Open onboarding page for new installs in production
if (import.meta.env.PROD) {
chrome.tabs.create({ url: "https://www.claudechrome.com" });
}
} else if (details.reason === "update") {
console.log(
"AIPex extension updated to version",
@@ -530,6 +535,102 @@ chrome.runtime.onMessageExternal.addListener(
return true; // Keep message channel open for async response
}
// Handle user manual replay request from website
if (message.request === "REPLAY_USER_MANUAL") {
const { manualId, startFromStep, steps } = message as {
manualId?: unknown;
startFromStep?: unknown;
steps?: unknown;
};
// Validate required fields
if (
typeof manualId !== "number" ||
!Array.isArray(steps) ||
steps.length === 0
) {
sendResponse({
success: false,
error: "Invalid replay data: manualId (number) and non-empty steps (array) are required",
});
return true;
}
// Validate step entries have required shape and bounded size
const MAX_STEPS = 500;
if (steps.length > MAX_STEPS) {
sendResponse({
success: false,
error: `Too many replay steps (max ${MAX_STEPS})`,
});
return true;
}
const ALLOWED_EVENT_TYPES = ["click", "navigation"];
const stepsValid = steps.every((s: unknown) => {
if (s === null || typeof s !== "object") return false;
const rec = s as Record<string, unknown>;
if (!rec.event || typeof rec.event !== "object") return false;
const event = rec.event as Record<string, unknown>;
return (
typeof event.type === "string" &&
ALLOWED_EVENT_TYPES.includes(event.type)
);
});
if (!stepsValid) {
sendResponse({
success: false,
error:
"Invalid replay steps: each step must contain an event with type 'click' or 'navigation'",
});
return true;
}
const resolvedStartFromStep =
typeof startFromStep === "number" && startFromStep >= 0
? startFromStep
: 0;
// Open sidepanel then forward replay data
const windowId = sender.tab?.windowId;
if (!windowId) {
sendResponse({ success: false, error: "No window ID available" });
return true;
}
chrome.sidePanel
.open({ windowId })
.then(() => {
// Wait for sidepanel to initialize before forwarding
setTimeout(() => {
chrome.runtime
.sendMessage({
request: "NAVIGATE_AND_SETUP_REPLAY",
data: {
manualId,
startFromStep: resolvedStartFromStep,
steps,
},
})
.catch(() => {
// Sidepanel may not yet have a listener acceptable race
});
}, 500);
sendResponse({ success: true });
})
.catch((error) => {
sendResponse({
success: false,
error: error instanceof Error ? error.message : String(error),
});
});
return true;
}
sendResponse({ success: false, error: "Unknown action" });
return true;
},

View File

@@ -0,0 +1,113 @@
import { describe, expect, it, vi } from "vitest";
import { createAIProvider } from "./ai-provider";
// Provide minimal mock for import.meta.env
vi.stubGlobal("import", { meta: { env: { PROD: false } } });
describe("createAIProvider", () => {
describe("URL validation", () => {
it("accepts valid https URLs", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
aiHost: "https://api.openai.com/v1",
}),
).not.toThrow();
});
it("accepts undefined aiHost", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
}),
).not.toThrow();
});
it("accepts empty string aiHost", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
aiHost: "",
}),
).not.toThrow();
});
it("rejects invalid URLs", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
aiHost: "not-a-url",
}),
).toThrow("Invalid aiHost URL");
});
it("rejects non-http protocols", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
aiHost: "ftp://evil.com",
}),
).toThrow("Unsupported protocol");
});
it("rejects javascript: protocol", () => {
expect(() =>
createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
// eslint-disable-next-line no-script-url
aiHost: "javascript:alert(1)",
}),
).toThrow("Unsupported protocol");
});
});
describe("provider creation", () => {
it("creates openai provider by default", () => {
const provider = createAIProvider({
aiProvider: "openai",
aiToken: "sk-test",
});
expect(provider).toBeDefined();
});
it("creates anthropic provider", () => {
const provider = createAIProvider({
aiProvider: "anthropic",
aiToken: "sk-test",
});
expect(provider).toBeDefined();
});
it("creates google provider", () => {
const provider = createAIProvider({
aiProvider: "google",
aiToken: "sk-test",
});
expect(provider).toBeDefined();
});
it("requires baseURL for custom providers", () => {
expect(() =>
createAIProvider({
aiProvider: "custom" as any,
aiToken: "sk-test",
}),
).toThrow("requires aiHost");
});
it("creates custom provider with valid baseURL", () => {
const provider = createAIProvider({
aiProvider: "custom" as any,
aiToken: "sk-test",
aiHost: "https://my-proxy.example.com",
});
expect(provider).toBeDefined();
});
});
});

View File

@@ -31,10 +31,52 @@ export interface ProviderConfig {
* const model = provider("gpt-4");
* ```
*/
/**
* Validate that a user-provided host URL is safe to use.
* Rejects private/internal addresses to mitigate SSRF risks.
*/
function validateHostUrl(url: string | undefined): string | undefined {
if (!url) return undefined;
let parsed: URL;
try {
parsed = new URL(url);
} catch {
throw new Error(`Invalid aiHost URL: ${url}`);
}
// Only allow http/https schemes
if (parsed.protocol !== "https:" && parsed.protocol !== "http:") {
throw new Error(
`Unsupported protocol in aiHost: ${parsed.protocol} (only http/https allowed)`,
);
}
// Block common internal/private hostnames
const hostname = parsed.hostname.toLowerCase();
const blocked = [
"localhost",
"127.0.0.1",
"0.0.0.0",
"[::1]",
"metadata.google.internal",
"169.254.169.254",
];
// In production, block private addresses
if (import.meta.env.PROD && blocked.includes(hostname)) {
throw new Error(
`aiHost points to a restricted address: ${hostname}`,
);
}
return parsed.origin + parsed.pathname.replace(/\/+$/, "");
}
export function createAIProvider(settings: AppSettings) {
const provider = settings.aiProvider ?? "openai";
const apiKey = settings.aiToken ?? "";
const baseURL = settings.aiHost || undefined;
const baseURL = validateHostUrl(settings.aiHost || undefined);
switch (provider) {
case "anthropic":

View File

@@ -5,14 +5,30 @@
import { useChatContext } from "@aipexstudio/aipex-react/components/chatbot";
import { Button } from "@aipexstudio/aipex-react/components/ui/button";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@aipexstudio/aipex-react/components/ui/dropdown-menu";
import { useTranslation } from "@aipexstudio/aipex-react/i18n/context";
import { getRuntime } from "@aipexstudio/aipex-react/lib/runtime";
import { cn } from "@aipexstudio/aipex-react/lib/utils";
import type { HeaderProps } from "@aipexstudio/aipex-react/types";
import { conversationStorage } from "@aipexstudio/browser-runtime";
import { KeyboardIcon, MicIcon, PlusIcon, SettingsIcon } from "lucide-react";
import {
KeyboardIcon,
MicIcon,
MoreHorizontalIcon,
PlusIcon,
SettingsIcon,
Share2Icon,
SparklesIcon,
} from "lucide-react";
import { useCallback, useEffect, useRef, useState } from "react";
import { UserProfile, useAuth } from "../auth";
import { shareConversation } from "../services/share-conversation";
import { ConversationHistory } from "./conversation-history";
import { useInputMode } from "./input-mode-context";
import { fromStorageFormat, toStorageFormat } from "./message-adapter";
@@ -27,7 +43,7 @@ export function BrowserChatHeader({
}: HeaderProps) {
const { t } = useTranslation();
const runtime = getRuntime();
const { messages, setMessages, interrupt } = useChatContext();
const { messages, setMessages, interrupt, sendMessage } = useChatContext();
const { user, login, isLoading: isAuthLoading } = useAuth();
const [currentConversationId, setCurrentConversationId] = useState<
@@ -133,6 +149,34 @@ export function BrowserChatHeader({
setInputMode(inputMode === "voice" ? "text" : "voice");
}, [inputMode, setInputMode]);
// Share conversation
const [isSharing, setIsSharing] = useState(false);
const handleShare = useCallback(async () => {
if (isSharing) return;
const nonSystemMessages = messages.filter((m) => m.role !== "system");
if (nonSystemMessages.length === 0) return;
setIsSharing(true);
try {
const { url } = await shareConversation(messages);
chrome.tabs.create({ url });
} catch (error) {
console.error(
"[Share] Failed:",
error instanceof Error ? error.message : String(error),
);
} finally {
setIsSharing(false);
}
}, [messages, isSharing]);
// Save as Skill — instructs the AI to create a skill from the conversation
const handleSaveAsSkill = useCallback(() => {
sendMessage("use skill-creator skill to save the conversation");
}, [sendMessage]);
return (
<div
className={cn(
@@ -180,8 +224,39 @@ export function BrowserChatHeader({
/>
</div>
{/* Right side - New Chat and User Profile */}
{/* Right side - More menu, New Chat, User Profile */}
<div className="flex items-center gap-1">
{/* More actions dropdown (Share, Save as Skill) */}
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="ghost" size="icon" className="size-8">
<MoreHorizontalIcon className="size-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
<DropdownMenuItem
onClick={handleShare}
disabled={
isSharing ||
messages.filter((m) => m.role !== "system").length === 0
}
>
<Share2Icon className="mr-2 size-4" />
{isSharing ? "Sharing..." : "Share Conversation"}
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem
onClick={handleSaveAsSkill}
disabled={
messages.filter((m) => m.role !== "system").length === 0
}
>
<SparklesIcon className="mr-2 size-4" />
Save as Skill
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
<Button
variant="ghost"
size="sm"

View File

@@ -10,6 +10,7 @@ import { I18nProvider } from "@aipexstudio/aipex-react/i18n/context";
import type { Language } from "@aipexstudio/aipex-react/i18n/types";
import { ThemeProvider } from "@aipexstudio/aipex-react/theme/context";
import type { Theme } from "@aipexstudio/aipex-react/theme/types";
import { ErrorBoundary } from "@aipexstudio/aipex-react/components/error/ErrorBoundary";
import { ChromeStorageAdapter } from "@aipexstudio/browser-runtime";
import React, { useCallback, useEffect, useRef, useState } from "react";
import ReactDOM from "react-dom/client";
@@ -36,6 +37,76 @@ import { UpdateBannerWrapper } from "../../lib/update-banner-wrapper";
const i18nStorageAdapter = new ChromeStorageAdapter<Language>();
const themeStorageAdapter = new ChromeStorageAdapter<Theme>();
// ---------------------------------------------------------------------------
// Replay setup listener
// ---------------------------------------------------------------------------
/** Replay step shape coming from the external website */
interface ReplayStepData {
id?: number;
event: { type: string; [key: string]: unknown };
url?: string | null;
aiTitle?: string | null;
aiSummary?: string | null;
}
/**
* Listens for `NAVIGATE_AND_SETUP_REPLAY` messages forwarded by the
* background service worker after an external `REPLAY_USER_MANUAL` request.
*
* The replay steps are persisted to `chrome.storage.local` under
* `aipex-pending-replay` so they can be consumed by the use-case system
* when it is available.
*/
function useReplaySetup() {
useEffect(() => {
const handler = (message: Record<string, unknown>) => {
if (message?.request !== "NAVIGATE_AND_SETUP_REPLAY") return;
const data = message.data as {
manualId?: number;
startFromStep?: number;
steps?: ReplayStepData[];
} | undefined;
if (!data || !Array.isArray(data.steps) || data.steps.length === 0) {
console.warn("[ReplaySetup] Invalid or empty replay data received");
return;
}
// Persist replay data for future use-case system consumption
chrome.storage.local
.set({
"aipex-pending-replay": {
manualId: data.manualId,
startFromStep: data.startFromStep ?? 0,
steps: data.steps,
receivedAt: Date.now(),
},
})
.catch(() => {
/* storage may be unavailable */
});
console.log(
"[ReplaySetup] Replay data stored:",
data.steps.length,
"steps for manual",
data.manualId,
);
};
chrome.runtime.onMessage.addListener(handler);
return () => {
chrome.runtime.onMessage.removeListener(handler);
};
}, []);
}
// ---------------------------------------------------------------------------
// Pending prompt
// ---------------------------------------------------------------------------
/**
* Reads and consumes a pending prompt saved by the openWithPrompt external
* message handler in the background service worker. Prompts older than 5 s
@@ -141,6 +212,7 @@ function ChatApp() {
const pendingInput = usePendingPrompt();
const heartbeat = useConversationHeartbeat();
useReplaySetup();
const handleStatusChange = useCallback(
(status: string) => {
@@ -242,13 +314,15 @@ export function renderChatApp() {
}
const App = () => (
<I18nProvider storageAdapter={i18nStorageAdapter}>
<ThemeProvider storageAdapter={themeStorageAdapter}>
<AuthProvider>
<ChatApp />
</AuthProvider>
</ThemeProvider>
</I18nProvider>
<ErrorBoundary>
<I18nProvider storageAdapter={i18nStorageAdapter}>
<ThemeProvider storageAdapter={themeStorageAdapter}>
<AuthProvider>
<ChatApp />
</AuthProvider>
</ThemeProvider>
</I18nProvider>
</ErrorBoundary>
);
ReactDOM.createRoot(rootElement).render(

View File

@@ -7,6 +7,33 @@ export {
WEBSITE_HOST,
WEBSITE_ORIGIN,
} from "../config/website";
// Recording upload
export {
uploadRecordingSession,
type UploadRecordingSessionPayload,
type UploadRecordingSessionResult,
type UploadRecordingStepPayload,
} from "./recording-upload";
// Replay controller
export {
ManualReplayController,
type ClickEvent,
type ExecutionResult,
type NavigationEvent,
type ReplayEventCallback,
type ReplayStatus,
type ReplayStep,
} from "./replay-controller";
// Screenshot upload
export {
uploadScreenshot,
type UploadScreenshotResult,
} from "./screenshot-upload";
// Share conversation
export {
shareConversation,
type ShareResult,
} from "./share-conversation";
// Sound effects
export {
playSoundEffect,
@@ -34,6 +61,16 @@ export {
toolManager,
unregisterDynamicTool,
} from "./tool-manager";
// User manuals API
export {
deleteUserManual,
fetchMyUserManuals,
fetchUserManualDetail,
type FetchUserManualDetailResponse,
type UserManualDetail,
type UserManualListItem,
type UserManualStep,
} from "./user-manuals-api";
// Version checking
export {
checkVersion,

View File

@@ -0,0 +1,81 @@
/**
* Recording Upload Service
* Uploads a complete recording session (steps + metadata) to the hosted API.
* Only used for non-BYOK user scenarios.
*/
import { WEBSITE_URL } from "../config/website";
import { getAuthCookieHeader } from "./web-auth";
/** Payload shape for a single recorded step */
export interface UploadRecordingStepPayload {
index: number;
timestamp: number;
url: string;
event: { type: string; [key: string]: unknown };
aiTitle?: string;
aiSummary?: string;
pageSnapshotId?: string;
screenshotUrl?: string | null;
screenshotKey?: string | null;
}
/** Full session payload sent to the server */
export interface UploadRecordingSessionPayload {
useCaseId: string;
params: Record<string, unknown>;
startedAt?: number;
completedAt?: number;
steps: UploadRecordingStepPayload[];
}
/** Server response after a successful upload */
export interface UploadRecordingSessionResult {
id?: number;
stepsInserted?: number;
slug?: string;
/** @deprecated Use `slug` instead */
sessionId?: string;
}
/**
* Upload a recording session to the server.
* Uses per-step `screenshotUrl` / `screenshotKey` references — raw base64
* screenshot data is NOT included in this request.
*/
export async function uploadRecordingSession(
payload: UploadRecordingSessionPayload,
): Promise<UploadRecordingSessionResult> {
const cookieHeader = await getAuthCookieHeader();
const response = await fetch(`${WEBSITE_URL}/api/recordings/sessions`, {
method: "POST",
headers: {
"Content-Type": "application/json",
...(cookieHeader ? { Cookie: cookieHeader } : {}),
},
body: JSON.stringify(payload),
credentials: "include",
});
if (!response.ok) {
const text = await response.text().catch(() => "");
throw new Error(
`Recording upload failed (${response.status}): ${text || "Unknown error"}`,
);
}
const json = (await response.json().catch(() => ({}))) as Record<
string,
unknown
>;
return {
id: typeof json.id === "number" ? json.id : undefined,
stepsInserted:
typeof json.stepsInserted === "number" ? json.stepsInserted : undefined,
slug: typeof json.slug === "string" ? json.slug : undefined,
sessionId:
typeof json.sessionId === "string" ? json.sessionId : undefined,
};
}

View File

@@ -0,0 +1,133 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import {
ManualReplayController,
type ReplayEventCallback,
type ReplayStep,
} from "./replay-controller";
// Enhance chrome mock for tabs used by replay controller
beforeEach(() => {
vi.useFakeTimers();
// Reset chrome.tabs mocks
const mockTabs = globalThis.chrome.tabs as any;
mockTabs.create = vi.fn().mockResolvedValue({ id: 100 });
mockTabs.update = vi.fn().mockResolvedValue({});
mockTabs.get = vi.fn().mockResolvedValue({
id: 100,
url: "https://example.com",
});
mockTabs.query = vi
.fn()
.mockResolvedValue([{ id: 100, url: "https://example.com" }]);
mockTabs.sendMessage = vi.fn().mockResolvedValue({ success: true });
mockTabs.onUpdated = {
addListener: vi.fn((cb: Function) => {
// Immediately call with 'complete' to simulate page load
setTimeout(() => cb(100, { status: "complete" }), 0);
}),
removeListener: vi.fn(),
};
});
afterEach(() => {
vi.useRealTimers();
});
function makeNavigationStep(url: string): ReplayStep {
return {
event: { type: "navigation", url },
url,
aiTitle: "Navigate",
aiSummary: `Navigate to ${url}`,
};
}
function makeClickStep(): ReplayStep {
return {
event: {
type: "click",
selector: "#btn",
textSnippet: "Click me",
rect: { x: 10, y: 20, width: 100, height: 40 },
value: {
tagName: "BUTTON",
id: "btn",
classes: [],
attributes: {},
elementDescription: "A button",
},
},
url: "https://example.com",
aiTitle: "Click button",
aiSummary: "Click the button",
};
}
describe("ManualReplayController", () => {
it("initializes with idle status", () => {
const controller = new ManualReplayController([makeNavigationStep("https://example.com")]);
expect(controller.getStatus()).toBe("idle");
expect(controller.getCurrentStepIndex()).toBe(0);
});
it("emits progress event on start", async () => {
const callback: ReplayEventCallback = vi.fn();
const steps = [makeNavigationStep("https://example.com")];
const controller = new ManualReplayController(steps, callback);
const startPromise = controller.start();
await vi.advanceTimersByTimeAsync(2000);
await startPromise;
expect(callback).toHaveBeenCalledWith(
expect.objectContaining({
type: "progress",
currentStep: 0,
totalSteps: 1,
}),
);
});
it("can be stopped", () => {
const steps = [makeNavigationStep("https://example.com")];
const controller = new ManualReplayController(steps);
controller.stop();
expect(controller.getStatus()).toBe("idle");
});
it("can be paused", () => {
const steps = [
makeNavigationStep("https://example.com"),
makeClickStep(),
];
const controller = new ManualReplayController(steps);
controller.pause();
expect(controller.getStatus()).toBe("paused");
});
it("resume does nothing when not paused", async () => {
const steps = [makeNavigationStep("https://example.com")];
const controller = new ManualReplayController(steps);
// Status is "idle", resume should be a no-op
await controller.resume();
expect(controller.getStatus()).toBe("idle");
});
it("skip current step increments index", async () => {
const steps = [makeNavigationStep("https://a.com"), makeClickStep()];
const controller = new ManualReplayController(steps);
expect(controller.getCurrentStepIndex()).toBe(0);
const skipPromise = controller.skipCurrentStep();
await vi.advanceTimersByTimeAsync(5000);
await skipPromise;
// Should have advanced past step 0
expect(controller.getCurrentStepIndex()).toBeGreaterThanOrEqual(1);
});
});

View File

@@ -0,0 +1,405 @@
/**
* User Manual Replay Controller
*
* Manages the execution of user manual replay steps,
* including navigation and click events.
*/
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
export interface ReplayStep {
id?: number;
event: NavigationEvent | ClickEvent | { type: string; [key: string]: unknown };
url: string | null;
aiTitle: string | null;
aiSummary: string | null;
}
export interface ClickEvent {
type: "click";
selector: string;
textSnippet: string;
rect: { x: number; y: number; width: number; height: number };
value: {
tagName: string;
id: string;
classes: string[];
attributes: Record<string, string>;
elementDescription: string;
};
}
export interface NavigationEvent {
type: "navigation";
url: string;
}
export interface ExecutionResult {
success: boolean;
error?: string;
details?: unknown;
}
export type ReplayStatus =
| "idle"
| "running"
| "paused"
| "completed"
| "error";
export type ReplayEventCallback = (event: {
type:
| "progress"
| "complete"
| "error"
| "step-start"
| "step-complete"
| "step-error";
currentStep?: number;
totalSteps?: number;
error?: string;
stepData?: ReplayStep;
result?: ExecutionResult;
}) => void;
// ---------------------------------------------------------------------------
// Controller
// ---------------------------------------------------------------------------
export class ManualReplayController {
private steps: ReplayStep[];
private currentStepIndex = 0;
private status: ReplayStatus = "idle";
private eventCallback?: ReplayEventCallback;
private targetTabId?: number;
private abortController?: AbortController;
constructor(steps: ReplayStep[], eventCallback?: ReplayEventCallback) {
this.steps = steps;
this.eventCallback = eventCallback;
}
// -----------------------------------------------------------------------
// Public API
// -----------------------------------------------------------------------
/** Start replay from a specific step */
async start(startFromStep = 0): Promise<void> {
if (this.status === "running") return;
this.currentStepIndex = startFromStep;
this.status = "running";
this.abortController = new AbortController();
this.emitEvent({
type: "progress",
currentStep: this.currentStepIndex,
totalSteps: this.steps.length,
});
try {
await this.executeNextStep();
} catch (error) {
this.status = "error";
this.emitEvent({
type: "error",
error: error instanceof Error ? error.message : String(error),
});
}
}
/** Stop replay */
stop(): void {
this.status = "idle";
this.abortController?.abort();
this.hideSpotlight();
}
/** Pause replay */
pause(): void {
this.status = "paused";
}
/** Resume replay from paused state */
async resume(): Promise<void> {
if (this.status !== "paused") return;
this.status = "running";
await this.executeNextStep();
}
/** Retry current step */
async retryCurrentStep(): Promise<void> {
this.status = "running";
await this.executeNextStep();
}
/** Skip current step and move to next */
async skipCurrentStep(): Promise<void> {
this.currentStepIndex++;
this.status = "running";
await this.executeNextStep();
}
getStatus(): ReplayStatus {
return this.status;
}
getCurrentStepIndex(): number {
return this.currentStepIndex;
}
// -----------------------------------------------------------------------
// Internal
// -----------------------------------------------------------------------
private async executeNextStep(): Promise<void> {
if (this.abortController?.signal.aborted) return;
if (this.currentStepIndex >= this.steps.length) {
this.status = "completed";
this.emitEvent({ type: "complete" });
this.hideSpotlight();
return;
}
if (this.status === "paused") return;
const step = this.steps[this.currentStepIndex]!;
this.emitEvent({
type: "step-start",
currentStep: this.currentStepIndex,
totalSteps: this.steps.length,
stepData: step,
});
try {
const result = await this.executeStep(step);
if (result.success) {
this.emitEvent({
type: "step-complete",
currentStep: this.currentStepIndex,
totalSteps: this.steps.length,
result,
});
this.currentStepIndex++;
await new Promise((r) => setTimeout(r, 1500));
await this.executeNextStep();
} else {
this.status = "paused";
this.emitEvent({
type: "step-error",
currentStep: this.currentStepIndex,
totalSteps: this.steps.length,
error: result.error,
result,
});
}
} catch (error) {
this.status = "paused";
this.emitEvent({
type: "step-error",
currentStep: this.currentStepIndex,
totalSteps: this.steps.length,
error: error instanceof Error ? error.message : String(error),
});
}
}
private async executeStep(step: ReplayStep): Promise<ExecutionResult> {
const event = step.event as { type: string; [key: string]: unknown };
if (!event?.type) {
return { success: false, error: "Invalid event data" };
}
switch (event.type) {
case "navigation":
return this.executeNavigation(event as NavigationEvent, step);
case "click":
return this.executeClick(event as ClickEvent, step);
default:
return { success: false, error: `Unknown event type: ${event.type}` };
}
}
// --- Navigation ---
private async executeNavigation(
event: NavigationEvent,
step: ReplayStep,
): Promise<ExecutionResult> {
try {
const url = event.url || step.url;
if (!url) return { success: false, error: "No URL provided for navigation" };
if (this.targetTabId) {
await chrome.tabs.update(this.targetTabId, { url, active: true });
} else {
const tab = await chrome.tabs.create({ url, active: true });
this.targetTabId = tab.id;
}
await this.waitForTabLoad();
return { success: true };
} catch (error) {
return {
success: false,
error: error instanceof Error ? error.message : String(error),
};
}
}
// --- Click ---
private async executeClick(
event: ClickEvent,
step: ReplayStep,
): Promise<ExecutionResult> {
try {
let tabId = this.targetTabId;
if (!tabId) {
const tabs = await chrome.tabs.query({
active: true,
currentWindow: true,
});
if (!tabs[0]?.id) {
return { success: false, error: "No active tab found" };
}
tabId = tabs[0].id;
this.targetTabId = tabId;
}
// Validate tab is accessible
let tab: chrome.tabs.Tab;
try {
tab = await chrome.tabs.get(tabId);
} catch {
return {
success: false,
error: "Target tab no longer exists. Please ensure the tab is open.",
};
}
if (
tab.url &&
!tab.url.startsWith("http://") &&
!tab.url.startsWith("https://")
) {
return {
success: false,
error:
"Cannot execute on this page. Only http/https pages are supported.",
};
}
// Navigate to correct URL if needed
if (step.url && tab.url && !tab.url.startsWith(step.url)) {
await chrome.tabs.update(tabId, { url: step.url });
await this.waitForTabLoad();
}
// Send click to content script with retry
const maxRetries = 3;
const retryDelay = 1000;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await chrome.tabs.sendMessage(tabId, {
request: "REPLAY_EXECUTE_CLICK",
eventData: event,
explanation: step.aiSummary || step.aiTitle,
stepUrl: step.url,
});
if (response?.success) {
return { success: true, details: response };
}
return {
success: false,
error: response?.error || "Click execution failed",
details: response,
};
} catch (error) {
const msg =
error instanceof Error ? error.message : String(error);
if (
(msg.includes("Receiving end does not exist") ||
msg.includes("Could not establish connection")) &&
attempt < maxRetries
) {
await new Promise((r) => setTimeout(r, retryDelay));
continue;
}
if (attempt === maxRetries) {
return {
success: false,
error:
"Could not connect to the page. The page may have been closed or navigated away.",
};
}
return { success: false, error: msg };
}
}
return {
success: false,
error: "Failed to execute click after multiple attempts",
};
} catch (error) {
return {
success: false,
error: error instanceof Error ? error.message : String(error),
};
}
}
// --- Helpers ---
private waitForTabLoad(): Promise<void> {
return new Promise((resolve) => {
const listener = (
tabId: number,
changeInfo: chrome.tabs.TabChangeInfo,
) => {
if (tabId === this.targetTabId && changeInfo.status === "complete") {
chrome.tabs.onUpdated.removeListener(listener);
resolve();
}
};
chrome.tabs.onUpdated.addListener(listener);
// Timeout after 30 s
setTimeout(() => {
chrome.tabs.onUpdated.removeListener(listener);
resolve();
}, 30_000);
});
}
private hideSpotlight(): void {
chrome.tabs
.query({ active: true, currentWindow: true })
.then((tabs) => {
if (tabs[0]?.id) {
chrome.tabs
.sendMessage(tabs[0].id, {
request: "ui-guider-hide-spotlight",
})
.catch(() => {});
}
})
.catch(() => {});
}
private emitEvent(event: Parameters<ReplayEventCallback>[0]): void {
try {
this.eventCallback?.(event);
} catch {
// Prevent callback errors from breaking the controller
}
}
}

View File

@@ -0,0 +1,66 @@
/**
* Screenshot Upload Service
* Uploads screenshots to the hosted storage service.
* Only used for non-BYOK user scenarios.
*/
import { WEBSITE_URL } from "../config/website";
import { getAuthCookieHeader } from "./web-auth";
export interface UploadScreenshotResult {
url: string;
key: string;
}
/**
* Upload a screenshot (base64 data URL) to server-side storage.
* @param base64 - Either a data URL (`data:image/...;base64,...`) or raw base64 string
* @returns The uploaded screenshot's public URL and storage key
*/
export async function uploadScreenshot(
base64: string,
): Promise<UploadScreenshotResult> {
if (!base64) {
throw new Error("Empty screenshot data");
}
// Ensure data URL format
const payloadBase64 = base64.startsWith("data:image")
? base64
: `data:image/png;base64,${base64}`;
const cookieHeader = await getAuthCookieHeader();
const response = await fetch(`${WEBSITE_URL}/api/storage/upload`, {
method: "POST",
headers: {
"Content-Type": "application/json",
...(cookieHeader ? { Cookie: cookieHeader } : {}),
},
body: JSON.stringify({
base64: payloadBase64,
}),
credentials: "include",
});
if (!response.ok) {
const text = await response.text().catch(() => "");
throw new Error(
`Screenshot upload failed (${response.status}): ${text || "Unknown error"}`,
);
}
const json = (await response.json().catch(() => ({}))) as Record<
string,
unknown
>;
if (typeof json.url !== "string" || typeof json.key !== "string") {
throw new Error("Invalid upload response: missing url or key");
}
return {
url: json.url,
key: json.key,
};
}

View File

@@ -0,0 +1,170 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { shareConversation } from "./share-conversation";
// Mock web-auth
vi.mock("./web-auth", () => ({
getAuthCookieHeader: vi.fn().mockResolvedValue("session=abc123"),
}));
// Mock website config
vi.mock("../config/website", () => ({
WEBSITE_URL: "https://www.claudechrome.com",
}));
describe("shareConversation", () => {
const originalFetch = globalThis.fetch;
beforeEach(() => {
globalThis.fetch = vi.fn();
});
afterEach(() => {
globalThis.fetch = originalFetch;
});
it("throws when there are no non-system messages", async () => {
const messages = [
{
id: "1",
role: "system" as const,
parts: [{ type: "text" as const, text: "You are helpful" }],
},
];
await expect(shareConversation(messages)).rejects.toThrow(
"No messages to share",
);
});
it("filters out system messages before sharing", async () => {
const mockFetch = globalThis.fetch as ReturnType<typeof vi.fn>;
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ url: "https://www.claudechrome.com/share/abc" }),
});
const messages = [
{
id: "1",
role: "system" as const,
parts: [{ type: "text" as const, text: "System prompt" }],
},
{
id: "2",
role: "user" as const,
parts: [{ type: "text" as const, text: "Hello" }],
},
{
id: "3",
role: "assistant" as const,
parts: [{ type: "text" as const, text: "Hi there!" }],
},
];
await shareConversation(messages);
expect(mockFetch).toHaveBeenCalledTimes(1);
const callBody = JSON.parse(mockFetch.mock.calls[0][1].body);
// Should only have 2 messages (user + assistant), not the system one
expect(callBody.messages).toHaveLength(2);
expect(callBody.messages[0].role).toBe("user");
expect(callBody.messages[1].role).toBe("assistant");
});
it("returns the share URL on success", async () => {
const mockFetch = globalThis.fetch as ReturnType<typeof vi.fn>;
mockFetch.mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({ url: "https://www.claudechrome.com/share/xyz" }),
});
const messages = [
{
id: "1",
role: "user" as const,
parts: [{ type: "text" as const, text: "Hello" }],
},
];
const result = await shareConversation(messages);
expect(result.url).toBe("https://www.claudechrome.com/share/xyz");
});
it("throws on server error", async () => {
const mockFetch = globalThis.fetch as ReturnType<typeof vi.fn>;
mockFetch.mockResolvedValue({
ok: false,
status: 500,
text: () => Promise.resolve("Internal Server Error"),
});
const messages = [
{
id: "1",
role: "user" as const,
parts: [{ type: "text" as const, text: "Hello" }],
},
];
await expect(shareConversation(messages)).rejects.toThrow(
"Share failed (500)",
);
});
it("throws when no URL is returned", async () => {
const mockFetch = globalThis.fetch as ReturnType<typeof vi.fn>;
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ success: true }),
});
const messages = [
{
id: "1",
role: "user" as const,
parts: [{ type: "text" as const, text: "Hello" }],
},
];
await expect(shareConversation(messages)).rejects.toThrow(
"no URL was returned",
);
});
it("strips screenshot data from tool parts", async () => {
const mockFetch = globalThis.fetch as ReturnType<typeof vi.fn>;
mockFetch.mockResolvedValue({
ok: true,
json: () =>
Promise.resolve({ url: "https://www.claudechrome.com/share/abc" }),
});
const messages = [
{
id: "1",
role: "assistant" as const,
parts: [
{
type: "tool" as const,
toolName: "capture_screenshot",
toolCallId: "call-1",
input: {},
output: { success: true },
state: "completed" as const,
screenshot: "data:image/png;base64,HUGESCREENSHOTDATA",
},
],
},
];
await shareConversation(messages);
const callBody = JSON.parse(mockFetch.mock.calls[0][1].body);
const toolPart = callBody.messages[0].content.parts[0];
// screenshot field should NOT be included in the shared payload
expect(toolPart.screenshot).toBeUndefined();
expect(toolPart.toolName).toBe("capture_screenshot");
});
});

View File

@@ -0,0 +1,129 @@
/**
* Share Conversation Service
* Uploads the current conversation to the hosted share API and returns
* a shareable URL.
*/
import type { UIMessage } from "@aipexstudio/aipex-react/types";
import { WEBSITE_URL } from "../config/website";
import { getAuthCookieHeader } from "./web-auth";
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
interface ShareablePart {
type: string;
[key: string]: unknown;
}
interface ShareableMessage {
id: string;
role: string;
content: { parts: ShareablePart[] };
metadata?: Record<string, unknown>;
}
export interface ShareResult {
url: string;
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
/**
* Convert a UI message part to a shareable format.
* Strips large binary payloads (screenshots) and internal-only fields.
*/
function toShareablePart(part: UIMessage["parts"][number]): ShareablePart {
switch (part.type) {
case "text":
return { type: "text", text: part.text };
case "file":
return {
type: "file",
mediaType: part.mediaType,
filename: part.filename,
url: part.url,
};
case "context":
return {
type: "context",
contextType: part.contextType,
label: part.label,
value: part.value,
metadata: part.metadata,
};
case "source-url":
return { type: "source-url", url: part.url };
case "reasoning":
return { type: "reasoning", text: part.text };
case "tool":
return {
type: "tool",
toolName: part.toolName,
input: part.input,
output: part.output,
state: part.state,
errorText: part.errorText,
toolCallId: part.toolCallId,
// Note: screenshot is intentionally omitted from share payload
};
default:
return part as ShareablePart;
}
}
// ---------------------------------------------------------------------------
// Public API
// ---------------------------------------------------------------------------
/**
* Share a conversation by uploading messages to the share API.
* @returns The shareable URL on success
* @throws On network or server errors
*/
export async function shareConversation(
messages: UIMessage[],
): Promise<ShareResult> {
// Filter out system messages
const messagesToShare = messages.filter((m) => m.role !== "system");
if (messagesToShare.length === 0) {
throw new Error("No messages to share");
}
const shareableMessages: ShareableMessage[] = messagesToShare.map((msg) => ({
id: msg.id,
role: msg.role,
content: { parts: msg.parts.map(toShareablePart) },
metadata: msg.metadata,
}));
const cookieHeader = await getAuthCookieHeader();
const response = await fetch(`${WEBSITE_URL}/api/share`, {
method: "POST",
headers: {
"Content-Type": "application/json",
...(cookieHeader ? { Cookie: cookieHeader } : {}),
},
body: JSON.stringify({ messages: shareableMessages }),
});
if (!response.ok) {
const text = await response.text().catch(() => "");
throw new Error(
`Share failed (${response.status}): ${text || "Unknown error"}`,
);
}
const result = (await response.json()) as Record<string, unknown>;
if (typeof result.url !== "string") {
throw new Error("Share succeeded but no URL was returned");
}
return { url: result.url };
}

View File

@@ -0,0 +1,136 @@
/**
* User Manuals API Service
* CRUD operations for user manuals stored on the website backend.
*/
import { WEBSITE_URL } from "../config/website";
import { getAuthCookieHeader } from "./web-auth";
// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------
export interface UserManualListItem {
id: number;
title: string | null;
slug: string | null;
createdAt: string;
public: boolean;
stepCount: number;
}
export interface UserManualStep {
id: number;
index: number | null;
url: string;
event: unknown;
aiTitle: string | null;
aiSummary: string | null;
screenshotUrl: string | null;
}
export interface UserManualDetail {
id: number;
title: string | null;
createdAt: string;
public: boolean;
params: unknown;
}
export interface FetchUserManualDetailResponse {
manual: UserManualDetail;
steps: UserManualStep[];
}
// ---------------------------------------------------------------------------
// API functions
// ---------------------------------------------------------------------------
/**
* Fetch the current user's user manuals.
* Returns an empty array when the user is not authenticated.
*/
export async function fetchMyUserManuals(): Promise<UserManualListItem[]> {
const cookieHeader = await getAuthCookieHeader();
if (!cookieHeader) {
return [];
}
const response = await fetch(`${WEBSITE_URL}/api/user-manuals/my`, {
method: "GET",
headers: {
"Content-Type": "application/json",
Cookie: cookieHeader,
},
credentials: "include",
});
if (!response.ok) {
if (response.status === 401) {
return [];
}
const text = await response.text().catch(() => "");
throw new Error(
`Fetch user manuals failed (${response.status}): ${text || "Unknown error"}`,
);
}
const json = (await response.json()) as { manuals: UserManualListItem[] };
return json.manuals;
}
/**
* Fetch a specific user manual with all its steps.
*/
export async function fetchUserManualDetail(
id: number,
): Promise<FetchUserManualDetailResponse> {
const cookieHeader = await getAuthCookieHeader();
const response = await fetch(`${WEBSITE_URL}/api/user-manuals/${id}`, {
method: "GET",
headers: {
"Content-Type": "application/json",
...(cookieHeader ? { Cookie: cookieHeader } : {}),
},
credentials: "include",
});
if (!response.ok) {
const text = await response.text().catch(() => "");
throw new Error(
`Fetch manual detail failed (${response.status}): ${text || "Unknown error"}`,
);
}
return (await response.json()) as FetchUserManualDetailResponse;
}
/**
* Delete a user manual by ID.
* Requires authentication.
*/
export async function deleteUserManual(id: number): Promise<void> {
const cookieHeader = await getAuthCookieHeader();
if (!cookieHeader) {
throw new Error("User not logged in");
}
const response = await fetch(`${WEBSITE_URL}/api/user-manuals/${id}`, {
method: "DELETE",
headers: {
"Content-Type": "application/json",
Cookie: cookieHeader,
},
credentials: "include",
});
if (!response.ok) {
const text = await response.text().catch(() => "");
throw new Error(
`Delete manual failed (${response.status}): ${text || "Unknown error"}`,
);
}
}