LLM/manbot
1
0
Fork 0
mirror of https://github.com/larchanka/manbot.git synced 2026-05-18 07:56:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(AO-01): standardize BaseProcess lifecycle and implement heartbeats

Browse Source
This commit is contained in:
larchanka
2026-02-22 22:17:07 +01:00
parent 62f7801caa
commit 1cd4f1f58f
6 changed files with 502 additions and 196 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
_board/PLAN.md
View File

@@ -1,23 +1,102 @@
# Architecture Overhaul: Process Isolation & Cron-Driven AI
# 🗂️ File Processing Pipeline — Implementation Plan
This plan describes the transition to an "Advanced" architecture where all components are managed by a dedicated Supervisor, communication is routed via a formalized Message Bus, and Cron jobs can trigger full AI task pipelines.
## Overview
Add file processing capabilities to ManBot. Users can send text documents, photos, and audio messages via Telegram. The system processes each file type independently: text is read and injected into context (or indexed into RAG if large), images are OCR'd or described using Ollama vision (`glm-ocr:q8_0`), and audio is transcribed using Whisper (`nodejs-whisper` + `ffmpeg-static`). All processing happens in a new independent `file-processor` process, consistent with the existing process-isolation architecture.
## Goals
- **Fault Tolerance**: Standardize process lifecycle and implement auto-restart.
- **Observability**: formalized IPC routing and real-time dashboard monitoring.
- **Autonomous Action**: Enable scheduled AI queries (Synthetic User Input).
---
## Phase 1: Infrastructure & Supervision (AO-01 to AO-03)
We will enhance `BaseProcess` to support heartbeats and health reporting, then implement a Supervisor role in the Orchestrator to monitor and restart crashed child processes.
## Architecture Decisions
## Phase 2: Router & Message Bus (AO-04 to AO-05)
Introduce a dedicated `Router` process to handle message distribution, decoupling the Orchestrator's supervision logic from its routing logic.
| Decision | Choice | Rationale |
|---|---|---|
| Audio format conversion | `ffmpeg-static` (npm) | No system ffmpeg installed; bundled binary is self-contained |
| Audio transcription | `nodejs-whisper` (base.en) | ~200MB, runs locally, no cloud dependency |
| Image model | `glm-ocr:q8_0` via Ollama | Already connected to local Ollama; on-demand |
| Image behaviour | Auto OCR + describe | Model decides: text → extract verbatim, no text → describe |
| Long text handling | Chunk → Summarize → RAG | Preserves content accessible via semantic search |
| Large file failures | Warn user, continue | Partial failures don't block the task pipeline |
| File cleanup | Delete after processing | Stateless processor; no persistent upload storage |
## Phase 3: Advanced Cron AI Queries (AO-08 to AO-10)
Extend the Cron system to support `ai_query` tasks that feed directly into the Planner, allowing the agent to perform scheduled research or maintenance autonomously.
---
## Phase 4: Monitoring & Control (AO-06, AO-07, AO-11)
Upgrade the Dashboard to provide a "Mission Control" experience, including process metrics, real-time IPC logs, and cron management.
## Phases
## Phase 5: Verification (AO-12 to AO-14)
Comprehensive testing of the new supervisor patterns and the end-to-end cron-to-ai flow.
### Phase 1 — Foundation (Tasks FP-01 to FP-03)
Set up infrastructure: new npm packages, config types, TypeScript interfaces.
No runtime changes — pure setup.
### Phase 2 — Core Services (Tasks FP-04 to FP-06)
Build the three low-level utilities:
- `OllamaAdapter.chatWithImage()` for vision
- `convertToWav()` for audio conversion
- `transcribeAudio()` for Whisper transcription
Each utility is independently testable and has no dependency on the new process.
### Phase 3 — File Processor Process (Tasks FP-07 to FP-08)
Build the `file-processor` service as a new `BaseProcess` subprocess and register it in the Orchestrator. At this point the service exists and responds to `file.process` envelopes.
### Phase 4 — Telegram Integration (Tasks FP-09 to FP-11)
Wire the Telegram side: file detection, download, `file.ingest` emission, and the Orchestrator handler that calls `file-processor`, collects results, handles warnings, and builds the enriched goal. Long-text RAG indexing pipeline implemented here.
### Phase 5 — Planner Integration + Cleanup (Tasks FP-12 to FP-13)
Update the planner prompt to handle enriched goals and indexed file references. Set up upload directory lifecycle (init + orphan cleanup).
### Phase 6 — Documentation and Verification (Tasks FP-14 to FP-15)
Update all docs to reflect the new subsystem, then perform end-to-end manual verification across all file types and edge cases.
---
## Data Flow
```
User sends file on Telegram
↓
[telegram-adapter]
Detect attachment → download → classify → emit file.ingest
↓
[core / Orchestrator]
file.ingest handler
↓ (parallel)
┌─ text → [file-processor] → inline or text_long
│ └─ text_long → chunk → summarize (model-router) → insert (rag-service)
├─ image → [file-processor] → OllamaAdapter.chatWithImage → OCR/description
└─ audio → [file-processor] → convertToWav → transcribeAudio → transcript
↓
Build enrichedGoal
↓
runTaskPipeline → Planner → Executor → ...
↓
[telegram-adapter] → User
```
---
## New Files
- `src/services/file-processor.ts` — new independent service process
- `src/utils/audio-converter.ts` — ffmpeg-static conversion utility
- `src/utils/whisper-transcriber.ts` — nodejs-whisper transcription utility
## Modified Files
- `src/services/ollama-adapter.ts` — add `chatWithImage()`
- `src/adapters/telegram-adapter.ts` — file detection, download, `file.ingest`
- `src/core/orchestrator.ts` — register file-processor, add `file.ingest` handler, long-text pipeline
- `src/agents/prompts/planner.ts` — file context instructions + examples
- `src/shared/config.ts` — `WhisperConfig`, `FileProcessorConfig`
- `config.json`, `config.json.example` — new config sections
- `_docs/COMPONENTS.md`, `_docs/TECH.md`, `_docs/MESSAGE PROTOCOL SPEC.md`, `README.md`
---
## New npm Dependencies
| Package | Type | Purpose |
|---|---|---|
| `nodejs-whisper` | runtime | Speech-to-text transcription |
| `ffmpeg-static` | runtime | Audio format conversion (ogg → wav) |
---
## Risk Notes
- **Whisper model download**: ~200MB on first audio request. User will see a delay and a descriptive error on first attempt. Subsequent uses are fast.
- **OCR model availability**: `glm-ocr:q8_0` must be pulled in Ollama before use (`ollama pull glm-ocr:q8_0`). If missing, image processing returns an Ollama error — the Orchestrator warns the user and skips.
- **Context length**: Inline file content is capped to prevent planner prompt overflow. Long files go through RAG instead.

174
_board/TASKS.md
View File

@@ -1,93 +1,135 @@
# 📋 Tasks — Architecture Overhaul (AO)
# 📋 Tasks — File Processing Pipeline
All tasks are prefixed `AO-` (Architecture Overhaul). They are ordered by implementation dependency.
All tasks are prefixed `FP-` (File Processing). They are ordered by implementation dependency.
---
## Phase 1 — Infrastructure & Supervision
## Phase 1 — Foundation
### AO-01 Standardize BaseProcess Lifecycle
**File**: `src/shared/base-process.ts`
Add health checks, heartbeats, and standardized status reporting.
→ [AO-01_BASE_PROCESS_LIFECYCLE.md](./TASKS/AO-01_BASE_PROCESS_LIFECYCLE.md)
### FP-01 Add npm Dependencies
**File**: `package.json`
**Deps**: None
Install `nodejs-whisper` and `ffmpeg-static` runtime packages.
→ [FP-01_ADD_DEPENDENCIES.md](./TASKS/FP-01_ADD_DEPENDENCIES.md)
### AO-02 Implement Process Supervisor
---
### FP-02 Add Config Types and Defaults
**File**: `src/shared/config.ts`, `config.json`
**Deps**: None
Add `WhisperConfig` and `FileProcessorConfig` interfaces, defaults, and env var overrides.
→ [FP-02_CONFIG_TYPES.md](./TASKS/FP-02_CONFIG_TYPES.md)
---
### FP-03 Define File Processing Protocol Types
**File**: `src/shared/protocol.ts`
**Deps**: FP-02
Define `FileDescriptor`, `FileIngestPayload`, `FileProcessRequest`, `ProcessedFile` interfaces.
→ [FP-03_PROTOCOL_TYPES.md](./TASKS/FP-03_PROTOCOL_TYPES.md)
---
## Phase 2 — Core Services
### FP-04 Extend OllamaAdapter with Vision Support
**File**: `src/services/ollama-adapter.ts`
**Deps**: FP-02, FP-03
Add `chatWithImage(messages, model, imagePath)` method using Ollama multimodal API.
→ [FP-04_OLLAMA_VISION.md](./TASKS/FP-04_OLLAMA_VISION.md)
---
### FP-05 Implement Audio Conversion Utility
**File**: `src/utils/audio-converter.ts`
**Deps**: FP-01, FP-02
`convertToWav(inputPath, outputPath)` — wraps `ffmpeg-static` to produce 16kHz mono WAV.
→ [FP-05_AUDIO_CONVERTER.md](./TASKS/FP-05_AUDIO_CONVERTER.md)
---
### FP-06 Implement Whisper Transcription Utility
**File**: `src/utils/whisper-transcriber.ts`
**Deps**: FP-01, FP-02, FP-05
`transcribeAudio(wavPath)` — calls `nodejs-whisper` with configured model and language.
→ [FP-06_WHISPER_TRANSCRIBER.md](./TASKS/FP-06_WHISPER_TRANSCRIBER.md)
---
## Phase 3 — File Processor Process
### FP-07 Build the File Processor Service
**File**: `src/services/file-processor.ts`
**Deps**: FP-03, FP-04, FP-05, FP-06
New `BaseProcess` subprocess. Routes files by category: text read, image OCR, audio transcription, unknown ignored. Deletes files after processing.
→ [FP-07_FILE_PROCESSOR_SERVICE.md](./TASKS/FP-07_FILE_PROCESSOR_SERVICE.md)
---
### FP-08 Register File Processor in Orchestrator
**File**: `src/core/orchestrator.ts`
Implement monitoring and auto-restart logic for child processes.
→ [AO-02_PROCESS_SUPERVISOR.md](./TASKS/AO-02_PROCESS_SUPERVISOR.md)
### AO-03 CLI Interactive Mode
**File**: `src/shared/base-process.ts`
Support `--interactive` mode to allow manual stdin/stdout testing of any service.
→ [AO-03_CLI_INTERACTIVE_MODE.md](./TASKS/AO-03_CLI_INTERACTIVE_MODE.md)
**Deps**: FP-07
Add `file-processor` to `PROCESS_SCRIPTS` and spawn it at startup.
→ [FP-08_REGISTER_FILE_PROCESSOR.md](./TASKS/FP-08_REGISTER_FILE_PROCESSOR.md)
---
## Phase 2 — Message Bus & Router
## Phase 4 — Telegram Integration
### AO-04 Standalone Router Service
**File**: `src/core/router-service.ts`
Create a lightweight dedicated process for message routing.
→ [AO-04_ROUTER_SERVICE.md](./TASKS/AO-04_ROUTER_SERVICE.md)
### FP-09 Telegram Adapter — File Detection and Download
**File**: `src/adapters/telegram-adapter.ts`
**Deps**: FP-02, FP-03
Detect photo/document/voice/audio, download to sandbox, classify MIME type, emit `file.ingest`.
→ [FP-09_TELEGRAM_FILE_DOWNLOAD.md](./TASKS/FP-09_TELEGRAM_FILE_DOWNLOAD.md)
### AO-05 Integrate Router in Orchestrator
---
### FP-10 Orchestrator — file.ingest Handler and Context Building
**File**: `src/core/orchestrator.ts`
Refactor Orchestrator to use the Router for IPC distribution.
→ [AO-05_INTEGRATE_ROUTER.md](./TASKS/AO-05_INTEGRATE_ROUTER.md)
**Deps**: FP-08, FP-09
Handle `file.ingest`: dispatch to file-processor in parallel, collect results, warn on failures, build enrichedGoal, call `runTaskPipeline`.
→ [FP-10_ORCHESTRATOR_FILE_INGEST.md](./TASKS/FP-10_ORCHESTRATOR_FILE_INGEST.md)
---
## Phase 3 — Cron-Driven AI Queries
### AO-08 SQLite Schema Update for Cron
**File**: `src/services/cron-manager.ts`
Add `ai_query` task type support to the database and types.
→ [AO-08_CRON_SCHEMA_UPDATE.md](./TASKS/AO-08_CRON_SCHEMA_UPDATE.md)
### AO-09 CronManager AI Query Support
**File**: `src/services/cron-manager.ts`
Implement `event.cron.ai_query` emission when scheduled query fires.
→ [AO-09_CRON_AI_QUERY.md](./TASKS/AO-09_CRON_AI_QUERY.md)
### AO-10 Orchestrator Synthetic Task Pipeline
### FP-11 Long Text Chunking, Summarization, and RAG Indexing
**File**: `src/core/orchestrator.ts`
Implement `handleCronAiQuery` to trigger full AI task pipeline from cron events.
→ [AO-10_SYNTHETIC_TASK_PIPELINE.md](./TASKS/AO-10_SYNTHETIC_TASK_PIPELINE.md)
**Deps**: FP-10, rag-service
Chunk long text → summarize each chunk via model-router → insert summaries into rag-service with file metadata.
→ [FP-11_LONG_TEXT_RAG_INDEXING.md](./TASKS/FP-11_LONG_TEXT_RAG_INDEXING.md)
---
## Phase 4 — Monitoring & UI
## Phase 5 — Planner Integration and Cleanup
### AO-06 Dashboard Process Health Monitoring
**File**: `src/services/dashboard-service.ts`
Extend UI to show status, restarts, and metrics for all child processes.
→ [AO-06_DASHBOARD_HEALTH.md](./TASKS/AO-06_DASHBOARD_HEALTH.md)
### AO-07 Real-time IPC Log Viewer
**File**: `src/services/dashboard-service.ts`
Add a live log streaming interface to view cross-process communication.
→ [AO-07_IPC_LOG_VIEWER.md](./TASKS/AO-07_IPC_LOG_VIEWER.md)
### AO-11 Cron Job Management UI
**File**: `src/services/dashboard-service.ts`
Add UI section to list, add, and manage scheduled AI queries.
→ [AO-11_CRON_MGMT_UI.md](./TASKS/AO-11_CRON_MGMT_UI.md)
### FP-12 Update Planner Prompt for File Context Awareness
**File**: `src/agents/prompts/planner.ts`
**Deps**: FP-10
Add `<file_context_instructions>` section and two new few-shot examples (inline context, indexed file).
→ [FP-12_PLANNER_PROMPT_FILE_CONTEXT.md](./TASKS/FP-12_PLANNER_PROMPT_FILE_CONTEXT.md)
---
## Phase 5 — Verification
### FP-13 Upload Directory Initialization and Cleanup
**File**: `src/core/orchestrator.ts`
**Deps**: FP-02, FP-07
Create upload dir at startup; clear orphaned files (older than 1h) at startup.
→ [FP-13_UPLOAD_DIR_CLEANUP.md](./TASKS/FP-13_UPLOAD_DIR_CLEANUP.md)
### AO-12 Test: Supervisor Auto-Restart
**File**: `src/tests/supervisor.test.ts`
Verify that killing a process triggers an automatic restart by the supervisor.
→ [AO-12_TEST_AUTO_RESTART.md](./TASKS/AO-12_TEST_AUTO_RESTART.md)
---
### AO-13 Test: Cron-Driven AI Task
**File**: `src/tests/cron-ai.test.ts`
Verify the full flow from cron trigger to task completion and Telegram notification.
→ [AO-13_TEST_CRON_AI_FLOW.md](./TASKS/AO-13_TEST_CRON_AI_FLOW.md)
## Phase 6 — Documentation and Verification
### AO-14 E2E Verification
**File**: Manual
Final validation of all "Advanced Architecture" features.
→ [AO-14_E2E_VERIFICATION.md](./TASKS/AO-14_E2E_VERIFICATION.md)
### FP-14 Update Documentation
**Files**: `_docs/COMPONENTS.md`, `_docs/TECH.md`, `_docs/MESSAGE PROTOCOL SPEC.md`, `README.md`
**Deps**: FP-07, FP-09, FP-10
Document the new subsystem in all architecture and user-facing docs.
→ [FP-14_UPDATE_DOCS.md](./TASKS/FP-14_UPDATE_DOCS.md)
---
### FP-15 End-to-End Verification
**Files**: Manual testing
**Deps**: FP-01 through FP-14
Manual verification of all file types: text (short + long), image (OCR + description), audio (voice + mp3), mixed, and edge/failure cases.
→ [FP-15_E2E_VERIFICATION.md](./TASKS/FP-15_E2E_VERIFICATION.md)

250
_board/_BOARD.md
View File

@@ -2,119 +2,147 @@
## To Do
### AO-01 Standardize BaseProcess Lifecycle
- tags: [todo, infra, core]
- defaultExpanded: true
```md
Add health checks, heartbeats, and standardized status reporting to BaseProcess.
Source: _board/TASKS/AO-01_BASE_PROCESS_LIFECYCLE.md
```
### AO-02 Implement Process Supervisor
- tags: [todo, orchestrator, core]
- defaultExpanded: false
```md
Implement monitoring and auto-restart logic for child processes in Orchestrator.
Source: _board/TASKS/AO-02_PROCESS_SUPERVISOR.md
```
### AO-03 CLI Interactive Mode
- tags: [todo, shared, devtools]
- defaultExpanded: false
```md
Support --interactive flag for manual service testing via stdin.
Source: _board/TASKS/AO-03_CLI_INTERACTIVE_MODE.md
```
### AO-04 Standalone Router Service
- tags: [todo, service, bus]
- defaultExpanded: false
```md
Create a lightweight dedicated process for message routing.
Source: _board/TASKS/AO-04_ROUTER_SERVICE.md
```
### AO-05 Integrate Router in Orchestrator
- tags: [todo, orchestrator, bus]
- defaultExpanded: false
```md
Refactor Orchestrator to use the Router for IPC distribution.
Source: _board/TASKS/AO-05_INTEGRATE_ROUTER.md
```
### AO-06 Dashboard Process Health Monitoring
- tags: [todo, dashboard, admin]
- defaultExpanded: false
```md
Extend UI to show status, restarts, and metrics for all child processes.
Source: _board/TASKS/AO-06_DASHBOARD_HEALTH.md
```
### AO-07 Real-time IPC Log Viewer
- tags: [todo, dashboard, debug]
- defaultExpanded: false
```md
Add a live log streaming interface to dashboard for IPC debugging.
Source: _board/TASKS/AO-07_IPC_LOG_VIEWER.md
```
### AO-08 SQLite Schema Update for Cron
- tags: [todo, services, database]
- defaultExpanded: false
```md
Add ai_query task type support to the cron database.
Source: _board/TASKS/AO-08_CRON_SCHEMA_UPDATE.md
```
### AO-09 CronManager AI Query Support
- tags: [todo, services, cron]
- defaultExpanded: false
```md
Implement event.cron.ai_query emission for scheduled AI tasks.
Source: _board/TASKS/AO-09_CRON_AI_QUERY.md
```
### AO-10 Orchestrator Synthetic Task Pipeline
- tags: [todo, orchestrator, ai]
- defaultExpanded: false
```md
Connect cron AI events to the full agent task pipeline.
Source: _board/TASKS/AO-10_SYNTHETIC_TASK_PIPELINE.md
```
### AO-11 Cron Job Management UI
- tags: [todo, dashboard, cron]
- defaultExpanded: false
```md
Add UI section to manage scheduled AI queries.
Source: _board/TASKS/AO-11_CRON_MGMT_UI.md
```
### AO-12 Test: Supervisor Auto-Restart
- tags: [todo, testing, qa]
- defaultExpanded: false
```md
Automated verification of process restart capability.
Source: _board/TASKS/AO-12_TEST_AUTO_RESTART.md
```
### AO-13 Test: Cron-Driven AI Task
- tags: [todo, testing, integration]
- defaultExpanded: false
```md
Verify full flow from cron trigger to autonomous AI execution.
Source: _board/TASKS/AO-13_TEST_CRON_AI_FLOW.md
```
### AO-14 E2E Verification
- tags: [todo, testing, e2e]
- defaultExpanded: false
```md
Final manual verification of the new architecture and features.
Source: _board/TASKS/AO-14_E2E_VERIFICATION.md
```
## In Progress
### FP-15 End-to-End Verification
- tags: [in-progress, qa, e2e]
- defaultExpanded: true
```md
Manual verification of all file types, edge cases, and failure scenarios via Telegram.
Source: FP-15_E2E_VERIFICATION.md
```
## Done
*(Previous tasks moved to archive or deleted as per request)*
### FP-14 Update Documentation
- tags: [done, docs]
- defaultExpanded: false
```md
Updated COMPONENTS.md, TECH.md, MESSAGE PROTOCOL SPEC.md, and README.md.
Source: FP-14_UPDATE_DOCS.md
```
### FP-13 Upload Directory Init and Cleanup
- tags: [done, orchestrator, infra]
- defaultExpanded: false
```md
initUploadDirectory() creates upload dir and purges orphaned files (>1h) on startup.
Source: FP-13_UPLOAD_DIR_CLEANUP.md
```
### FP-12 Update Planner Prompt for File Context
- tags: [done, planner, prompt]
- defaultExpanded: false
```md
Added <file_context_awareness> block to PLANNER_SYSTEM_PROMPT.
Documents text/image/audio/indexed file fences and guidance.
Source: FP-12_PLANNER_PROMPT_FILE_CONTEXT.md
```
### FP-11 Long Text Chunking and RAG Indexing
- tags: [done, orchestrator, rag]
- defaultExpanded: false
```md
indexLongText(): 2k-char chunks, 3-at-a-time summarisation, RAG insert with metadata.
Source: FP-11_LONG_TEXT_RAG_INDEXING.md
```
### FP-10 Orchestrator — file.ingest Handler
- tags: [done, orchestrator, core]
- defaultExpanded: false
```md
handleFileIngest(): parallel processing, enrichedGoal builder, 32k char cap, user warnings.
Source: FP-10_ORCHESTRATOR_FILE_INGEST.md
```
### FP-09 Telegram Adapter — File Detection and Download
- tags: [done, telegram, adapter]
- defaultExpanded: false
```md
Detects photo/document/voice/audio, size-guards, downloads, classifies, emits file.ingest.
Source: FP-09_TELEGRAM_FILE_DOWNLOAD.md
```
### FP-08 Register File Processor in Orchestrator
- tags: [done, orchestrator, infra]
- defaultExpanded: false
```md
Added 'file-processor' to PROCESS_SCRIPTS; spawned at startup alongside other services.
Source: FP-08_REGISTER_FILE_PROCESSOR.md
```
### FP-07 Build the File Processor Service
- tags: [done, service, core]
- defaultExpanded: false
```md
file-processor.ts BaseProcess: routes text/image/audio/unknown, deletes files, emits audit events.
Source: FP-07_FILE_PROCESSOR_SERVICE.md
```
### FP-06 Implement Whisper Transcription Utility
- tags: [done, util, audio]
- defaultExpanded: false
```md
Created src/utils/whisper-transcriber.ts. transcribeAudio() with 5-min timeout,
auto-download, first-run UX. Build clean, 156 tests pass.
Source: FP-06_WHISPER_TRANSCRIBER.md
```
### FP-05 Implement Audio Conversion Utility
- tags: [done, util, audio]
- defaultExpanded: false
```md
Created src/utils/audio-converter.ts. convertToWav() with ffmpeg-static,
60s timeout, stderr capture. Build clean, 156 tests pass.
Source: FP-05_AUDIO_CONVERTER.md
```
### FP-04 Extend OllamaAdapter with Vision Support
- tags: [done, service, ollama]
- defaultExpanded: false
```md
Added chatWithImage() with base64 image injection into Ollama multimodal messages.
Reuses fetchWithRetry. Build clean, 156 tests pass.
Source: FP-04_OLLAMA_VISION.md
```
### FP-03 Define File Processing Protocol Types
- tags: [done, infra, protocol]
- defaultExpanded: false
```md
Created src/shared/file-protocol.ts with all shared types and classifyMimeType() helper.
Updated MESSAGE PROTOCOL SPEC.md. Build and tests pass.
Source: FP-03_PROTOCOL_TYPES.md
```
### FP-02 Add Config Types and Defaults
- tags: [done, infra, config]
- defaultExpanded: false
```md
Added WhisperConfig and FileProcessorConfig interfaces, defaults, env var overrides.
Updated config.json.example. All 156 tests pass.
Source: FP-02_CONFIG_TYPES.md
```
### FP-01 Add npm Dependencies
- tags: [done, infra, deps]
- defaultExpanded: false
```md
Installed nodejs-whisper ^0.2.9, ffmpeg-static ^5.3.0, @types/ffmpeg-static ^5.1.0.
Both confirmed ESM-compatible. Build passes.
Source: FP-01_ADD_DEPENDENCIES.md
```
### DB-07 Orchestrator Integration & Notion UI
- tags: [done, ui, orchestrator]
- defaultExpanded: false
```md
Converted the dashboard to a TypeScript service, integrated it into the Orchestrator, added IPC logging, and implemented a Notion-like UI with light/dark theme support.
Source: src/services/dashboard-service.ts
```

65
src/shared/__tests__/base-process.test.ts Normal file
View File

@@ -0,0 +1,65 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { BaseProcess } from "../base-process.js";
import { envelopeSchema } from "../../protocol.js";
describe("BaseProcess", () => {
let bp: BaseProcess;
const processName = "test-process";
beforeEach(() => {
bp = new BaseProcess({ processName, heartbeatInterval: 100 });
});
afterEach(() => {
bp.stop();
});
it("should start with 'starting' status", () => {
// @ts-ignore - reaching into protected for test
expect(bp.status).toBe("starting");
});
it("should change status to 'ready' on start", () => {
bp.start();
// @ts-ignore
expect(bp.status).toBe("ready");
});
it("should emit heartbeats periodically", async () => {
const stdoutSpy = vi.spyOn(process.stdout, "write").mockImplementation(() => true);
bp.start();
// Wait for at least two heartbeats (one on start, one after 100ms)
await new Promise(resolve => setTimeout(resolve, 150));
expect(stdoutSpy).toHaveBeenCalled();
const output = stdoutSpy.mock.calls.map(call => call[0].toString());
const heartbeats = output.map(line => JSON.parse(line))
.filter(env => env.type === "event.system.heartbeat");
expect(heartbeats.length).toBeGreaterThanOrEqual(2);
expect(heartbeats[0].payload.status).toBe("ready");
expect(heartbeats[0].from).toBe(processName);
stdoutSpy.mockRestore();
});
it("should include memory and uptime in heartbeat", async () => {
const stdoutSpy = vi.spyOn(process.stdout, "write").mockImplementation(() => true);
bp.start();
await new Promise(resolve => setTimeout(resolve, 50));
const line = stdoutSpy.mock.calls[0][0].toString();
const env = JSON.parse(line);
expect(env.payload.memory).toBeDefined();
expect(env.payload.memory.rss).toBeGreaterThan(0);
expect(env.payload.uptime).toBeGreaterThanOrEqual(0);
stdoutSpy.mockRestore();
});
});

79
src/shared/base-process.ts
View File

@@ -5,12 +5,15 @@
import { EventEmitter } from "node:events";
import { createInterface } from "node:readline";
import type { Envelope } from "./protocol.js";
import { envelopeSchema } from "./protocol.js";
import { randomUUID } from "node:crypto";
import type { Envelope, ProcessStatus, HeartbeatPayload } from "./protocol.js";
import { envelopeSchema, PROTOCOL_VERSION } from "./protocol.js";
export interface BaseProcessOptions {
/** Process name used as default `from` in outgoing messages. */
processName: string;
/** How often to send heartbeats in ms. Default 10s. */
heartbeatInterval?: number;
}
export interface BaseProcessEvents {
@@ -21,26 +24,95 @@ export interface BaseProcessEvents {
/**
* Base process: reads JSONL from stdin, validates with Zod, emits messages;
* writes validated JSONL to stdout.
* Includes lifecycle management and periodic heartbeats.
*/
export class BaseProcess extends EventEmitter {
readonly processName: string;
protected status: ProcessStatus = "starting";
private rl: ReturnType<typeof createInterface> | null = null;
private running = false;
private heartbeatTimer: NodeJS.Timeout | null = null;
private readonly heartbeatInterval: number;
private readonly startTime: number;
constructor(options: BaseProcessOptions) {
super();
this.processName = options.processName;
this.heartbeatInterval = options.heartbeatInterval ?? 10000;
this.startTime = Date.now();
}
/**
* Start reading stdin. Call once after setting up message handler.
* Start reading stdin and begin heartbeats.
*/
start(): void {
if (this.running) return;
this.running = true;
this.status = "ready";
this.rl = createInterface({ input: process.stdin, terminal: false });
this.rl.on("line", (line: string) => this.handleLine(line));
this.rl.on("close", () => this.handleClose());
this.startHeartbeat();
}
/**
* Stop reading and stop heartbeats.
*/
stop(): void {
this.status = "stopping";
this.stopHeartbeat();
if (this.rl) {
this.rl.close();
this.rl = null;
}
this.running = false;
}
protected setStatus(status: ProcessStatus): void {
this.status = status;
this.sendHeartbeat(); // Immediate heartbeat on status change
}
private startHeartbeat(): void {
this.stopHeartbeat();
this.sendHeartbeat();
this.heartbeatTimer = setInterval(() => this.sendHeartbeat(), this.heartbeatInterval);
}
private stopHeartbeat(): void {
if (this.heartbeatTimer) {
clearInterval(this.heartbeatTimer);
this.heartbeatTimer = null;
}
}
private sendHeartbeat(): void {
const memory = process.memoryUsage();
const payload: HeartbeatPayload = {
status: this.status,
uptime: Math.floor((Date.now() - this.startTime) / 1000),
memory: {
rss: memory.rss,
heapTotal: memory.heapTotal,
heapUsed: memory.heapUsed,
external: memory.external,
},
version: "1.0.1", // TODO: pull from package.json or config
};
const envelope: Envelope<HeartbeatPayload> = {
id: randomUUID(),
timestamp: Date.now(),
from: this.processName,
to: "core", // System events usually go to core/supervisor
type: "event.system.heartbeat",
version: PROTOCOL_VERSION,
payload,
};
this.send(envelope);
}
/**
@@ -72,6 +144,7 @@ export class BaseProcess extends EventEmitter {
private handleClose(): void {
this.running = false;
this.status = "stopping";
}
/**

19
src/shared/protocol.ts
View File

@@ -68,6 +68,25 @@ export const eventSchema = envelopeSchema.extend({
export type Event<T = unknown> = z.infer<typeof eventSchema> & { payload: T };
// --- System Events ---
export const processStatusSchema = z.enum(["starting", "ready", "degraded", "stopping"]);
export type ProcessStatus = z.infer<typeof processStatusSchema>;
export const heartbeatPayloadSchema = z.object({
status: processStatusSchema,
uptime: z.number(),
memory: z.object({
rss: z.number(),
heapTotal: z.number(),
heapUsed: z.number(),
external: z.number(),
}),
version: z.string(),
});
export type HeartbeatPayload = z.infer<typeof heartbeatPayloadSchema>;
// --- Helpers ---
export const PROTOCOL_VERSION = VERSION;