feat: implement research skill with lynx and ddg

_board/PLAN.md

@@ -587,4 +587,46 @@ The strategy involves two paths:
 1.  **Time Skill Test**: Use the "time" skill and verify that the Planner either:
     - Creates a `shell (date)` node followed by a `skill (time)` node.
     - Or the \`skill (time)\` node executes the tool itself without hallucinating a shell error.
-2.  **Complex Skill Test**: Create a skill that requires web search and verify it correctly gathers data or requests the search tool.
+# Phase P11: Research Skill with Lynx & DuckDuckGo
+
+## Overview
+
+Implement a new persistent skill called `research` that leverages the `lynx` text-based browser and DuckDuckGo's HTML interface. This skill enables deep web research by allowing the agent to:
+1.  Perform web searches via `https://html.duckduckgo.com/html?q={QUERY}`.
+2.  Navigate recursively through links found in search results.
+3.  Gather and consolidate information from multiple pages into a single high-quality response.
+
+The skill will be implemented using the existing Dynamic Skills System (Phase P9), meaning it will consist of a manifest entry and a specialized `SKILL.md` instruction file.
+
+## Proposed Changes
+
+### Component 1: Environment Guard
+- **File**: `src/core/orchestrator.ts`
+- **Action**: Add `verifySystemDependencies()` method to check for `lynx` availability at startup.
+- **Goal**: Early failure if dependencies are missing on the host machine.
+
+### Component 2: Research Skill Manifest
+- **File**: `skills/CONFIG.md`
+- **Action**: Add an entry for the `research` skill:
+  `| research | Deep web research using lynx. Use this for fact-checking, gathering news, or deep dives into specific topics. |`
+
+### Component 3: Research Skill Instructions
+- **NEW File**: `skills/research/SKILL.md`
+- **Content**: 
+    - Search syntax: `lynx -dump "https://html.duckduckgo.com/html?q={QUERY}"`.
+    - Browsing syntax: `lynx -dump {URL}`.
+    - Protocol for link discovery: Scanning the `References` section of the dump.
+    - Instructions for recursive depth (depth 1-2 standard).
+    - Requirement to consolidate findings before final response.
+
+### Component 4: Planner Optimization
+- **File**: `src/agents/prompts/planner.ts`
+- **Action**: Add a few-shot example that uses the `research` skill for a multi-step investigation.
+
+## Verification Plan
+
+### Manual Verification
+1.  **Dependency Test**: Uninstall `lynx` (if possible/safe) or mock failure to ensure Orchestrator logs a warning.
+2.  **Basic Search**: Ask "Search for the current price of Bitcoin" and verify it uses the `research` skill.
+3.  **Deep Dive**: Ask "Research the history of the Svelte framework and its key contributors" and verify it follows links to Wikipedia or GitHub.
+4.  **Consolidation**: Verify that the final response is a well-formatted summary, not raw dump output.

_board/TASKS.md

@@ -676,4 +676,45 @@ Improve LLM outputs by ensuring that data gathered from tools (search, shell, we
 **Description**: Investigate and implement a way for `skill` nodes to perform their own tool calls if the model supports it.
 **Acceptance Criteria**:
 - Defined protocol for tool-calling within the GeneratorService.
-- Updated ExecutorAgent to support multi-turn skill execution if needed.
+# Research Skill Tasks
+
+## Phase 11: Core Research Skill Implementation
+
+### Task SK-RS-01: Verify Lynx Dependency in Orchestrator
+**File**: `src/core/orchestrator.ts`
+**Description**: Implement a check at startup to ensure `lynx` is installed on the system.
+**Acceptance Criteria**:
+- Add `verifySystemDependencies()` method.
+- Check `lynx --version` or `command -v lynx`.
+- Log an error or warning if missing.
+- Orchestrator continues to run but skill functionality is flagged as degraded.
+
+### Task SK-RS-02: Create Research Skill Prompt
+**File**: `skills/research/SKILL.md`
+**Description**: Create the instruction file for the research skill.
+**Acceptance Criteria**:
+- Define DuckDuckGo search URL: `https://html.duckduckgo.com/html?q={QUERY}`.
+- Instruct LLM to use `lynx -dump` for both searching and browsing.
+- Provide clear instructions on link extraction from `References` section.
+- Set recursive depth limits and summarization quality standards.
+
+### Task SK-RS-03: Register Research Skill in Manifest
+**File**: `skills/CONFIG.md`
+**Description**: Register the new skill in the global skills configuration.
+**Acceptance Criteria**:
+- Add `research` entry with description.
+- Ensure the description highlights its capability for deep web searches.
+
+### Task SK-RS-04: Update Planner with Research Guidance
+**File**: `src/agents/prompts/planner.ts`
+**Description**: Add a few-shot example to the planner to encourage the use of the `research` skill for information gathering.
+**Acceptance Criteria**:
+- New example showing task -> search -> browse -> summarize.
+- Instructions to prefer `research` over generic `http_search` for complex queries.
+
+### Task SK-RS-05: E2E Verification of Research Skill
+**Description**: Perform full end-to-end testing of the research capability.
+**Acceptance Criteria**:
+- Ask the agent complex questions requiring multi-step web browsing.
+- Verify the agent follows links and consolidates findings.
+- Verify the final response is informative and well-reasoned.

_board/TASKS/SK-RS-01_VERIFY_LYNX.md

@@ -0,0 +1,16 @@
+# Task: Verify Lynx Dependency in Orchestrator
+
+## Description
+Implement a check at startup in the Orchestrator to ensure `lynx` is installed on the system. This prevents runtime errors when the `research` skill is invoked.
+
+## File
+- `src/core/orchestrator.ts`
+
+## Dependencies
+- None
+
+## Acceptance Criteria
+- [ ] Add `verifySystemDependencies()` method to `Orchestrator` class.
+- [ ] Execute `lynx --version` or `command -v lynx` during startup.
+- [ ] Log a high-visibility warning `[Core] ⚠️ Warning: 'lynx' not found. Research skill will be non-functional.` if missing.
+- [ ] Ensure the application continues to start regardless of the check result.

_board/TASKS/SK-RS-02_RESEARCH_PROMPT.md

@@ -0,0 +1,18 @@
+# Task: Create Research Skill Prompt
+
+## Description
+Create the `SKILL.md` file for the `research` skill, containing detailed instructions for the LLM on how to perform web research using `lynx` and DuckDuckGo.
+
+## File
+- `skills/research/SKILL.md`
+
+## Dependencies
+- Phase P9 (Dynamic Skills System)
+
+## Acceptance Criteria
+- [ ] Create `skills/research/` directory.
+- [ ] Define DuckDuckGo HTML search URL: `https://html.duckduckgo.com/html?q={SEARCH+QUERY}`.
+- [ ] Instruct LLM to use `lynx -dump` to get text representation of pages.
+- [ ] Include detailed instructions for link navigation via the `References` section of the dump.
+- [ ] Define standard browsing depth (e.g., recursive depth of 2).
+- [ ] Mandate a final synthesis step to merge all gathered data.

_board/TASKS/SK-RS-03_REGISTER_SKILL.md

@@ -0,0 +1,15 @@
+# Task: Register Research Skill in Manifest
+
+## Description
+Register the `research` skill in the global skills configuration file so the Planner and Executor can discover it.
+
+## File
+- `skills/CONFIG.md`
+
+## Dependencies
+- SK-RS-02
+
+## Acceptance Criteria
+- [ ] Add a row to the skills table in `skills/CONFIG.md`.
+- [ ] Name: `research`.
+- [ ] Description: `Deep web research using lynx. Use this for fact-checking, gathering news, or deep dives into specific topics.`

_board/TASKS/SK-RS-04_PLANNER_GUIDANCE.md

@@ -0,0 +1,15 @@
+# Task: Update Planner with Research Guidance
+
+## Description
+Add a few-shot example to the Planner Agent's prompt to guide it on when and how to use the `research` skill effectively for complex information-gathering tasks.
+
+## File
+- `src/agents/prompts/planner.ts`
+
+## Dependencies
+- SK-RS-03
+
+## Acceptance Criteria
+- [ ] Update `buildPlannerPrompt` to include a specific research use case in the examples.
+- [ ] Ensure the planner understands that `research` involves multiple recursive steps (search -> links -> summarize).
+- [ ] Add instructions to prefer the `research` skill over simple search tools for open-ended queries.

_board/TASKS/SK-RS-05_E2E_VERIFICATION.md

@@ -0,0 +1,13 @@
+# Task: E2E Verification of Research Skill
+
+## Description
+Perform manual and automated end-to-end testing of the `research` skill to ensure it correctly navigates the web and consolidates information.
+
+## Dependencies
+- SK-RS-04
+
+## Acceptance Criteria
+- [ ] Verify `lynx` search works via shell.
+- [ ] Verify the agent can follow links from the search result.
+- [ ] Test with a complex query (e.g., "Research the latest advancements in solid-state batteries as of 2024").
+- [ ] Confirm the final response is informative and formatted in readable Markdown.

_board/_BOARD.md

@@ -12,8 +12,63 @@
 
 ## In Progress
 
+
+
+
+
+
 ## Done
 
+### SK-RS-05 E2E Verification of Research Skill
+  - tags: [done, qa]
+  - defaultExpanded: false
+    ```md
+    Perform end-to-end testing of the web research capability.
+    
+    Source: SK-RS-05_E2E_VERIFICATION.md
+    ```
+
+
+### SK-RS-04 Update Planner with Research Guidance
+  - tags: [done, planner]
+  - defaultExpanded: false
+    ```md
+    Add research-focused few-shot examples to the planner prompt.
+    
+    Source: SK-RS-04_PLANNER_GUIDANCE.md
+    ```
+
+
+### SK-RS-03 Register Research Skill in Manifest
+  - tags: [done, skill]
+  - defaultExpanded: false
+    ```md
+    Add the research skill to skills/CONFIG.md.
+    
+    Source: SK-RS-03_REGISTER_SKILL.md
+    ```
+
+
+### SK-RS-02 Create Research Skill Prompt
+  - tags: [done, skill]
+  - defaultExpanded: false
+    ```md
+    Create the SKILL.md instruction file for web research using lynx and DuckDuckGo.
+    
+    Source: SK-RS-02_RESEARCH_PROMPT.md
+    ```
+
+
+### SK-RS-01 Verify Lynx Dependency in Orchestrator
+  - tags: [done, core]
+  - defaultExpanded: false
+    ```md
+    Implement a startup check in orchestrator.ts to ensure lynx is installed on the host.
+    
+    Source: SK-RS-01_VERIFY_LYNX.md
+    ```
+
+
 ### SK-09 Planner Dependency Rule for Skills
   - tags: [done]
   - defaultExpanded: false

skills/CONFIG.md

@@ -3,3 +3,4 @@
 | Name | Description |
 | --- | --- |
 | weather | ALWAYS use this skill to get the current weather and forecasts. NEVER use any other tool for this purpose. |
+| research | Deep web research using lynx. Use this for fact-checking, gathering news, or deep dives into specific topics. |

skills/research/SKILL.md

@@ -0,0 +1,79 @@
+# Research Skill
+
+Deep web research using the text-based browser `lynx` and DuckDuckGo HTML interface.
+
+## When to Use
+
+✅ **USE this skill when:**
+
+- You need current information from the web.
+- You need to deep dive into a topic by following multiple links.
+- Fact-checking or verifying news.
+- Researching technical documentation or complex subjects.
+
+❌ **DON'T use this skill when:**
+
+- The information is already available in your training data (unless it's time-sensitive).
+- You can solve the task with a simple search and the snippet is enough.
+
+## Strategy
+
+Web research is a multi-step process. Do not stop at the first page of search results.
+
+1.  **Search**: Start with a precise DuckDuckGo query.
+2.  **Analyze**: Scan the search results. Identify 2-3 most promising links.
+3.  **Browse**: Visit those links one by one.
+4.  **Recurse**: If a page contains a "References" section or promising links to deeper info, follow them.
+5.  **Summarize**: Gather all key findings and provide a comprehensive response.
+
+## Commands
+
+### 1. Perform a Search
+
+Use DuckDuckGo's HTML or Lite interface with a standard User-Agent to avoid blocks.
+
+```bash
+lynx -useragent="Lynx/2.8.9rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/1.1.1g" -dump "https://html.duckduckgo.com/html?q=YOUR+SEARCH+QUERY"
+```
+
+### 2. Browse a URL
+
+```bash
+lynx -useragent="Lynx/2.8.9rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/1.1.1g" -dump "https://example.com/some/path"
+```
+
+## How to Navigate Links & Redirects
+
+### Links
+When you use `lynx -dump`, the output contains a **References** section at the bottom.
+In the main text, numbers in brackets like `[1]` correspond to these links.
+
+### Redirects (e.g., from DuckDuckGo)
+DuckDuckGo search results often point to redirect URLs. If `lynx -dump` returns a page with a "REFRESH" link or just a single link, you MUST follow it to get the actual content.
+
+**Example of DDG redirect output:**
+```text
+   REFRESH(0 sec):
+   [1]https://actual-destination.com/article
+```
+In this case, execute a new search on the URL provided in `[1]`.
+
+**Protocol**:
+- List the URLs from the References section that you want to visit.
+- Execute new `lynx -dump` commands for those URLs.
+
+## Guidelines
+
+- **Max Depth**: Typically go 1-2 levels deep from the search results.
+- **Data Extraction**: Extract only relevant text. Ignore obvious navigation menus or ads.
+- **Synthesis**: Your final output should not be a raw data dump. It must be a structured, factual answer to the user's initial goal.
+- **Parallelism**: You can plan multiple `lynx` commands in parallel for different URLs if the Planner allows it.
+
+## Example Workflow
+
+User Goal: "Research the current status of the RISC-V ecosystem in 2024."
+
+1. `lynx -dump "https://html.duckduckgo.com/html?q=RISC-V+ecosystem+status+2024"`
+2. Identify links for "RISC-V International news", "Phoronix RISC-V benchmarks", and "Wikipedia RISC-V".
+3. `lynx -dump "URL_FROM_PREVIOUS_STEP"` for each.
+4. Synthesize findings into a categorized report (Hardware, Software Support, Benchmarks, Corporate Adoption).

src/agents/prompts/analyzer.ts

@@ -18,7 +18,8 @@ Your goal is to synthesize raw tool outputs into a clear response optimized for
    - *Bold*: *text* or **text**
    - _Italic_: _text_
    - \`Code\`: \`inline code\` or \`\`\`language\n pre-formatted block \`\`\`
-   - > Quotes: Use for citations.
+   - > Quotes: Use for highlighting important information or citations. 
+   - For simple charts or graphs, use \`\`\`language\n pre-formatted block \`\`\`.
 4. **Links**: Use [title](url) syntax.
 
 ## ANALYSIS GUIDELINES:

src/agents/prompts/planner.ts

@@ -95,6 +95,37 @@ User: "who won the F1 race today?"
     { "from": "f1-search", "to": "f1-report" }
   ]
 }
+
+## Example: Deep Research
+User: "Deep dive into the current status of the RISC-V ecosystem."
+{
+  "taskId": "task-riscv",
+  "complexity": "large",
+  "reflectionMode": "OFF",
+  "nodes": [
+    {
+      "id": "research-eco",
+      "type": "skill",
+      "service": "executor",
+      "input": { 
+        "skillName": "research", 
+        "task": "Investigate RISC-V hardware, software support, and corporate adoption in 2024. Use search first, then follow key documentation links." 
+      }
+    },
+    {
+      "id": "final-report",
+      "type": "generate_text",
+      "service": "model-router",
+      "input": {
+        "prompt": "Consolidate the RISC-V research into a comprehensive report.",
+        "system_prompt": "analyzer"
+      }
+    }
+  ],
+  "edges": [
+    { "from": "research-eco", "to": "final-report" }
+  ]
+}
 </examples>`;
 
 export interface PlannerPromptOptions {

src/core/orchestrator.ts

@@ -5,7 +5,8 @@
  */
 
 import { createInterface } from "node:readline";
-import { spawn, type ChildProcess } from "node:child_process";
+import { spawn, exec, type ChildProcess } from "node:child_process";
+import { promisify } from "node:util";
 import { randomUUID } from "node:crypto";
 import { join, dirname } from "node:path";
 import { fileURLToPath } from "node:url";
@@ -644,7 +645,21 @@ export class Orchestrator {
     });
   }
 
+  private async verifySystemDependencies(): Promise<void> {
+    const execAsync = promisify(exec);
+    try {
+      await execAsync("lynx --version");
+      ConsoleLogger.info("core", "Dependency check passed: lynx found.");
+    } catch (err) {
+      ConsoleLogger.warn("core", "Dependency check failed: 'lynx' not found. Research skill will be non-functional.");
+    }
+  }
+
   start(): void {
+    this.verifySystemDependencies().catch((err) => {
+      ConsoleLogger.error("core", "Dependency verification error", err);
+    });
+
     for (const [name, scriptPath] of Object.entries(PROCESS_SCRIPTS)) {
       this.spawnProcess(name, scriptPath);
     }