mirror of
https://github.com/browseros-ai/BrowserOS.git
synced 2026-05-20 12:30:04 +00:00
231bd6821d
* chore(eval): pin agisdk version to prevent silent dataset drift `pip install agisdk` previously fetched whatever version pip resolved at CI time. If agisdk publishes a new version with changed task definitions or grader behavior, the weekly eval silently shifts under our feet — making "did the score move because of code or data?" unanswerable. Pin to agisdk==0.3.5 (the version we currently develop against). Bump intentionally with a documented re-baseline run. * fix(eval): exclude 4 more tasks identified by 8-trial never-passing audit After 8 trials across K2.5 + Opus 4.6 (Phase 1 and Phase 2), 5 tasks never passed. Per-task root-cause investigation via parallel deep-dive subagents flagged 4 of them as fundamentally unfixable in the eval pipeline as it stands; the 5th (`dashdish-5`) is a prompt-rule fix that stays in. - gocalendar-7: goal/grader contradiction. Goal says "move event to July 19, 10 AM"; grader expects `eventsDiff.updated.*.start == "2024-07-18T17:00Z"` (= July 18, 10 AM PDT — same day, 1 hour shift). Even after the Phase 2 HTML5 dnd dispatch fix correctly populates `eventsDiff.updated`, the values are July 19 (matching the goal), which the grader rejects. - staynb-5: grader hardcodes literal `'Oct 13 2025'` and `'Oct 23 2025'` year strings. The staynb date picker interprets bare "Oct 13" as the most-recent-past instance (currently 2024 since today is 2026), not 2025. No agent can produce a persisted date string containing 2025. - staynb-9: under-specified task. Goal says "maximum number of guests supported"; grader requires the very specific string "32 Guests, 16 Infants" — encoding UI knowledge (Adults+Children=Guests display, Infants render separately, per-category cap=16, Pets excluded) that isn't in the prompt. Even Opus 4.6 stopped at 16 across 3 trials. - opendining-3: grader requires `contains(booking.date, '2024-07-20')` but the React-controlled date textbox flakily no-ops on `fill`. 3/8 trial pass rate is essentially coin-flip noise driven by tool-fidelity variance rather than agent capability. Removing to reduce score noise; Phase 2 fill post-validate warning helps when it does work, but the task's signal-to-noise is too low for the eval set. Dataset goes from 40 -> 36 tasks. Total EXCLUDED_TASKS now 11 entries. Validated by 8-trial pass-record audit; deep-dive notes saved to plans/audits/.