fix(eval): exclude 5 more tasks where pipeline (not agent) fails

Extends EXCLUDED_TASKS to 7 entries based on the K2.5 + Opus 4.6 head-to-head deep-dive on the 2026-04-28 runs. The exclusion rule: remove a task only if it is unsolvable for any agent — either the task data is invalid, the eval site is broken, or the grader penalizes correct work. Tasks that fail because of our agent's tool fidelity (drag, custom-widget fill, click on React submit, etc.) STAY in — those are real capability gaps the team should see in the score. New exclusions: - fly-unified-9: goal references "Dec 18 2024 at 10:00" but the live eval site has only 2025 inventory and no 10:00 slot. Both models successfully booked the closest available flight and were penalized on a grader expectation that can never be met. - fly-unified-4: eval site stores wall-clock flight times as bare UTC (T08:00:00.000Z) while the grader expects them shifted by 8h (T16:00:00.000Z = 8 AM PST). Opus 4.6 completed the entire booking correctly. Eval-site TZ-storage bug. - gomail-8: goal says "Clear all emails from GitHub in the inbox", but criterion 3 expects exactly 1 email updated. Both K2.5 and Opus correctly cleared all 4 GitHub emails. Grader contradicts goal. - networkin-6: goal says "Choose a random person you haven't connected with"; grader hardcodes profilesDiff.updated."4".connectionGrade. Both models randomized correctly and missed id 4. Grader contradicts goal. - networkin-9: eval site's searchHistoryDiff doesn't record queries submitted via the autocomplete + Enter path. Opus 4.6 completed the task end-to-end (Stanford alum, connection request, message); only failed because the search-history criterion was never written server-side. Eval-site bug. Dataset goes from 45 -> 40 tasks. Score impact (same K2.5/Opus runs, recomputed against the cleaned 40-task denominator): K2.5: 21/45 (46.7%) -> 21/40 (52.5%) Opus 4.6: 28/45 (62.2%) -> 28/40 (70.0%) Δ: 15.6 pp -> 17.5 pp (real model gap, less pipeline noise)
feat(eval): add lenient-strings grader softening
2026-05-13 23:53:25 +00:00 · 2026-04-28 21:56:03 +05:30 · 2026-04-28 20:29:48 +05:30 · 2026-04-28 20:29:47 +05:30
3 changed files with 138 additions and 16 deletions
--- a/packages/browseros-agent/apps/eval/data/agisdk-real.jsonl
+++ b/packages/browseros-agent/apps/eval/data/agisdk-real.jsonl
@@ -1,20 +1,16 @@
 {"query_id": "agisdk-dashdish-10", "dataset": "agisdk-real", "query": "Place an order from \"Souvla\" for a \"Medium Classic Cheeseburger\" and a \"Small Bacon Double Cheeseburger\" with \"Standard Delivery\" as the method with the default charged options.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-10", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}
-{"query_id": "agisdk-fly-unified-5", "dataset": "agisdk-real", "query": "Find me the cheapest fare for a flight from Orlando to Milwaukee on December 5th, 2024 and book it.\nPassenger: John Doe\nDate of Birth: 01/01/1990\nSex: Male\nSeat Selection: No\nPayment: Credit Card (378342143523967), Exp: 12/25, Security Code: 420 Address: 123 Main St, San Francisco, CA, 94105, USA, Phone: 555-123-4567, Email: johndoe@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-5", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "United Airlines"}}}
+{"query_id": "agisdk-fly-unified-5", "dataset": "agisdk-real", "query": "Find me the cheapest fare for a flight from Orlando to Milwaukee on December 5th, 2024 and book it.\nPassenger: John Doe\nDate of Birth: 01/01/1990\nSex: Male\nSeat Selection: No\nPayment: Credit Card (378342143523967), Exp: 12/30, Security Code: 420 Address: 123 Main St, San Francisco, CA, 94105, USA, Phone: 555-123-4567, Email: johndoe@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-5", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "United Airlines"}}}
 {"query_id": "agisdk-udriver-10", "dataset": "agisdk-real", "query": "Order me a ride for 4pm, I'll be at the de Young muesum headed to the Waterbar, fanciest option possible please.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-10", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-10", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
 {"query_id": "agisdk-udriver-9", "dataset": "agisdk-real", "query": "Book me a ride from the thai restaurant I last took a ride to for later today at 2pm, I'll be at 333 Apartments on Fremont", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-9", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-9", "challenge_type": "retrieval-action", "difficulty": "hard", "similar_to": "Uber"}}}
 {"query_id": "agisdk-topwork-4", "dataset": "agisdk-real", "query": "Create a job post for a UI/UX Designer with expertise in Figma, Sketch, and Adobe Creative Suite, including project details, timeline, and required skills (Wireframing, Prototyping, Responsive Design).", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-4", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
 {"query_id": "agisdk-gocalendar-4", "dataset": "agisdk-real", "query": "Change the \"Team Check-In\" event on July 18, 2024, name to \"Project Kickoff\" and update the location to \"Zoom\"", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-4", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Google Calendar"}}}
 {"query_id": "agisdk-staynb-6", "dataset": "agisdk-real", "query": "Find and book the stay with the best value for money (cheapest stay with the best reviews) for 1 day. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-6", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-6", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Airbnb"}}}
-{"query_id": "agisdk-fly-unified-9", "dataset": "agisdk-real", "query": "Book me a flight from San Francisco to Chicago in Basic Economy on December 18th at 10:00. Ensure no seat selection is made.\nPassenger: David Lee\nDate of Birth: 07/22/1985\nSex: Male\nSeat Selection: No\nPayment: Credit Card (9999 8888 7777), Exp: 03/30, Address: 987 Cedar St, Chicago, IL, 60601, USA, Phone: 555-987-1234, Email: davidlee@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-9", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-9", "challenge_type": "action", "difficulty": "hard", "similar_to": "United Airlines"}}}
-{"query_id": "agisdk-networkin-9", "dataset": "agisdk-real", "query": "Find a professional who attended Stanford and send them a connection request and a message.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-9", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-9", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
 {"query_id": "agisdk-udriver-11", "dataset": "agisdk-real", "query": "I need to go from Pacific Catch on Chestnut back home to 333 Fremont now. If the fancy version is within ten dollars of the regular one, book that.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-udriver.vercel.app", "metadata": {"original_task_id": "udriver-11", "website": "UDriver", "category": "agisdk-real", "additional": {"agisdk_task_id": "udriver-11", "challenge_type": "action", "difficulty": "hard", "similar_to": "Uber"}}}
-{"query_id": "agisdk-fly-unified-4", "dataset": "agisdk-real", "query": "Book me a round-trip flight from Providence (Rhode Island) to Indianapolis, departing on December 5th, 2024 at 08:00 and returning on December 9th at 14:00.\nPassenger: Jane Smith\nDate of Birth: 02/14/1995\nSex: Female\nSeat Selection: Yes (Window seat)\nPayment: Credit Card (378342143523967), Exp: 06/26, security code: 345 Address: 456 Elm St, Miami, FL, 33101, USA, Phone: 555-987-6543, Email: janesmith@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-4", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "United Airlines"}}}
 {"query_id": "agisdk-networkin-5", "dataset": "agisdk-real", "query": "Send a connection request to John Smith.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-5", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-5", "challenge_type": "action", "difficulty": "easy", "similar_to": "LinkedIn"}}}
 {"query_id": "agisdk-zilloft-6", "dataset": "agisdk-real", "query": "Select a property listed in San Francisco as \"Condos\" within a price range under $300,000 and request a tour for tomorrow at 4:00 PM. Use these contact details: Name: Sarah Brown, Email: sarahbrown@example.com, Phone: 555-987-6543.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-zilloft.vercel.app", "metadata": {"original_task_id": "zilloft-6", "website": "Zilloft", "category": "agisdk-real", "additional": {"agisdk_task_id": "zilloft-6", "challenge_type": "action", "difficulty": "medium", "similar_to": "Zillow"}}}
 {"query_id": "agisdk-topwork-2", "dataset": "agisdk-real", "query": "Create a job posting for a Backend Developer specializing in Python, Django, and Flask to develop a high-performance web application. Include project details such as required skills (PostgreSQL, Docker, AWS, CI/CD), estimated project timeline, and budget.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-2", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-2", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
 {"query_id": "agisdk-gocalendar-3", "dataset": "agisdk-real", "query": "Delete the event titled \"Breakfast Meeting with Client\" scheduled for July 19, 2024", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-3", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-3", "challenge_type": "action", "difficulty": "easy", "similar_to": "Google Calendar"}}}
 {"query_id": "agisdk-topwork-3", "dataset": "agisdk-real", "query": "Create a job listing for a Full-Stack Developer with expertise in Java, Spring Boot, and Angular, outlining the project scope, estimated duration, and required skills (MySQL, Docker, Kubernetes, and Jenkins). The ideal candidate should have experience in enterprise-level applications and building scalable microservices. After creating the job post, please describe what you included in the job listing.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-3", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-3", "challenge_type": "retrieval", "difficulty": "medium", "similar_to": "Upwork"}}}
-{"query_id": "agisdk-fly-unified-2", "dataset": "agisdk-real", "query": "Book me a one-way flight from Indiana to New York on December 2nd 2024 at 12:00.\nPassenger: John Doe\nDate of Birth: 01/01/1990\nSex: Male\nSeat Selection: No\nPayment: Credit Card (378342143523967), Exp: 12/25, Security Code: 245, Address: 123 Main St, San Francisco, CA, 94105, USA, Phone: 555-123-4567, Email: johndoe@example.com.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-fly-unified.vercel.app", "metadata": {"original_task_id": "fly-unified-2", "website": "Fly Unified", "category": "agisdk-real", "additional": {"agisdk_task_id": "fly-unified-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "United Airlines"}}}
 {"query_id": "agisdk-dashdish-7", "dataset": "agisdk-real", "query": "Select \"Express Delivery\" for an order from \"DragonEats\" of \"Mushroom Swiss Burger\" and complete the checkout with the pre-loaded Visa card.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-7", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-7", "challenge_type": "action", "difficulty": "hard", "similar_to": "Doordash"}}}
 {"query_id": "agisdk-networkin-3", "dataset": "agisdk-real", "query": "Write a post inviting users to a networking event, including details about the event's purpose, date, and target audience.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-3", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-3", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
 {"query_id": "agisdk-gomail-7", "dataset": "agisdk-real", "query": "Delete the email with the subject \"New Leadership Articles You Can't Miss\" from the Inbox.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-7", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-7", "challenge_type": "retrieval-action", "difficulty": "hard", "similar_to": "Gmail"}}}
@@ -23,16 +19,13 @@
 {"query_id": "agisdk-staynb-2", "dataset": "agisdk-real", "query": "Click on one of the stays displayed on the homepage and book it for a family of 4 (2 adults and 2 children). For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-2", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Airbnb"}}}
 {"query_id": "agisdk-opendining-10", "dataset": "agisdk-real", "query": "Check the menus of all restaurants for vegetarian options and make a reservation at the one with the most vegetarian choices. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-10", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-10", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "OpenTable"}}}
 {"query_id": "agisdk-opendining-4", "dataset": "agisdk-real", "query": "Use the search bar to search for a restaurant on September 2nd at 4:30 PM for 7 people, using \"Japanese\" as the search term, and book the first result. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-4", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-4", "challenge_type": "action", "difficulty": "hard", "similar_to": "OpenTable"}}}
-{"query_id": "agisdk-gomail-8", "dataset": "agisdk-real", "query": "Clear all emails from \"GitHub\" in the inbox to trash.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-8", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-8", "challenge_type": "action", "difficulty": "medium", "similar_to": "Gmail"}}}
 {"query_id": "agisdk-dashdish-4", "dataset": "agisdk-real", "query": "Schedule a delivery order from \"Taco Bell\" adding a \"Classic Cheeseburger\" large size for later and add the note \"Leave at the front door\".", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-4", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Doordash"}}}
 {"query_id": "agisdk-networkin-1", "dataset": "agisdk-real", "query": "Create a new text post for the feed with a professional update about AI trends in 2025, mentioning three key advancements and their impact on the job market.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-1", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-1", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
 {"query_id": "agisdk-dashdish-5", "dataset": "agisdk-real", "query": "Add three \"Loaded Bacon Cheese Fries\" to the shopping cart from \"Man vs. Fries\". Proceed to checkout and select \"Pickup\" as the delivery method.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-5", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Doordash"}}}
 {"query_id": "agisdk-opendining-5", "dataset": "agisdk-real", "query": "Scroll through the homepage carousel until \"Ocean Breeze\" is visible, select the second available time slot, and complete the reservation. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-opendining.vercel.app", "metadata": {"original_task_id": "opendining-5", "website": "OpenDining", "category": "agisdk-real", "additional": {"agisdk_task_id": "opendining-5", "challenge_type": "action", "difficulty": "medium", "similar_to": "OpenTable"}}}
-{"query_id": "agisdk-topwork-1", "dataset": "agisdk-real", "query": "Create a new job post for a Frontend Developer with expertise in React and TypeScript, specifying project details such as estimated duration, required skills, and budget.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-topwork.vercel.app", "metadata": {"original_task_id": "topwork-1", "website": "TopWork", "category": "agisdk-real", "additional": {"agisdk_task_id": "topwork-1", "challenge_type": "action", "difficulty": "medium", "similar_to": "Upwork"}}}
 {"query_id": "agisdk-gocalendar-1", "dataset": "agisdk-real", "query": "Create a new event titled \"Team Meeting\" on July 19, 2024, from 2 PM to 2:30 PM, and include \"Conference Room A\" as the location", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gocalendar.vercel.app", "metadata": {"original_task_id": "gocalendar-1", "website": "GoCalendar", "category": "agisdk-real", "additional": {"agisdk_task_id": "gocalendar-1", "challenge_type": "action", "difficulty": "medium", "similar_to": "Google Calendar"}}}
 {"query_id": "agisdk-gomail-5", "dataset": "agisdk-real", "query": "Schedule an email to jane.doe@example.com with the subject \"Weekly Update\" to be sent next Monday at 9:00 AM.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-5", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-5", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Gmail"}}}
 {"query_id": "agisdk-staynb-4", "dataset": "agisdk-real", "query": "Book a stay for 2 children with 1 adult. For fields you don't know the answer for, just fill them in with anything of your choice.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-4", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-4", "challenge_type": "action", "difficulty": "medium", "similar_to": "Airbnb"}}}
-{"query_id": "agisdk-networkin-6", "dataset": "agisdk-real", "query": "Choose a random person who you haven't connected with, connect with them, and send them a message saying, 'howdy, partner'.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-networkin.vercel.app", "metadata": {"original_task_id": "networkin-6", "website": "Networkin", "category": "agisdk-real", "additional": {"agisdk_task_id": "networkin-6", "challenge_type": "action", "difficulty": "medium", "similar_to": "LinkedIn"}}}
 {"query_id": "agisdk-dashdish-2", "dataset": "agisdk-real", "query": "Add a \"Medium Pepperoni Pizza\" from the restaurant \"Papa Johns Pizza\" to the shopping cart and purchase it.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-dashdish.vercel.app", "metadata": {"original_task_id": "dashdish-2", "website": "DashDish", "category": "agisdk-real", "additional": {"agisdk_task_id": "dashdish-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Doordash"}}}
 {"query_id": "agisdk-staynb-8", "dataset": "agisdk-real", "query": "Scroll through the homepage and book the last stay located in Paris.", "graders": ["agisdk_state_diff"], "start_url": "https://evals-staynb.vercel.app", "metadata": {"original_task_id": "staynb-8", "website": "StayNB", "category": "agisdk-real", "additional": {"agisdk_task_id": "staynb-8", "challenge_type": "retrieval-action", "difficulty": "medium", "similar_to": "Airbnb"}}}
 {"query_id": "agisdk-gomail-2", "dataset": "agisdk-real", "query": "Mark the first email in the Inbox as \"read\".", "graders": ["agisdk_state_diff"], "start_url": "https://evals-gomail.vercel.app", "metadata": {"original_task_id": "gomail-2", "website": "GoMail", "category": "agisdk-real", "additional": {"agisdk_task_id": "gomail-2", "challenge_type": "action", "difficulty": "easy", "similar_to": "Gmail"}}}
--- a/packages/browseros-agent/apps/eval/scripts/agisdk-evaluate.py
+++ b/packages/browseros-agent/apps/eval/scripts/agisdk-evaluate.py
@@ -10,12 +10,39 @@ Input format:

 Output format:
    {"reward": 0.0, "pass": false, "message": "...", "per_criterion": [...]}
+
+Lenient string matching is enabled by default: a failed criterion where
+expected_value is a clean substring of actual_value (both strings) is
+re-marked as a softened pass. This handles AGISDK tasks where the model
+adds harmless decoration to a title or note (e.g. topwork-3, topwork-4).
+Set AGISDK_STRICT_STRINGS=1 to disable and recover the strict score.
 """

 import json
+import os
 import sys


+_STRICT = os.environ.get("AGISDK_STRICT_STRINGS", "").lower() in ("1", "true", "yes")
+
+
+def _soft_string_match(detail: object) -> bool:
+    """Return True iff `detail` is `{actual_value, expected_value}` with both
+    strings and a non-empty `expected_value` that is contained in `actual_value`
+    (case-insensitive). Otherwise False — the criterion stays failed.
+    """
+    if not isinstance(detail, dict):
+        return False
+    actual = detail.get("actual_value")
+    expected = detail.get("expected_value")
+    if not isinstance(actual, str) or not isinstance(expected, str):
+        return False
+    expected_stripped = expected.strip()
+    if not expected_stripped:
+        return False
+    return expected_stripped.lower() in actual.lower()
+
+
 def main():
    data = json.loads(sys.stdin.read())
    task_id = data["task_id"]
@@ -54,17 +81,35 @@ def main():

        reward_val = float(reward_val) if reward_val is not None else 0.0
        results = info.get("results", [])
-        per_criterion = [
-            {"passed": r[0], "detail": str(r[1]) if len(r) > 1 else ""}
-            for r in results
-        ]
+
+        per_criterion = []
+        softened_count = 0
+        for r in results:
+            passed = bool(r[0])
+            detail = r[1] if len(r) > 1 else ""
+            entry: dict = {"passed": passed, "detail": str(detail)}
+            if not _STRICT and not passed and _soft_string_match(detail):
+                entry["passed"] = True
+                entry["softened"] = True
+                softened_count += 1
+            per_criterion.append(entry)
+
+        # Recompute pass/reward after softening: if every criterion now passes,
+        # the task counts as a soft pass.
+        all_pass = all(c["passed"] for c in per_criterion) and bool(per_criterion)
+        if all_pass and reward_val != 1.0:
+            reward_val = 1.0
+
+        out_message = str(message)
+        if softened_count and all_pass:
+            out_message = f"Task passed (with {softened_count} softened string criterion/criteria)."

        print(
            json.dumps(
                {
                    "reward": reward_val,
                    "pass": reward_val == 1.0,
-                    "message": str(message),
+                    "message": out_message,
                    "per_criterion": per_criterion,
                }
            )
--- a/packages/browseros-agent/apps/eval/scripts/build-agisdk-dataset.py
+++ b/packages/browseros-agent/apps/eval/scripts/build-agisdk-dataset.py
@@ -11,12 +11,86 @@ Usage:
 """

 import json
+import re
 import sys
+from datetime import date

 # evals-omnizon.vercel.app was DMCA-takedown'd by Vercel (HTTP 451). Every task
 # on that site fails grading with "Failed to fetch /finish endpoint".
 EXCLUDED_WEBSITES = {"omnizon"}

+# Tasks where either the task itself is invalid (data rot, eval site broken)
+# or the grader penalizes correct work. We do NOT exclude tasks where the
+# agent system genuinely fails (e.g. broken MCP tools) — those are real
+# capability gaps the team needs to see in the score.
+#
+# Each entry below was confirmed via head-to-head deep-dive on the 2026-04-28
+# K2.5 + Opus 4.6 runs; see plans/audits/.
+EXCLUDED_TASKS = {
+    # evals-topwork.vercel.app throws Minified React error #185
+    # ("Maximum update depth exceeded") on every form submit; the page renders
+    # "Application error: a client-side exception has occurred" instead of
+    # saving the job post. Eval site is broken.
+    "topwork-1",
+    # Hardcodes `Exp: 12/25` in both the goal text and a jmespath grader
+    # criterion (`paymentInfo.expDate`). Freshening the goal alone leaves the
+    # grader expecting the original (now-expired) value; freshening both would
+    # require monkey-patching agisdk's TaskConfig at runtime. Unsolvable
+    # without two-sided patching.
+    "fly-unified-2",
+    # Goal says "Dec 18 2024 at 10:00", but the live eval site only has 2025
+    # inventory and no 10:00 slot at all. Both K2.5 and Opus successfully
+    # booked the closest flight; neither could match the grader's expected
+    # timestamp. Data rot.
+    "fly-unified-9",
+    # Eval site stores selected flight times as bare-UTC wall time
+    # (`T08:00:00.000Z`) but the grader expects them shifted by 8h
+    # (`T16:00:00.000Z` = 8 AM PST). Opus 4.6 completed the booking
+    # correctly and was penalized only on the timestamp criteria.
+    # Eval-site TZ-storage bug.
+    "fly-unified-4",
+    # Goal says "Clear all emails from GitHub in the inbox" but the third
+    # grader criterion expects exactly 1 update. Both models correctly
+    # interpreted "all" and were penalized for it. Grader contradicts goal.
+    "gomail-8",
+    # Goal says "Choose a random person you haven't connected with" but the
+    # grader hardcodes `profilesDiff.updated."4".connectionGrade`. Both models
+    # picked someone other than profile id 4 (correctly random) and were
+    # penalized. Grader contradicts goal.
+    "networkin-6",
+    # Eval site's `searchHistoryDiff` doesn't record search queries submitted
+    # via the autocomplete + Enter path. Opus 4.6 completed the entire task
+    # correctly (sent connection request + message to a Stanford alumna) but
+    # the grader's first criterion (search history contains "stanford") was
+    # never triggered server-side. Eval-site bug.
+    "networkin-9",
+}
+
+# Far-future replacement used by `freshen_goal_dates` when a task's hardcoded
+# credit-card expiration is in the past (or expires within the next 6 months).
+_FRESH_EXP = "Exp: 12/30"
+_EXP_PATTERN = re.compile(r"Exp:\s*(\d{2})/(\d{2})\b")
+
+
+def freshen_goal_dates(goal: str) -> str:
+    """Roll any `Exp: MM/YY` date forward when it's within 6 months of today.
+
+    Several AGISDK tasks (e.g., fly-unified-{2,5,12}) hardcode credit-card
+    expirations like `Exp: 12/25`. The eval-site checkout forms reject expired
+    cards; once the wall clock passes the hardcoded date, those tasks become
+    unsolvable. Two-digit years are interpreted as 20YY.
+    """
+    today_yyyymm = date.today().year * 12 + date.today().month
+
+    def replace(match: re.Match[str]) -> str:
+        mo, yr = int(match.group(1)), int(match.group(2))
+        exp_yyyymm = (2000 + yr) * 12 + mo
+        if exp_yyyymm <= today_yyyymm + 6:
+            return _FRESH_EXP
+        return match.group(0)
+
+    return _EXP_PATTERN.sub(replace, goal)
+

 def has_llm_eval(task: dict) -> bool:
    return any(e.get("type") == "llm_boolean" for e in task.get("evals", []))
@@ -36,6 +110,8 @@ def main():
    skipped_infeasible = 0
    skipped_llm = 0
    skipped_excluded = 0
+    skipped_tasks = 0
+    freshened = 0

    for task in all_tasks:
        if not task.get("possible", True):
@@ -46,13 +122,20 @@ def main():
            skipped_llm += 1
            continue

+        task_id = task["id"]
+        if task_id in EXCLUDED_TASKS:
+            skipped_tasks += 1
+            continue
+
        website = task.get("website", {})
        if website.get("id") in EXCLUDED_WEBSITES:
            skipped_excluded += 1
            continue

-        task_id = task["id"]
-        goal = task.get("goal", "")
+        original_goal = task.get("goal", "")
+        goal = freshen_goal_dates(original_goal)
+        if goal != original_goal:
+            freshened += 1
        start_url = website.get("url", "")

        if not start_url or not goal:
@@ -83,7 +166,8 @@ def main():

    print(
        f"Generated {count} tasks (skipped {skipped_infeasible} infeasible, "
-        f"{skipped_llm} llm_boolean, {skipped_excluded} excluded sites)",
+        f"{skipped_llm} llm_boolean, {skipped_excluded} excluded sites, "
+        f"{skipped_tasks} excluded tasks; freshened {freshened} expired card dates)",
        file=sys.stderr,
    )