mirror of
https://github.com/browseros-ai/BrowserOS.git
synced 2026-05-13 15:46:22 +00:00
29056226bb
- Add hover_at, type_at, drag_at coordinate tools to server - Add hoverAt, typeAt, dragAt methods to Browser class - Export server internals (browser, tool-loop, registry) for eval imports - Copy eval app from enterprise repo with agents, graders, runner, dashboard - Nest eval-targets inside apps/eval - Adapt sessionExecutionDir → workingDir for current server API - Add biome ignore for dashboard HTML to prevent lint breaking onclick handlers
157 KiB
157 KiB
| 1 | ID | Starting URL | Category | Difficulty | Task | HITL_Operator_ID | operator_hitl_eval | operator_hitl_reasoning | operator_hitl_link |
|---|---|---|---|---|---|---|---|---|---|
| 2 | 2 | https://www.acehardware.com | READ | easy | Search for "LED light bulbs" on AceHardware.com and provide the titles and sale prices of the first 5 products that appear. Only use http://acehardware.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-1 | Success | Navigates and finds correct items via search | https://www.loom.com/share/85352b1277d84363b16101550ee27766 |
| 3 | 19 | https://www.airbnb.com | DELETE | easy | Log in to your Airbnb account, save a Guest Favorite property to your wishlist, and then go to your wishlist and remove the property you previously added. Only use http://airbnb.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-4 | Success | AI Agent successfully selected guest favorite property listing, saved it to wishlist, and deleted it with confirmation from human. | https://www.loom.com/share/d72f7868dc2e4172bbc020fbfe17cbdc?sid=06ba2a9b-89c1-4b29-982d-afd857147870 |
| 4 | 20 | https://www.airbnb.com | DELETE | hard | Log in to Airbnb, send a query to a host of a New York City property, and then delete your message confirming that the message does not appear on your profile. Only use http://airbnb.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-3 | Failure | AI Agent failed to delete. could not find delete option. Instead chose to archive the whole message, failing task. | https://www.loom.com/share/60832434f452439c8e1d9fc12ab3ea4c |
| 5 | 22 | https://www.airbnb.com | READ | easy | Browse the “Play” section in Paris and extract the titles and brief descriptions of the top five properties. Only use http://airbnb.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-2 | Success | Agent succesfully goes and finds options and clicks in to extract info from each listing | https://www.loom.com/share/93627d5d5cd543edbc98f6372f711dcc |
| 6 | 33 | https://www.alibaba.com | READ | easy | Search for "smartphones" on Alibaba and note how many listings are available. Only use http://alibaba.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-7 | Success | AI Agent successfully locates a number for the amount of listings of smartphones on Alibaba. | https://www.loom.com/share/bb3f8b6626124e35a42b862c35bf2e02 |
| 7 | 35 | https://www.alibaba.com | READ | hard | Search for "RFID tags", filter by a minimum order quantity of less than 1000, and list the first 3 supplier names. Only use http://alibaba.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-5 | Success | AI Agent successfully completed the task. Interesting to note, Agent first searched by category instead of going to search bar, but corrected. Additionally, Agent answer included the same supplier twice, as it was listed with a different product. Answer is technically correct. | https://www.loom.com/share/c64d0336840e41f4bee5f5b690a059e7 |
| 8 | 51 | https://www.allrecipes.com | CREATE | easy | Log in, write a detailed review for the "Lemon Chicken Orzo Soup" recipe including a 5-star rating and your comments on taste and presentation. Only use http://allrecipes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-8 | Failure | Hits "Signal - Not Acceptable" error when navigating into the webpage for the recipe. Tries navigating to incognito and via search but both paths fail | https://www.loom.com/share/e2d32b9e72d34c0788a85fadd95a9718 |
| 9 | 56 | https://www.allrecipes.com | CREATE | easy | Log in, create a new recipe collection titled "Quick Weeknight Dinners" and populate it with 5 chosen recipes. Only use http://allrecipes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-11 | Failure | Agent created a new collection titled correctly. Agent checked back with Favorites Tab to check if it had properly favorited an item before continuing to search for recipes. Agent added 4 recipes, and noticed that there was only four recipes in the collection. Agent misidentified the missing recipe, and attempted to add that recipe again. The agent checked for that specific recipe to be in the collection, yet ignored checking that there were 5 recipes in the collection overall. Thus, it misidentified the number of recipes, and populated a collection with 4 recipes instead of 5. | https://www.loom.com/share/39cbb83e3c90456781acae3726f934f7 |
| 10 | 63 | https://www.allrecipes.com | READ | easy | Search for a seasonal, fall-appropriate recipe and note the recommended serving size mentioned. Only use http://allrecipes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-10 | Success | Agent successfully found a serving size for a recipe designated for the fall time of the year. Interestingly, the Agent identified Halloween as a Fall time of year as opposed to searching in the search bar for "Fall". | https://www.loom.com/share/4cd22d81732c4d69be2ca3f4ffe7c581 |
| 11 | 67 | https://www.allrecipes.com | READ | easy | Locate a "chocolate chip cookie" recipe page and summarize the calorie count from its nutritional information. Only use http://allrecipes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-9 | Success | Agent successfully navigated and found the calorie count. Notably, Agent could have clicked "Jump to nutritional facts" but scrolled instead. | https://www.loom.com/share/b1be71530e1043ce98cf0c5c1b80c7a9 |
| 12 | 73 | https://www.alltrails.com | DELETE | easy | Log in to your account, add the "Canyon Loop" trail to your saved trails, and then delete it—confirm that it no longer appears in your list. Only use http://alltrails.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-12 | Success | Agent saved the trail and successfully deleted it. Agent needed human assistance to log in, as well as navigate to browser settings to allow third-party cookies for log-in portal. Agent also needed human assistance to confirm the save and delete functions. | https://www.loom.com/share/da0184a08aa347938290fc0f21e9b25d |
| 13 | 80 | https://www.alltrails.com | UPDATE | easy | Log in to your account and update your adventure profile by changing your favorite activity from "hiking" to "mountain biking." Only use http://alltrails.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-13 | Success | Agent selected "hiking," as it wasn't previously saved as a favorite. Then it changed its response to "mountain biking," constituting a success. | https://www.loom.com/share/bd57ff75e59f4e20a4045c1b3a4d81f1 |
| 14 | 93 | https://www.apa.org | READ | hard | Use the website’s search function to find articles published in 2022 related to "clinical psychology" and output the titles of the first four matches. Only use http://apa.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-15 | Success | Agent filters for all the necessary criteria, and lists the desired output. Agent first searched for the titles within the year 2022, and then narrowed the search by filtering for articles alone. | https://www.loom.com/share/138e4d0f4d9c4ce4828cbdef5d7ee6d8 |
| 15 | 102 | https://apkpure.com | READ | easy | Search for the app "Spotify" on APKPure and list the latest version number along with its release date as shown on the version history section. Only use http://apkpure.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-18 | Success | Agent was able to find the necessary details for Spotify's version release date and latest version number. Agent needed human assistance to complete CAPTCHA verification. | https://www.loom.com/share/79b47ee0d25846e5b28a6386d73dc134 |
| 16 | 104 | https://apnews.com | READ | easy | Use the website’s search bar to look for articles containing "climate change" and list the titles of the first five articles returned. Only use http://apnews.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-19 | Success | Agent was able to successfully return the top article names for "climate change" search. | https://www.loom.com/share/84350e0bf560465682535f1acf37abbe |
| 17 | 113 | https://archive.org | READ | easy | Use advanced search for ”Space images" on archive.org and output the capture dates and titles of the first 10 images listed. Only use http://archive.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-20 | Success | Agent successfully navigated to the Advanced Search function and input "space" as the description field, and media type as "image". Then it returned the capture date and titles of the first ten images. Agent did require two searches and approval from human to proceed with searches. | https://www.loom.com/share/3948c0065c4f4d9295d73f72d90d1800 |
| 18 | 125 | http://www.barnesandnoble.com | DELETE | easy | Log in, add the physical book "To Kill a Mockingbird" to your wishlist, then remove it and verify its deletion. Only use http://barnesandnoble.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-23 | Success | Agent requested human assistance to proceed with security risk, and also log in. It identified the proper book, asked for confirmation to add it to the wishlist, and checked to make sure it was in the wishlist. Agent struggled to add the book to the wishlist on its first 3 attempts(figuring out which functions add the books to the wishlist), and had to re-search the book a few times. It then properly identified the method to add books to the wishlist and with confirmation, proceeded. Then Agent asked for confirmation to delete it before completing. | https://www.loom.com/share/28d123708a074a6c8d788714df11e925 |
| 19 | 129 | http://www.barnesandnoble.com | READ | easy | Use the store locator by entering ZIP code 90210, then display the address, store hours, and contact details of the nearest location. Only use http://barnesandnoble.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-22 | Success | Agent successfully identified the closest B&N to zip code 90210 when presented with multiple options. | https://www.loom.com/share/a5512e8360ba4b99bdcaf99c86a3b745 |
| 20 | 139 | https://www.bbb.org | READ | easy | Search for any companies on bbb.org and output the BBB rating of the first business listed in the results. Only use http://bbb.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-24 | Success | Agent did successfuly find the first restaurant listed and identified where the rating would be, but this specific restaurant did not have a rating. It asked if it should keep looking, and after browsing more, decided to filter for restaurants with ratings. The drawback of this method/question is that the BBB rating filter only allows you to select one rating at a time, so the Agent was only able to find restaurants with the given rating that it selected in the filter. | https://www.loom.com/share/e73525e3052a440597389d10a88c6d5d |
| 21 | 140 | https://www.bbb.org | READ | hard | Search for any businesses in Los Angeles with a BBB rating of A+ and list the names of the first five businesses displayed. Only use http://bbb.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-25 | Success | Agent was able to successfully identify the first 5 businesses listed with an A+ rating | https://www.loom.com/share/0d63cc5a6aaf4e5c819d8d5e57ff3ebe |
| 22 | 149 | https://www.bestbuy.com | CREATE | easy | Log into your Best Buy account and create a new wishlist titled "Holiday Shopping," then add the "Sony 65-inch 4K Ultra HD TV" to that wishlist. Only use http://bestbuy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-29 | Success | Agent was able to create a wishlist in the account and add the desired TV to the wishlist. Agent needed human assistance to log in several times and would frequently get logged out. Agent also misclicked multiple times before successfully saving the desired TV. | https://www.loom.com/share/88f9901daf4e4431b40ac59835212350 |
| 23 | 151 | https://www.bestbuy.com | CREATE | easy | On the "Sony 75-inch 4K TV" product page, post a new question in the Q&A section asking, "Does this TV support VESA wall mounting?" Only use http://bestbuy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-26 | Failure | Gateway error | https://www.loom.com/share/52c28e65062149fa90e974d254b6dfa1 |
| 24 | 159 | https://www.bestbuy.com | READ | easy | Locate the guide titled "How to Choose the Best 4K TV: The Essential Guide" and summarize the top 3 tips provided. Only use http://bestbuy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-27 | Failure | Agent did not source the guide directly from BestBuy; rather, Agent looked up the name of the guide in a separate tab on Bing. Agent also tried to identify tips by searching for tips as a key word, and then tried to locate a Help or How-To section. Lastly, it summarized 3 succinct tips for selecting a TV. | https://www.loom.com/share/2367be1048104d73a0c10d5cd6de7741 |
| 25 | 162 | https://www.bestbuy.com | UPDATE | hard | Log into your account, add an item to your cart, then switch the delivery option from standard shipping to in-store pickup and confirm that the order details update accordingly. Only use http://bestbuy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-28 | Success | Agent successfully added the item to the cart for delivery and changed it to pickup at the nearest address. | https://www.loom.com/share/a5d861d524c9459fad82b3c562182286 |
| 26 | 172 | https://www.booking.com | READ | hard | Search for 8 review score or above hotels in Manhattan, New York for next weekend with free cancellation, free Wi-Fi, and breakfast included. Compare at least four options and pick the one closest to the Empire State Building. Only use http://booking.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-30 | Success | Agent was able to find two hotels that meet the desired criteria. However, Agent's method was not the most efficient, as it filtered for hotels with a balcony at one point, among other inefficient search criteria. | https://www.loom.com/share/8b64cc7fc3794d7093cb0151cf7a161d |
| 27 | 180 | https://www.buzzfeed.com | CREATE | easy | Using the Community submission portal, draft a BuzzFeed listicle titled “Top 10 Internet Memes of 2023” that includes five list items, each with a short description. Only use http://buzzfeed.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-33 | Success | Agent created and saved the post successfully. Agent asked to include thumbnail, which human told agent to continue without. Agent also needed assistance to log in. | https://www.loom.com/share/3dc0b5de3c6546f399fbf63748e19dcb |
| 28 | 183 | https://www.buzzfeed.com | DELETE | hard | Log in, post a comment on one of the front page news articles on BuzzFeed, then locate this previously posted comment and delete the comment, ensuring it is removed from the article page. Only use http://buzzfeed.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-34 | Success | Agent successfully commented and deleted its comment, and requested approval from human before commenting and deleting. | https://www.loom.com/share/1e5957bb1324419e9ca9d85b61902ea5 |
| 29 | 198 | https://www.canada.ca | READ | easy | Navigate to the COVID-19 information section on Canada.ca and note three public health guidelines currently recommended. Only use http://canada.ca to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-36 | Failure | Agent noted two guidelines rather than outlining three distinct ones. One could argue that the agent noted more than three, but in either case, agent failed to outline three distinct guidelines. | https://www.loom.com/share/a86c6c1e8f184fabb0e33bda6cc392f7 |
| 30 | 206 | https://www.caranddriver.com | DELETE | hard | Log in to your Car and Drive account, save and article to your bookmarks, then delete the previously bookmarked article from your "My Bookmarks" list, and confirm that the article is no longer stored. Only use http://caranddriver.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-37 | Success | Agent was able to identify an article, bookmark it, and then unbookmark it and confirm. However, it needed human assistance to verify all of these actions. | https://www.loom.com/share/1262049f0dc8478f95a3c78934473e7e |
| 31 | 209 | https://www.caranddriver.com | READ | easy | Navigate to Car and Driver’s homepage and list the titles of the top three featured car reviews. Only use http://caranddriver.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-39 | Success | Agent was able to find 3 featured reviews on the front page. | https://www.loom.com/share/4f73dbc16010432390ac460edd4386c1 |
| 32 | 213 | https://www.cars.com | READ | easy | In the "New Cars" section, check the pricing for 2024 BMW 3 Series 330e and list at least two pricing components (e.g., MSRP, incentives) that are displayed. Only use http://cars.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-40 | Success | Agent was able to find multiple pricing/incentive components for the BMW series 3 330e. | https://www.loom.com/share/49099a8e9d944d52ade7f37246b3af13 |
| 33 | 220 | https://www.cbr.com | READ | easy | Navigate to the CBR homepage and list the titles of the top five most recent articles in the news section. Only use http://cbr.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-42 | Success | Agent was able to locate and report 5 top articles from the news section. | https://www.loom.com/share/38c7e87875204c9ba2e37a897d813fbf |
| 34 | 221 | https://www.cbr.com | READ | easy | Navigate to an article featuring an embedded movie trailer and extract its title, description, and the link to the full trailer (if available). Only use http://cbr.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-41 | Failure | Agent consistently tried to play video when it should have collected the link to the videos instead. | https://www.loom.com/share/67f242fa08154ddf85941e5f1dc3f582 |
| 35 | 222 | https://www.cbr.com | READ | easy | Use the site’s search function to look up "Star Wars" and list the first five article titles with a brief note on each article’s focus. Only use http://cbr.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-43 | Success | Agent successfully found Star Wars-related articles and summarized them efficiently. | https://www.loom.com/share/0da9b1fea9854df9a52b7480e53f1deb |
| 36 | 228 | https://www.cbssports.com | READ | easy | Check the live stream schedule and list the next two sports events along with their start times and the channels on which they will be broadcast. Only use http://cbssports.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-44 | Success | Agent successfully navigated through CBS Sports and found the next two events streaming. | https://www.loom.com/share/81b96de835d246d6bf13ad15083c416c |
| 37 | 231 | https://www.cdc.gov | READ | easy | Find the latest measles advisory issued by the CDC, then note the publication date along with a brief summary of the advisory. Only use http://cdc.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-45 | Success | Agent successfully identified the most recent advisory and summarized it succinctly. | https://www.loom.com/share/91dc8d0139ee482086b1519694a3598d |
| 38 | 236 | https://my.clevelandclinic.org | READ | easy | Navigate to the Health Education section and list the top 5 articles on heart disease available on the Cleveland Clinic website. Only use http://clevelandclinic.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-46 | Success | Agent successfully identified the top 5 most relevant articles on heart disease as prompted. | https://www.loom.com/share/c509ec8303b6495680a001a68298f94b |
| 39 | 247 | https://www.cnet.com | READ | easy | Search for a CNET column on artificial intelligence and list any recommended products that the article highlights. Only use http://cnet.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-50 | Failure | Agent failed to access the site, and human could access on same browser. | https://www.loom.com/share/a9ecb32027af465bb06317193e2ff5bd |
| 40 | 249 | https://www.cnet.com | READ | easy | Browse any Deals or Offers section available and list the details of at least one affiliate deal mentioned. Only use http://cnet.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-47 | Failure | Agent failed to access the site, and human could access on same browser. | https://www.loom.com/share/46abbfeba2da45e885a1809d1366f8a4 |
| 41 | 256 | https://collider.com | DELETE | hard | Log into your Collider newsletter subscription settings, ensure that communications have been subscribed to, and unsubscribe from all communications, then verify that a confirmation message is displayed. Only use http://collider.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-52 | Success | Agent managed to unsubscribe to all notifications successfully. | https://www.loom.com/share/54916f41fbf5427ebdcdccf57ff132bb |
| 42 | 261 | https://collider.com | READ | easy | Search for Collider’s coverage on a war film and provide a summary of the review highlights. Only use http://collider.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-51 | Failure | Agent used the browser search function rather than the site search function, and also left the site to search bing before returning to the website. | https://www.loom.com/share/2ae30e7b3e914f2a9cf875446f02c1d0 |
| 43 | 264 | https://www.collinsdictionary.com | READ | easy | Look up the word "onomatopoeia" and extract the main definition and one usage example from the Collins Corpus. Only use http://collinsdictionary.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-53 | Success | Agent managed to find a definition and usage of the word onomatopoeia. | https://www.loom.com/share/6e506344b31a49d591eda6fe9267500b |
| 44 | 266 | https://www.columbia.edu | READ | hard | Use the search functionality to locate pages detailing tuition and fees, then extract the published tuition fee information for undergraduate programs. Only use http://columbia.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-54 | Success | Agent managed to find a multitude of information on Columbia's tuition pricing, and provided a link as well. | https://www.loom.com/share/fb921d5142914eeb8734f426dbfde0b1 |
| 45 | 267 | https://www.columbia.edu | READ | easy | Identify and list the dates for the upcoming Columbia University campus events from the events calendar. Only use http://columbia.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-55 | Success | Agent identified a plethora of upcoming events listed on the Columbia website. | https://www.loom.com/share/8196ada8a6fb474480e74aa90f8e6e67 |
| 46 | 269 | https://www.commonsensemedia.org | READ | easy | Use the search feature to find the expert review of the animated film “Frozen” and extract its recommended age range and one key critique. Only use http://commonsensemedia.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-56 | Success | Agent identified a review include age range for the movie Frozen as well as a key critique. | https://www.loom.com/share/9168436354d74afebaa4eb104eb20cfe |
| 47 | 292 | https://newyork.craigslist.org | CREATE | easy | Log in to Craigslist, search for a baby stroller in your area in the "for sale" section, and save it to your favorites. Only use http://craigslist.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-57 | Success | Agent was able to identify a stroller in human's area and save it to favorites and confirm. | https://www.loom.com/share/d0f26c072dd7457c9318af3bd7013aa9 |
| 48 | 299 | https://www.crunchbase.com | CREATE | hard | Log in to Crunchbase and create a new list titled "Tech Unicorns," then add the companies Uber, Airbnb, and Stripe to the list. Only use http://crunchbase.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-65 | Failure | Agent got blocked from accessing crunchbase. | https://www.loom.com/share/02ecc12a4a604f75999a8f2a503ba2fc |
| 49 | 306 | https://www.crunchbase.com | CREATE | easy | Add "Tesla Inc." to your followed companies list so you receive future updates on funding and corporate changes. Only use http://crunchbase.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-63 | Failure | Agent got blocked from accessing crunchbase. | https://www.loom.com/share/e09d1209d3014b00a3bba87b5942efd1 |
| 50 | 308 | https://www.crunchbase.com | CREATE | easy | Create a search for companies in Los Angeles and filter for companies in the E-commerce sector and save the updated query. Only use http://crunchbase.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-62 | Success | Agent was able to identify Los Angeles companies in the e-commerce sector. Agent notably passed CAPTCHA verification without human assistance. | https://www.loom.com/share/92bc9f263e624e3baf9d0baa9ff530e2 |
| 51 | 323 | https://www.crunchyroll.com | CREATE | hard | Access your account and set up a watchlist called "Classic Anime" by adding at least four classic anime series from the catalog. Only use http://crunchyroll.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-71 | Success | Agent successfully created list with 4 Anime shows that it located. | https://www.loom.com/share/500982124b3648569659f01dba50fa33 |
| 52 | 328 | https://www.crunchyroll.com | DELETE | easy | Sign in to your profile, create a new watchlist titled "Action-packed Series," then remove the watchlist titled "Action-packed Series," confirming its deletion from your account. Only use http://crunchyroll.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-67 | Success | Agent was able to create the list and delete it, both requiring human approval to do so. | https://www.loom.com/share/b02c3a4e547e4d15afe52c31f4a3d54b |
| 53 | 331 | https://www.crunchyroll.com | READ | easy | Visit the news or blog section and summarize the details of an upcoming Crunchyroll event. Only use http://crunchyroll.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-68 | Success | Agent successfully found and reported an upcoming Crunchyroll event including dates, | https://www.loom.com/share/00b0b0259f204bf38d6de46be226428a |
| 54 | 332 | https://www.crunchyroll.com | READ | easy | Navigate to the details page of the anime "One Piece" and extract the available language and subtitle options. Only use http://crunchyroll.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-66 | Success | Agent was able to successfully locate and report the possible language options. | https://www.loom.com/share/449c22aa6baa472cbd696e42a163e550 |
| 55 | 334 | https://www.crunchyroll.com | UPDATE | easy | Log in to your account, create a "Weekend Binge" watchlist and add "Attack on Titan." Then open the "Weekend Binge" watchlist, and update it by adding the anime "Hunter x Hunter" to the list. Only use http://crunchyroll.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-69 | Success | Agent successfully created the playlist, and then added both tv shows to update it. | https://www.loom.com/share/4e1ca68f414b4c81bccc694349c67996 |
| 56 | 364 | https://www.dickssportinggoods.com | CREATE | easy | Register a new user account on Dick’s Sporting Goods by filling out the required personal information and confirm the registration via the on-screen confirmation message. Only use http://dickssportinggoods.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-74 | Failure | Agent was not able to proceed with mock information within a reasonable time - user could have completed the task much quicker with minimal wait. | https://www.loom.com/share/1db5725203ee4b3cbe476f78abba1179 |
| 57 | 368 | https://www.dickssportinggoods.com | CREATE | easy | Log in, write a detailed product review for a Wilson baseball glove including a star rating, a title, and your experience with the product, then submit the review. Only use http://dickssportinggoods.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-77 | Success | Agent was able to add a review and submit with all qualifications outlined in the prompt. | https://www.loom.com/share/6f5fd9e56446493b8261871c22475fa8 |
| 58 | 375 | https://www.dickssportinggoods.com | UPDATE | easy | Add a pair of Nike running shoes to your cart, then go to your cart and modify the quantity of the Nike running shoes from 1 to 2. Only use http://dickssportinggoods.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-76 | Success | Agent was able to add a random shoe, and then up the quantity to 2 in the cart. | https://www.loom.com/share/311482f6e0fd4d8cb60fa14942d836a7 |
| 59 | 376 | https://www.dickssportinggoods.com | UPDATE | easy | Log in to your profile and update your address by adding your new apartment details. Only use http://dickssportinggoods.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-75 | Success | Agent was able to update users address, with much help from human input | https://www.loom.com/share/e2b9b25e9c744b0889bcb842c75ad6cc |
| 60 | 377 | https://www.dictionary.com | READ | easy | Look up the word "pronunciation," play its audio sample, and note the phonetic transcription displayed. Only use http://dictionary.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-79 | Success | Agent was able to play the audio source and display the phonetic transcription. | https://www.loom.com/share/bb555d03e16b488786aae1166305f5a2 |
| 61 | 378 | https://www.dictionary.com | READ | easy | Look up the word "quintessential" and record both its pronunciation guide and the available audio sample. Only use http://dictionary.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-78 | Failure | Agent was unable to record or download the audio file for the pronounciation of the word. | https://www.loom.com/share/beb87dbf0a3f4d808d132cb23ce9a87f |
| 62 | 391 | https://www.dw.com | READ | easy | Visit the DW homepage and list the headline and publication time of the top news article featured in the main section. Only use http://dw.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-80 | Success | Agent was able to navigate to the lead article and return the title. | https://www.loom.com/share/2964071f21e0484baca64e7b16e870c7 |
| 63 | 397 | https://www.eater.com | READ | easy | Locate a feature article on international dining scenes and list two featured cities mentioned in the piece. Only use http://eater.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-81 | Success | Agent was able to find an article that mentioned multiple eating destinations. | https://www.loom.com/share/1f941216787e40ee812c2534f8bd9583 |
| 64 | 403 | https://www.ebay.com | READ | hard | Find three different sellers offering the same "Blue Tooth Speaker" item. Compare the total cost (including shipping and taxes), seller ratings, return policies, and estimated delivery times. Create a table to compare the various speakers and recommend the best purchase option if I'm purchasing from New York City, NY. Only use http://ebay.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-82 | Success | Agent was able to successfully find and compare the same speaker sold by three different vendors with all qualities required. | https://www.loom.com/share/757118633b2d490e94c3d01363499b1c |
| 65 | 412 | https://economictimes.indiatimes.com | READ | easy | Navigate to the Markets section and record the current BSE and NSE index values as displayed on the dashboard. Only use http://economictimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-83 | Success | Agent was able to successfully navigate and find the correct index values as tasked. | https://www.loom.com/share/5245862c0a2f41d7ba984ba73f68f9f4 |
| 66 | 417 | https://www.elle.com | READ | easy | Use the site’s search function to find articles about “Sustainable Fashion” and record the publication dates of the top 3 results. Only use http://elle.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-85 | Success | Agent was able to find three articles that were relevant and pertinent when searching for sustainable fashion. | https://www.loom.com/share/bae3d8c7c9c6430499d438e28d75678f |
| 67 | 418 | https://www.elle.com | READ | easy | Identify the digital magazine section and record the publication date of the most recent issue available. Only use http://elle.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-84 | Failure | Agent timed out - 10 minutes to find a publication was more than enough time to navigate the website, and users would likely search themselves instead of tasking agent. | https://www.loom.com/share/65565384a21a4e4786284617cdf2256f |
| 68 | 420 | https://en.wikipedia.org/wiki/Main_Page | READ | easy | What does the 'Climate Change' article say is driving the current rise in global temperatures? Only use http://en.wikipedia.org/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-86 | Success | Agent was able to navigate Wikipedia and find a proper explanation for the question posed. | https://www.loom.com/share/b33585301d194478b855cf802bf8662b |
| 69 | 421 | https://en.wikipedia.org/wiki/Main_Page | READ | easy | Select the 'Page information' option under the 'Tools' dropdown to see how many page views the 'COVID-19' article has in the past 30 days. Only use http://en.wikipedia.org/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-87 | Success | Agent was able to find the correct number of page views as instructed. | https://www.loom.com/share/49d16910832a45968204d07aa43f68f4 |
| 70 | 424 | https://www.encyclopedia.com | READ | easy | Use the search bar to find and list the title along with the publisher of the encyclopedia entry on "Climate Change." Only use http://encyclopedia.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-88 | Success | Agent was able to find and identify the publisher of the page on climate change. | https://www.loom.com/share/cb915cc84f2247e4910d34b66f7b1b7a |
| 71 | 425 | https://www.encyclopedia.com | READ | easy | Search for "Artificial Intelligence" and list any related news or magazine articles or media referenced on the entry. Only use http://encyclopedia.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-89 | Success | Agent was able to identify the resources section, although it took the agent several tries to identify where on the page it was. It was also able to distinguish between news articles and books. | https://www.loom.com/share/a0912cb6c0b746158fb8ad6011f78f04 |
| 72 | 430 | https://www.eonline.com | READ | easy | Search for articles mentioning "Beyoncé" and provide the headlines and publication dates for the first three results. Only use http://eonline.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-90 | Success | Agent was able to find and list 3 articles related to Beyonce. | https://www.loom.com/share/4727f42511514081a3f8553a4b57c304 |
| 73 | 432 | https://www.epa.gov | READ | easy | Find the latest update on air quality data for Los Angeles on the EPA’s Air Quality System page and record the reported index value. Only use http://epa.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-92 | Failure | Agent struggled to locate Los Angeles, and scrolled too far to the point where "select a city" was off the screen, and it started to try and select a county. This was inconvenient for the user, so designated this task as a "fail." | https://www.loom.com/share/4ee8fe23b9e04c379aec8472e2c5e4b0 |
| 74 | 433 | https://www.epa.gov | READ | easy | Locate the SmartWay program page and list two benefits that it offers for businesses. Only use http://epa.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-91 | Success | Agent was able to scan the page and list two relevant benefits for businesses. | https://www.loom.com/share/097e21b69661497ca51b11b0f12c34d0 |
| 75 | 445 | https://www.esquire.com | DELETE | hard | Create a comment on a recent article as described in a previous task, then delete that same comment and verify that it is no longer visible. Only use http://esquire.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-98 | Failure | Agent got logged out twice, and struggled to post the comment multiple times. Agent notably input a username longer than 15 letters, which was the maximum character limit, and struggled to reduce the count to only 15 characters. | https://www.loom.com/share/e824bbbe7ae04027a8fb45e016d47c56 |
| 76 | 446 | https://www.esquire.com | READ | hard | Navigate through the archive by decade and note the publication year of one highlighted article from each decade (e.g., 1980s, 1990s, 2000s, etc.). Only use http://esquire.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-95 | Success | Agent found an article from each of the decades required. | https://www.loom.com/share/cf4d5c1cacf64dc2a86c9142ad7403d4 |
| 77 | 449 | https://www.esquire.com | READ | easy | Navigate to the Esquire homepage and list the headlines of the top 3 featured articles. Only use http://esquire.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-93 | Success | Agent managed to find the top 3 features articles on the website. | https://www.loom.com/share/25a2efd933534b9c82b7ec7498954bf5 |
| 78 | 453 | https://www.etsy.com | READ | easy | Sign in to your Etsy account using your email and password. Only use http://etsy.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-99 | Success | Agent was able to log in, with human guidance on which account to use. | https://www.loom.com/share/26f183a04728493795e08fd0e21e727d |
| 79 | 460 | https://www.eventbrite.com | READ | easy | Access the Eventbrite Help Center to locate guidelines on organizing virtual events; then list the key steps and recommendations provided. Only use http://eventbrite.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-100 | Success | Agent was able to access the help center and find the necessary guidelines as tasked. | https://www.loom.com/share/ba891de7228f41d6bc4b47a9a2cc4899 |
| 80 | 481 | https://www.expedia.com/ | READ | easy | Search for flights from New York’s JFK to Los Angeles’ LAX on June 15th with a return on June 20th, and list the top three cheapest fares. Only use http://expedia.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-103 | Success | Agent was able to find and list three flights as desired. | https://www.loom.com/share/29d80c80a38a4018ada37ec7896651fb |
| 81 | 488 | https://www.facebook.com | CREATE | hard | Log in and set up a new Facebook Page for a fictitious business called “Eco-Groceries” complete with a short business description. Only use http://facebook.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-108 | Failure | Agent was unable to create the page after an error of multiple failed attempts to create a page. | https://www.loom.com/share/399e3a4c5bad44469cf74499a5656fb3 |
| 82 | 495 | https://www.facebook.com | CREATE | easy | Log in, go to the Videos section to see Reels, and save the first reel that will appear on the page. Only use http://facebook.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-111 | Success | Agent was able to navigate to reels and save the first reel it encountered. | https://www.loom.com/share/a595266e0e9b4b6982e24534afcc0206 |
| 83 | 505 | https://www.fandom.com | READ | hard | Find an image posted to the Stranger Things Wiki forum and identify the dimensions of the largest downloadable option Only use http://fandom.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-112 | Success | Agent was able to locate an image and provide a link to its dimensions. Agent needed human confirmation to move to the next image after the first image did not provide its dimensions. | https://www.loom.com/share/747a723f645f4190ba5140e84872b072 |
| 84 | 508 | https://www.fda.gov | READ | easy | Locate the section dedicated to FDA-regulated foods and extract the risk factors listed in the Food Safety sub‐section. Only use http://fda.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-113 | Failure | Human timed the agent out due to inconvenience of time taken by agent to figure out task. | https://www.loom.com/share/0b2a0d95155c4fabbb7273ba20d945aa |
| 85 | 517 | https://www.firstcry.com | READ | easy | Search for baby strollers on FirstCry and list the prices, key features, and customer ratings of the top 3 models displayed. Only use http://firstcry.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-115 | Success | Agent stalled for a bit but then was quickly able to identify three strollers as tasked. | https://www.loom.com/share/c615374eded1429493b67b77fbb662d9 |
| 86 | 519 | https://www.firstcry.com | READ | easy | Browse the maternity care products section and extract details (price, customer rating, and availability) for the "Maternity Support Belt." Only use http://firstcry.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-117 | Success | Agent was able to find a "maternity support belt" and extract pricing and additional details. | https://www.loom.com/share/9e70f36e9647421aad3501fc7764fd4e |
| 87 | 521 | https://www.firstcry.com | READ | easy | Check the "FirstCry Prime" membership page and note down the benefits offered, including free shipping and early sale access details. Only use http://firstcry.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-114 | Success | Agent was able to identify that Firstcry Prime had changed to Firstcry Plus, and provided benefits on it. | https://www.loom.com/share/fac08f80b2664778b77715b54bf00cca |
| 88 | 537 | https://flickr.com | READ | easy | Search Flickr for photos tagged "sunset" and list the titles and usernames of the first 5 results. Only use http://flickr.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-120 | Success | Agent was able to return the names of the first few titles labelled "Sunset." | https://www.loom.com/share/edf3b136bfe148dca626537dd56e78d6 |
| 89 | 538 | https://flickr.com | READ | easy | Open the details of a selected photo and extract its metadata details (title, description, and tags) as displayed on its page. Only use http://flickr.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-121 | Success | Agent was able to extract metadata details of the photo it selected. | https://www.loom.com/share/18dd9b1455e3439795564e4e2609f51d |
| 90 | 543 | https://www.flipkart.com | READ | hard | In the "Laptops" section, apply the filter for "Dell" and extract the average discount percentage on the first 3 Dell laptops displayed. Only use http://flipkart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-124 | Success | Agent was able to aggregate the total of the three discount it saw, and averaged it out. | https://www.loom.com/share/0e9958e2a67a44dd96a951f42b11c7f7 |
| 91 | 545 | https://www.flipkart.com/ | READ | hard | Navigate to the "Mobiles" category, search for "iPhone", and list the product names, prices, and discount details of the top 5 search results. Only use http://flipkart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-123 | Success | Agent was able to locate and report top 5 results including price and other details as asked. | https://www.loom.com/share/ce2bf908e90c4c939adfac9137198b09 |
| 92 | 552 | https://food52.com | READ | hard | Log in, use the community forum search on Food52 to find posts tagged with “grilling” and list the titles of the first three posts along with the respective usernames of the posters. Only use http://food52.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-127 | Success | Agent found the first three post titles and usernames related to "grilling." | https://www.loom.com/share/9c629e4960074aa2a2a26ed050947e86 |
| 93 | 553 | https://food52.com | READ | easy | Browse the “A Few of Our Faves” section and list three recipes featured this week, including their titles. Only use http://food52.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-125 | Failure | Agent was not able to find the "A few of our faves" section. Human timed it out, because agent was headed down an incorrect path that was unlikely to succeed. | https://www.loom.com/share/953b2059719c459880db4f0080aa1d11 |
| 94 | 554 | https://food52.com | READ | easy | Use the Food52 search function to find a recipe that includes both lemon and thyme in the ingredients, then list the step-by-step cooking instructions of the first matching result. Only use http://food52.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-126 | Success | Agent found a recipe with lemon and thyme ingredients and listed out the steps to make it. | https://www.loom.com/share/8e5452cae0da45fcb9c17dc0352b4fc3 |
| 95 | 564 | https://www.ford.com | CREATE | hard | Establish a new saved configuration for a Ford Mustang Mach-E with added options such as an extended battery and panoramic sunroof, then note the generated build reference number. Only use http://ford.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-130 | Success | Agent was able to create the custom build and save it to the collection, and return the build reference as a url. | https://www.loom.com/share/9f42fa69206c4df49514eb2e08dbfcf4 |
| 96 | 565 | https://www.ford.com | CREATE | hard | Utilize the financing calculator tool to simulate a Ford Credit financing option for a Ford Fusion, and output the estimated monthly payment along with the term details. Only use http://ford.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-129 | Success | Agent was able to report out a credit financing option for a Ford Fusion including monthly payment and term details. | https://www.loom.com/share/83139f880a1641e4a8197440377c29c3 |
| 97 | 567 | https://www.ford.com | DELETE | hard | Login, create a new saved vehicle configuration, then access your saved vehicle configurations and delete it, and confirm that it is no longer present in your saved builds list. Only use http://ford.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-128 | Success | Agent successfully designed and saved a build, and proceeded to delete it afterwards. Agent asked for human to submit zip code and log in. | https://www.loom.com/share/0645b4da79f64544bf4ba0bff6aa9e09 |
| 98 | 574 | https://fortune.com | READ | easy | Navigate to the Fortune homepage and list the titles of the top 5 featured business articles. Only use http://fortune.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-132 | Success | Agent was able to find and return the titles of the top 5 features business articles. | https://www.loom.com/share/819fe591f0a8460199c7e89b9ed01ccc |
| 99 | 577 | https://www.foxnews.com | READ | hard | Use the website’s search function to find an article related to the "US Economy" and extract the main points from the introduction. Only use http://foxnews.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-134 | Success | Agent was able to find an article related to US economy and summarize the main points from the introduction. | https://www.loom.com/share/486e862425cc49e4ad39f823560dd2f9 |
| 100 | 578 | https://www.foxnews.com | READ | easy | Retrieve an archived article on "immigration policy" from the site’s archive; summarize its publication date and primary focus. Only use http://foxnews.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-133 | Success | Agent was able to find an archived article about "immigration policy." | https://www.loom.com/share/2f259f6f60f446ee920aa0da0171f4a2 |
| 101 | 582 | https://www.foxsports.com | READ | easy | Browse the video highlights section and list the titles of the five most recent NBA highlight videos. Only use http://foxsports.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-135 | Success | Agent was able to find the nba highlights, but had to have human instruct it to navigate to the NBA channel before it successfully identified the highlights, as it was stuck beforehand. | https://www.loom.com/share/efeacefe21494adc9e4888955fd1da45 |
| 102 | 591 | https://www.freepik.com | READ | easy | Navigate to the Reimagine tool page and note the supported input file formats along with any usage requirements mentioned. Only use http://freepik.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-136 | Failure | Agent failed to locate any details on file format availability, even after human prompting in the right direction. | https://www.loom.com/share/37cd1870ec1641a3a8cd85b90ccd2b6d |
| 103 | 604 | https://gamerant.com | READ | easy | Use the search function to find articles about "Cyberpunk 2077" and provide the headlines of the first five results. Only use http://gamerant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-137 | Success | Agent was able to find and return 5 articles related to "Cyberpunk 2077." | https://www.loom.com/share/57a04d8a25764f0ea9979bf60d4dfcf9 |
| 104 | 608 | https://gamerant.com | READ | easy | Search for latest PS5 review on Game Rant and extract the publication date along with a brief summary of the main criticisms. Only use http://gamerant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-139 | Failure | Agent got stuck and then navigated to bing to search, which is out of bounds for the task. | https://www.loom.com/share/a5f12c5975384a8196fd2729ae0fbd91 |
| 105 | 609 | https://gamerant.com | UPDATE | hard | Log in to your account, post a comment on "Monster Hunter Wilds Review." Then navigate to your comment and update it with additional details. Only use http://gamerant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-138 | Failure | Agent was able to create and post a comment on the desired article, but GameRant has to approve the comment before it can be updated, which takes ~24 hours. Due to the fact that the comment could not be updated, task is failed. | https://www.loom.com/share/e87f32e12dbd44f68ff2754626d46e79 |
| 106 | 612 | https://gamerant.com | UPDATE | easy | Change your display name in your account settings to include a gaming tag "rockstar36458" and confirm that the update is applied. Only use http://gamerant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-140 | Success | Agent was able to change the display name to the desired name as tasked. | https://www.loom.com/share/d9cb3628ad184695871b8d053468ecc0 |
| 107 | 615 | https://www.gamespot.com | READ | easy | Select the review article for "The Legend of Zelda: Tears of the Kingdom" and extract the review score along with three key highlights mentioned in the review. Only use http://gamespot.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-141 | Success | Agent was able to locate the score of the review and list three key elements of the review as well. | https://www.loom.com/share/0beadeff71c44d84839bfddf7d6dd011 |
| 108 | 622 | https://www.gamesradar.com | READ | easy | Use the website’s search function to look for articles on "comics" and list the titles of the first three results. Only use http://gamesradar.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-142 | Success | Agent returned the top three relevant articles on "comics" as tasked, | https://www.loom.com/share/168a6d1d9e53440f9be47031b35b2e0f |
| 109 | 652 | https://github.com | CREATE | easy | Fork the repository "microsoft/vscode" to your GitHub account and verify that the fork appears in your profile repository list. Only use http://github.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-144 | Success | Agent was able to fork the repository to the account as tasked. | https://www.loom.com/share/a6478d018af24dd68aaf215bef6e6d56 |
| 110 | 660 | https://github.com | DELETE | easy | Log in, create a repository called "OldProject" on your Github account then delete the repository named "OldProject" from your GitHub account and verify its no longer there. Only use http://github.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-146 | Success | Agent created the repository and then deleted it. Notably, agent asked for confirmation 3/4 times, each time prompted by the website to do so,. | https://www.loom.com/share/40dbbae0e02c42c2baccd5637e9ecd92 |
| 111 | 661 | https://github.com | DELETE | hard | Log in, create a repository called "sample-repo" with a new branch "old-feature" then delete the "old-feature" branch using GitHub’s branch management options. Only use http://github.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-145 | Failure | Agent was not able to locate branch options within a reasonable timeframe, so user timed out the agent. | https://www.loom.com/share/0cb1f053ca034e9cab31aff0d783b22b |
| 112 | 672 | https://github.com | UPDATE | hard | Create a repository named "sample-repo-2" then change the branch protection rules for the "main" branch in your repository to require at least one approving review before merging any changes. Only use http://github.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-147 | Success | Agent created the repository and required one review before merging changes. | https://www.loom.com/share/67ba45175b1c4e15b00bdddabb47fef2 |
| 113 | 680 | https://www.glassdoor.com/index.htm | CREATE | easy | Post a new discussion in a Glassdoor community Bowl about work-life balance at tech startups, providing your insights and suggestions. Only use http://glassdoor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-150 | Success | Agent was successfully able to create a post about work-life balance on glassdoor | https://www.loom.com/share/61816c93c75a40e5a8adba39fdf642fe |
| 114 | 687 | https://www.glassdoor.com/index.htm | DELETE | hard | Log in to your Glassdoor account, post a new comment in a community discussion about job search tips, then delete your comment and verify that it is no longer displayed. Only use http://glassdoor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-151 | Success | Agent was successfully able to post a comment and then delete it later as well, surprising thing is the delete operation was so fast that i was not able to notice it. | https://www.loom.com/share/335ba4a0fa954959a95b684f362f3d67 |
| 115 | 688 | https://www.glassdoor.com/index.htm | READ | easy | Navigate to the New York City Bowl and display the number of members as well as the bio of the bowl. Only use http://glassdoor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-149 | Failure | Agent used the Bing search tab, rather than attempting to navigate the website. | https://www.loom.com/share/96ec82fe51374df38abc166dc1ac5ab0 |
| 116 | 689 | https://www.glassdoor.com/index.htm | READ | hard | Navigate to the "Interviews" section, search for interview experiences at "Tesla", and report the most frequently mentioned interview question. Only use http://glassdoor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-148 | Success | Agent looked at multiple interview accounts to aggregate interview question data, and returned the most common question asked by these accounts. | https://www.loom.com/share/b5bf8a992a1841429d4ff4ee64149b97 |
| 117 | 695 | https://www.goal.com/en-us | READ | easy | Find the match analysis of the last Champions League game and write down the main tactical observation discussed in the article. Only use http://goal.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-152 | Failure | Agent failed to look for summary or insights or key movements of the match, it only focused on find the key words which weren't directly available | https://www.loom.com/share/6794d3e02fb04525ba483cd019a8f6d2 |
| 118 | 703 | https://www.goodhousekeeping.com | DELETE | hard | Log in to your GoodHousekeeping.com account, add one recipe to "Weekend Cooking" list, and then remove one recipe from your "Weekend Cooking" list, and verify that it no longer appears in your saved items. Only use http://goodhousekeeping.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-153 | Success | Agent successfully added a receipe and then deleted it as well, however there was no option available to create a personalized list | https://www.loom.com/share/7c43978aa58f4f1290ed7fcabd1edf8e |
| 119 | 710 | https://www.goodhousekeeping.com | READ | easy | In the Beauty category, filter for articles discussing skincare routines and list the publication dates of the first three posts. Only use http://goodhousekeeping.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-156 | Success | Agent was successfully able to extract required information | https://www.loom.com/share/a187daae33434e05bc1d2df78c553270 |
| 120 | 715 | https://www.goodreads.com | READ | easy | Search for the book "To Kill a Mockingbird" on Goodreads, view its ratings, and list the top 3 user reviews. Only use http://goodreads.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-157 | Success | Agent was successfully able to extract top three user reviews from the given book | https://www.loom.com/share/55b2719fa11c460fbd89e98d991e1e1f |
| 121 | 746 | https://www.grubhub.com | READ | easy | Navigate to Catering, and summarize 5 of the FAQs, as well as the associated answers. Only use http://grubhub.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-159 | Success | Agent was successfully able to find and summarize 5 FAQ's from the catering section | https://www.loom.com/share/2e378793790c4e78aa5c68eb5917958e |
| 122 | 751 | https://www.harpersbazaar.com | READ | easy | Search for the article on "Spring Fashion Trends" and extract the names of the five luxury brands mentioned in it. Only use http://harpersbazaar.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-161 | Success | Agent successfully found the article and it did extract the required information | https://www.loom.com/share/90298b7270d946b380a489e52bcf82e7 |
| 123 | 756 | https://www.healthline.com | READ | easy | Use the search tool to look for “diabetes management” and output the titles of the first five articles displayed. Only use http://healthline.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-163 | Success | Agent successfully extracted the information requested to it | https://www.loom.com/share/46c6b7729b1847b89cff7890971ed76f |
| 124 | 758 | https://www.healthline.com | READ | easy | Navigate to Healthline’s homepage and list the titles of the top five featured health articles. Only use http://healthline.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-164 | Success | Agent did pull the 5 articles from featured section not directly from the homepage but it is still somewhat correct | https://www.loom.com/share/245febbd34d94cf9afabcf4ac05ca51e |
| 125 | 765 | https://www2.hm.com/en_us/index.html | READ | easy | View the "Sale" section and record the discount percentage displayed on the first product. Only use http://hm.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-165 | Success | Agent was successfully able to find the sale section and pull the discount percentage on the first product | https://www.loom.com/share/1c8bc36e7d5a46c9bedfadf2b58d4b9b |
| 126 | 771 | https://www.homedepot.com | CREATE | easy | Log in to your Home Depot account and create a new project list titled “Spring Garden,” then add the product “Husky Outdoor Planter Box” to the list. Only use http://homedepot.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-166 | Failure | Agent failed to find the exact product mentioned in the task | https://www.loom.com/share/55ffa3c0146842a394abb279a153ecc2 |
| 127 | 775 | https://www.homedepot.com | CREATE | easy | Submit a “Contact Us” inquiry regarding delivery options for a “Samsung Family Hub Refrigerator,” including your contact details and a brief question. Only use http://homedepot.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-167 | Success | Agent surprisingly had a live chat with an live agent on the other side by first providing the contact information and then asking the question with a brief description. | https://www.loom.com/share/a40b11fbe8554963914064c4bf09710c |
| 128 | 782 | https://www.homedepot.com | READ | easy | Search for “porch lumber” on Home Depot and extract the pricing details for the 2x4 pressure-treated lumber available for delivery. Only use http://homedepot.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-168 | Success | Agent did filter the luber according to the requirements and at the end gave an convincing answer | https://www.loom.com/share/5481befa7b4947ba907891426fc46a41 |
| 129 | 793 | https://www.hotels.com | READ | hard | Filter search results for properties in Paris available next month that offer spa amenities and bars, and list the amenities of the first three hotels. Only use http://hotels.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-169 | Success | Agent successfully filtered hotels with required amenities and extracted top 3 hotels info | https://www.loom.com/share/c3729852497f47e2b529b03d5a94a417 |
| 130 | 799 | https://www.housebeautiful.com | CREATE | easy | Sign up for notifications on new articles by entering your email and opting in for alerts on interior design trends. Only use http://housebeautiful.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-170 | Success | Agent successfully found the signup section for receiving new articles and signed up for it | https://www.loom.com/share/849ba26d01154bc5ac3d7c31df1fcab3 |
| 131 | 804 | https://www.housebeautiful.com | READ | easy | Navigate to the homepage and list the titles of the top 3 featured articles on modern interior design. Only use http://housebeautiful.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-172 | Failure | Agent failed to open the webpage | https://www.loom.com/share/d6b85a1b2a4b4a9ab72cb7d4547e4694 |
| 132 | 819 | https://www.ign.com | READ | easy | Navigate to the Guides section, Locate IGN’s GTA V guide, read the cheat codes section, and provide a summary of at least three cheat codes mentioned. Only use http://ign.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-176 | Failure | Agent failed to access the website directly due to an error "Site Unavailable" | https://www.loom.com/share/294a578b2cf946d9b85887eddda150a4 |
| 133 | 827 | https://www.ikea.com | CREATE | easy | Log in and post a review for the "BILLY bookcase" emphasizing its ease of assembly, then share your review on the product page. Only use http://ikea.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-178 | Success | Agent successfully found the product page and submitted a positive review as mentioned | https://www.loom.com/share/0bc81cc284914ee3b3074e3a1606815d |
| 134 | 839 | https://www.ikea.com | UPDATE | easy | Access your profile, navigate to your saved addresses, and update your default delivery address to include specific instructions for building access. Only use http://ikea.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-177 | Success | Agent succcessfully navigated to the account settings and updated the instructions in saved address | https://www.loom.com/share/35d20a7ec5794273b0544dc5122cb4ce |
| 135 | 850 | http://indeed.com | READ | easy | Check if a "Delivery Driver" job in Orlando provides 401k and paid time off. If it does, add it to my save list. Only use http://indeed.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-180 | Success | Agent successfully found the job as mentioned and added to saved list | https://www.loom.com/share/984904cca7f84df9bc088cf5ad36a8b3 |
| 136 | 854 | https://www.independent.co.uk/us | READ | easy | Navigate to the "World" news section on Independent.co.uk and list the headlines of the top 3 articles. Only use http://independent.co.uk to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-181 | Success | Agent successfully finds the title of the articles | https://www.loom.com/share/f3cbc4362f954493b55d605e27d44f4f |
| 137 | 856 | https://www.india.com | READ | easy | Locate the article covering the latest economic policy announcement in the Business section and note its publication date and headline. Only use http://india.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-182 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/71f0e766d2ed49ccbc74d94784284774 |
| 138 | 860 | https://www.indiamart.com | READ | easy | Look up "LED lighting systems" and note any indicators of supplier verification and lead response times that appear on the product pages. Only use http://indiamart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-183 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/ba1f8594e18540bc8c3c0608430235da |
| 139 | 861 | https://www.indiamart.com | READ | easy | Use the search function to find "organic tea leaves" and list the top 5 supplier names along with their location details. Only use http://indiamart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-184 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/0f04c3cd677740f088ed0feb10bf6d3f |
| 140 | 873 | https://www.instacart.com | CREATE | easy | Search for "Peanut Butter and Jelly", add it to your cart, and then go to your cart to check the estimated delivery time for the order. Only use http://instacart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-186 | Success | Agent successfully found a item which did contain peanut butter and jelly and added it to the cart to display the estimated delivery time | https://www.loom.com/share/ef977079aac6451c8da3598b5f1d1ac9 |
| 141 | 875 | https://www.instacart.com | CREATE | easy | Log in to Instacart and add “Organic Carrots” to your shopping cart for immediate checkout. Only use http://instacart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-185 | Success | Agent successfully found the item requested and added it to the shopping cart | https://www.loom.com/share/e2109f74db3644998bbf71c6412fbe0b |
| 142 | 883 | https://www.instacart.com | READ | hard | Search for organic bananas on Instacart and list the top 3 prices along with their retailer names. Only use http://instacart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-187 | Success | Agent successfully found the organic bananas from three different stores and retured there prices respectively | https://www.loom.com/share/07ae754082344723a4f90db27488013d |
| 143 | 885 | https://www.instacart.com | UPDATE | easy | Log in to your Instacart account, add 6 bunches of organic bananas to your cart, and then access your cart and update the quantity of your organic bananas from 6 to 10 bunches. Only use http://instacart.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-188 | Success | Agent successfully updated the organic bananas bunches from 6 to 10 | https://www.loom.com/share/b57cc255ea1149bf82799e03edef7f62 |
| 144 | 890 | https://www.instructables.com | READ | easy | Filter projects tagged with "woodworking" and list the titles and authors of the first 5 Instructables. Only use http://instructables.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-189 | Success | Agent successfully found the 5 projects tagged with woodworking | https://www.loom.com/share/dcdb3ec864ed4f1ba70f723901cc59f2 |
| 145 | 895 | https://www.investopedia.com | READ | easy | Go to the market analysis section, locate the latest update on US economic indicators, and extract the names of at least three key indicators mentioned. Only use http://investopedia.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-190 | Success | Agent successfully found the article realted to economic damage and summed it up in 3 key indicators | https://www.loom.com/share/d825089ef52d45719057803977e9a1da |
| 146 | 897 | https://www.irs.gov | READ | easy | Go to the Help page on IRS.gov that explains Direct Pay options for individuals and list the available payment methods mentioned. Only use http://irs.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-191 | Success | Agent successfully found the payment options for Direct Pay | https://www.loom.com/share/5e8a28ddfa734b39a60d3ccf29da7384 |
| 147 | 905 | https://www.istockphoto.com | READ | easy | Filter search results for "business meeting" by horizontal orientation, then list the first 5 image descriptions or titles displayed. Only use http://istockphoto.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-192 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/0f1f50c3b2844eba879d47e824b57ea4 |
| 148 | 913 | https://www.jagranjosh.com | READ | easy | Search for the "Previous Year Question Papers" section and report the number of downloadable papers available for the UPSC exam. Only use http://jagranjosh.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-194 | Failure | Agent failed to return the number of papers it found as it was trying to search for more papers and it kept going even after I asked it to stop | https://www.loom.com/share/1b61261b0127485ab6ed17075e156b30 |
| 149 | 916 | https://www.jagranjosh.com | READ | easy | Check the academic calendar section for upcoming exam dates and list the next three major exams along with their schedules. Only use http://jagranjosh.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-193 | Success | Agent successfully found the upcoming exams from three different sections | https://www.loom.com/share/487078d29f4e45ffb969cc786ded1b3f |
| 150 | 919 | https://www.jstor.org | READ | easy | Search for an article on "Innovation in Urban Design" and list the keywords associated with it from the metadata. Only use http://jstor.org/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-195 | Failure | Agent failed to find the article and so it was not able to find the keywords as well | https://www.loom.com/share/7345b55a14ce440a95169ef78b0c4949 |
| 151 | 927 | https://www.justdial.com | READ | easy | Locate the “24x7 Emergency Medical Clinic” in Bangalore and display its operating hours along with a brief description of services. Only use http://justdial.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-196 | Failure | Agent failed to access the information of operating hours of the medical clinic | https://www.loom.com/share/8b463ace45b0459fa85e2503b70f7f87 |
| 152 | 943 | https://www.khanacademy.org | READ | easy | Search for "Algebra" courses on Khan Academy and list the titles and short descriptions of the first five results. Only use http://khanacademy.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-199 | Failure | Agent failed to find the first five courses related to algebra as it only returned the top 4 | https://www.loom.com/share/363fc6ee19dd4777b443dd6b681b4fe6 |
| 153 | 945 | https://www.khanacademy.org | READ | easy | On the Khan Academy homepage, list all available interface languages and note the total number of languages offered. Only use http://khanacademy.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-197 | Success | Agent successfully found the number of languages offered | https://www.loom.com/share/78714670fd4842a1907cbb69ebc3eb02 |
| 154 | 948 | https://www.khanacademy.org | UPDATE | easy | Log in to your student account and update your monthly learning goal on the progress dashboard to reflect a new target for completed exercises. Only use http://khanacademy.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-198 | Failure | Agent failed find an option to update the monthly learning goal in the available sections. The account seems to have no progress or goals set yet. | https://www.loom.com/share/197c4e4f2f694983adf7c61031880e2e |
| 155 | 953 | https://kidshealth.org | READ | easy | Navigate to the "Parents" section and locate an article about managing screen time for children; then, provide a brief summary of the article title and its main tips. Only use http://kidshealth.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-200 | Failure | Agent stopped abruptly with no reason mentioned and even after retrying it was not able to contiue working on the task | https://www.loom.com/share/481db56ebd554f77b578ec10ccaeec57 |
| 156 | 958 | https://www.kmart.com.au | CREATE | hard | Log in to your Kmart account, create a new wishlist titled "Holiday Gifts," and add 3 items representing gifts for him, for her, and for kids. Only use http://kmart.com.au/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-201 | Success | Agent successfully added the items to a shopping wishlist | https://www.loom.com/share/6766580910f648259831b5654e147bfc |
| 157 | 960 | https://www.kmart.com.au | DELETE | easy | Log in to your Kmart account, add a "Camera" to your shopping cart, then remove the "Camera" item from your shopping cart, and confirm its deletion from the cart. Only use http://kmart.com.au/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-202 | Success | Agent successfully added an item then removed it as suggested | https://www.loom.com/share/51653ffc4d7d4c05ad87ad3bc35cb3ed |
| 158 | 977 | https://www.latimes.com | READ | easy | Locate a feature article with a photo gallery and list the captions of the photos presented within the gallery. Only use http://latimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-204 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/80cae66f97e1433e8e020c9fbb9eaf0d |
| 159 | 980 | https://www.latimes.com | READ | easy | Use the search function to locate articles on “Los Angeles” and “immigration”, then provide the titles of the first three results. Only use http://latimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-205 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/f3cdc6edfd114710acb99454d23cfc28 |
| 160 | 981 | https://www.latimes.com | READ | easy | Browse the entertainment section and list the top three celebrity news headlines along with their publication dates. Only use http://latimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-206 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/bdd2650c636b47e8af9d0539034be088 |
| 161 | 992 | https://letterboxd.com | CREATE | hard | Compose a detailed film review diary entry for "The Grand Budapest Hotel," focusing on its cinematography, and rate it 5 stars. Only use http://letterboxd.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-207 | Success | Agent successfully found the film and posted a review as suggested | https://www.loom.com/share/673c45ce024443ac95dcfedf73cf1b35 |
| 162 | 995 | https://letterboxd.com | CREATE | easy | Update your profile with a public update listing your top 3 favorite films of all time. Only use http://letterboxd.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-208 | Success | Agent successfully added 3 films to the profile in Favorite section | https://www.loom.com/share/0d9baa35e54d463b86f1dff9c0af03d2 |
| 163 | 1015 | https://www.linkedin.com | CREATE | easy | Add a job history entry to your profile titled Analyst at LabRite, from June 2021 - September 2023. Only use http://linkedin.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-213 | Success | Agent successfully added the experience as suggested | https://www.loom.com/share/333a5756433d45339ad0009a6285ac91 |
| 164 | 1017 | https://www.linkedin.com | CREATE | easy | Comment on a recent article posted by Google, ensuring that your comment has a positive and professional connotation. Only use http://linkedin.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-210 | Success | Agent was successfully able to post a comment on a google's recent post | https://www.loom.com/share/54cb8853a8ab4ebabe518b30458261b2 |
| 165 | 1021 | https://www.linkedin.com | CREATE | easy | Compose a text-based post asking your network for advice on upskilling in data analytics and mention two trending skills you wish to learn. Only use http://linkedin.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-209 | Success | Agent successfully composed a post on linkedIn | https://www.loom.com/share/9d13551e90e446e19796ea70d628dc53 |
| 166 | 1033 | https://www.linkedin.com | UPDATE | easy | Make a post thanking several colleagues. Then edit the post to add an additional relevant hashtag before resubmitting it. Only use http://linkedin.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-211 | Success | Agent successfully added a new post and later edited it to update the hashtags | https://www.loom.com/share/3936ddd915434ccaad7076c4b701fc80 |
| 167 | 1041 | https://www.livejournal.com | DELETE | easy | Log in, create a temporary journal entry titled "Temporary Post," then delete this entry and confirm that it no longer appears on your journal. Only use http://livejournal.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-215 | Success | Agent successfully added a temporary journal and then deleted it | https://www.loom.com/share/bb03606157d4498599208795ab21c735 |
| 168 | 1043 | https://www.livejournal.com | DELETE | hard | Login, create a friends list and add 3 individuals, then from your friends list, remove a specific friend by selecting their profile and confirming their deletion from your contact list. Only use http://livejournal.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-214 | Success | Agent successfully added three friends and removed one at the end | https://www.loom.com/share/a877a6d12fac42f79f650e904f893d64 |
| 169 | 1050 | https://www.livejournal.com | UPDATE | hard | Log in and create a journal entry and schedule send it for 12 hours later. Then modify a scheduled journal entry by changing its posting time to an earlier hour on the same day as originally planned. Only use http://livejournal.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-216 | Failure | Agent failed to update the timings of the post it already posted some time ago | https://www.loom.com/share/6a0dd71a149348c8832890b4232e13c8 |
| 170 | 1052 | https://www.livescience.com | READ | easy | Use the LiveScience search bar to look up "volcano research" and list the titles of the relevant articles that appear. Only use http://livescience.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-217 | Success | Agent successfully found the information as requested | https://www.loom.com/share/26e467a2863b4abb8525e59240a6777d |
| 171 | 1053 | https://www.livescience.com | READ | easy | Navigate to the "Space" category and extract the title, author (if available), and publication date of the first article. Only use http://livescience.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-218 | Success | Agent successfully found the information as requested | https://www.loom.com/share/1ccc5308df4849249c6c76c3b77e6922 |
| 172 | 1057 | https://www.loc.gov | FILE_MANIPULATION | easy | Download a digitized manuscript related to early American literature from the catalog and verify that the preview includes the word "manuscript." Only use http://loc.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-219 | Success | Agent successfully found a manuscript and was able to download it as well | https://www.loom.com/share/fa15a26c1aea4e3b9a47bcceef48098d |
| 173 | 1063 | https://www.lonelyplanet.com | READ | easy | Use the search function to find articles on “Nepal trekking” and extract the titles of at least three related articles. Only use http://lonelyplanet.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-220 | Success | Agent successfully found the title of top three articles as requested | https://www.loom.com/share/1fcb4e5d27c64eb5963ac45decf4099a |
| 174 | 1084 | https://www.macys.com | READ | easy | Find the product description and reviews for a specific item. Only use http://macys.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-225 | Success | Agent successfully found the information about a dress with its price and reviews | https://www.loom.com/share/4a906832ef2f4e09b4f046f3998d2e45 |
| 175 | 1087 | https://www.made-in-china.com | READ | hard | Navigate to the suppliers profiles section, select a verified supplier offering "electronic components," and extract the certification details provided on their profile. Only use http://made-in-china.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-226 | Failure | Agent failed to access the website due to an error "No Internet" | https://www.loom.com/share/1ff0ecb867a849cd953149c08f34b25b |
| 176 | 1094 | https://www.marksandspencer.com | READ | easy | Navigate to the "Food" section and list the names and prices of the first five ready-meal products. Only use http://marksandspencer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-227 | Failure | Agent failed to access the website due to an error "No Internet" | https://www.loom.com/share/3400ff8530194203ab619ee969eea485 |
| 177 | 1096 | https://www.marksandspencer.com | READ | easy | Go to the "Help" or "Customer Service" section and list the top five frequently asked questions (FAQs) displayed. Only use http://marksandspencer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-228 | Failure | Agent failed to access the website due to an error "No Internet" | https://www.loom.com/share/ea3fb3918a36463b851c5cf6148e7415 |
| 178 | 1106 | https://www.maryland.gov/Pages/default.aspx | READ | easy | Use the instant search feature to look up "public records retention policy" and provide the URL of the first matching result. Only use http://maryland.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-230 | Success | Agent successfully found the info as requested | https://www.loom.com/share/397201cdce494fa98c258f5e15c387a0 |
| 179 | 1107 | https://www.maryland.gov/Pages/default.aspx | READ | easy | Explore the "Feedback" section and summarize the process given for submitting citizen input. Only use http://maryland.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-229 | Failure | Agent failed to access the website due to an error "No Internet" | https://www.loom.com/share/8d039cf44a2a4c6d975b4d44794b1373 |
| 180 | 1109 | https://mashable.com | READ | easy | Find an article that includes a section on upcoming events or festivals, and list the event names mentioned within that section. Only use http://mashable.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-231 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/38f68cd13611440db5c8e9b282d96216 |
| 181 | 1110 | https://mashable.com | READ | easy | Browse the Mashable homepage and list the top three trending headlines along with their publication dates. Only use http://mashable.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-232 | Failure | Agent failed to access the website due to an error "Site Unavailable" | https://www.loom.com/share/502ec5f7ab164a5682f94570bc4e4b76 |
| 182 | 1126 | https://www.medscape.com | READ | easy | Use the search feature to find articles on "COVID-19 long-haulers" and list the first five article titles that appear. Only use http://medscape.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-233 | Success | Agent successfully found the top 5 articles as requested | https://www.loom.com/share/9e4bb4d0353f4ce0956dbd9dcdec3e1d |
| 183 | 1134 | https://www.mercari.com | READ | hard | Browse the electronics category and calculate the average price of used smartphones shown in the listings. Only use http://mercari.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-234 | Success | Agent successfully found the average pricing of the used smartphones as requested | https://www.loom.com/share/eaa30d4d25cd43b78c8dccf1bf5d3c41 |
| 184 | 1144 | https://www.metacritic.com | READ | easy | Use the site search to find reviews for “The Shawshank Redemption” and extract both the critic consensus and the overall user score. Only use http://metacritic.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-236 | Success | Agent successfully found the information from an article as requested | https://www.loom.com/share/a8707a343aec4115b21f2c186991e234 |
| 185 | 1147 | https://www.metacritic.com | READ | easy | Use the search bar to locate reviews for the film “Titanic” and find the publication date of one of the critic reviews. Only use http://metacritic.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-235 | Success | Agent successfully found the article | https://www.loom.com/share/0eff7fb79e12408ca68077595a3b6a83 |
| 186 | 1154 | https://www.michigan.gov/som | READ | easy | Locate the most recent press release on state economic development on Michigan.gov and extract the release date along with the main headline. Only use http://michigan.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-237 | Success | Agent successfully found the ecomonic department and from that found the latest article as requested | https://www.loom.com/share/d22558965f504499abe058d817cabb31 |
| 187 | 1166 | https://www.mountsinai.org | READ | easy | Search for information on virtual urgent care service and outline the steps a patient should follow to access a virtual consultation. Only use http://mountsinai.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-238 | Success | Agent successfully found the information in virtual urgent care steps | https://www.loom.com/share/05b48a9158b7482994f02beb68013156 |
| 188 | 1197 | https://www.ndtv.com | READ | easy | Locate the regional news segment for Rajasthan and list the top three headlines along with their brief introductions. Only use http://ndtv.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-240 | Failure | Agent failed to find news from Rajasthan directly from the website so it tried to search it on another domain, which is not allowed | https://www.loom.com/share/d079531976a440538f97f2b549212a15 |
| 189 | 1198 | https://www.ndtv.com | READ | easy | Navigate to the Business section in the footer and list the titles of the five most recent articles along with their publication dates. Only use http://ndtv.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-239 | Success | Agent successfully found the top-5 articles from the business section | https://www.loom.com/share/9ff022ceaa954e5d93e273aa8ec480d5 |
| 190 | 1210 | https://www.newegg.com | READ | easy | Search for "external SSD" in the storage category, apply a capacity filter for 1TB, and list the top three product models with their prices. Only use http://newegg.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-242 | Success | Agent successfully found the top 3 ssd as requested | https://www.loom.com/share/6ae46e333a1643629f0e395ffba48f80 |
| 191 | 1212 | https://www.newegg.com | READ | easy | Visit the Newegg Premier membership page, read through the listed benefits, and summarize at least five key membership perks. Only use http://newegg.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-243 | Success | Agent successfully found the information on premium membership | https://www.loom.com/share/7edb7cff50b143f4b2b62d75bf8ca029 |
| 192 | 1224 | https://www.nj.com | READ | easy | Locate the "Food & Dining" guide and extract the names of three recommended local restaurants highlighted in the guide. Only use http://nj.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-244 | Success | Agent successfully found the recommendation on 3 restaurant from an article in Food & Dining section | https://www.loom.com/share/bd51787434e645919c64a87e7b0ff293 |
| 193 | 1230 | https://www.nordstrom.com | DELETE | easy | Log in, add the "Ray-Ban Aviator Sunglasses" to your wish list, then remove that item and confirm its deletion from your account. Only use http://nordstrom.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-245 | Failure | Agent failed to login even after multiple help from my side | https://www.loom.com/share/04d3dfff320640639a3f67370e1dcd06 |
| 194 | 1231 | https://www.nordstrom.com | DELETE | easy | Log in, create a wish list titled "Past Purchases", and then permanently delete that wish list. Only use http://nordstrom.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-246 | Failure | Agent failed to login even after multiple help from my side | https://www.loom.com/share/9516c4e3ed17496f9d7bfe093a57aa7a |
| 195 | 1233 | https://www.nordstrom.com | READ | easy | Use the search bar to locate BOSS Menswear Suit and record the product name, price, and available sizes displayed. Only use http://nordstrom.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-247 | Success | Agent successfully found the suit with its price and available sizes | https://www.loom.com/share/ae1fa80395ce4e5e873bd986f2f81d75 |
| 196 | 1236 | https://www.nordstrom.com | READ | easy | Navigate to the "Under $100" section in the Women's Handbags category and record the names, prices, and availability of the first three items. Only use http://nordstrom.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-248 | Success | Agent successfully found the information on handbags from the mentioned section | https://www.loom.com/share/f931815495a14faeb5dc6ae247b43551 |
| 197 | 1240 | https://www.nps.gov/index.htm | READ | easy | Navigate to the homepage and search find a park by state in Wyoming on the first page Only use http://nps.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-249 | Success | Agent successfully found all the parks mentioned in the given location | https://www.loom.com/share/eec83e432f2a46d1aec4627e9abae55b |
| 198 | 1241 | https://www.ny.gov | READ | easy | Search for official guidelines regarding the renewal of a New York State driver’s license, and note the contact information provided. Only use http://ny.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-250 | Success | Agent successfully found the contact information for renewal of a license | https://www.loom.com/share/8638e6a0e10449f4a04394c30c02bf85 |
| 199 | 1245 | https://www.nyc.gov | READ | easy | Search for the FAQ page on recycling and waste management services and extract two common questions along with their answers. Only use http://nyc.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-252 | Success | Agent successfully found two questions on waste management as requested | https://www.loom.com/share/c06b7df265ba404a9b12039c6ad268c4 |
| 200 | 1246 | https://www.nyc.gov | READ | easy | Navigate to the "Benefits & Support" section and list the names of three programs available to NYC residents. Only use http://nyc.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-251 | Success | Agent successfully found the information on programs in NYC as requested | https://www.loom.com/share/11e5ea1be38d48ae925eb31334cc8e0e |
| 201 | 1254 | https://nypost.com | READ | easy | Browse the News section and summarize the leading article by noting its headline, a brief excerpt, and the publication time. Only use http://nypost.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-255 | Success | Agent successfully found the information on a recent article | https://www.loom.com/share/f5fa0ae2e8f249648084269594efc66a |
| 202 | 1257 | https://nypost.com | READ | easy | Go to the Business section and extract the headline of the article discussing the latest market trends. Only use http://nypost.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-254 | Success | Agent successfully found the article from buisness section | https://www.loom.com/share/5435771d5b224c90ae240a622bcd8847 |
| 203 | 1259 | https://nypost.com | UPDATE | hard | Log in to the NY Post comment section, leave a comment on an article from the front page, then locate this comment, and edit it to add more information. Only use http://nypost.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-253 | Success | Agent successfully wrote a comment on a recent article and edited it later | https://www.loom.com/share/6f00c56e88c64ad292f1dbe394cb8ff1 |
| 204 | 1270 | https://www.oed.com/?tl=true | READ | hard | Filter search results to show only entries for words first used from 1500 to 1600 and list the headwords of the first 10 results. Only use http://oed.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-256 | Success | Agent successfully filtered words and gave the top 10 as requested | https://www.loom.com/share/c8f96832553f4a2db45e3e8ef34d5396 |
| 205 | 1307 | https://www.oregon.gov/Pages/index.aspx | READ | easy | Identify the page for the Oregon agency responsible for environmental protection and list the contact details provided on that page. Only use http://oregon.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-262 | Success | Agent successfully found the environmental quality page and contact details mentioned in there | https://www.loom.com/share/485c6f09a91a41ba97be3f5c604c6b49 |
| 206 | 1321 | https://www.pcgamer.com | DELETE | hard | Log in to your pcgamer.com account, post a comment on one of the front page articles, then locate your recently posted comment and delete it. Only use http://pcgamer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-266 | Success | Agent successfully wrote a comment on a recent article and deleted it later | https://www.loom.com/share/a96709ac8a8d46dfa2303c374550afbd |
| 207 | 1324 | https://www.pcgamer.com | READ | easy | Navigate to the "Reviews" section and list the titles of the three most recent PC hardware review articles. Only use http://pcgamer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-267 | Success | Agent successfully found the top-3 articles on hardware reviews | https://www.loom.com/share/14cc95ec8a104b8c9555a3d97821a795 |
| 208 | 1356 | https://pixabay.com | READ | hard | Go to the "Videos" section and output the file sizes from the download section and resolutions for the top 3 trending videos. Only use http://pixabay.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-269 | Success | Agent successfully found the top-3 trending videos along with their download sizes | https://www.loom.com/share/a6484b32718840698ee376f4bfb5e533 |
| 209 | 1359 | https://pixabay.com | READ | easy | Search for "vintage" imagery and list the publication dates (if provided) and contributor ids (if provided) for the top 5 results. Only use http://pixabay.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-270 | Success | Agent successfully found the information on top-5 results as requested | https://www.loom.com/share/0f66394869894683ac3d621d3984a523 |
| 210 | 1360 | https://pixabay.com | READ | easy | Search for "abstract background" in the vectors/illustrations category and output the contributor usernames and at least three tags from each of the first 5 results. Only use http://pixabay.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-268 | Success | Agent successfully found the top-5 results and information related to it | https://www.loom.com/share/db0b0eba3cac4798a0c22d4908fa1bff |
| 211 | 1364 | https://www.playstation.com/en-us | CREATE | easy | Subscribe to the newsletters by entering your email to receive monthly updates on PlayStation news, deals, and events. Only use http://playstation.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-271 | Failure | Agent couldn't find the newsletter subscription page, so it tried to perform a Microsoft Bing search to retrieve it | https://www.loom.com/share/f8b00cbdbd3c4580a29f41396a052ea9 |
| 212 | 1365 | https://www.playstation.com/en-us | CREATE | easy | In the support section, submit a formal inquiry about issues with remote downloads, specifying your console model and error details. Only use http://playstation.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-272 | Success | Agent initiated a conversation with a helper bot, thus submitting a formal inquiry. | https://www.loom.com/share/830fb1cc90c540498792a41261950f2b |
| 213 | 1369 | https://www.playstation.com/en-us | READ | easy | Browse the homepage and identify the main featured release; provide its headline and a brief summary. Only use http://playstation.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-273 | Success | Agent was able to browse the homepage and locate the main article release, returning its headline and summary. | https://www.loom.com/share/b647d674a63a4b9691b7dafb47be3997 |
| 214 | 1377 | https://www.polygon.com | READ | easy | Find any top 19 list of recommended PC games and pull out the first three game titles from that list. Only use http://polygon.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-274 | Success | Agent found three game titles from a recommended games list. | https://www.loom.com/share/7ce851a7c0df4dc6bfa0dac0dc116ea6 |
| 215 | 1392 | https://www.purdue.edu | READ | easy | Locate the section on study abroad programs and list two destination countries where Purdue students can study. Only use http://purdue.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-275 | Success | Agent was able to find and return two locations for Purdue students to study abroad. | https://www.loom.com/share/6be02622f0414fa6a7570531c78e2159 |
| 216 | 1395 | https://quizlet.com | READ | easy | Log in, visit the Quizlet Help Center, navigate to the FAQ section on creating your questions, and summarize the first three steps listed. Only use http://quizlet.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-276 | Success | Agent was able to navigate to the FAQ section on creating your questions, and summarize the first three steps listed. | https://www.loom.com/share/3c5f6915ba0d4746bda46ddade3b72c5 |
| 217 | 1400 | https://www.radiotimes.com | CREATE | easy | Use the social media sharing buttons on a Radiotimes article to generate a shareable link and create a custom message endorsing the article. Only use http://radiotimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-278 | Success | Agent was able to create a shareable link and embed it within its generated message. | https://www.loom.com/share/264fd2c005bd407db10fe4f428c592b0 |
| 218 | 1407 | https://www.radiotimes.com | READ | hard | Locate tonight's featured TV schedule on Radiotimes, and list the titles of shows airing on both BBC and ITV. Only use http://radiotimes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-279 | Failure | Agent timed out - human determined that 10 minutes was inefficient for the task. | https://www.loom.com/share/d33ff5659aae427a803e2bb5a248f7e4 |
| 219 | 1413 | https://www.realsimple.com | READ | easy | Visit the Travel & Lifestyle section under Shopping and record the author’s name and publication date for the leading article. Only use http://realsimple.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-280 | Success | Agent was able to visit the Shopping section and record the author’s name and publication date for the leading article. | https://www.loom.com/share/d745ea240f334e7c959f33d576395048 |
| 220 | 1416 | https://www.realtor.com | READ | easy | Filter apartments for rent in New York City with a monthly rent under $2,500, and list the first three listings’ addresses, rental prices, and highlighted amenities. Only use http://realtor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-281 | Failure | Agent timed out - human determined that 10 minutes was inefficient for the task. | https://www.loom.com/share/d7350e00b4cc49b49fc29b885a8e9464 |
| 221 | 1417 | https://www.realtor.com | READ | easy | Go to the “Find a Realtor” section, enter criteria for a buyer in Phoenix, AZ, and list the names and contact details of the first five matching real estate professionals. Only use http://realtor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-282 | Success | Agent was able to input criteria and find 5 agents as well as their contact information as tasked. | https://www.loom.com/share/1bd7b6ee39484efe808b22b3ccc0f4b0 |
| 222 | 1428 | https://www.redbubble.com | READ | easy | Look up “Floral Patterns” on Redbubble and list the product titles that appear in the search results. Only use http://redbubble.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-283 | Success | Agent was able to find 20 product titles for a search of floral pattern. | https://www.loom.com/share/51324eb381cf40c99b0ab32f104cad5f |
| 223 | 1438 | https://www.rei.com | READ | easy | Search for "hiking boots" on REI.com and list the names, prices, and ratings of the top three results. Only use http://rei.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-284 | Success | Agent was able to find the three highest rated hiking boots and return them. | https://www.loom.com/share/00e1e801f97e4782b1bd8b1492e193ca |
| 224 | 1448 | https://restaurantguru.com | READ | easy | Search for Italian restaurants in New York City and list the top 3 restaurants by user rating. Only use http://restaurantguru.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-285 | Success | Agent was able to find the three highest rated Italian restaurants in NYC. | https://www.loom.com/share/21fa843105bc49e29b5ff388ee70672d |
| 225 | 1456 | https://resy.com | CREATE | easy | Create a reservation request for a large group of eight at a restaurant in "NoMad Diner", including a note requesting a private dining area. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-292 | Success | Agent was able to reserve an 8 person table at the desired restaurant. | https://www.loom.com/share/cee0abe0c39a4c50b47693ffd5775ac5 |
| 226 | 1464 | https://resy.com | CREATE | easy | Create a personalized list titled "Romantic Dinners" on Resy and add the restaurants "Le Bernardin," "Jeju Noodle Bar,," and "Eleven Madison Park" to it. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-290 | Success | Agent was successfully able to add all of the restaurants to a saved list. | https://www.loom.com/share/3ab447ddcff04c90878fff8b6d605f25 |
| 227 | 1466 | https://resy.com | DELETE | easy | Login, add "Passerine" as favoriated in your account. then access your favorites list on Resy and remove it from the list. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-291 | Success | Agent was able to add the restaurant to the hit list and then remove it. | https://www.loom.com/share/faa8b6702bd445fb9cd9c7e2d8ba9640 |
| 228 | 1468 | https://resy.com | READ | easy | How many $'s does the Carbone in Greenwich Village, New York have? Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-293 | Success | Agent was able to identify that Carbone has a 4 $ rating. | https://www.loom.com/share/a1d96a35c6804f3999e66e54cd8956c0 |
| 229 | 1469 | https://resy.com | READ | hard | Search for available reservation times for a French restaurant in New York City on an upcoming Friday evening. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-286 | Success | Agent was able to find available reservation times for French restaurants in New York City tonight. | https://www.loom.com/share/a985dbf577804361b0b32f0d4f0b3641 |
| 230 | 1470 | https://resy.com | READ | easy | Find Arden, in New York. List the available dining times for tonight. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-294 | Success | Agent was able to find reservation times for Arden in NYC. | https://www.loom.com/share/b864f98aad8141d19f04806932a4112c |
| 231 | 1472 | https://resy.com | UPDATE | easy | Make a reservation at "Tacombi - Williamsburg" for 7:00 PM on a given night. Modify your "Tacombi - Williamsburg" reservation by changing the dinner time from 7:00 PM to 8:00 PM. Only use http://resy.com/ to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-288 | Failure | Agent was unable to alter the reservation directly, but attempted to cancel the reservation and then create a new one. | https://www.loom.com/share/55ce718465024a4d98555ba006043ffe |
| 232 | 1478 | https://www.reverso.net/text-translation | READ | easy | Use the collaborative dictionary to translate 'proud' from English to Arabic and return the result Only use http://reverso.net to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-295 | Success | Agent was able to find the Arabic translation for the word proud. | https://www.loom.com/share/339e91e6305a44f29d68b655e73b5c1d |
| 233 | 1491 | https://www.samsung.com/us | READ | easy | Navigate to the Samsung Galaxy S23 product page and list its key technical specifications such as processor, battery capacity, and display resolution. Only use http://samsung.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-296 | Failure | Human timed agent out - it couldn't find the specs for the Galaxy s23 after multiple attempts. | https://www.loom.com/share/f49f2cba8ff1445593e097f66d89ff19 |
| 234 | 1503 | https://screenrant.com | READ | easy | What is the description of the game 'Marvel Rivals'? Only use http://screenrant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-299 | Failure | Agent couldn't find a Marvel Rivals synopsis, so it tried to perform a Microsoft Bing search to retrieve it | https://www.loom.com/share/ff30845bcd4e4733829821ac939bcd19 |
| 235 | 1504 | https://screenrant.com | READ | easy | Find the article '10 TV Shows That Went Downhill After A Major Cliffhanger' and summarize why 'The Flash' is included. Only use http://screenrant.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-298 | Success | Agent was able to find the article and explained why "The Flash" was included in the article. | https://www.loom.com/share/f5c685cf4fb24487869db2635ca900fe |
| 236 | 1522 | https://us.shein.com/?ref=www&rep=dir&ret=us | UPDATE | easy | While logged in, add a "gold hoop earrings" product to your cart, update its quantity to 2 units, and verify that the total price is adjusted accordingly. Only use http://shein.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-301 | Failure | Agent was unable to add the item to the cart, so this task was a failure. | https://www.loom.com/share/1a5013c1ea74468ab53dd84a0c687d1a |
| 237 | 1520 | https://us.shein.com/?ref=www&rep=dir&ret=us | READ | hard | Browse the "New In" section and list the product names, prices, and available colors of the top 5 most popular items. Only use http://shein.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-300 | Failure | Agent was unable to find the most popular items and list the respective details. | https://www.loom.com/share/928f7200485743ba8841c34c20a732cc |
| 238 | 1525 | https://www.si.com | READ | easy | Locate an SI.com feature on college sports and provide a brief summary outlining its key discussion points. Only use http://si.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-302 | Success | Agent found an article and returned a short summary as well. | https://www.loom.com/share/a1f2648de3a64e6689376fa3e9c94e40 |
| 239 | 1528 | https://www.si.edu | READ | easy | Navigate the homepage to locate the "Smithsonian Open Access" section and list the titles of the first five featured digital assets. Only use http://si.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-303 | Success | Agent found the first five titles of the Smithsonian digital assets. | https://www.loom.com/share/e7b149504f994800a153b6778127d892 |
| 240 | 1539 | https://www.simplyrecipes.com | UPDATE | hard | Login, post a comment praising a Lentil Soup recipe, then edit your previously submitted comment with additional recommendations for pairings with this dish. Only use http://simplyrecipes.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-304 | Success | Agent was able to comment and then edit and save its comment. Human had to log in and confirm both actions. | https://www.loom.com/share/f51eaff07d374cfe88c92d30b32f354f |
| 241 | 1545 | https://www.skysports.com | READ | easy | Identify and read the latest article under the rugby section covering match recaps, and list the names of the two teams that played. Only use http://skysports.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-306 | Success | Agent found the most recent match recap and summarized it for the human. | https://www.loom.com/share/3e29ef3d27c244f2992495027c3d9e0a |
| 242 | 1546 | https://www.skysports.com | READ | easy | Browse the homepage for any breaking news related to the "Football" and provide a short text summary of the update. Only use http://skysports.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-305 | Success | Agent was able to find relevant news related to Football and provide a summary. | https://www.loom.com/share/bc43d8b0b5e04d0c90ce3f5c6ecb4162 |
| 243 | 1550 | https://www.slideshare.net | READ | easy | Search for presentations on "digital marketing trends 2023" and list the titles and authors of the first five results. Only use http://slideshare.net to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-308 | Success | Agent was able to list the titles and authors of the first five results. | https://www.loom.com/share/f9a501a33b794e1cbe1c1d08836c7278 |
| 244 | 1551 | https://www.slideshare.net | READ | easy | Browse the "Business" category and extract the title of the most-viewed presentation. Only use http://slideshare.net to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-307 | Success | Agent successfully found the title of the most recent presentation. | https://www.loom.com/share/2ae5a74a414b4c83a24116390d8cd2c0 |
| 245 | 1553 | https://www.slideshare.net | READ | easy | Identify three presentations on "sustainable energy" sorted by popularity and list their titles. Only use http://slideshare.net to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-309 | Success | Agent was able to find and return three presentations on sustainable energy. | https://www.loom.com/share/4b8f7349ae15476f87ba9cacbe266191 |
| 246 | 1555 | https://www.smithsonianmag.com | READ | hard | Browse the History category and record the title and publication date of the most popular article (based on visible engagement or share counts). Only use http://smithsonianmag.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-310 | Success | Agent identified an article based on its mention in a 2024 most popular article. | https://www.loom.com/share/16d834dc65be4a5ba208fc6b0adb71d1 |
| 247 | 1566 | https://www.sofascore.com | READ | hard | Explore the heatmap visualization on a live football game page and specify the zone where the highest concentration of shots occurred. Only use http://sofascore.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-312 | Success | Agent identified the portion of the pitch that received the most attention by looking at the heat map. | https://www.loom.com/share/957445d3027046f985113c2b4c172bd5 |
| 248 | 1573 | https://www.sportskeeda.com | CREATE | easy | Log in to your Sportskeeda account and post a comment on the latest "UEFA Champions League" article sharing your opinion on the match outcome. Only use http://sportskeeda.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-314 | Success | Agent was able to post a relevant comment that it generated, as tasked. | https://www.loom.com/share/b3fc2bee55c4494f926830bc6e2b6184 |
| 249 | 1586 | https://www.sportskeeda.com | UPDATE | easy | Access your account settings on Sportskeeda and update your bio to better reflect your interests in both traditional sports and esports. Only use http://sportskeeda.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-313 | Success | Agent successfully updated the bio to reflect the desired changes and saved it. | https://www.loom.com/share/3c42863cfeb84a19baf5ddda3145106b |
| 250 | 1591 | https://open.spotify.com | CREATE | easy | Create a podcast playlist by starting a new list and adding one episode from a podcast in the "Tech Trends" category. Only use http://spotify.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-315 | Success | Agent created a playlist with the first episode from a tech podcast. | https://www.loom.com/share/1eba6becd83c4c6a82a50498836297be |
| 251 | 1601 | https://open.spotify.com | DELETE | easy | Create a temporary playlist titled "Test Playlist," add two tracks, then delete the playlist and confirm its removal from your library. Only use http://spotify.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-317 | Success | Agent was able to create the playlist, add two tracks, and delete it successfully. | https://www.loom.com/share/ceff1d6bbe04424e8e72037cd23836cb |
| 252 | 1605 | https://open.spotify.com | FILE_MANIPULATION | hard | Login and find out how to request extended streaming data, and follow the steps to request this data. Only use http://spotify.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-316 | Failure | Interface failed to respond multiple times when human took control of browser to log in. | https://www.loom.com/share/c1d4746287004768b7f5d725948eec1c |
| 253 | 1609 | https://open.spotify.com | READ | easy | Browse the Spotify home page and list the names of the top 5 trending playlists currently featured. Only use http://spotify.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-318 | Success | Agent identified the 5 top trending playlists on Spotify. | https://www.loom.com/share/9463ed7671f94dae96979bbe0ab57b50 |
| 254 | 1625 | https://www.springer.com/us | READ | hard | Find the "Philosophy" articles page and summarize the abstract of the most recent article posted. Only use http://springer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-320 | Success | Agent identified and summarized a recent philosophy article with a scientific twist as well. | https://www.loom.com/share/d5ae495541a04de182591ad3395108eb |
| 255 | 1635 | https://stackexchange.com | CREATE | easy | Log in, navigate to the Data Science community, and post an answer on a question about "feature engineering techniques" including at least two bullet points. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-325 | Success | Agent was able to log in and post an answer to a question about "feature engineering techniques." | https://www.loom.com/share/b354f8d8c30841b88e7bf1b3e04591bd |
| 256 | 1640 | https://stackexchange.com | CREATE | hard | Post an answer in the Super User community to a question regarding "SSD vs HDD performance" and include relevant benchmark numbers. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-331 | Success | Agent posted a response to the question regarding "SSD vs HDD performance" and included data as well. | https://www.loom.com/share/f9cf7b71abbb42c1a5960b0b94f9d790 |
| 257 | 1642 | https://stackexchange.com | CREATE | easy | Log in and post a new question in the Academia community titled "How to balance research and teaching responsibilities?" with contextual examples from your experience. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-328 | Success | Agent was able to create the post successfully. | https://www.loom.com/share/f08de55d6b704fe492ee6be07a7fbdc0 |
| 258 | 1644 | https://stackexchange.com | CREATE | hard | Using the Markdown editor, log in and create a comprehensive guide titled "How to use StackExchange effectively" on the Meta site. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-326 | Success | Agent was able to post a guide titled "How to use StackExchange effectively" on the Meta site. | https://www.loom.com/share/7924cdabcf1a4f7db85f18df5c2692af |
| 259 | 1658 | https://stackexchange.com | DELETE | easy | In the Super User community, post a temporary answer labeled "Test answer for deletion" to an existing question and then delete it, ensuring it no longer appears in your contributions. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-322 | Failure | After CAPTCHA popped up 4 times after human solved it again and again, it became clear there was a connection issue between the browser and the agent. | https://www.loom.com/share/7f523d79a2784f38abb3f206068ff728 |
| 260 | 1661 | https://stackexchange.com | READ | easy | Visit the Meta site of StackExchange and list the titles of the 3 most recent community discussion topics. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-330 | Success | Agent was able to find three most recent community discussion topics. | https://www.loom.com/share/2e9ca161cabc4774b4a5c87ac1baf3f9 |
| 261 | 1662 | https://stackexchange.com | READ | easy | Use the main search bar to find questions tagged with "python" in the Stack Overflow community and output the titles of the first 5 results. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-323 | Failure | CAPTCHA didn't process no matter how many times human solved it. | https://www.loom.com/share/1a1b44bd39b94e089e36efaf322cde6f |
| 262 | 1669 | https://stackexchange.com | UPDATE | easy | Log in and post a question in the Cross Validated community titled "Statistical Significance in A/B Testing." Then update the title of this question to "Understanding Statistical Significance in A/B Testing." Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-329 | Success | Agent was able to post the question, and then revise and update the title of the question. | https://www.loom.com/share/b31278e480a141f4b47e90c7f0365648 |
| 263 | 1671 | https://stackexchange.com | UPDATE | easy | Log in to the Web Applications community and answer a question "increasing website engagement." Then update your answer with additional examples and an emoji. Only use http://stackexchange.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-327 | Failure | Agent was not able to find the question, so this task is a fail. | https://www.loom.com/share/b387f365140e493db24bec79f2ba678d |
| 264 | 1676 | https://stackoverflow.com/questions | CREATE | hard | Provide an answer to a question regarding best practices for REST API development in Ruby on Rails, including a sample code implementation. Only use http://stackoverflow.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-335 | Failure | Agent found the answer it needed to provide, and returned it instead of commenting on the question. | https://www.loom.com/share/b9f2ab60e2974cf5a1aa9ce361ef728d |
| 265 | 1690 | https://stackoverflow.com/questions | DELETE | easy | Log in, post a question about Node.js with the tag "node" and then go back to delete the tag from your posted question on Node.js, ensuring that the removal is confirmed. Only use http://stackoverflow.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-332 | Success | Agent was able to post about Node.js with the tag "node" and then go back to delete it. | https://www.loom.com/share/0630fede78604a67b23aa532140f599f |
| 266 | 1692 | https://stackoverflow.com/questions | READ | easy | Browse through the "Help Center" to locate the page on editing posts and list the top three tips mentioned. Only use http://stackoverflow.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-333 | Success | Agent identified three tips from the Help Center page and returned them. | https://www.loom.com/share/24b1f5545eae49ebb97037cdbb3714e7 |
| 267 | 1698 | https://stackoverflow.com/questions | UPDATE | hard | Log in to your account, and post a question regarding web security, then update the tags on your question about web security to also include "sql-injection" Only use http://stackoverflow.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-334 | Failure | Instead of creating a new post and editing that, it updated a previously posted post. | https://www.loom.com/share/73a77d54805b4fc4b31d7142efdd71b1 |
| 268 | 1704 | https://stardewvalleywiki.com/Stardew_Valley_Wiki | READ | easy | Analyze which villagers have birthdays in Summer using the calendar data Only use http://stardewvalleywiki.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-337 | Success | Agent was able to identify which villagers have summer birthdays successfully. | https://www.loom.com/share/283932ed63fa4d109ac36aeef9252be5 |
| 269 | 1705 | https://stardewvalleywiki.com/Stardew_Valley_Wiki | READ | easy | Find and list all the gifts that Sebastian "loves" according to his gift preferences page. Only use http://stardewvalleywiki.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-336 | Success | Agent identified a list of Sebastian's "loved" gifts. | https://www.loom.com/share/fc1aa4d991b248c8a0b01aee44a47f77 |
| 270 | 1713 | https://www.statista.com | READ | easy | Filter the available reports by the sports industry and list the titles of the first three annual reports presented. Only use http://statista.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-338 | Success | Agent was able to filter by the sports industry and list the top three reports mentioned. | https://www.loom.com/share/e719c65807a8478fb145fa99d95af9fc |
| 271 | 1723 | https://store.steampowered.com | READ | hard | Go to the Top Sellers section and list all games currently discounted at more than 50%. Only use http://steampowered.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-340 | Success | Agent was able to browse the list of top sellers and provide a list of 50% or more discounted games. Human asked the agent to stop at some point because it was scrolling, which was both exhaustive but redundant. | https://www.loom.com/share/0de4035fa39743968e020b42cacb72c1 |
| 272 | 1729 | https://www.studocu.com | CREATE | easy | Create a new account and then create a course name of "Introduction to Economics." Ensure that a confirmation message appears. Only use http://studocu.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-341 | Failure | Agent could not find a way to name the specific course "Introduction to Economics." | https://www.loom.com/share/75aff4cdb4ca42f5addc4ad03eeb420a |
| 273 | 1732 | https://www.studocu.com | CREATE | hard | Assemble a new study plan by creating a "Revision Pack" StudyList that curates documents on key course topics and then use the site’s sharing feature to share it with a friend. Only use http://studocu.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-342 | Success | Agent was able to create the study list and generate a link to be able to share. | https://www.loom.com/share/3308ade497a342e6986bcbb8ba3ab067 |
| 274 | 1745 | https://www.target.com | READ | easy | Check how long shipping takes for an ipad Air to be delivered to the New York City 10003 Zip Code Only use http://target.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-343 | Success | Agent updated the zip code to 10003 and found the time that it would take to ship, as well as the shipping cost. | https://www.loom.com/share/d9ff2cfb7d81444aaef88fbd4c3ed0da |
| 275 | 1750 | https://www.techradar.com | READ | easy | Search for articles related to “energy efficient computers” and extract a recommendation mentioned in one of the articles. Only use http://techradar.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-344 | Success | Agent found a relevant suggestion about power saving for a vehicle. | https://www.loom.com/share/ef83fbea84c64c66b9731d0ebc17f1d0 |
| 276 | 1761 | https://www.telegraph.co.uk/us | DELETE | easy | Log in to your Telegraph account, find an article on the front page and save it to your reading list, then access your reading list, delete the saved articles, and verify that it no longer appears. Only use http://telegraph.co.uk to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-346 | Success | Agent was able to add an article from the front page, save it, and then delete it from the saved list. | https://www.loom.com/share/e3484f134eb94f99a9a2f4cc1b81bb05 |
| 277 | 1765 | https://www.telegraph.co.uk/us | READ | easy | Use the search function to find an article about a “Royal Ceremony” and provide the headline and publication date of the first result. Only use http://telegraph.co.uk to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-345 | Failure | Agent found an article for a red carpet walk, not a royal ceremony, which would imply British Royalty/wedding. | https://www.loom.com/share/4ce3c3bd6502402cbf26bd11b387046d |
| 278 | 1768 | https://www.telegraph.co.uk/us | UPDATE | easy | Log in to your Telegraph account, navigate to your profile settings, and change your newsletter subscription preference from daily updates to weekly updates. Only use http://telegraph.co.uk to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-347 | Failure | Agent subscribed to a daily newsletter, and when prompted by human to change the preference to a weekly cadence, the agent tried to subscribe to a daily newsletter rather than changing the existing preference. | https://www.loom.com/share/3ccc1849528b471ba9a65bff0f6a302c |
| 279 | 1769 | https://testbook.com | READ | easy | Search the articles section for "SSC exam pattern changes" and extract the key updates mentioned. Only use http://testbook.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-349 | Success | Agent was able to find and provide a sufficient amount of key exam updates. | https://www.loom.com/share/0bc6633f51ff4e56a57074db149ae4c0 |
| 280 | 1774 | https://www.texas.gov | READ | easy | In the "Find Services" section, locate the official guidelines on vehicle registration and extract the checklist of submission requirements shown. Only use http://texas.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-350 | Success | Agent was able to find the "Find Services" section and provide all the registration details demanded. | https://www.loom.com/share/f31fa648c64b46d7b7f9f8ed533f845c |
| 281 | 1778 | https://www.theatlantic.com | READ | easy | Navigate to The Atlantic homepage and list the titles of the top three featured articles displayed. Only use http://theatlantic.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-352 | Success | Agent found the top three featured articles displayed and returned them. | https://www.loom.com/share/7dd0b06515b54d259bdf2dadea67bf90 |
| 282 | 1780 | https://www.theatlantic.com | READ | easy | Visit the subscription page and identify the differences in benefits offered between the digital, digital and print subscriptions. Only use http://theatlantic.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-351 | Success | Agent quickly identified the differences between each subscription model. | https://www.loom.com/share/d55ef8b0e09d48d7aad9f92471d4c225 |
| 283 | 1784 | https://www.thegamer.com | READ | easy | Search for reviews and list the headlines of the top five game review articles currently featured. Only use http://thegamer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-353 | Success | Agent successfully found and returned the top five featured articles on game reviews. | https://www.loom.com/share/3379ab3407fd4143a66ebcd5dbad0692 |
| 284 | 1785 | https://www.thegamer.com | READ | hard | Locate an Opinion or Cultural Commentary article discussing modern gaming culture and summarize its central argument in one or two sentences. Only use http://thegamer.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-354 | Failure | Agent found an article and summarized the synopsis, but failed to summarize the argument. | https://www.loom.com/share/70ea4cce4329430e8dc54bfb6eba3989 |
| 285 | 1792 | https://www.thehindu.com | READ | easy | Search the Health section for COVID-19 related news, and output the title and publication time of the first article that appears. Only use http://thehindu.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-355 | Success | Agent found the first article related to COVID-19. | https://www.loom.com/share/9ab426bd8e894276a180e4c2eaf6e66c |
| 286 | 1798 | https://www.themoviedb.org | READ | easy | Browse the "Trending" section on themoviedb.org and extract the titles of the top 5 trending movies at the moment. Only use http://themoviedb.org to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-356 | Success | Agent found the top 5 trending movies on the website. | https://www.loom.com/share/06ffe60589af45858c49584180b81caf |
| 287 | 1801 | https://www.thepioneerwoman.com | CREATE | easy | Log in to the site and save your favorite recipe from the "Comfort Food" category to your saved recipes collection. Only use http://thepioneerwoman.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-358 | Success | Agent was able to find a recipe in the comfort food section and save it. | https://www.loom.com/share/8a5f9a820e34432486b570749858a4a6 |
| 288 | 1813 | https://www.thepioneerwoman.com | READ | easy | Locate a tutorial featuring Ree Drummond’s signature dish. Only use http://thepioneerwoman.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-357 | Success | Agent found a tutorial recipe with Ree Drummonds signature cookies, and provided a link as well. | https://www.loom.com/share/b4360731f56a44d6935f8e751ffdea1f |
| 289 | 1816 | https://www.thesaurus.com | READ | easy | Look up the synonyms for "rapid" and list the first five synonyms that appear. Only use http://thesaurus.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-359 | Success | Agent found the first 5 synonyms for "rapid." | https://www.loom.com/share/239c8aa13b4d4b3f8a76bc35a0cd5356 |
| 290 | 1819 | https://www.thespruce.com | READ | easy | Search for "DIY kitchen backsplash ideas" on TheSpruce and list the titles of the first five articles. Only use http://thespruce.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-360 | Success | Agent found the first five articles that came up when it searched for "DIY Kitchen Backsplash Ideas." | https://www.loom.com/share/0dcc2ec986ac44318a7a29dca2b4ce22 |
| 291 | 1828 | https://www.thoughtco.com | READ | easy | Browse the ThoughtCo homepage and list the titles of the three newest articles displayed on the landing page. Only use http://thoughtco.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-361 | Success | Agent found and returned the first three articles on the homepage. | https://www.loom.com/share/440863d345144980b3edaa6eee82fabc |
| 292 | 1829 | https://www.threads.net | READ | hard | Use the search function to find threads mentioning “sustainability” and summarize the main discussion points from the first three results. Only use http://threads.net to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-362 | Success | Agent was able to find 3 threads and summarize them as tasked. | https://www.loom.com/share/d92d6951a60e4a13a4d5b4173809702c |
| 293 | 1832 | https://www.ticketmaster.com | READ | hard | Visit the New York Giants' page, and find an available game to find tickets for. Only use http://ticketmaster.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-364 | Success | Agent was technically correct in the sense that no tickets are available for NFL games because as of April 29th, 2025, the NFL schedule has not yet been released. | https://www.loom.com/share/eada76b1d70f49cd931efb28b4860e6f |
| 294 | 1836 | https://www.tiktok.com/explore | READ | easy | Browse the For You feed and list the usernames and view counts of the first 5 videos displayed. Only use http://tiktok.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-365 | Failure | Agent provided the like count, but not the view count. | https://www.loom.com/share/4a06f6f93707479790ed02c4428fd1f9 |
| 295 | 1843 | https://time.com | READ | easy | Navigate to the "Science" section and summarize the key findings of the leading article in 2–3 sentences. Only use http://time.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-366 | Success | Agent was able to find the first article and provide a 2-3 sentence summary effectively. | https://www.loom.com/share/5a21fd851cdd4eb98aaf4bb32f4d13ea |
| 296 | 1848 | https://www.timeanddate.com | READ | easy | Browse the API section to retrieve the Date Calculator API Pricing details including the package prices and number of credits Only use http://timeanddate.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-367 | Success | Agent was able to find pricing details and return them for Time and Date. | https://www.loom.com/share/e91057e3b75b4338bee8d57ed1e60601 |
| 297 | 1853 | https://www.today.com | CREATE | easy | Save an article about summer recipes to your reading list and add a brief note explaining why you found it interesting. Only use http://today.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-371 | Failure | Agent was able to save the recipe, but couldn't access the bookmarks folder and din't provide a note as to why it liked the recipe. | https://www.loom.com/share/17ce764a1d834ed7859ed6b9c9d3e06f |
| 298 | 1858 | https://www.today.com | READ | hard | Scroll through the infinite content feed in the News category and record how many new articles load after an extended scroll. Only use http://today.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-370 | Success | Agent was able to scroll until it couldn't anymore and recorded 120 different articles. | https://www.loom.com/share/c636b561c7cd4dd6946ac1b6bb171494 |
| 299 | 1860 | https://www.today.com | READ | easy | Search for articles on “healthy breakfast recipes” and display the title and summary of the top result. Only use http://today.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-369 | Success | Agent was able to return the title and summary of the top article when searching for “healthy breakfast recipes." | https://www.loom.com/share/dbf78b03714a46d4aae535a50265884e |
| 300 | 1861 | https://www.today.com | READ | easy | Navigate to the homepage and locate the schedule for the live Today show broadcast, then record the start time of today’s episode. Only use http://today.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-372 | Success | Agent was able to find the schedule and return the time of today's episode. | https://www.loom.com/share/265886ec04224d089e729129f9d21886 |
| 301 | 1868 | https://www.tomsguide.com | FILE_MANIPULATION | easy | Locate the downloadable PDF version of the "How to Enable 2FA" guide on Tom's Guide, download it, and verify that the filename contains the text "2FA". Only use http://tomsguide.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-373 | Failure | Agent was unable to locate the guide with a downloadable pdf. | https://www.loom.com/share/c77b6f9390c34dffa93e47e71695e8cb |
| 302 | 1873 | https://www.tomsguide.com | READ | easy | In the Reviews section, filter for gaming review and note down the headline of the top-listed review article. Only use http://tomsguide.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-374 | Success | Agent was able to filter for reviews and listed the top review headline. | https://www.loom.com/share/f3719ce945f44dcd98290a56bc3c3b41 |
| 303 | 1883 | https://www.travelweekly.com | READ | easy | Browse the homepage and list the top five featured travel industry news headlines. Only use http://travelweekly.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-375 | Success | Agent was able to navigate to the homepage and list the top five featured travel industry news headlines. | https://www.loom.com/share/1dadc99a54c14323893106588b3e13a0 |
| 304 | 1885 | https://www.travelweekly.com | READ | easy | Use the search tool to look for upcoming webinar events on travel technology and output the titles of the events. Only use http://travelweekly.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-376 | Failure | Agent wasn't able to find any webinars related to travel technology. | https://www.loom.com/share/4d2604581b404397a7370915155ba012 |
| 305 | 1888 | https://us.trip.com/?locale=en-us | READ | easy | Browse the Trip homepage and list the top 5 trending featured properties Only use http://trip.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-377 | Success | Agent was able to browse the homepage and list the top 5 trending properties. | https://www.loom.com/share/6cca7c0d64554d4eb4168cd7ead7097b |
| 306 | 1890 | https://us.trip.com/?locale=en-us | READ | easy | Search for tours available in Rome tomorrow and list the top three tours sorted by "Top Rated". Only use http://trip.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-378 | Success | Agent was able to search for tours available in Rome tomorrow and list the top three tours sorted by "Top Rated". | https://www.loom.com/share/ccb756ffbdf54e02a8de3c71468f729b |
| 307 | 1892 | https://www.tripadvisor.com | READ | hard | Identify the top-rated hotel in Paris, verify if it offers free cancellation, and analyze at least three recent guest reviews to see if they mention staff helpfulness. Only use http://tripadvisor.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-379 | Success | Agent was able to filter by the necessary constraints, locate the highest rated hotel, and provide multiple anecdotes of the hotel's service. | https://www.loom.com/share/c4ac4e017b24491a8032bae7c60096bb |
| 308 | 1897 | https://www.trustpilot.com | READ | hard | Use Trustpilot’s search function to filter HR & Recruiting located in "London", then list the review summaries for the top three highest‑rated companies. Only use http://trustpilot.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-380 | Success | Agent filtered by HR & Recruiting located in "London", then listed the review summaries for the top three highest‑rated companies. | https://www.loom.com/share/9de36385ab79453ab0f274add78374b7 |
| 309 | 1911 | https://www.twitch.tv | UPDATE | easy | Modify your alert settings by updating the text that appears when someone donates Bits, ensuring your channel name is featured prominently in the alert. Only use http://twitch.tv to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-381 | Success | Agent was able to create a bit donation message, and then update it to include a name, which human provided. Name was able to be different than account name. | https://www.loom.com/share/855c3569496444eb81a9de7f67a060e5 |
| 310 | 1914 | https://www.ucdavis.edu | READ | easy | Navigate to the Admissions section and extract the key deadline dates for Fall 2025 undergraduate applications. Only use http://ucdavis.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-382 | Success | Agent was able to screen the Admissions section and extract the key deadline dates for Fall 2025 undergraduate applications. | https://www.loom.com/share/09ad9976c8af4334bd50bd1a7554f53e |
| 311 | 1915 | https://www.ucdavis.edu | READ | easy | Search for the UC Davis library page and retrieve the opening hours and contact information for the main library. Only use http://ucdavis.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-383 | Success | Agent was able to identify hours and contact information at the UC Davis main library. | https://www.loom.com/share/84bfda4fbd9145a5a2db75f6bf1ec4c0 |
| 312 | 1917 | https://www.ucdavis.edu | READ | easy | Go to the Financial Aid and Scholarships page and extract the main eligibility criteria and deadlines for applying for aid. Only use http://ucdavis.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-384 | Success | Agent was able to extract the main eligibility criteria and deadlines for applying for aid. | https://www.loom.com/share/30d15d920aa34ec18ba6103fa5927a50 |
| 313 | 1924 | https://www.ucla.edu | READ | easy | Search for information on UCLA’s athletic programs or sports teams and list the sports teams mentioned on the page. Only use http://ucla.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-385 | Success | Agent was able to identify all of the sports teams at UCLA. | https://www.loom.com/share/d3271a21051f49518bf9c23f3fbc487d |
| 314 | 1930 | https://www.ufl.edu | READ | easy | Use the site’s search bar to find information on UF research computing facilities and note two primary services offered. Only use http://ufl.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-386 | Success | Agent looked at UF research computing facilities and noted two primary services offered. | https://www.loom.com/share/f45244fe07934c09975f4556b58405cf |
| 315 | 1931 | https://www.ufl.edu | READ | easy | Browse the Academic Departments listing and extract key details on the Computer & Information Science & Engineering department’s research focus areas. Only use http://ufl.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-387 | Success | Agent was able to scan the Computer Science department and identify research domains. | https://www.loom.com/share/81c1d23815894d9cab853bdb774ee881 |
| 316 | 1938 | https://www.ultimate-guitar.com | READ | easy | Find the guitar tabs for "Hotel California" by the Eagles. Only use http://ultimate-guitar.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-388 | Failure | Agent was unable to access the site on the browser. | https://www.loom.com/share/480a5947a89146ac95a74bc6e8dce1f0 |
| 317 | 1939 | https://umich.edu | READ | easy | Browse the University of Michigan homepage and list the top three headlines featured in the News section. Only use http://umich.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-389 | Success | Agent was able to identify the top headlines on the front page. | https://www.loom.com/share/36139c1a70bb4846af8046461601d115 |
| 318 | 1942 | https://umich.edu | READ | easy | Explore the “Michigan Online” section and list two online courses or learning platforms mentioned on that page. Only use http://umich.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-390 | Success | Agent listed two online courses or learning platforms mentioned on the "Michigan Online" page. | https://www.loom.com/share/79a08f6474f84015bdae6b111bf55eb5 |
| 319 | 1951 | https://www.uniqlo.com/us/en | DELETE | hard | Log in to your Uniqlo account, add the BlockTech Shirt to a wishlist, then remove it from the wishlist, confirming that the product no longer appears on the list. Only use http://uniqlo.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-391 | Failure | Agent was unable to find the BlockTech shirt, so this was a failure. | https://www.loom.com/share/6d3a572b9c074f6e9001140f575a245c |
| 320 | 1966 | https://www.upenn.edu | READ | easy | Search the website for information on interdisciplinary research centers and list two names or brief descriptions provided on the page. Only use http://upenn.edu to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-392 | Success | Agent was able to find information on interdisciplinary research centers and list two names provided. | https://www.loom.com/share/2b1df8ae1dc54e71a9f3c61f4b15f364 |
| 321 | 1968 | https://www.urbandictionary.com | READ | easy | Find the definition for “woke” and list any descriptive tags or keywords that appear alongside the definition. Only use http://urbandictionary.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-393 | Success | Agent was able to find the definition of "woke" and listed "cultural marxism" as keywords. | https://www.loom.com/share/88c744c8639b4ee4b9ac54c779dd9a73 |
| 322 | 1969 | https://www.urbandictionary.com | READ | hard | Use the search bar to look up “stan” and compare the top two definitions by summarizing the contrasting perspectives or nuances in their descriptions. Only use http://urbandictionary.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-394 | Success | Agent was able to find the definition for "stan," and briefly summarized the differences for the top two definitions. | https://www.loom.com/share/e0336d8b997743f1b0ca0528cbbf8ad9 |
| 323 | 1974 | https://www.usa.gov | FILE_MANIPULATION | easy | Find and download the most recent government guide on disaster preparedness from USA.gov, then verify that the file is in PDF format. Only use http://usa.gov to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-395 | Success | Agent found the pdf, downloaded it, confirmed it was in pdf form, and offered a link to access the pdf. | https://www.loom.com/share/3b6f8a412b2b42098883c826c3b49c82 |
| 324 | 2021 | https://www.usnews.com | READ | easy | Search the site for "economic trends 2023" and list the titles and publication dates of the first five matching articles. Only use http://usnews.com to achieve the task. Don't go to any other site. The task is achievable with just navigation from this site. | HITL-Operator-400 | Success | Agent listed the titles and publication dates of the first five matching articles for "economic trends 2023." | https://www.loom.com/share/87a65e2ad36b463cac7fd32f72e5b95c |