{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/mind2web","name":"Mind2Web","full_name":null,"description_markdown":"Dataset Summary\r\n\r\nMind2Web is a dataset for developing and evaluating generalist agents for the web that can follow language instructions to complete complex tasks on any website. Existing datasets for web agents either use simulated websites or only cover a limited set of websites and tasks, thus not suitable for generalist web agents. With over 2,000 open-ended tasks collected from 137 websites spanning 31 domains and crowdsourced action sequences for the tasks, Mind2Web provides three necessary ingredients for building generalist web agents: 1. diverse domains, websites, and tasks, 2. use of real-world websites instead of simulated and simplified ones, and 3. a broad spectrum of user interaction patterns.\r\nDataset Structure\r\nData Fields\r\n\r\n    \"annotation_id\" (str): unique id for each task\r\n    \"website\" (str): website name\r\n    \"domain\" (str): website domain\r\n    \"subdomain\" (str): website subdomain\r\n    \"confirmed_task\" (str): task description\r\n    \"action_reprs\" (list[str]): human readable string representation of the action sequence\r\n    \"actions\" (list[dict]): list of actions (steps) to complete the task\r\n        \"action_uid\" (str): unique id for each action (step)\r\n        \"raw_html\" (str): raw html of the page before the action is performed\r\n        \"cleaned_html\" (str): cleaned html of the page before the action is performed\r\n        \"operation\" (dict): operation to perform\r\n            \"op\" (str): operation type, one of CLICK, TYPE, SELECT\r\n            \"original_op\" (str): original operation type, contain additional HOVER and ENTER that are mapped to CLICK, not used\r\n            \"value\" (str): optional value for the operation, e.g., text to type, option to select\r\n        \"pos_candidates\" (list[dict]): ground truth elements. Here we only include positive elements that exist in \"cleaned_html\" after our preprocessing, so \"pos_candidates\" might be empty. The original labeled element can always be found in the \"raw_html\".\r\n            \"tag\" (str): tag of the element\r\n            \"is_original_target\" (bool): whether the element is the original target labeled by the annotator\r\n            \"is_top_level_target\" (bool): whether the element is a top level target find by our algorithm. please see the paper for more details.\r\n            \"backend_node_id\" (str): unique id for the element\r\n            \"attributes\" (str): serialized attributes of the element, use json.loads to convert back to dict\r\n        \"neg_candidates\" (list[dict]): other candidate elements in the page after preprocessing, has similar structure as \"pos_candidates\"\r\n\r\nData Splits\r\n\r\n    train: 1,009 instances\r\n    test: (To prevent potential data leakage, please check our repo for information on obtaining the test set.)\r\n        Cross Task: 252 instances, tasks from the same website are seen during training\r\n        Cross Website: 177 instances, websites are not seen during training\r\n        Cross Domain: 9,12 instances, entire domains are not seen during training","description_withheld":null,"homepage":"https://huggingface.co/datasets/osunlp/Mind2Web","introduced_date":"2023-06-09","introduced_date_note":null,"introduced_by":{"paper":"/paper/mind2web-towards-a-generalist-agent-for-the-1","title":"Mind2Web: Towards a Generalist Agent for the Web","first_author":"Xiang Deng","url":null},"license":{"name":"CC BY 4.0 Deed","url":"https://creativecommons.org/licenses/by/4.0/"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"task","url":null,"datasets_with_task":"/datasets/task/task"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Mind2Web"],"data_loaders":[],"num_papers_in_archive":110,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/on-mind2web","task":"","dataset_variant":"Mind2Web","rows":1,"metrics":["Element Accuracy","Operation F1 score","Step Success Rate"],"first_row_in_archive_order":{"model":"HTML-T5-XL","paper":"/paper/a-real-world-webagent-with-planning-long","metrics":{"Element Accuracy":"73","Operation F1 score":"75.6","Step Success Rate":"67.1"},"code_links":[]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":1,"of":1,"counts":{"papers_with_a_benchmark_row":1,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":1,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/a-real-world-webagent-with-planning-long","slug":"a-real-world-webagent-with-planning-long","title":"A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis","date":"2023-07-24","arxiv_id":"2307.12856","rows_on_this_dataset":1,"code_links":0,"syntology":null}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/a-real-world-webagent-with-planning-long","slug":"a-real-world-webagent-with-planning-long","title":"A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis","date":"2023-07-24","arxiv_id":"2307.12856","rows_on_this_dataset":1,"code_links":0,"syntology":null}],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":6,"papers_placed":0,"judged_by":{"model":"Claude Sonnet 4.5","models_as_recorded":["global.anthropic.claude-sonnet-4-5-20250929-v1:0"],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[{"leaderboard":"/sota/on-mind2web","since_archive":"/sota/on-mind2web#since-judged","papers_judged":6,"placed":0}]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"966cf27d43100c9e3e41c4a204cddc32c532d8bb31ad6ee3d424569b490ad89e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}