{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/dialogue-state-tracking-challenge","name":"Dialogue State Tracking Challenge","full_name":"Dialogue State Tracking Challenge","description_markdown":"The Dialog State Tracking Challenges 2 & 3 (DSTC2&3) were research challenge focused on improving the state of the art in tracking the state of spoken dialog systems. State tracking, sometimes called belief tracking, refers to accurately estimating the user's goal as a dialog progresses. Accurate state tracking is desirable because it provides robustness to errors in speech recognition, and helps reduce ambiguity inherent in language within a temporal process like dialog.\r\nIn these challenges, participants were given labelled corpora of dialogs to develop state tracking algorithms. The trackers were then evaluated on a common set of held-out dialogs, which were released, un-labelled, during a one week period.\r\n\r\nThe corpus was collected using Amazon Mechanical Turk, and consists of dialogs in two domains: restaurant information, and tourist information. Tourist information subsumes restaurant information, and includes bars, cafés etc. as well as multiple new slots. There were two rounds of evaluation using this data:\r\n\r\nDSTC 2 released a large number of training dialogs related to restaurant search. Compared to DSTC (which was in the bus timetables domain), DSTC 2 introduces changing user goals, tracking 'requested slots' as well as the new restaurants domain. Results from DSTC 2 were presented at SIGDIAL 2014.\r\nDSTC 3 addressed the problem of adaption to a new domain - tourist information. DSTC 3 releases a small amount of labelled data in the tourist information domain; participants will use this data plus the restaurant data from DSTC 2 for training.\r\nDialogs used for training are fully labelled; user transcriptions, user dialog-act semantics and dialog state are all annotated. (This corpus therefore is also suitable for studies in Spoken Language Understanding.)\r\n\r\nSource: [https://github.com/matthen/dstc](https://github.com/matthen/dstc)\r\nImage Source: [https://www.aclweb.org/anthology/W13-4065.pdf](https://www.aclweb.org/anthology/W13-4065.pdf)","description_withheld":null,"homepage":"https://github.com/matthen/dstc","introduced_date":"2013-01-01","introduced_date_note":null,"introduced_by":{"paper":"/paper/the-dialog-state-tracking-challenge","title":"The Dialog State Tracking Challenge","first_author":"Jason Williams","url":null},"license":{"name":"Unknown","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"},{"name":"Dialog","url":"/datasets/modality/dialog"}],"tasks":[{"name":"Intent Detection","url":"/task/intent-detection","datasets_with_task":"/datasets/task/intent-detection"},{"name":"Deblurring","url":"/task/deblurring","datasets_with_task":"/datasets/task/deblurring"},{"name":"Visual Tracking","url":"/task/visual-tracking","datasets_with_task":"/datasets/task/visual-tracking"},{"name":"Slot Filling","url":"/task/slot-filling","datasets_with_task":"/datasets/task/slot-filling"},{"name":"Spoken Language Understanding","url":"/task/spoken-language-understanding","datasets_with_task":"/datasets/task/spoken-language-understanding"},{"name":"Dialogue State Tracking","url":"/task/dialogue-state-tracking","datasets_with_task":"/datasets/task/dialogue-state-tracking"},{"name":"domain classification","url":"/task/domain-classification","datasets_with_task":"/datasets/task/domain-classification"},{"name":"Spoken Dialogue Systems","url":"/task/spoken-dialogue-systems","datasets_with_task":"/datasets/task/spoken-dialogue-systems"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Second dialogue state tracking challenge","Dialogue State Tracking Challenge"],"data_loaders":[{"repo":"https://github.com/matthen/dstc","url":"https://github.com/matthen/dstc","frameworks":[]}],"num_papers_in_archive":33,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/dialogue-state-tracking-on-second-dialogue","task":"Dialogue State Tracking","dataset_variant":"Second dialogue state tracking challenge","rows":7,"metrics":["Joint","Area","Food","Price","Request"],"first_row_in_archive_order":{"model":"Seq2Seq-DU-w/oSchema","paper":"/paper/a-sequence-to-sequence-approach-to-dialogue","metrics":{"Joint":"85"},"code_links":[{"title":"sweetalyssum/Seq2Seq-DU","url":"https://github.com/sweetalyssum/Seq2Seq-DU"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/deblurring-on-second-dialogue-state-tracking","task":"Deblurring","dataset_variant":"Second dialogue state tracking challenge","rows":1,"metrics":["MAE"],"first_row_in_archive_order":{"model":"Ours","paper":"/paper/attention-stay-focus-1","metrics":{"MAE":"0.0377"},"code_links":[{"title":"tuvovan/ATTSF","url":"https://github.com/tuvovan/ATTSF"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/intent-detection-on-dialogue-state-tracking","task":"Intent Detection","dataset_variant":"Dialogue State Tracking Challenge","rows":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"MIDAS","paper":"/paper/midas-multi-level-intent-domain-and-slot","metrics":{"Accuracy":"94.27"},"code_links":[{"title":"adlnlp/Midas","url":"https://github.com/adlnlp/Midas"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/slot-filling-on-dialogue-state-tracking","task":"Slot Filling","dataset_variant":"Dialogue State Tracking Challenge","rows":1,"metrics":["F1 score"],"first_row_in_archive_order":{"model":"MIDAS","paper":"/paper/midas-multi-level-intent-domain-and-slot","metrics":{"F1 score":"98.56"},"code_links":[{"title":"adlnlp/Midas","url":"https://github.com/adlnlp/Midas"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/visual-tracking-on-second-dialogue-state","task":"Visual Tracking","dataset_variant":"Second dialogue state tracking challenge","rows":1,"metrics":["Score"],"first_row_in_archive_order":{"model":"MDNet","paper":"/paper/learning-multi-domain-convolutional-neural","metrics":{"Score":"0.64"},"code_links":[{"title":"HyeonseobNam/py-MDNet","url":"https://github.com/HyeonseobNam/py-MDNet"},{"title":"FelixOliver/PROYECTO-FINAL-VOT","url":"https://github.com/FelixOliver/PROYECTO-FINAL-VOT"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":9,"of":9,"counts":{"papers_with_a_benchmark_row":9,"with_a_code_link":8,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":9,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":1,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":1,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/midas-multi-level-intent-domain-and-slot","slug":"midas-multi-level-intent-domain-and-slot","title":"MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU","date":"2024-08-15","arxiv_id":"2408.08144","rows_on_this_dataset":2,"code_links":1,"syntology":null},{"paper":"/paper/effective-sequence-to-sequence-dialogue-state","slug":"effective-sequence-to-sequence-dialogue-state","title":"Effective Sequence-to-Sequence Dialogue State Tracking","date":"2021-08-31","arxiv_id":"2108.13990","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/attention-stay-focus-1","slug":"attention-stay-focus-1","title":"Attention! Stay Focus!","date":"2021-04-16","arxiv_id":"2104.07925","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/a-sequence-to-sequence-approach-to-dialogue","slug":"a-sequence-to-sequence-approach-to-dialogue","title":"A Sequence-to-Sequence Approach to Dialogue State Tracking","date":"2020-11-18","arxiv_id":"2011.09553","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/towards-universal-dialogue-state-tracking","slug":"towards-universal-dialogue-state-tracking","title":"Towards Universal Dialogue State Tracking","date":"2018-10-22","arxiv_id":"1810.09587","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/global-locally-self-attentive-dialogue-state","slug":"global-locally-self-attentive-dialogue-state","title":"Global-Locally Self-Attentive Dialogue State Tracker","date":"2018-05-19","arxiv_id":"1805.09655","rows_on_this_dataset":1,"code_links":2,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":8,"samples_ran":7,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":7,"official":{"repos":["salesforce/glad"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/global-locally-self-attentive-dialogue-state#ran","syntology_url":"https://syntology.ai/paper/1805.09655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.09655"}}}},{"paper":"/paper/dialogue-learning-with-human-teaching-and","slug":"dialogue-learning-with-human-teaching-and","title":"Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems","date":"2018-04-18","arxiv_id":"1804.06512","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/neural-belief-tracker-data-driven-dialogue","slug":"neural-belief-tracker-data-driven-dialogue","title":"Neural Belief Tracker: Data-Driven Dialogue State Tracking","date":"2016-06-12","arxiv_id":"1606.03777","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/learning-multi-domain-convolutional-neural","slug":"learning-multi-domain-convolutional-neural","title":"Learning Multi-Domain Convolutional Neural Networks for Visual Tracking","date":"2015-10-27","arxiv_id":"1510.07945","rows_on_this_dataset":1,"code_links":2,"syntology":null}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/midas-multi-level-intent-domain-and-slot","slug":"midas-multi-level-intent-domain-and-slot","title":"MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU","date":"2024-08-15","arxiv_id":"2408.08144","rows_on_this_dataset":2,"code_links":1,"syntology":null},{"paper":"/paper/effective-sequence-to-sequence-dialogue-state","slug":"effective-sequence-to-sequence-dialogue-state","title":"Effective Sequence-to-Sequence Dialogue State Tracking","date":"2021-08-31","arxiv_id":"2108.13990","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/attention-stay-focus-1","slug":"attention-stay-focus-1","title":"Attention! Stay Focus!","date":"2021-04-16","arxiv_id":"2104.07925","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/a-sequence-to-sequence-approach-to-dialogue","slug":"a-sequence-to-sequence-approach-to-dialogue","title":"A Sequence-to-Sequence Approach to Dialogue State Tracking","date":"2020-11-18","arxiv_id":"2011.09553","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/towards-universal-dialogue-state-tracking","slug":"towards-universal-dialogue-state-tracking","title":"Towards Universal Dialogue State Tracking","date":"2018-10-22","arxiv_id":"1810.09587","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/global-locally-self-attentive-dialogue-state","slug":"global-locally-self-attentive-dialogue-state","title":"Global-Locally Self-Attentive Dialogue State Tracker","date":"2018-05-19","arxiv_id":"1805.09655","rows_on_this_dataset":1,"code_links":2,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":8,"samples_ran":7,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":7,"official":{"repos":["salesforce/glad"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/global-locally-self-attentive-dialogue-state#ran","syntology_url":"https://syntology.ai/paper/1805.09655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.09655"}}}},{"paper":"/paper/dialogue-learning-with-human-teaching-and","slug":"dialogue-learning-with-human-teaching-and","title":"Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems","date":"2018-04-18","arxiv_id":"1804.06512","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/neural-belief-tracker-data-driven-dialogue","slug":"neural-belief-tracker-data-driven-dialogue","title":"Neural Belief Tracker: Data-Driven Dialogue State Tracking","date":"2016-06-12","arxiv_id":"1606.03777","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/learning-multi-domain-convolutional-neural","slug":"learning-multi-domain-convolutional-neural","title":"Learning Multi-Domain Convolutional Neural Networks for Visual Tracking","date":"2015-10-27","arxiv_id":"1510.07945","rows_on_this_dataset":1,"code_links":2,"syntology":null}],"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/dataset/dialogue-state-tracking-challenge/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[{"order":"ran","page":1,"of":1,"url":"/dataset/dialogue-state-tracking-challenge/papers/ran/1","json":"/dataset/dialogue-state-tracking-challenge/papers/ran/1.json","rows":1}],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":1,"samples_harvested":8,"samples_ran":7,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":0,"samples_unverified":1,"pointer_only_for_licence":7,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":1,"papers_with_a_run_with_no_instrument_failure":1,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"45e2ac3d67d3cc82c644d9e4ab65b3ae8b8be1c158be0b6899895410a5d80f7f","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}