{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/enigma","slug":"enigma","name":"ENIGMA","full_name":"ENIGMA","full_name_withheld":false,"description_markdown":"**ENIGMA** is an evaluation framework for dialog systems based on Pearson and Spearman's rank correlations between the estimated rewards and the true rewards.  ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data (see details in Section 2), which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach","paper":"/paper/towards-automatic-evaluation-of-dialog","first_author":"Haoming Jiang","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/towards-automatic-evaluation-of-dialog"},"source":{"url":"https://arxiv.org/abs/2102.10242v3","title":"Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Dialog System Evaluation","url":"/methods/category/dialog-system-evaluation","pwc_aliases":[]}],"n_papers_tagged":7,"archive_num_papers":7,"papers_newest_first":[{"paper":null,"slug":"mizar-60-for-mizar-50","title":"MizAR 60 for Mizar 50","date":"2023-03-12","arxiv_id":"2303.06686","n_code_links":0,"syntology":null},{"paper":null,"slug":"global-multi-site-benchmark-classification-of","title":"Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures","date":"2022-06-16","arxiv_id":"2206.08122","n_code_links":0,"syntology":null},{"paper":"/paper/the-isabelle-enigma","slug":"the-isabelle-enigma","title":"The Isabelle ENIGMA","date":"2022-05-04","arxiv_id":"2205.01981","n_code_links":1,"syntology":null},{"paper":"/paper/learning-theorem-proving-components","slug":"learning-theorem-proving-components","title":"Learning Theorem Proving Components","date":"2021-07-21","arxiv_id":"2107.10034","n_code_links":1,"syntology":null},{"paper":"/paper/fast-and-slow-enigmas-and-parental-guidance","slug":"fast-and-slow-enigmas-and-parental-guidance","title":"Fast and Slow Enigmas and Parental Guidance","date":"2021-07-14","arxiv_id":"2107.06750","n_code_links":1,"syntology":null},{"paper":null,"slug":"new-techniques-that-improve-enigma-style","title":"Improving ENIGMA-Style Clause Selection While Learning From History","date":"2021-02-26","arxiv_id":"2102.13564","n_code_links":0,"syntology":null},{"paper":"/paper/towards-automatic-evaluation-of-dialog","slug":"towards-automatic-evaluation-of-dialog","title":"Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach","date":"2021-02-20","arxiv_id":"2102.10242","n_code_links":1,"syntology":null}],"papers_shown":7,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":7,"with_a_code_link":4,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":7,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[],"tasks":[{"task":"/task/automated-theorem-proving","name":"Automated Theorem Proving","papers":2},{"task":"/task/classification-1","name":"Classification","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/graph-neural-network","name":"Graph Neural Network","papers":1},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":1},{"task":"/task/off-policy-evaluation","name":"Off-policy evaluation","papers":1},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":1},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":1}],"tasks_shown":11,"n_tasks":11,"usage_by_year":[{"year":"2021","papers":4},{"year":"2022","papers":2},{"year":"2023","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/enigma"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=ENIGMA","read_at":"2026-10-05T00:55:38+00:00","query":"ENIGMA","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"ENIGMA","origin_arxiv_id":"2102.10242"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"same_paper","archive_paper_arxiv_id":"2102.10242","repositories":{"listed":2,"in_graph":2,"list_truncated":false,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"outranked","may_be_wrong_sense":true,"bearers_found":6,"served":{"arxiv_id":"2102.10242","citations":20},"leading_other_bearer":{"arxiv_id":"1701.06532","title":"ENIGMA: Efficient Learning-based Inference Guiding Machine","citations":101,"named_by":["title_prefix"],"bearer_name":"ENIGMA: Efficient Learning-based Inference Guiding Machine","ask_with":"have('1701.06532')"},"n_other_bearers_listed":5,"note":"'ENIGMA' is ALSO the name of the paper 'ENIGMA: Efficient Learning-based Inference Guiding Machine' (arXiv:1701.06532, 101 citations), which is cited FAR more than the paper this answer is about (20). This answer may be about the wrong ENIGMA. If you meant the better-known one, ask again with that entry of `also_named`."},"name_competition_text":"The same name also belongs to the paper “ENIGMA: Efficient Learning-based Inference Guiding Machine” (arXiv:1701.06532, 101 citations), which the lookup finds cited far more than the paper this answer is about (20 citations), so this answer may be about a different sense of the name.","near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"2102.10242","page":"/paper/towards-automatic-evaluation-of-dialog","syntology_url":"https://syntology.ai/paper/2102.10242","in_graph_layer":false,"samples":0},"line":"repository known, no verified implementation (declared_code; repositories: 2 listed of 2 in the graph, strongest link: official). This answer is about arXiv:2102.10242, the paper the archive credits with this method.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"ca44576e568d957f26672dda18a228a14960770bd54ebe65c525579488ebd7fa","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}