{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/farstail-a-persian-natural-language-inference","title":"FarsTail: A Persian Natural Language Inference Dataset","arxiv_id":"2009.08820","date":"2020-09-18","proceeding":null,"authors":["Hossein Amirkhani","Mohammad AzariJafari","Zohreh Pourjafari","Soroush Faridan-Jahromi","Zeinab Kouhkan","Azadeh Amirak"],"abstract":"Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been devoted to develop more diverse datasets for different languages. In this paper, we present a new dataset for the NLI task in the Persian language, also known as Farsi, which is one of the dominant languages in the Middle East. This dataset, named FarsTail, includes 10,367 samples which are provided in both the Persian language as well as the indexed format to be useful for non-Persian researchers. The samples are generated from 3,539 multiple-choice questions with the least amount of annotator interventions in a way similar to the SciTail dataset. A carefully designed multi-step process is adopted to ensure the quality of the dataset. We also present the results of traditional and state-of-the-art methods on FarsTail including different embedding methods such as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid baseline for the future research. The best obtained test accuracy is 83.38% which shows that there is a big room for improving the current methods to be useful for real-world NLP applications in different languages. We also investigate the extent to which the models exploit superficial clues, also known as dataset biases, in FarsTail, and partition the test set into easy and hard subsets according to the success of biased models. The dataset is available at https://github.com/dml-qom/FarsTail.","url_abs":"https://arxiv.org/abs/2009.08820v2","url_pdf":"https://arxiv.org/pdf/2009.08820v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"farstail-a-persian-natural-language-inference","repo_url":"https://github.com/dml-qom/FarsTail","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"}],"methods":[{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bigru","method_name":"BiGRU"},{"method_slug":"cbow-word2vec","method_name":"CBoW Word2Vec"},{"method_slug":"elmo","method_name":"ELMo"},{"method_slug":"esim","method_name":"ESIM"},{"method_slug":"hbmp","method_name":"HBMP"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"skip-gram-word2vec","method_name":"Skip-gram Word2Vec"},{"method_slug":"ulmfit","method_name":"ULMFiT"},{"method_slug":"fasttext","method_name":"fastText"}],"datasets_introduced":[{"slug":"farstail","name":"FarsTail","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"mBERT","rank_in_archive_order":1,"of":10,"metrics":{"% Test Accuracy":"83.38"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"ParsBERT","rank_in_archive_order":2,"of":10,"metrics":{"% Test Accuracy":"82.99"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"Translate-Source + fastText","rank_in_archive_order":3,"of":10,"metrics":{"% Test Accuracy":"78.13"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"LSTM + BERT (concat)","rank_in_archive_order":4,"of":10,"metrics":{"% Test Accuracy":"75.83"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"ESIM + BERT (FarsTail, MultiNLI)","rank_in_archive_order":5,"of":10,"metrics":{"% Test Accuracy":"74.62"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"ULMFiT","rank_in_archive_order":6,"of":10,"metrics":{"% Test Accuracy":"72.44"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"ESIM + fastText","rank_in_archive_order":7,"of":10,"metrics":{"% Test Accuracy":"71.16"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"Translate-Target + fastText","rank_in_archive_order":8,"of":10,"metrics":{"% Test Accuracy":"70.46"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"Decomposable Attention Model + word2vec","rank_in_archive_order":9,"of":10,"metrics":{"% Test Accuracy":"66.62"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-farstail","task":"Natural Language Inference","dataset":"FarsTail","model":"HBMP + word2vec","rank_in_archive_order":10,"of":10,"metrics":{"% Test Accuracy":"66.04"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2009.08820","atlas_url":"https://app.syntology.ai/?focus=2009.08820","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2009.08820"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2009.08820\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2009.08820"},"call":"get_concepts_for_paper(arxiv_id=\"2009.08820\")"}],"by_repo":[{"repo":"dml-qom/FarsTail","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"fb370b1b5a86a360496931e7cb863703a2d0e9fc0e3226c5b2888164104d7e83","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}