{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/rucola-russian-corpus-of-linguistic","title":"RuCoLA: Russian Corpus of Linguistic Acceptability","arxiv_id":"2210.12814","date":"2022-10-23","proceeding":null,"authors":["Vladislav Mikhailov","Tatiana Shamardina","Max Ryabinin","Alena Pestova","Ivan Smurov","Ekaterina Artemova"],"abstract":"Linguistic acceptability (LA) attracts the attention of the research community due to its many uses, such as testing the grammatical knowledge of language models and filtering implausible texts with acceptability classifiers. However, the application scope of LA in languages other than English is limited due to the lack of high-quality resources. To this end, we introduce the Russian Corpus of Linguistic Acceptability (RuCoLA), built from the ground up under the well-established binary LA approach. RuCoLA consists of $9.8$k in-domain sentences from linguistic publications and $3.6$k out-of-domain sentences produced by generative models. The out-of-domain set is created to facilitate the practical use of acceptability for improving language generation. Our paper describes the data collection protocol and presents a fine-grained analysis of acceptability classification experiments with a range of baseline approaches. In particular, we demonstrate that the most widely used language models still fall behind humans by a large margin, especially when detecting morphological and semantic errors. We release RuCoLA, the code of experiments, and a public leaderboard (rucola-benchmark.com) to assess the linguistic competence of language models for Russian.","url_abs":"https://arxiv.org/abs/2210.12814v1","url_pdf":"https://arxiv.org/pdf/2210.12814v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rucola-russian-corpus-of-linguistic","repo_url":"https://github.com/russiannlp/rucola","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[{"slug":"rucola","name":"RuCoLA","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/linguistic-acceptability-on-cola","task":"Linguistic Acceptability","dataset":"CoLA","model":"RemBERT","rank_in_archive_order":43,"of":43,"metrics":{"MCC":"0.6"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-itacola","task":"Linguistic Acceptability","dataset":"ItaCoLA","model":"XLM-R","rank_in_archive_order":2,"of":4,"metrics":{"MCC":"0.52"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-itacola","task":"Linguistic Acceptability","dataset":"ItaCoLA","model":"mBERT","rank_in_archive_order":4,"of":4,"metrics":{"MCC":"0.36"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"ruRoBERTa","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"79.34","MCC":"0.53"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"RemBERT","rank_in_archive_order":4,"of":9,"metrics":{"Accuracy":"75.06","MCC":"0.44"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"ruBERT","rank_in_archive_order":5,"of":9,"metrics":{"Accuracy":"74.3","MCC":"0.42"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"ruGPT-3","rank_in_archive_order":6,"of":9,"metrics":{"Accuracy":"53.82","MCC":"0.30"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"ruT5","rank_in_archive_order":7,"of":9,"metrics":{"Accuracy":"68.41","MCC":"0.25"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"mBERT","rank_in_archive_order":8,"of":9,"metrics":{"MCC":"0.15"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-rucola","task":"Linguistic Acceptability","dataset":"RuCoLA","model":"XLM-R","rank_in_archive_order":9,"of":9,"metrics":{"Accuracy":"61.13","MCC":"0.13"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2210.12814","atlas_url":"https://app.syntology.ai/?focus=2210.12814","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2210.12814"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2210.12814\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2210.12814"},"call":"get_concepts_for_paper(arxiv_id=\"2210.12814\")"}],"by_repo":[{"repo":"russiannlp/rucola","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"c1379e957dbc8e7413f12238aa7482713cca6f26793f1213d2e81990d737e859","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}