{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/mgpt-few-shot-learners-go-multilingual","title":"mGPT: Few-Shot Learners Go Multilingual","arxiv_id":"2204.07580","date":"2022-04-15","proceeding":null,"authors":["Oleh Shliazhko","Alena Fenogenova","Maria Tikhonova","Vladislav Mikhailov","Anastasia Kozlova","Tatiana Shavrina"],"abstract":"Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models. This paper introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus. We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism; Deepspeed and Megatron frameworks allow us to parallelize the training and inference steps effectively. The resulting models show performance on par with the recently released XGLM models by Facebook, covering more languages and enhancing NLP possibilities for low resource languages of CIS countries and Russian small nations. We detail the motivation for the choices of the architecture design, thoroughly describe the data preparation pipeline, and train five small versions of the model to choose the most optimal multilingual tokenization strategy. We measure the model perplexity in all covered languages and evaluate it on the wide spectre of multilingual tasks, including classification, generative, sequence labeling and knowledge probing. The models were evaluated with the zero-shot and few-shot methods. Furthermore, we compared the classification tasks with the state-of-the-art multilingual model XGLM. source code and the mGPT XL model are publicly released.","url_abs":"https://arxiv.org/abs/2204.07580v2","url_pdf":"https://arxiv.org/pdf/2204.07580v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mgpt-few-shot-learners-go-multilingual","repo_url":"https://github.com/ai-forever/mgpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"cross-lingual-natural-language-inference","task_name":"Cross-Lingual Natural Language Inference"},{"task_slug":"cross-lingual-paraphrase-identification","task_name":"Cross-Lingual Paraphrase Identification"},{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"few-shot-ner","task_name":"Few-shot NER"},{"task_slug":"knowledge-probing","task_name":"Knowledge Probing"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"part-of-speech-tagging","task_name":"Part-Of-Speech Tagging"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-lingual-natural-language-inference-on-4","task":"Cross-Lingual Natural Language Inference","dataset":"XNLI","model":"mGPT","rank_in_archive_order":5,"of":5,"metrics":{"Accuracy":"40.6"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-transfer-on-xcopa","task":"Cross-Lingual Transfer","dataset":"XCOPA","model":"mGPT","rank_in_archive_order":6,"of":6,"metrics":{"Accuracy":"55.5"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-ner-on-xglue","task":"Few-shot NER","dataset":"XGLUE","model":"mGPT","rank_in_archive_order":1,"of":1,"metrics":{"Avg F1":"0.85"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-xwino","task":"Natural Language Inference","dataset":"XWINO","model":"mGPT","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"0.562"},"uses_additional_data":false},{"leaderboard":"/sota/part-of-speech-tagging-on-xglue","task":"Part-Of-Speech Tagging","dataset":"XGLUE","model":"mGPT","rank_in_archive_order":1,"of":1,"metrics":{"Avg. F1":"0.56"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.07580","atlas_url":"https://app.syntology.ai/?focus=2204.07580","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2204.07580"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2204.07580\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2204.07580"},"call":"get_concepts_for_paper(arxiv_id=\"2204.07580\")"}],"by_repo":[{"repo":"ai-forever/mgpt","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"6e7e9ac578153183cc5b8ec1bf22882ac215e655d41f9d7c1979cb4fa0cfd1d2","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}