{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/pangu-a-large-scale-autoregressive-pretrained","title":"PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation","arxiv_id":"2104.12369","date":"2021-04-26","proceeding":null,"authors":["Wei Zeng","Xiaozhe Ren","Teng Su","Hui Wang","Yi Liao","Zhiwei Wang","Xin Jiang","ZhenZhang Yang","Kaisheng Wang","Xiaoda Zhang","Chen Li","Ziyan Gong","Yifan Yao","Xinjing Huang","Jun Wang","Jianfeng Yu","Qi Guo","Yue Yu","Yan Zhang","Jin Wang","Hengtao Tao","Dasen Yan","Zexuan Yi","Fang Peng","Fangqing Jiang","Han Zhang","Lingfeng Deng","Yehong Zhang","Zhe Lin","Chao Zhang","Shaojie Zhang","Mingyue Guo","Shanzhi Gu","Gaojun Fan","YaoWei Wang","Xuefeng Jin","Qun Liu","Yonghong Tian"],"abstract":"Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \\textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\\alpha$, with up to 200 billion parameters. PanGu-$\\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$\\alpha$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$\\alpha$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$\\alpha$ in performing various tasks under few-shot or zero-shot settings.","url_abs":"https://arxiv.org/abs/2104.12369v1","url_pdf":"https://arxiv.org/pdf/2104.12369v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pangu-a-large-scale-autoregressive-pretrained","repo_url":"https://github.com/sunzeyeah/rlhf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pangu-a-large-scale-autoregressive-pretrained","repo_url":"https://github.com/2023-MindSpore-1/ms-code-14/tree/main/Pangu_alpha","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"pangu-a-large-scale-autoregressive-pretrained","repo_url":"https://github.com/2023-MindSpore-1/ms-code-162","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"pangu-a-large-scale-autoregressive-pretrained","repo_url":"https://github.com/JoegameZhou/mPanGu-Alpha-53","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"pangu-a-large-scale-autoregressive-pretrained","repo_url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/pangu_alpha","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"cloze-multi-choices-few-shot","task_name":"Cloze (multi-choices) (Few-Shot)"},{"task_slug":"cloze-multi-choices-one-shot","task_name":"Cloze (multi-choices) (One-Shot)"},{"task_slug":"cloze-multi-choices-zero-shot","task_name":"Cloze (multi-choices) (Zero-Shot)"},{"task_slug":"common-sense-reasoning-few-shot","task_name":"Common Sense Reasoning (Few-Shot)"},{"task_slug":"common-sense-reasoning-one-shot","task_name":"Common Sense Reasoning (One-Shot)"},{"task_slug":"common-sense-reasoning-zero-shot","task_name":"Common Sense Reasoning (Zero-Shot)"},{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"few-shot-image-classification","task_name":"Few-Shot Image Classification"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-inference-few-shot","task_name":"Natural Language Inference (Few-Shot)"},{"task_slug":"natural-language-inference-one-shot","task_name":"Natural Language Inference (One-Shot)"},{"task_slug":"natural-language-inference-zero-shot","task_name":"Natural Language Inference (Zero-Shot)"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"reading-comprehension-few-shot","task_name":"Reading Comprehension (Few-Shot)"},{"task_slug":"reading-comprehension-one-shot","task_name":"Reading Comprehension (One-Shot)"},{"task_slug":"reading-comprehension-zero-shot","task_name":"Reading Comprehension (Zero-Shot)"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"pangu-a","method_name":"PanGu-$α$"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[{"slug":"pangu-a","name":"PanGu-$α$","full_name":"PanGu-$α$"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.12369","atlas_url":"https://app.syntology.ai/?focus=2104.12369","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2104.12369"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2104.12369\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2104.12369"},"call":"get_concepts_for_paper(arxiv_id=\"2104.12369\")"}],"by_repo":[{"repo":"sunzeyeah/rlhf","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2305.18290"],"harvested_for_calls":[{"arxiv_id":"2305.18290","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2305.18290"},"call":"get_harvested_code_for_paper(arxiv_id=\"2305.18290\")"}]},{"repo":"2023-MindSpore-1/ms-code-14","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"2023-MindSpore-1/ms-code-162","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"JoegameZhou/mPanGu-Alpha-53","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"mindspore-ai/models","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2208.08084"],"harvested_for_calls":[{"arxiv_id":"2208.08084","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2208.08084"},"call":"get_harvested_code_for_paper(arxiv_id=\"2208.08084\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"8d152451d273444a89cc8e5f37248ad36448ca04827439fca70b17098364ac01","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}