{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/nebula-i-a-general-framework-for","title":"Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters","arxiv_id":"2205.09470","date":"2022-05-19","proceeding":null,"authors":["Yang Xiang","Zhihua Wu","Weibao Gong","Siyu Ding","Xianjie Mo","Yuang Liu","Shuohuan Wang","Peng Liu","Yongshuai Hou","Long Li","Bin Wang","Shaohuai Shi","Yaqian Han","Yue Yu","Ge Li","Yu Sun","Yanjun Ma","dianhai yu"],"abstract":"The ever-growing model size and scale of compute have attracted increasing interests in training deep learning models over multiple nodes. However, when it comes to training on cloud clusters, especially across remote clusters, huge challenges are faced. In this work, we introduce a general framework, Nebula-I, for collaboratively training deep learning models over remote heterogeneous clusters, the connections between which are low-bandwidth wide area networks (WANs). We took natural language processing (NLP) as an example to show how Nebula-I works in different training phases that include: a) pre-training a multilingual language model using two remote clusters; and b) fine-tuning a machine translation model using knowledge distilled from pre-trained models, which run through the most popular paradigm of recent deep learning. To balance the accuracy and communication efficiency, in Nebula-I, parameter-efficient training strategies, hybrid parallel computing methods and adaptive communication acceleration techniques are jointly applied. Meanwhile, security strategies are employed to guarantee the safety, reliability and privacy in intra-cluster computation and inter-cluster communication. Nebula-I is implemented with the PaddlePaddle deep learning framework, which can support collaborative training over heterogeneous hardware, e.g. GPU and NPU. Experiments demonstrate that the proposed framework could substantially maximize the training efficiency while preserving satisfactory NLP performance. By using Nebula-I, users can run large-scale training tasks over cloud clusters with minimum developments, and the utility of existed large pre-trained models could be further promoted. We also introduced new state-of-the-art results on cross-lingual natural language inference tasks, which are generated based upon a novel learning framework and Nebula-I.","url_abs":"https://arxiv.org/abs/2205.09470v1","url_pdf":"https://arxiv.org/pdf/2205.09470v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nebula-i-a-general-framework-for","repo_url":"https://github.com/PaddlePaddle/Paddle","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"cross-lingual-natural-language-inference","task_name":"Cross-Lingual Natural Language Inference"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"distributed-computing","task_name":"Distributed Computing"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.09470","atlas_url":"https://app.syntology.ai/?focus=2205.09470","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2205.09470"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2205.09470\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2205.09470"},"call":"get_concepts_for_paper(arxiv_id=\"2205.09470\")"}],"by_repo":[{"repo":"PaddlePaddle/Paddle","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2410.01359"],"harvested_for_calls":[{"arxiv_id":"2410.01359","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2410.01359"},"call":"get_harvested_code_for_paper(arxiv_id=\"2410.01359\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"8a675e28b9a3ba502766382d041d2aadf95c972c11a2372fa9627c28d1ddafd4","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}