{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/mesh-tensorflow-deep-learning-for","title":"Mesh-TensorFlow: Deep Learning for Supercomputers","arxiv_id":"1811.02084","date":"2018-11-05","proceeding":"NeurIPS 2018 12","authors":["Noam Shazeer","Youlong Cheng","Niki Parmar","Dustin Tran","Ashish Vaswani","Penporn Koanantakool","Peter Hawkins","HyoukJoong Lee","Mingsheng Hong","Cliff Young","Ryan Sepassi","Blake Hechtman"],"abstract":"Batch-splitting (data-parallelism) is the dominant distributed Deep Neural\nNetwork (DNN) training strategy, due to its universal applicability and its\namenability to Single-Program-Multiple-Data (SPMD) programming. However,\nbatch-splitting suffers from problems including the inability to train very\nlarge models (due to memory constraints), high latency, and inefficiency at\nsmall batch sizes. All of these can be solved by more general distribution\nstrategies (model-parallelism). Unfortunately, efficient model-parallel\nalgorithms tend to be complicated to discover, describe, and to implement,\nparticularly on large clusters. We introduce Mesh-TensorFlow, a language for\nspecifying a general class of distributed tensor computations. Where\ndata-parallelism can be viewed as splitting tensors and operations along the\n\"batch\" dimension, in Mesh-TensorFlow, the user can specify any\ntensor-dimensions to be split across any dimensions of a multi-dimensional mesh\nof processors. A Mesh-TensorFlow graph compiles into a SPMD program consisting\nof parallel operations coupled with collective communication primitives such as\nAllreduce. We use Mesh-TensorFlow to implement an efficient data-parallel,\nmodel-parallel version of the Transformer sequence-to-sequence model. Using TPU\nmeshes of up to 512 cores, we train Transformer models with up to 5 billion\nparameters, surpassing state of the art results on WMT'14 English-to-French\ntranslation task and the one-billion-word language modeling benchmark.\nMesh-Tensorflow is available at https://github.com/tensorflow/mesh .","url_abs":"http://arxiv.org/abs/1811.02084v1","url_pdf":"http://arxiv.org/pdf/1811.02084v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mesh-tensorflow-deep-learning-for","repo_url":"https://github.com/tensorflow/mesh","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mesh-tensorflow","method_name":"Mesh-TensorFlow"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"mesh-tensorflow","name":"Mesh-TensorFlow","full_name":"Mesh-TensorFlow"}],"results":[{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"Mesh Tensorflow","rank_in_archive_order":12,"of":27,"metrics":{"Number of params":"4.9B","PPL":" 24.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1811.02084","atlas_url":"https://app.syntology.ai/?focus=1811.02084","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1811.02084"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1811.02084\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1811.02084"},"call":"get_concepts_for_paper(arxiv_id=\"1811.02084\")"}],"by_repo":[{"repo":"tensorflow/mesh","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2101.03961"],"harvested_for_calls":[{"arxiv_id":"2101.03961","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2101.03961"},"call":"get_harvested_code_for_paper(arxiv_id=\"2101.03961\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"89ed07db225da5237182125b14fb5efae8fcb434c2ef44d3254e060dbdafdab4","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}