{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/sparknet-training-deep-networks-in-spark","title":"SparkNet: Training Deep Networks in Spark","arxiv_id":"1511.06051","date":"2015-11-19","proceeding":null,"authors":["Philipp Moritz","Robert Nishihara","Ion Stoica","Michael. I. Jordan"],"abstract":"Training deep networks is a time-consuming process, with networks for object\nrecognition often requiring multiple days to train. For this reason, leveraging\nthe resources of a cluster to speed up training is an important area of work.\nHowever, widely-popular batch-processing computational frameworks like\nMapReduce and Spark were not designed to support the asynchronous and\ncommunication-intensive workloads of existing distributed deep learning\nsystems. We introduce SparkNet, a framework for training deep networks in\nSpark. Our implementation includes a convenient interface for reading data from\nSpark RDDs, a Scala interface to the Caffe deep learning framework, and a\nlightweight multi-dimensional tensor library. Using a simple parallelization\nscheme for stochastic gradient descent, SparkNet scales well with the cluster\nsize and tolerates very high-latency communication. Furthermore, it is easy to\ndeploy and use with no parameter tuning, and it is compatible with existing\nCaffe models. We quantify the dependence of the speedup obtained by SparkNet on\nthe number of machines, the communication frequency, and the cluster's\ncommunication overhead, and we benchmark our system's performance on the\nImageNet dataset.","url_abs":"http://arxiv.org/abs/1511.06051v4","url_pdf":"http://arxiv.org/pdf/1511.06051v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparknet-training-deep-networks-in-spark","repo_url":"https://github.com/amplab/SparkNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1511.06051","atlas_url":"https://app.syntology.ai/?focus=1511.06051","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1511.06051"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1511.06051\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1511.06051"},"call":"get_concepts_for_paper(arxiv_id=\"1511.06051\")"}],"by_repo":[{"repo":"amplab/SparkNet","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"d1cab05a7896a5782e745deceaa4634f11bc22e1583afcaeac655794a4310b0a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}