{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/online-batch-selection-for-faster-training-of","title":"Online Batch Selection for Faster Training of Neural Networks","arxiv_id":"1511.06343","date":"2015-11-19","proceeding":null,"authors":["Ilya Loshchilov","Frank Hutter"],"abstract":"Deep neural networks are commonly trained using stochastic non-convex\noptimization procedures, which are driven by gradient information estimated on\nfractions (batches) of the dataset. While it is commonly accepted that batch\nsize is an important parameter for offline tuning, the benefits of online\nselection of batches remain poorly understood. We investigate online batch\nselection strategies for two state-of-the-art methods of stochastic\ngradient-based optimization, AdaDelta and Adam. As the loss function to be\nminimized for the whole dataset is an aggregation of loss functions of\nindividual datapoints, intuitively, datapoints with the greatest loss should be\nconsidered (selected in a batch) more frequently. However, the limitations of\nthis intuition and the proper control of the selection pressure over time are\nopen questions. We propose a simple strategy where all datapoints are ranked\nw.r.t. their latest known loss value and the probability to be selected decays\nexponentially as a function of rank. Our experimental results on the MNIST\ndataset suggest that selecting batches speeds up both AdaDelta and Adam by a\nfactor of about 5.","url_abs":"http://arxiv.org/abs/1511.06343v4","url_pdf":"http://arxiv.org/pdf/1511.06343v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-batch-selection-for-faster-training-of","repo_url":"https://github.com/Lasagne/Lasagne","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"adadelta","method_name":"AdaDelta"},{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1511.06343","atlas_url":"https://app.syntology.ai/?focus=1511.06343","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1511.06343"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1511.06343\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1511.06343"},"call":"get_concepts_for_paper(arxiv_id=\"1511.06343\")"}],"by_repo":[{"repo":"Lasagne/Lasagne","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"02ca4d6602b003444f35d6ad4372d042a61aec32374b3380e283f39568dfa418","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}