{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/fast-large-scale-optimization-by-unifying","title":"Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods","arxiv_id":"1311.2115","date":"2013-11-09","proceeding":null,"authors":["Jascha Sohl-Dickstein","Ben Poole","Surya Ganguli"],"abstract":"We present an algorithm for minimizing a sum of functions that combines the\ncomputational efficiency of stochastic gradient descent (SGD) with the second\norder curvature information leveraged by quasi-Newton methods. We unify these\ndisparate approaches by maintaining an independent Hessian approximation for\neach contributing function in the sum. We maintain computational tractability\nand limit memory requirements even for high dimensional optimization problems\nby storing and manipulating these quadratic approximations in a shared, time\nevolving, low dimensional subspace. Each update step requires only a single\ncontributing function or minibatch evaluation (as in SGD), and each step is\nscaled using an approximate inverse Hessian and little to no adjustment of\nhyperparameters is required (as is typical for quasi-Newton methods). This\nalgorithm contrasts with earlier stochastic second order techniques that treat\nthe Hessian of each contributing function as a noisy approximation to the full\nHessian, rather than as a target for direct estimation. We experimentally\ndemonstrate improved convergence on seven diverse optimization problems. The\nalgorithm is released as open source Python and MATLAB packages.","url_abs":"http://arxiv.org/abs/1311.2115v7","url_pdf":"http://arxiv.org/pdf/1311.2115v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-large-scale-optimization-by-unifying","repo_url":"https://github.com/Sohl-Dickstein/Sum-of-Functions-Optimizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1311.2115","atlas_url":"https://app.syntology.ai/?focus=1311.2115","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1311.2115"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1311.2115\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1311.2115"},"call":"get_concepts_for_paper(arxiv_id=\"1311.2115\")"}],"by_repo":[{"repo":"Sohl-Dickstein/Sum-of-Functions-Optimizer","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0aee5b88975bf4ba530cd4b3a05a8bbf39ba4ff19cb20e5f6efb05bb579be9a4","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}