{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/optimus-an-efficient-dynamic-resource","title":"Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters","arxiv_id":null,"date":"2018-04-26","proceeding":"EuroSys 2018 4","authors":["Yanghua Peng"],"abstract":"Deep learning workloads are common in today’s production clusters due to the proliferation of deep learning-driven AI services\r\n(e.g., speech recognition, machine translation). A deep learning\r\ntraining job is resource-intensive and time-consuming. Efficient\r\nresource scheduling is the key to the maximal performance of a\r\ndeep learning cluster. Existing cluster schedulers are largely not tailored to deep learning jobs, and typically specifying a fixed amount\r\nof resources for each job, prohibiting high resource efficiency and\r\njob performance. This paper proposes Optimus, a customized job\r\nscheduler for deep learning clusters, which minimizes job training\r\ntime based on online resource-performance models. Optimus uses\r\nonline fitting to predict model convergence during training and\r\nsets up performance models to accurately estimate training speed\r\nas a function of allocated resources in each job. Based on the models,\r\na simple yet effective method is designed and used for dynamically\r\nallocating resources and placing deep learning tasks to minimize\r\njob completion time. We implement Optimus on top of Kubernetes,\r\na cluster manager for container orchestration, and experiment on a\r\ndeep learning cluster with 7 CPU servers and 6 GPU servers, run\u0002ning 9 training jobs using the MXNet framework. Results show that\r\nOptimus outperforms representative cluster schedulers by about\r\n139% and 63% in terms of job completion time and makespan, respectively","url_abs":"https://dl.acm.org/doi/abs/10.1145/3190508.3190517","url_pdf":"https://dl.acm.org/doi/abs/10.1145/3190508.3190517","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimus-an-efficient-dynamic-resource","repo_url":"https://github.com/pengyanghua/optimus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mxnet","reach":null}],"tasks":[{"task_slug":null,"task_name":"CPU"},{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"scheduling","task_name":"Scheduling"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"pengyanghua/optimus","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4648692303c18de91cb8525fc7b27e91d076dcd04b37dbba246072aa1ef43531","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}