{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/composable-deep-reinforcement-learning-for","title":"Composable Deep Reinforcement Learning for Robotic Manipulation","arxiv_id":"1803.06773","date":"2018-03-19","proceeding":null,"authors":["Tuomas Haarnoja","Vitchyr Pong","Aurick Zhou","Murtaza Dalal","Pieter Abbeel","Sergey Levine"],"abstract":"Model-free deep reinforcement learning has been shown to exhibit good\nperformance in domains ranging from video games to simulated robotic\nmanipulation and locomotion. However, model-free methods are known to perform\npoorly when the interaction time with the environment is limited, as is the\ncase for most real-world robotic tasks. In this paper, we study how maximum\nentropy policies trained using soft Q-learning can be applied to real-world\nrobotic manipulation. The application of this method to real-world manipulation\nis facilitated by two important features of soft Q-learning. First, soft\nQ-learning can learn multimodal exploration strategies by learning policies\nrepresented by expressive energy-based models. Second, we show that policies\nlearned with soft Q-learning can be composed to create new policies, and that\nthe optimality of the resulting policy can be bounded in terms of the\ndivergence between the composed policies. This compositionality provides an\nespecially valuable tool for real-world manipulation, where constructing new\npolicies by composing existing skills can provide a large gain in efficiency\nover training from scratch. Our experimental evaluation demonstrates that soft\nQ-learning is substantially more sample efficient than prior model-free deep\nreinforcement learning methods, and that compositionality can be performed for\nboth simulated and real-world tasks.","url_abs":"http://arxiv.org/abs/1803.06773v1","url_pdf":"http://arxiv.org/pdf/1803.06773v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"composable-deep-reinforcement-learning-for","repo_url":"https://github.com/haarnoja/softqlearning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1803.06773","atlas_url":"https://app.syntology.ai/?focus=1803.06773","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1803.06773"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1803.06773\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1803.06773"},"call":"get_concepts_for_paper(arxiv_id=\"1803.06773\")"}],"by_repo":[{"repo":"haarnoja/softqlearning","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"5481f48b7fb91fbce3be6ce086e66390743daf5f61f068405a0853b59f0d75c2","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}