{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/3d-diffuser-actor-policy-diffusion-with-3d","title":"3D Diffuser Actor: Policy Diffusion with 3D Scene Representations","arxiv_id":null,"date":"2024-02-18","proceeding":null,"authors":["Tsung-Wei Ke*","Nikolaos Gkanatsios*","and Katerina Fragkiadaki"],"abstract":"We marry diffusion policies and 3D scene representations for robot manipulation. Diffusion policies learn the action distribution conditioned on the robot and environment state using conditional diffusion models. They have recently shown to outperform both deterministic and alternative state-conditioned action distribution learning methods. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy architecture that, given a language instruction, builds a 3D representation of the visual scene and conditions on it to iteratively denoise 3D rotations and translations for the robot's end-effector. At each denoising iteration, our model represents end-effector pose estimates as 3D scene tokens and predicts the 3D translation and rotation error for each of them, by featurizing them using 3D relative attention to other 3D visual and language tokens. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 16.3% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it outperforms the current SOTA in the setting of zero-shot unseen scene generalization by being able to successfully run 0.2 more tasks, a 7% relative increase. It also works in the real world from a handful of demonstrations. We ablate our model's architectural design choices, such as 3D scene featurization and 3D relative attentions, and show they all help generalization. Our results suggest that 3D scene representations and powerful generative modeling are keys to efficient robot learning from demonstrations.","url_abs":"https://arxiv.org/abs/2402.10885","url_pdf":"https://arxiv.org/pdf/2402.10885.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-diffuser-actor-policy-diffusion-with-3d","repo_url":"https://github.com/nickgkan/3d_diffuser_actor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"robot-manipulation-generalization","task_name":"Robot Manipulation Generalization"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/robot-manipulation-on-calvin","task":"Robot Manipulation","dataset":"CALVIN","model":"3DDA","rank_in_archive_order":12,"of":19,"metrics":{"avg. sequence length (D to D)":"3.35"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-on-calvin","task":"Robot Manipulation","dataset":"CALVIN","model":"3D Diffusor Actor","rank_in_archive_order":14,"of":19,"metrics":{"avg. sequence length (D to D)":"3.27"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-on-rlbench","task":"Robot Manipulation","dataset":"RLBench","model":"3D Diffuser Actor","rank_in_archive_order":6,"of":18,"metrics":{"Input Image Size":"256","Succ. Rate (18 tasks, 100 demo/task)":"81.3","Training Time (A100 x hour)":"936","Training Time (V100 x 8 x day)":"8"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-generalization-on-gembench","task":"Robot Manipulation Generalization","dataset":"GEMBench","model":"3D diffuser actor","rank_in_archive_order":3,"of":6,"metrics":{"Average Success Rate":"44.0","Average Success Rate (L1)":"91.9±0.8","Average Success Rate (L2)":"43.4±2.8","Average Success Rate (L3)":"37.0±2.2","Average Success Rate (L4)":"0.0±0.0"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-generalization-on-the","task":"Robot Manipulation Generalization","dataset":"The COLOSSEUM","model":"3D Diffuser Actor","rank_in_archive_order":4,"of":9,"metrics":{"Average decrease average across all perturbations":"-15.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-generalization-on-calvin","task":"Zero-shot Generalization","dataset":"CALVIN","model":"3D Diffuser Actor","rank_in_archive_order":4,"of":5,"metrics":{"Avg. sequence length":"3.27"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"nickgkan/3d_diffuser_actor","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2402.10885"],"harvested_for_calls":[{"arxiv_id":"2402.10885","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2402.10885"},"call":"get_harvested_code_for_paper(arxiv_id=\"2402.10885\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"abbe363b76d386972fd450e779e89f47d43eb2b0c67925a93762e3ec677040b1","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}