{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/the-reactor-a-fast-and-sample-efficient-actor","title":"The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning","arxiv_id":"1704.04651","date":"2017-04-15","proceeding":"ICLR 2018 1","authors":["Audrunas Gruslys","Will Dabney","Mohammad Gheshlaghi Azar","Bilal Piot","Marc Bellemare","Remi Munos"],"abstract":"In this work we present a new agent architecture, called Reactor, which\ncombines multiple algorithmic and architectural contributions to produce an\nagent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al.,\n2016) and Categorical DQN (Bellemare et al., 2017), while giving better\nrun-time performance than A3C (Mnih et al., 2016). Our first contribution is a\nnew policy evaluation algorithm called Distributional Retrace, which brings\nmulti-step off-policy updates to the distributional reinforcement learning\nsetting. The same approach can be used to convert several classes of multi-step\npolicy evaluation algorithms designed for expected value evaluation into\ndistributional ones. Next, we introduce the \\b{eta}-leave-one-out policy\ngradient algorithm which improves the trade-off between variance and bias by\nusing action values as a baseline. Our final algorithmic contribution is a new\nprioritized replay algorithm for sequences, which exploits the temporal\nlocality of neighboring observations for more efficient replay prioritization.\nUsing the Atari 2600 benchmarks, we show that each of these innovations\ncontribute to both the sample efficiency and final agent performance. Finally,\nwe demonstrate that Reactor reaches state-of-the-art performance after 200\nmillion frames and less than a day of training.","url_abs":"http://arxiv.org/abs/1704.04651v2","url_pdf":"http://arxiv.org/pdf/1704.04651v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"distributional-reinforcement-learning","task_name":"Distributional Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"a3c","method_name":"A3C"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dqn","method_name":"DQN"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"retrace","method_name":"Retrace"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-2600-alien","task":"Atari Games","dataset":"Atari 2600 Alien","model":"Reactor 500M","rank_in_archive_order":11,"of":49,"metrics":{"Score":"12689.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-amidar","task":"Atari Games","dataset":"Atari 2600 Amidar","model":"Reactor 500M","rank_in_archive_order":25,"of":48,"metrics":{"Score":"1015.8"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-assault","task":"Atari Games","dataset":"Atari 2600 Assault","model":"Reactor 500M","rank_in_archive_order":20,"of":45,"metrics":{"Score":"8323.3"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-asterix","task":"Atari Games","dataset":"Atari 2600 Asterix","model":"Reactor 500M","rank_in_archive_order":19,"of":49,"metrics":{"Score":"205914.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-asteroids","task":"Atari Games","dataset":"Atari 2600 Asteroids","model":"Reactor 500M","rank_in_archive_order":18,"of":43,"metrics":{"Score":"3726.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-atlantis","task":"Atari Games","dataset":"Atari 2600 Atlantis","model":"Reactor 500M","rank_in_archive_order":34,"of":42,"metrics":{"Score":"302831.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-bank-heist","task":"Atari Games","dataset":"Atari 2600 Bank Heist","model":"Reactor 500M","rank_in_archive_order":14,"of":45,"metrics":{"Score":"1259.7"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-battle-zone","task":"Atari Games","dataset":"Atari 2600 Battle Zone","model":"Reactor 500M","rank_in_archive_order":11,"of":47,"metrics":{"Score":"64070.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-beam-rider","task":"Atari Games","dataset":"Atari 2600 Beam Rider","model":"Reactor 500M","rank_in_archive_order":31,"of":49,"metrics":{"Score":"11033.4"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-berzerk","task":"Atari Games","dataset":"Atari 2600 Berzerk","model":"Reactor 500M","rank_in_archive_order":15,"of":39,"metrics":{"Score":"2303.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-bowling","task":"Atari Games","dataset":"Atari 2600 Bowling","model":"Reactor 500M","rank_in_archive_order":16,"of":44,"metrics":{"Score":"81.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-boxing","task":"Atari Games","dataset":"Atari 2600 Boxing","model":"Reactor 500M","rank_in_archive_order":17,"of":45,"metrics":{"Score":"99.4"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-breakout","task":"Atari Games","dataset":"Atari 2600 Breakout","model":"Reactor 500M","rank_in_archive_order":20,"of":58,"metrics":{"Score":"514.8"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-centipede","task":"Atari Games","dataset":"Atari 2600 Centipede","model":"Reactor 500M","rank_in_archive_order":42,"of":45,"metrics":{"Score":"3422.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-chopper-command","task":"Atari Games","dataset":"Atari 2600 Chopper Command","model":"Reactor 500M","rank_in_archive_order":9,"of":45,"metrics":{"Score":"107779.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-crazy-climber","task":"Atari Games","dataset":"Atari 2600 Crazy Climber","model":"Reactor 500M","rank_in_archive_order":6,"of":49,"metrics":{"Score":"236422.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-defender","task":"Atari Games","dataset":"Atari 2600 Defender","model":"Reactor 500M","rank_in_archive_order":10,"of":21,"metrics":{"Score":"223025.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-demon-attack","task":"Atari Games","dataset":"Atari 2600 Demon Attack","model":"Reactor 500M","rank_in_archive_order":16,"of":46,"metrics":{"Score":"115154.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-double-dunk","task":"Atari Games","dataset":"Atari 2600 Double Dunk","model":"Reactor 500M","rank_in_archive_order":10,"of":43,"metrics":{"Score":"23.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-enduro","task":"Atari Games","dataset":"Atari 2600 Enduro","model":"Reactor 500M","rank_in_archive_order":13,"of":48,"metrics":{"Score":"2224.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1704.04651","atlas_url":"https://app.syntology.ai/?focus=1704.04651","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1704.04651"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1704.04651\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1704.04651"},"call":"get_concepts_for_paper(arxiv_id=\"1704.04651\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"e8de137938cd2db37a92e16f9ff8d2902353cd8d4aaf39452f5bd0a7d81c6258","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}