{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/understanding-multi-step-deep-reinforcement","title":"Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target","arxiv_id":"1901.07510","date":"2019-01-22","proceeding":null,"authors":["J. Fernando Hernandez-Garcia","Richard S. Sutton"],"abstract":"Multi-step methods such as Retrace($\\lambda$) and $n$-step $Q$-learning have\nbecome a crucial component of modern deep reinforcement learning agents. These\nmethods are often evaluated as a part of bigger architectures and their\nevaluations rarely include enough samples to draw statistically significant\nconclusions about their performance. This type of methodology makes it\ndifficult to understand how particular algorithmic details of multi-step\nmethods influence learning. In this paper we combine the $n$-step action-value\nalgorithms Retrace, $Q$-learning, Tree Backup, Sarsa, and $Q(\\sigma)$ with an\narchitecture analogous to DQN. We test the performance of all these algorithms\nin the mountain car environment; this choice of environment allows for faster\ntraining times and larger sample sizes. We present statistical analyses on the\neffects of the off-policy correction, the backup length parameter $n$, and the\nupdate frequency of the target network on the performance of these algorithms.\nOur results show that (1) using off-policy correction can have an adverse\neffect on the performance of Sarsa and $Q(\\sigma)$; (2) increasing the backup\nlength $n$ consistently improved performance across all the different\nalgorithms; and (3) the performance of Sarsa and $Q$-learning was more robust\nto the effect of the target network update frequency than the performance of\nTree Backup, $Q(\\sigma)$, and Retrace in this particular task.","url_abs":"http://arxiv.org/abs/1901.07510v2","url_pdf":"http://arxiv.org/pdf/1901.07510v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-multi-step-deep-reinforcement","repo_url":"https://github.com/kochlisGit/autonomous-vehicles-agent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dqn","method_name":"DQN"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"retrace","method_name":"Retrace"},{"method_slug":"sarsa","method_name":"Sarsa"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1901.07510","atlas_url":"https://app.syntology.ai/?focus=1901.07510","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1901.07510"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1901.07510\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1901.07510"},"call":"get_concepts_for_paper(arxiv_id=\"1901.07510\")"}],"by_repo":[{"repo":"kochlisGit/autonomous-vehicles-agent","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"27649e6e341591fa3748f70a3dcf80127431d03f9432c67fd00fe8cd944404ba","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}