{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/true-online-temporal-difference-learning","title":"True Online Temporal-Difference Learning","arxiv_id":"1512.04087","date":"2015-12-13","proceeding":null,"authors":["Harm van Seijen","A. Rupam Mahmood","Patrick M. Pilarski","Marlos C. Machado","Richard S. Sutton"],"abstract":"The temporal-difference methods TD($\\lambda$) and Sarsa($\\lambda$) form a\ncore part of modern reinforcement learning. Their appeal comes from their good\nperformance, low computational cost, and their simple interpretation, given by\ntheir forward view. Recently, new versions of these methods were introduced,\ncalled true online TD($\\lambda$) and true online Sarsa($\\lambda$), respectively\n(van Seijen & Sutton, 2014). These new versions maintain an exact equivalence\nwith the forward view at all times, whereas the traditional versions only\napproximate it for small step-sizes. We hypothesize that these true online\nmethods not only have better theoretical properties, but also dominate the\nregular methods empirically. In this article, we put this hypothesis to the\ntest by performing an extensive empirical comparison. Specifically, we compare\nthe performance of true online TD($\\lambda$)/Sarsa($\\lambda$) with regular\nTD($\\lambda$)/Sarsa($\\lambda$) on random MRPs, a real-world myoelectric\nprosthetic arm, and a domain from the Arcade Learning Environment. We use\nlinear function approximation with tabular, binary, and non-binary features.\nOur results suggest that the true online methods indeed dominate the regular\nmethods. Across all domains/representations the learning speed of the true\nonline methods are often better, but never worse than that of the regular\nmethods. An additional advantage is that no choice between traces has to be\nmade for the true online methods. Besides the empirical results, we provide an\nin-depth analysis of the theory behind true online temporal-difference\nlearning. In addition, we show that new true online temporal-difference methods\ncan be derived by making changes to the online forward view and then rewriting\nthe update equations.","url_abs":"http://arxiv.org/abs/1512.04087v2","url_pdf":"http://arxiv.org/pdf/1512.04087v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"true-online-temporal-difference-learning","repo_url":"https://github.com/armahmood/totd-rndmdp-experiments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1512.04087","atlas_url":"https://app.syntology.ai/?focus=1512.04087","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1512.04087"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1512.04087\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1512.04087"},"call":"get_concepts_for_paper(arxiv_id=\"1512.04087\")"}],"by_repo":[{"repo":"armahmood/totd-rndmdp-experiments","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0cbee0b2a8ad64697b0cefe0533acf8ae3c1b85a012612f8cd87f2cc6d03982d","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}