Papers › A Distributional Perspective on Reinforcement Learning

A Distributional Perspective on Reinforcement Learning

21 Jul 2017ICML 2017 8arXiv:1707.06887archive 2025-07-28

Marc G. Bellemare, Will Dabney, Rémi Munos

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

PaperPDFConference PDFCodeCode Syntology ran

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 1 code sample harvested from 1 repository linked to this paper: 0 ran · 1 unverified.

By repository: community (archive-listed): 1 sample from 1 repository, 0 ran (0 with no instrument failure; 0 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

No sample is linked to this paper from 8 of its repositories that Syntology harvested for other papers:

22 repositories listed; official and paper-mentioned ones first.

Abdelhamid-bouzid/Distributional-RL mentioned on GitHubpytorchSyntology: not harvested report
BY571/DQN-Atari-Agents mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
Curt-Park/rainbow-is-all-you-need mentioned on GitHubMITSyntology: not harvested report
Kchu/DeepRL_CK mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to QUOTA: The Quantile Option Architecture for Reinforcement Learning (arXiv:1811.02073; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
NervanaSystems/coach mentioned on GitHubtfSyntology: not harvested report
Remtasya/DDPG-Actor-Critic-Reinforcement-Learning-Reacher-Environment mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Asynchronous Methods for Deep Reinforcement Learning (arXiv:1602.01783; that paper's own run record: 60 ran (of which 20 constructed an object rather than computing a result; 50 with no instrument failure: 2 honoured, 1 violated, 47 with no contract checked; 10 where Syntology's instrument failed) · 35 unverified); nothing checks that they implement this paper's method. report
chainer/chainerrl mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ChainerRL: A Deep Reinforcement Learning Library (arXiv:1912.03905; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
eric-yim/fin_map mentioned on GitHubtfSyntology: not harvested report
facebookresearch/Horizon mentioned on GitHubpytorchSyntology: not harvested report
facebookresearch/ReAgent mentioned on GitHubpytorchSyntology: not harvested report
guillaumeboniface/bananaland mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Deep Reinforcement Learning with Double Q-learning (arXiv:1509.06461; that paper's own run record: 56 ran (of which 38 constructed an object rather than computing a result; 55 with no instrument failure: 0 honoured, 0 violated, 55 with no contract checked; 1 where Syntology's instrument failed) · 50 unverified); nothing checks that they implement this paper's method. report
kochlisGit/autonomous-vehicles-agent mentioned on GitHubtfSyntology: not harvested report
marload/DistRL-TensorFlow2 mentioned on GitHubtfApache-2.0Syntology: not harvested report
marload/dist-rl-tf2 mentioned on GitHubtfApache-2.0Syntology: not harvested report
mindspore-courses/Rainbow-MindSpore mentioned on GitHubmindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Prioritized Experience Replay (arXiv:1511.05952; that paper's own run record: 80 ran (of which 62 constructed an object rather than computing a result; 72 with no instrument failure: 4 honoured, 0 violated, 68 with no contract checked; 8 where Syntology's instrument failed) · 31 unverified); nothing checks that they implement this paper's method. report
parilo/gym_bipedal_walker_v2_solution mentioned on GitHubtfSyntology: not harvested report
pihey1995/DistributionalRL mentioned on GitHubpytorchSyntology: not harvested report
shuli0808/DQN mentioned on GitHubpytorchSyntology: not harvested report
Stable-Baselines-Team/stable-baselines3-contrib pytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to A Closer Look at Invalid Action Masking in Policy Gradient Algorithms (arXiv:2006.14171; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
chandar-lab/RLHive pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Playing Atari with Deep Reinforcement Learning (arXiv:1312.5602; that paper's own run record: 64 ran (of which 24 constructed an object rather than computing a result; 46 with no instrument failure: 5 honoured, 0 violated, 41 with no contract checked; 18 where Syntology's instrument failed) · 53 unverified) and Prioritized Experience Replay (arXiv:1511.05952; that paper's own run record: 80 ran (of which 62 constructed an object rather than computing a result; 72 with no instrument failure: 4 honoured, 0 violated, 68 with no contract checked; 8 where Syntology's instrument failed) · 31 unverified); nothing checks that they implement this paper's method. report
opendilab/DI-engine pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Hindsight Experience Replay (arXiv:1707.01495; that paper's own run record: 16 ran (of which 10 constructed an object rather than computing a result; 11 with no instrument failure: 1 honoured, 0 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified), Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (arXiv:1901.02860; that paper's own run record: 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified), Decision Transformer: Reinforcement Learning via Sequence Modeling (arXiv:2106.01345; that paper's own run record: 17 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
qgallouedec/deep_rl pytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

1 sample harvested: 0 ran · 1 unverified. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

1unverified

Licence: 1 of the 1 sample is pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from BY571/DQN-Atari-Agents. 1 sample whose link to this paper records no repository is listed under one of this paper's own repositories, because identical code was harvested from it for another paper. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity.

BY571/DQN-Atari-Agents community (archive-listed) 1 sample, 0 ran · 1 unverified
run in 143 papers run_atari_dqn.py identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own unverified licence of this copy not recorded · 34182f569fae27fe · report

Tasks

Atari GamesReinforcement LearningReinforcement Learning (RL)reinforcement-learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Atari Games Atari 2600 Alien C51 noop Score 3166.0 #26 of 49 Archive leaderboard report
Atari Games Atari 2600 Amidar C51 noop Score 1735.0 #14 of 48 Archive leaderboard report
Atari Games Atari 2600 Assault C51 noop Score 7203.0 #23 of 45 Archive leaderboard report
Atari Games Atari 2600 Asterix C51 noop Score 406211 #9 of 49 Archive leaderboard report
Atari Games Atari 2600 Asteroids C51 noop Score 1516.0 #33 of 43 Archive leaderboard report
Atari Games Atari 2600 Atlantis C51 noop Score 841075.0 #22 of 42 Archive leaderboard report
Atari Games Atari 2600 Bank Heist C51 noop Score 976.0 #26 of 45 Archive leaderboard report
Atari Games Atari 2600 Battle Zone C51 noop Score 28742.0 #28 of 47 Archive leaderboard report
Atari Games Atari 2600 Beam Rider C51 noop Score 14074.0 #26 of 49 Archive leaderboard report
Atari Games Atari 2600 Berzerk C51 noop Score 1645.0 #19 of 39 Archive leaderboard report
Atari Games Atari 2600 Bowling C51 noop Score 81.8 #15 of 44 Archive leaderboard report
Atari Games Atari 2600 Boxing C51 noop Score 97.8 #22 of 45 Archive leaderboard report
Atari Games Atari 2600 Breakout C51 noop Score 748.0 #13 of 58 Archive leaderboard report
Atari Games Atari 2600 Centipede C51 noop Score 9646.0 #19 of 45 Archive leaderboard report
Atari Games Atari 2600 Chopper Command C51 noop Score 15600.0 #14 of 45 Archive leaderboard report
Atari Games Atari 2600 Crazy Climber C51 noop Score 179877.0 #10 of 49 Archive leaderboard report
Atari Games Atari 2600 Demon Attack C51 noop Score 130955.0 #11 of 46 Archive leaderboard report
Atari Games Atari 2600 Double Dunk C51 noop Score 2.5 #18 of 43 Archive leaderboard report
Atari Games Atari 2600 Enduro C51 noop Score 3454.0 #4 of 48 Archive leaderboard report
Atari Games Atari 2600 Fishing Derby C51 noop Score 8.9 #33 of 44 Archive leaderboard report
Atari Games Atari 2600 Freeway C51 noop Score 33.9 #10 of 59 Archive leaderboard report
Atari Games Atari 2600 Frostbite C51 noop Score 3965.0 #21 of 53 Archive leaderboard report
Atari Games Atari 2600 Gopher C51 noop Score 33641.0 #19 of 43 Archive leaderboard report
Atari Games Atari 2600 Gravitar C51 noop Score 440.0 #35 of 53 Archive leaderboard report
Atari Games Atari 2600 HERO C51 noop Score 38874 #4 of 44 Archive leaderboard report
Atari Games Atari 2600 Ice Hockey C51 noop Score -3.5 #35 of 43 Archive leaderboard report
Atari Games Atari 2600 James Bond C51 noop Score 1909.0 #18 of 45 Archive leaderboard report
Atari Games Atari 2600 Kangaroo C51 noop Score 12853.0 #19 of 47 Archive leaderboard report
Atari Games Atari 2600 Krull C51 noop Score 9735.0 #20 of 45 Archive leaderboard report
Atari Games Atari 2600 Kung-Fu Master C51 noop Score 48192.0 #18 of 43 Archive leaderboard report
Atari Games Atari 2600 Ms. Pacman C51 noop Score 3415.0 #25 of 47 Archive leaderboard report
Atari Games Atari 2600 Name This Game C51 noop Score 12542.0 #20 of 43 Archive leaderboard report
Atari Games Atari 2600 Pong C51 noop Score 20.9 #19 of 52 Archive leaderboard report
Atari Games Atari 2600 Private Eye C51 noop Score 15095.0 #8 of 52 Archive leaderboard report
Atari Games Atari 2600 Q*Bert C51 noop Score 23784 #18 of 57 Archive leaderboard report
Atari Games Atari 2600 River Raid C51 noop Score 17322.0 #15 of 41 Archive leaderboard report
Atari Games Atari 2600 Road Runner C51 noop Score 55839.0 #22 of 44 Archive leaderboard report
Atari Games Atari 2600 Robotank C51 noop Score 52.3 #28 of 42 Archive leaderboard report
Atari Games Atari 2600 Seaquest C51 noop Score 266434.0 #10 of 57 Archive leaderboard report
Atari Games Atari 2600 Space Invaders C51 noop Score 5747.0 #22 of 55 Archive leaderboard report
Atari Games Atari 2600 Star Gunner C51 noop Score 49095.0 #33 of 43 Archive leaderboard report
Atari Games Atari 2600 Tennis C51 noop Score 23.1 #8 of 43 Archive leaderboard report
Atari Games Atari 2600 Time Pilot C51 noop Score 8329.0 #28 of 44 Archive leaderboard report
Atari Games Atari 2600 Tutankham C51 noop Score 280.0 #12 of 44 Archive leaderboard report
Atari Games Atari 2600 Up and Down C51 noop Score 15612.0 #33 of 44 Archive leaderboard report
Atari Games Atari 2600 Venture C51 noop Score 1520.0 #13 of 55 Archive leaderboard report
Atari Games Atari 2600 Video Pinball C51 noop Score 949604.0 #5 of 42 Archive leaderboard report
Atari Games Atari 2600 Wizard of Wor C51 noop Score 9300.0 #21 of 41 Archive leaderboard report
Atari Games Atari 2600 Zaxxon C51 noop Score 10513.0 #24 of 41 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections