Methods › Reinforcement Learning › Replay Memory › Prioritized Experience Replay
Prioritized Experience Replay
Introduced by Tom Schaul et al. in Prioritized Experience Replay
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Prioritized Experience Replay is a type of experience replay in reinforcement learning where we more frequently replay transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error. This prioritization can lead to a loss of diversity, which is alleviated with stochastic prioritization, and introduce bias, which can be corrected with importance sampling.
The stochastic sampling method interpolates between pure greedy prioritization and uniform random sampling. The probability of being sampled is ensured to be monotonic in a transition's priority, while guaranteeing a non-zero probability even for the lowest-priority transition. Concretely, define the probability of sampling transition i as
P(i) = (pᵢ^α)/(∑ₖ pₖ^α)
where pᵢ > 0 is the priority of transition i. The exponent α determines how much prioritization is used, with α=0 corresponding to the uniform case.
Prioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to. We can correct this bias by using importance-sampling (IS) weights:
wᵢ = (1/N·1/(P(i)))^β
that fully compensates for the non-uniform probabilities P(i) if β= 1. These weights can be folded into the Q-learning update by using wᵢδᵢ instead of δᵢ - weighted IS rather than ordinary IS. For stability reasons, we always normalize weights by 1/maxᵢwᵢ so that they only scale the update downwards.
The two types of prioritization are proportional based, where pᵢ = |δᵢ| + ϵ and rank-based, where pᵢ = 1/(rank(i)), the latter where rank(i) is the rank of transition i when the replay memory is sorted according to |δᵢ|, For proportional based, hyperparameters used were α= 0.7, β₀ = 0.5. For the rank-based variant, hyperparameters used were α= 0.6, β₀ = 0.4.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 77 in the graph, strongest link: mentioned on GitHub). The lookup matched 'PRIORITIZED EXPERIENCE REPLAY' ignoring case. This answer is about arXiv:1511.05952, the paper the archive credits with this method.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- Curious Replay · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Curious Replay') - Dark Experience Replay · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Dark Experience Replay') - a simple approach that learns a reward model from online experience · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='a simple approach that learns a reward model from online experience')
Origin paper of this answer, arXiv:1511.05952 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 80 ran (of which 62 constructed an object rather than computing a result; 72 with no instrument failure: 4 honoured, 0 violated, 68 with no contract checked; 8 where Syntology's instrument failed) · 31 unverified (of 111 harvested samples) · 43 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 138 · with a code link: 61 · where Syntology ran a sample: 19 (17 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (19 of 138 tagged: 17 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 138, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization 18 Jun 2025 · 1 repository · arXiv:2506.15654
-
Calibrated Value-Aware Model Learning with Stochastic Environment Models 28 May 2025 · 0 repositories · arXiv:2505.22772
-
Online Learning-based Adaptive Beam Switching for 6G Networks: Enhancing Efficiency and Resilience 12 May 2025 · 1 repository · arXiv:2505.08032
-
Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation 11 Apr 2025 · 0 repositories · arXiv:2504.08195
-
PER-DPP Sampling Framework and Its Application in Path Planning 10 Mar 2025 · 0 repositories · arXiv:2503.07411
-
OptionZero: Planning with Learned Options 23 Feb 2025 · 1 repository · arXiv:2502.16634Syntology official (archive's flag): 7 ran · 7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 7 samples that ran constructed an object rather than computing a result (of 8 harvested samples) · 8 pointer-only (licence)
-
Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations 14 Feb 2025 · 0 repositories · arXiv:2502.10303
-
Enhancing UAV Path Planning Efficiency Through Accelerated Learning 17 Jan 2025 · 0 repositories · arXiv:2501.10141
-
SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks 7 Jan 2025 · 0 repositories · arXiv:2501.03676
-
Evaluating World Models with LLM for Decision Making 13 Nov 2024 · 0 repositories · arXiv:2411.08794
-
Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping 7 Nov 2024 · 0 repositories · arXiv:2411.04915
-
Interpreting the Learned Model in MuZero Planning 7 Nov 2024 · 0 repositories · arXiv:2411.04580
-
Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC 6 Nov 2024 · 3 repositories · arXiv:2411.03820Syntology official (archive's flag): 3 ran · 16 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 9 unverified (of 25 harvested samples) · 21 pointer-only (licence)
-
Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay 16 Oct 2024 · 0 repositories · arXiv:2410.12236
-
Learning in complex action spaces without policy gradients 8 Oct 2024 · 0 repositories · arXiv:2410.06317
-
Investigating the Interplay of Prioritized Replay and Generalization 12 Jul 2024 · 0 repositories · arXiv:2407.09702
-
ROER: Regularized Optimal Experience Replay 4 Jul 2024 · 1 repository · arXiv:2407.03995
-
Combining AI Control Systems and Human Decision Support via Robustness and Criticality 3 Jul 2024 · 0 repositories · arXiv:2407.03210
-
Physics-informed Imitative Reinforcement Learning for Real-world Driving 18 Jun 2024 · 0 repositories · arXiv:2407.02508
-
Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction 2 Jun 2024 · 1 repository · arXiv:2406.00614
-
Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning 2 Jun 2024 · 0 repositories · arXiv:2406.00761
-
Efficient Multi-agent Reinforcement Learning by Planning 20 May 2024 · 1 repository · arXiv:2405.11778
-
ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze 25 Apr 2024 · 1 repository · arXiv:2404.16364
-
Laser Learning Environment: A new environment for coordination-critical multi-agent tasks 4 Apr 2024 · 1 repository · arXiv:2404.03596
-
An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning 12 Mar 2024 · 0 repositories · arXiv:2403.07566
-
Reinforcement learning to maximise wind turbine energy generation 17 Feb 2024 · 0 repositories · arXiv:2402.11384
-
Directly Attention Loss Adjusted Prioritized Experience Replay 24 Nov 2023 · 0 repositories · arXiv:2311.14390
-
Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part II: Control-Aware Radio Resource Allocation 19 Nov 2023 · 0 repositories · arXiv:2311.11280
-
Advancing Algorithmic Trading: A Multi-Technique Enhancement of Deep Q-Network Models 9 Nov 2023 · 0 repositories · arXiv:2311.05743
-
Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning 7 Nov 2023 · 0 repositories · arXiv:2311.03711
-
SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models 19 Oct 2023 · 1 repository · arXiv:2310.12494
-
Using Experience Classification for Training Non-Markovian Tasks 18 Oct 2023 · 0 repositories · arXiv:2310.11678
-
MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Games 17 Oct 2023 · 1 repository · arXiv:2310.11305Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction 10 Oct 2023 · 0 repositories · arXiv:2310.06513
-
A Long N-step Surrogate Stage Reward for Deep Reinforcement Learning 21 Sep 2023 · 0 repositories
-
Self-Predictive Universal AI 21 Sep 2023 · 0 repositories
-
AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning 7 Aug 2023 · 1 repository · arXiv:2308.03526Syntology 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples)
-
Safety Margins for Reinforcement Learning 25 Jul 2023 · 0 repositories · arXiv:2307.13642
-
λ-models: Effective Decision-Aware Reinforcement Learning with Latent Models 30 Jun 2023 · 0 repositories · arXiv:2306.17366
-
Curious Replay for Model-based Adaptation 28 Jun 2023 · 1 repository · arXiv:2306.15934
-
Decoupled Prioritized Resampling for Offline RL 8 Jun 2023 · 2 repositories · arXiv:2306.05412Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
What model does MuZero learn? 1 Jun 2023 · 0 repositories · arXiv:2306.00840
-
Attention-based QoE-aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality 15 May 2023 · 0 repositories · arXiv:2305.08569
-
Extracting Diagnosis Pathways from Electronic Health Records Using Deep Reinforcement Learning 10 May 2023 · 1 repository · arXiv:2305.06295
-
Quantum deep Q learning with distributed prioritized experience replay 19 Apr 2023 · 0 repositories · arXiv:2304.09648
-
Model Predictive Control with Self-supervised Representation Learning 14 Apr 2023 · 1 repository · arXiv:2304.07219
-
MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization 21 Feb 2023 · 1 repository · arXiv:2302.10418
-
Equivariant MuZero 9 Feb 2023 · 0 repositories · arXiv:2302.04798
-
Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning 28 Nov 2022 · 0 repositories · arXiv:2211.15612
-
Double Deep Q-Learning in Opponent Modeling 24 Nov 2022 · 0 repositories · arXiv:2211.15384
-
Epistemic Monte Carlo Tree Search 21 Oct 2022 · 0 repositories · arXiv:2210.13455
-
Efficient Offline Policy Optimization with a Learned Model 12 Oct 2022 · 1 repository · arXiv:2210.05980
-
Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation 22 Sep 2022 · 0 repositories · arXiv:2209.11094
-
Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks 16 Sep 2022 · 1 repository · arXiv:2209.07670
-
Human-level Atari 200x faster 15 Sep 2022 · 1 repository · arXiv:2209.07550Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples)
-
Actor Prioritized Experience Replay 1 Sep 2022 · 1 repository · arXiv:2209.00532
-
Transformers are Sample-Efficient World Models 1 Sep 2022 · 2 repositories · arXiv:2209.00588Syntology official (archive's flag): 17 ran · 17 ran (of which 13 constructed an object rather than computing a result; 16 with no instrument failure: 1 honoured, 0 violated, 15 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 26 harvested samples) · 26 pointer-only (licence)
-
Associative Memory Based Experience Replay for Deep Reinforcement Learning 16 Jul 2022 · 0 repositories · arXiv:2207.07791
-
Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments 4 Jul 2022 · 0 repositories · arXiv:2207.01251
-
An AlphaZero-Inspired Approach to Solving Search Problems 2 Jul 2022 · 0 repositories · arXiv:2207.00919
-
DNA: Proximal Policy Optimization with a Dual Network Architecture 20 Jun 2022 · 1 repository · arXiv:2206.10027
-
Introspective Experience Replay: Look Back When Surprised 7 Jun 2022 · 1 repository · arXiv:2206.03171
-
Impartial Games: A Challenge for Reinforcement Learning 25 May 2022 · 1 repository · arXiv:2205.12787
-
Optimal Adaptive Prediction Intervals for Electricity Load Forecasting in Distribution Systems via Reinforcement Learning 18 May 2022 · 1 repository · arXiv:2205.08698
-
Gamma and Vega Hedging Using Deep Distributional Reinforcement Learning 10 May 2022 · 1 repository · arXiv:2205.05614
-
Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods 25 Apr 2022 · 1 repository · arXiv:2204.11464
-
Reward Reports for Reinforcement Learning 22 Apr 2022 · 1 repository · arXiv:2204.10817
-
Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach 21 Apr 2022 · 1 repository · arXiv:2204.10256
-
MuZero with Self-competition for Rate Control in VP9 Video Compression 14 Feb 2022 · 0 repositories · arXiv:2202.06626
-
Procedural Generalization by Planning with Self-Supervised World Models 2 Nov 2021 · 0 repositories · arXiv:2111.01587
-
Mastering Atari Games with Limited Data 30 Oct 2021 · 3 repositories · arXiv:2111.00210Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Large Batch Experience Replay 4 Oct 2021 · 2 repositories · arXiv:2110.01528Syntology official (archive's flag): 5 ran · 7 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified (of 11 harvested samples)
-
Benchmarking Sample Selection Strategies for Batch Reinforcement Learning 29 Sep 2021 · 0 repositories
-
Planning in Stochastic Environments with a Learned Model 29 Sep 2021 · 2 repositories
-
Policy improvement by planning with Gumbel 29 Sep 2021 · 2 repositories
-
Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving 26 Sep 2021 · 1 repository · arXiv:2109.12516
-
Deep Reinforcement Learning at the Edge of the Statistical Precipice 30 Aug 2021 · 3 repositories · arXiv:2108.13264Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 4 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
Optimal Scheduling of Isolated Microgrids Using Automated Reinforcement Learning-based Multi-period Forecasting 15 Aug 2021 · 0 repositories · arXiv:2108.06764
-
Value-Based Reinforcement Learning for Continuous Control Robotic Manipulation in Multi-Task Sparse Reward Settings 28 Jul 2021 · 0 repositories · arXiv:2107.13356
-
Proper Value Equivalence 18 Jun 2021 · 1 repository · arXiv:2106.10316
-
Vector Quantized Models for Planning 8 Jun 2021 · 0 repositories · arXiv:2106.04615
-
Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization 27 May 2021 · 0 repositories · arXiv:2105.12991
-
Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning 6 May 2021 · 0 repositories · arXiv:2105.02580
-
Learning and Planning in Complex Action Spaces 13 Apr 2021 · 1 repository · arXiv:2104.06303
-
Online and Offline Reinforcement Learning by Planning with a Learned Model 13 Apr 2021 · 2 repositories · arXiv:2104.06294
-
A coevolutionary approach to deep multi-agent reinforcement learning 12 Apr 2021 · 1 repository · arXiv:2104.05610
-
Accelerating Online Reinforcement Learning via Model-Based Meta-Learning 15 Mar 2021 · 0 repositories
-
Visualizing MuZero Models 25 Feb 2021 · 1 repository · arXiv:2102.12924
-
Combining Off and On-Policy Training in Model-Based Reinforcement Learning 24 Feb 2021 · 0 repositories · arXiv:2102.12194
-
Accelerated Sim-to-Real Deep Reinforcement Learning: Learning Collision Avoidance from Human Player 21 Feb 2021 · 1 repository · arXiv:2102.10711
-
Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision 10 Feb 2021 · 2 repositories · arXiv:2102.05599
-
Revisiting Prioritized Experience Replay: A Value Perspective 5 Feb 2021 · 2 repositories · arXiv:2102.03261
-
Playing Atari with Capsule Networks: A systematic comparison of CNN and CapsNets-based agents. 1 Jan 2021 · 0 repositories
-
Playing Nondeterministic Games through Planning with a Learned Model 1 Jan 2021 · 0 repositories
-
Weighted Bellman Backups for Improved Signal-to-Noise in Q-Updates 1 Jan 2021 · 0 repositories
-
Reinforcement Learning for Robust Missile Autopilot Design 26 Nov 2020 · 0 repositories · arXiv:2011.12956
-
Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking 15 Nov 2020 · 1 repository · arXiv:2011.07537Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning 13 Nov 2020 · 0 repositories · arXiv:2011.06752
-
On the role of planning in model-based deep reinforcement learning 8 Nov 2020 · 0 repositories · arXiv:2011.04021
-
The Value Equivalence Principle for Model-Based Reinforcement Learning 6 Nov 2020 · 0 repositories · arXiv:2011.03506
Tasks archive 2025-07-28
20 shown of 82 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections