{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/prioritized-experience-replay","slug":"prioritized-experience-replay","name":"Prioritized Experience Replay","full_name":"Prioritized Experience Replay","full_name_withheld":false,"description_markdown":"**Prioritized Experience Replay** is a type of [experience replay](https://paperswithcode.com/method/experience-replay) in reinforcement learning where we more frequently replay transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error. This prioritization can lead to a loss of diversity, which is alleviated with stochastic prioritization, and introduce bias, which can be corrected with importance sampling.\r\n\r\nThe stochastic sampling method interpolates between pure greedy prioritization and uniform random sampling. The probability of being sampled is ensured to be monotonic in a transition's priority,  while guaranteeing a non-zero probability even for the lowest-priority transition. Concretely, define the probability of sampling transition $i$ as\r\n\r\n$$P(i) = \\frac{p_i^{\\alpha}}{\\sum_k p_k^{\\alpha}}$$\r\n\r\nwhere $p_i > 0$ is the priority of transition $i$. The exponent $\\alpha$ determines how much prioritization is used, with $\\alpha=0$ corresponding to the uniform case.\r\n\r\nPrioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to. We can correct this bias by using\r\nimportance-sampling (IS) weights:\r\n\r\n$$ w\\_{i} = \\left(\\frac{1}{N}\\cdot\\frac{1}{P\\left(i\\right)}\\right)^{\\beta} $$\r\n\r\nthat fully compensates for the non-uniform probabilities $P\\left(i\\right)$ if $\\beta = 1$. These weights can be folded into the [Q-learning](https://paperswithcode.com/method/q-learning) update by using $w\\_{i}\\delta\\_{i}$ instead of $\\delta\\_{i}$ - weighted IS rather than ordinary IS. For stability reasons, we always normalize weights by $1/\\max\\_{i}w\\_{i}$ so\r\nthat they only scale the update downwards.\r\n\r\nThe two types of prioritization are proportional based, where $p\\_{i} = |\\delta\\_{i}| + \\epsilon$ and rank-based, where $p\\_{i} = \\frac{1}{\\text{rank}\\left(i\\right)}$, the latter where $\\text{rank}\\left(i\\right)$ is the rank of transition $i$ when the replay memory is sorted according to |$\\delta\\_{i}$|, For proportional based, hyperparameters used were $\\alpha = 0.7$, $\\beta\\_{0} = 0.5$. For the rank-based variant, hyperparameters used were $\\alpha = 0.6$, $\\beta\\_{0} = 0.4$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Prioritized Experience Replay","paper":"/paper/prioritized-experience-replay","first_author":"Tom Schaul","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/prioritized-experience-replay"},"source":{"url":"http://arxiv.org/abs/1511.05952v4","title":"Prioritized Experience Replay","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Replay Memory","url":"/methods/category/replay-memory","pwc_aliases":[]}],"n_papers_tagged":138,"archive_num_papers":138,"papers_newest_first":[{"paper":"/paper/cawr-corruption-averse-advantage-weighted","slug":"cawr-corruption-averse-advantage-weighted","title":"CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization","date":"2025-06-18","arxiv_id":"2506.15654","n_code_links":1,"syntology":null},{"paper":null,"slug":"calibrated-value-aware-model-learning-with","title":"Calibrated Value-Aware Model Learning with Stochastic Environment Models","date":"2025-05-28","arxiv_id":"2505.22772","n_code_links":0,"syntology":null},{"paper":"/paper/online-learning-based-adaptive-beam-switching","slug":"online-learning-based-adaptive-beam-switching","title":"Online Learning-based Adaptive Beam Switching for 6G Networks: Enhancing Efficiency and Resilience","date":"2025-05-12","arxiv_id":"2505.08032","n_code_links":1,"syntology":null},{"paper":null,"slug":"graph-based-deep-reinforcement-learning-aided","title":"Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation","date":"2025-04-11","arxiv_id":"2504.08195","n_code_links":0,"syntology":null},{"paper":null,"slug":"per-dpp-sampling-framework-and-its","title":"PER-DPP Sampling Framework and Its Application in Path Planning","date":"2025-03-10","arxiv_id":"2503.07411","n_code_links":0,"syntology":null},{"paper":"/paper/optionzero-planning-with-learned-options","slug":"optionzero-planning-with-learned-options","title":"OptionZero: Planning with Learned Options","date":"2025-02-23","arxiv_id":"2502.16634","n_code_links":1,"syntology":{"ran":7,"of":8,"n_ran_checked":7,"n_instrument":0,"unverified":1,"pointer_only":8,"phrase":"7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 7 samples that ran constructed an object rather than computing a result","official":{"repos":["rlglab/optionzero"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":7,"n_ran_no_instrument_failure":7,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"reinforcement-learning-in-strategy-based-and","title":"Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations","date":"2025-02-14","arxiv_id":"2502.10303","n_code_links":0,"syntology":null},{"paper":null,"slug":"enhancing-uav-path-planning-efficiency","title":"Enhancing UAV Path Planning Efficiency Through Accelerated Learning","date":"2025-01-17","arxiv_id":"2501.10141","n_code_links":0,"syntology":null},{"paper":null,"slug":"sale-based-offline-reinforcement-learning","title":"SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks","date":"2025-01-07","arxiv_id":"2501.03676","n_code_links":0,"syntology":null},{"paper":null,"slug":"evaluating-world-models-with-llm-for-decision","title":"Evaluating World Models with LLM for Decision Making","date":"2024-11-13","arxiv_id":"2411.08794","n_code_links":0,"syntology":null},{"paper":null,"slug":"evaluating-robustness-of-reinforcement","title":"Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping","date":"2024-11-07","arxiv_id":"2411.04915","n_code_links":0,"syntology":null},{"paper":null,"slug":"interpreting-the-learned-model-in-muzero","title":"Interpreting the Learned Model in MuZero Planning","date":"2024-11-07","arxiv_id":"2411.04580","n_code_links":0,"syntology":null},{"paper":"/paper/beyond-the-rainbow-high-performance-deep","slug":"beyond-the-rainbow-high-performance-deep","title":"Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC","date":"2024-11-06","arxiv_id":"2411.03820","n_code_links":3,"syntology":{"ran":16,"of":25,"n_ran_checked":12,"n_instrument":4,"unverified":9,"pointer_only":21,"phrase":"16 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 9 unverified","official":{"repos":["viptankz/btr"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"paper":null,"slug":"enhancing-llm-agents-for-code-generation-with","title":"Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay","date":"2024-10-16","arxiv_id":"2410.12236","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-in-complex-action-spaces-without","title":"Learning in complex action spaces without policy gradients","date":"2024-10-08","arxiv_id":"2410.06317","n_code_links":0,"syntology":null},{"paper":null,"slug":"investigating-the-interplay-of-prioritized","title":"Investigating the Interplay of Prioritized Replay and Generalization","date":"2024-07-12","arxiv_id":"2407.09702","n_code_links":0,"syntology":null},{"paper":"/paper/roer-regularized-optimal-experience-replay","slug":"roer-regularized-optimal-experience-replay","title":"ROER: Regularized Optimal Experience Replay","date":"2024-07-04","arxiv_id":"2407.03995","n_code_links":1,"syntology":null},{"paper":null,"slug":"combining-ai-control-systems-and-human","title":"Combining AI Control Systems and Human Decision Support via Robustness and Criticality","date":"2024-07-03","arxiv_id":"2407.03210","n_code_links":0,"syntology":null},{"paper":null,"slug":"sample-efficient-imitative-multi-token","title":"Physics-informed Imitative Reinforcement Learning for Real-world Driving","date":"2024-06-18","arxiv_id":"2407.02508","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-monte-carlo-tree-search-via-on-the","slug":"efficient-monte-carlo-tree-search-via-on-the","title":"Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction","date":"2024-06-02","arxiv_id":"2406.00614","n_code_links":1,"syntology":null},{"paper":null,"slug":"shared-unique-features-and-task-aware","title":"Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning","date":"2024-06-02","arxiv_id":"2406.00761","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-multi-agent-reinforcement-learning","slug":"efficient-multi-agent-reinforcement-learning","title":"Efficient Multi-agent Reinforcement Learning by Planning","date":"2024-05-20","arxiv_id":"2405.11778","n_code_links":1,"syntology":null},{"paper":"/paper/rezero-boosting-mcts-based-algorithms-by-just","slug":"rezero-boosting-mcts-based-algorithms-by-just","title":"ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze","date":"2024-04-25","arxiv_id":"2404.16364","n_code_links":1,"syntology":null},{"paper":"/paper/laser-learning-environment-a-new-environment","slug":"laser-learning-environment-a-new-environment","title":"Laser Learning Environment: A new environment for coordination-critical multi-agent tasks","date":"2024-04-04","arxiv_id":"2404.03596","n_code_links":1,"syntology":null},{"paper":null,"slug":"an-improved-strategy-for-blood-glucose","title":"An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning","date":"2024-03-12","arxiv_id":"2403.07566","n_code_links":0,"syntology":null},{"paper":null,"slug":"reinforcement-learning-to-maximise-wind","title":"Reinforcement learning to maximise wind turbine energy generation","date":"2024-02-17","arxiv_id":"2402.11384","n_code_links":0,"syntology":null},{"paper":null,"slug":"directly-attention-loss-adjusted-prioritized","title":"Directly Attention Loss Adjusted Prioritized Experience Replay","date":"2023-11-24","arxiv_id":"2311.14390","n_code_links":0,"syntology":null},{"paper":null,"slug":"multi-timescale-control-and-communications-1","title":"Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part II: Control-Aware Radio Resource Allocation","date":"2023-11-19","arxiv_id":"2311.11280","n_code_links":0,"syntology":null},{"paper":null,"slug":"advancing-algorithmic-trading-a-multi","title":"Advancing Algorithmic Trading: A Multi-Technique Enhancement of Deep Q-Network Models","date":"2023-11-09","arxiv_id":"2311.05743","n_code_links":0,"syntology":null},{"paper":null,"slug":"mitigating-estimation-errors-by-twin-td","title":"Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning","date":"2023-11-07","arxiv_id":"2311.03711","n_code_links":0,"syntology":null},{"paper":"/paper/sdgym-low-code-reinforcement-learning","slug":"sdgym-low-code-reinforcement-learning","title":"SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models","date":"2023-10-19","arxiv_id":"2310.12494","n_code_links":1,"syntology":null},{"paper":null,"slug":"using-experience-classification-for-training","title":"Using Experience Classification for Training Non-Markovian Tasks","date":"2023-10-18","arxiv_id":"2310.11678","n_code_links":0,"syntology":null},{"paper":"/paper/minizero-comparative-analysis-of-alphazero","slug":"minizero-comparative-analysis-of-alphazero","title":"MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Games","date":"2023-10-17","arxiv_id":"2310.11305","n_code_links":1,"syntology":{"ran":6,"of":7,"n_ran_checked":6,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["rlglab/minizero"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"accelerating-monte-carlo-tree-search-with","title":"Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction","date":"2023-10-10","arxiv_id":"2310.06513","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-long-n-step-surrogate-stage-reward-for-deep","title":"A Long $N$-step Surrogate Stage Reward for Deep Reinforcement Learning","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"self-predictive-universal-ai","title":"Self-Predictive Universal AI","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/alphastar-unplugged-large-scale-offline","slug":"alphastar-unplugged-large-scale-offline","title":"AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning","date":"2023-08-07","arxiv_id":"2308.03526","n_code_links":1,"syntology":{"ran":4,"of":4,"n_ran_checked":4,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":null,"slug":"safety-margins-for-reinforcement-learning","title":"Safety Margins for Reinforcement Learning","date":"2023-07-25","arxiv_id":"2307.13642","n_code_links":0,"syntology":null},{"paper":null,"slug":"l-ac-learning-latent-decision-aware-models","title":"$λ$-models: Effective Decision-Aware Reinforcement Learning with Latent Models","date":"2023-06-30","arxiv_id":"2306.17366","n_code_links":0,"syntology":null},{"paper":"/paper/curious-replay-for-model-based-adaptation","slug":"curious-replay-for-model-based-adaptation","title":"Curious Replay for Model-based Adaptation","date":"2023-06-28","arxiv_id":"2306.15934","n_code_links":1,"syntology":null},{"paper":"/paper/offline-prioritized-experience-replay","slug":"offline-prioritized-experience-replay","title":"Decoupled Prioritized Resampling for Offline RL","date":"2023-06-08","arxiv_id":"2306.05412","n_code_links":2,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["sail-sg/oper","yueyang130/odpr"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"what-model-does-muzero-learn","title":"What model does MuZero learn?","date":"2023-06-01","arxiv_id":"2306.00840","n_code_links":0,"syntology":null},{"paper":null,"slug":"attention-based-qoe-aware-digital-twin","title":"Attention-based QoE-aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality","date":"2023-05-15","arxiv_id":"2305.08569","n_code_links":0,"syntology":null},{"paper":"/paper/extracting-diagnosis-pathways-from-electronic","slug":"extracting-diagnosis-pathways-from-electronic","title":"Extracting Diagnosis Pathways from Electronic Health Records Using Deep Reinforcement Learning","date":"2023-05-10","arxiv_id":"2305.06295","n_code_links":1,"syntology":null},{"paper":null,"slug":"quantum-deep-q-learning-with-distributed","title":"Quantum deep Q learning with distributed prioritized experience replay","date":"2023-04-19","arxiv_id":"2304.09648","n_code_links":0,"syntology":null},{"paper":"/paper/model-predictive-control-with-self-supervised","slug":"model-predictive-control-with-self-supervised","title":"Model Predictive Control with Self-supervised Representation Learning","date":"2023-04-14","arxiv_id":"2304.07219","n_code_links":1,"syntology":null},{"paper":"/paper/mac-po-multi-agent-experience-replay-via","slug":"mac-po-multi-agent-experience-replay-via","title":"MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization","date":"2023-02-21","arxiv_id":"2302.10418","n_code_links":1,"syntology":null},{"paper":null,"slug":"equivariant-muzero","title":"Equivariant MuZero","date":"2023-02-09","arxiv_id":"2302.04798","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-from-good-trajectories-in-offline","title":"Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning","date":"2022-11-28","arxiv_id":"2211.15612","n_code_links":0,"syntology":null},{"paper":null,"slug":"double-deep-q-learning-in-opponent-modeling","title":"Double Deep Q-Learning in Opponent Modeling","date":"2022-11-24","arxiv_id":"2211.15384","n_code_links":0,"syntology":null},{"paper":null,"slug":"planning-with-uncertainty-deep-exploration-in","title":"Epistemic Monte Carlo Tree Search","date":"2022-10-21","arxiv_id":"2210.13455","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-offline-policy-optimization-with-a","slug":"efficient-offline-policy-optimization-with-a","title":"Efficient Offline Policy Optimization with a Learned Model","date":"2022-10-12","arxiv_id":"2210.05980","n_code_links":1,"syntology":null},{"paper":null,"slug":"parallel-reinforcement-learning-simulation","title":"Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation","date":"2022-09-22","arxiv_id":"2209.11094","n_code_links":0,"syntology":null},{"paper":"/paper/reducing-variance-in-temporal-difference","slug":"reducing-variance-in-temporal-difference","title":"Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks","date":"2022-09-16","arxiv_id":"2209.07670","n_code_links":1,"syntology":null},{"paper":"/paper/human-level-atari-200x-faster","slug":"human-level-atari-200x-faster","title":"Human-level Atari 200x faster","date":"2022-09-15","arxiv_id":"2209.07550","n_code_links":1,"syntology":{"ran":1,"of":3,"n_ran_checked":0,"n_instrument":1,"unverified":2,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified","official":null}},{"paper":"/paper/actor-prioritized-experience-replay","slug":"actor-prioritized-experience-replay","title":"Actor Prioritized Experience Replay","date":"2022-09-01","arxiv_id":"2209.00532","n_code_links":1,"syntology":null},{"paper":"/paper/transformers-are-sample-efficient-world","slug":"transformers-are-sample-efficient-world","title":"Transformers are Sample-Efficient World Models","date":"2022-09-01","arxiv_id":"2209.00588","n_code_links":2,"syntology":{"ran":17,"of":26,"n_ran_checked":16,"n_instrument":1,"unverified":9,"pointer_only":26,"phrase":"17 ran (of which 13 constructed an object rather than computing a result; 16 with no instrument failure: 1 honoured, 0 violated, 15 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified","official":{"repos":["eloialonso/iris"],"state":"official (archive's flag): 17 ran","n_ran":17,"n_constructed":13,"n_ran_no_instrument_failure":16,"n_unverified":9,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"associative-memory-based-experience-replay","title":"Associative Memory Based Experience Replay for Deep Reinforcement Learning","date":"2022-07-16","arxiv_id":"2207.07791","n_code_links":0,"syntology":null},{"paper":null,"slug":"asynchronous-curriculum-experience-replay-a","title":"Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments","date":"2022-07-04","arxiv_id":"2207.01251","n_code_links":0,"syntology":null},{"paper":null,"slug":"an-alphazero-inspired-approach-to-solving","title":"An AlphaZero-Inspired Approach to Solving Search Problems","date":"2022-07-02","arxiv_id":"2207.00919","n_code_links":0,"syntology":null},{"paper":"/paper/dna-proximal-policy-optimization-with-a-dual","slug":"dna-proximal-policy-optimization-with-a-dual","title":"DNA: Proximal Policy Optimization with a Dual Network Architecture","date":"2022-06-20","arxiv_id":"2206.10027","n_code_links":1,"syntology":null},{"paper":"/paper/look-back-when-surprised-stabilizing-reverse","slug":"look-back-when-surprised-stabilizing-reverse","title":"Introspective Experience Replay: Look Back When Surprised","date":"2022-06-07","arxiv_id":"2206.03171","n_code_links":1,"syntology":null},{"paper":"/paper/impartial-games-a-challenge-for-reinforcement","slug":"impartial-games-a-challenge-for-reinforcement","title":"Impartial Games: A Challenge for Reinforcement Learning","date":"2022-05-25","arxiv_id":"2205.12787","n_code_links":1,"syntology":null},{"paper":"/paper/optimal-adaptive-prediction-intervals-for","slug":"optimal-adaptive-prediction-intervals-for","title":"Optimal Adaptive Prediction Intervals for Electricity Load Forecasting in Distribution Systems via Reinforcement Learning","date":"2022-05-18","arxiv_id":"2205.08698","n_code_links":1,"syntology":null},{"paper":"/paper/gamma-and-vega-hedging-using-deep","slug":"gamma-and-vega-hedging-using-deep","title":"Gamma and Vega Hedging Using Deep Distributional Reinforcement Learning","date":"2022-05-10","arxiv_id":"2205.05614","n_code_links":1,"syntology":null},{"paper":"/paper/towards-evaluating-adaptivity-of-model-based","slug":"towards-evaluating-adaptivity-of-model-based","title":"Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods","date":"2022-04-25","arxiv_id":"2204.11464","n_code_links":1,"syntology":null},{"paper":"/paper/reward-reports-for-reinforcement-learning","slug":"reward-reports-for-reinforcement-learning","title":"Reward Reports for Reinforcement Learning","date":"2022-04-22","arxiv_id":"2204.10817","n_code_links":1,"syntology":null},{"paper":"/paper/revisiting-gaussian-mixture-critic-in-off","slug":"revisiting-gaussian-mixture-critic-in-off","title":"Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach","date":"2022-04-21","arxiv_id":"2204.10256","n_code_links":1,"syntology":null},{"paper":null,"slug":"muzero-with-self-competition-for-rate-control","title":"MuZero with Self-competition for Rate Control in VP9 Video Compression","date":"2022-02-14","arxiv_id":"2202.06626","n_code_links":0,"syntology":null},{"paper":"/paper/procedural-generalization-by-planning-with-1","slug":"procedural-generalization-by-planning-with-1","title":"Procedural Generalization by Planning with Self-Supervised World Models","date":"2021-11-02","arxiv_id":"2111.01587","n_code_links":0,"syntology":null},{"paper":"/paper/mastering-atari-games-with-limited-data","slug":"mastering-atari-games-with-limited-data","title":"Mastering Atari Games with Limited Data","date":"2021-10-30","arxiv_id":"2111.00210","n_code_links":3,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["werner-duvaud/muzero-general","yewr/efficientzero"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/large-batch-experience-replay","slug":"large-batch-experience-replay","title":"Large Batch Experience Replay","date":"2021-10-04","arxiv_id":"2110.01528","n_code_links":2,"syntology":{"ran":7,"of":11,"n_ran_checked":4,"n_instrument":3,"unverified":4,"pointer_only":0,"phrase":"7 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified","official":{"repos":["sureli/laber"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":3,"n_ran_no_instrument_failure":3,"n_unverified":0,"ran_from_kinds":["listed","official"]}}},{"paper":null,"slug":"benchmarking-sample-selection-strategies-for","title":"Benchmarking Sample Selection Strategies for Batch Reinforcement Learning","date":"2021-09-29","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/planning-in-stochastic-environments-with-a","slug":"planning-in-stochastic-environments-with-a","title":"Planning in Stochastic Environments with a Learned Model","date":"2021-09-29","arxiv_id":null,"n_code_links":2,"syntology":null},{"paper":"/paper/policy-improvement-by-planning-with-gumbel","slug":"policy-improvement-by-planning-with-gumbel","title":"Policy improvement by planning with Gumbel","date":"2021-09-29","arxiv_id":null,"n_code_links":2,"syntology":null},{"paper":"/paper/prioritized-experience-based-reinforcement","slug":"prioritized-experience-based-reinforcement","title":"Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Driving","date":"2021-09-26","arxiv_id":"2109.12516","n_code_links":1,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-at-the-edge-of","slug":"deep-reinforcement-learning-at-the-edge-of","title":"Deep Reinforcement Learning at the Edge of the Statistical Precipice","date":"2021-08-30","arxiv_id":"2108.13264","n_code_links":3,"syntology":{"ran":5,"of":5,"n_ran_checked":5,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 4 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["google-research/rliable"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"optimal-scheduling-of-isolated-microgrids","title":"Optimal Scheduling of Isolated Microgrids Using Automated Reinforcement Learning-based Multi-period Forecasting","date":"2021-08-15","arxiv_id":"2108.06764","n_code_links":0,"syntology":null},{"paper":null,"slug":"value-based-reinforcement-learning-for","title":"Value-Based Reinforcement Learning for Continuous Control Robotic Manipulation in Multi-Task Sparse Reward Settings","date":"2021-07-28","arxiv_id":"2107.13356","n_code_links":0,"syntology":null},{"paper":"/paper/proper-value-equivalence","slug":"proper-value-equivalence","title":"Proper Value Equivalence","date":"2021-06-18","arxiv_id":"2106.10316","n_code_links":1,"syntology":null},{"paper":null,"slug":"vector-quantized-models-for-planning","title":"Vector Quantized Models for Planning","date":"2021-06-08","arxiv_id":"2106.04615","n_code_links":0,"syntology":null},{"paper":null,"slug":"optimistic-reinforcement-learning-by-forward","title":"Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization","date":"2021-05-27","arxiv_id":"2105.12991","n_code_links":0,"syntology":null},{"paper":null,"slug":"time-aware-q-networks-resolving-temporal","title":"Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning","date":"2021-05-06","arxiv_id":"2105.02580","n_code_links":0,"syntology":null},{"paper":"/paper/learning-and-planning-in-complex-action","slug":"learning-and-planning-in-complex-action","title":"Learning and Planning in Complex Action Spaces","date":"2021-04-13","arxiv_id":"2104.06303","n_code_links":1,"syntology":null},{"paper":"/paper/online-and-offline-reinforcement-learning-by","slug":"online-and-offline-reinforcement-learning-by","title":"Online and Offline Reinforcement Learning by Planning with a Learned Model","date":"2021-04-13","arxiv_id":"2104.06294","n_code_links":2,"syntology":null},{"paper":"/paper/a-coevolutionairy-approach-to-deep-multi","slug":"a-coevolutionairy-approach-to-deep-multi","title":"A coevolutionary approach to deep multi-agent reinforcement learning","date":"2021-04-12","arxiv_id":"2104.05610","n_code_links":1,"syntology":null},{"paper":null,"slug":"accelerating-online-reinforcement-learning-1","title":"Accelerating Online Reinforcement Learning via Model-Based Meta-Learning","date":"2021-03-15","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/visualizing-muzero-models","slug":"visualizing-muzero-models","title":"Visualizing MuZero Models","date":"2021-02-25","arxiv_id":"2102.12924","n_code_links":1,"syntology":null},{"paper":null,"slug":"combining-off-and-on-policy-training-in-model","title":"Combining Off and On-Policy Training in Model-Based Reinforcement Learning","date":"2021-02-24","arxiv_id":"2102.12194","n_code_links":0,"syntology":null},{"paper":"/paper/accelerated-sim-to-real-deep-reinforcement","slug":"accelerated-sim-to-real-deep-reinforcement","title":"Accelerated Sim-to-Real Deep Reinforcement Learning: Learning Collision Avoidance from Human Player","date":"2021-02-21","arxiv_id":"2102.10711","n_code_links":1,"syntology":null},{"paper":"/paper/improving-model-based-reinforcement-learning","slug":"improving-model-based-reinforcement-learning","title":"Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision","date":"2021-02-10","arxiv_id":"2102.05599","n_code_links":2,"syntology":null},{"paper":"/paper/revisiting-prioritized-experience-replay-a-1","slug":"revisiting-prioritized-experience-replay-a-1","title":"Revisiting Prioritized Experience Replay: A Value Perspective","date":"2021-02-05","arxiv_id":"2102.03261","n_code_links":2,"syntology":null},{"paper":null,"slug":"playing-atari-with-capsule-networks-a","title":"Playing Atari with Capsule Networks: A systematic comparison of CNN and CapsNets-based agents.","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"playing-nondeterministic-games-through","title":"Playing Nondeterministic Games through Planning with a Learned Model","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"weighted-bellman-backups-for-improved-signal","title":"Weighted Bellman Backups for Improved Signal-to-Noise in Q-Updates","date":"2021-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"reinforcement-learning-for-robust-missile","title":"Reinforcement Learning for Robust Missile Autopilot Design","date":"2020-11-26","arxiv_id":"2011.12956","n_code_links":0,"syntology":null},{"paper":"/paper/tonic-a-deep-reinforcement-learning-library","slug":"tonic-a-deep-reinforcement-learning-library","title":"Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking","date":"2020-11-15","arxiv_id":"2011.07537","n_code_links":1,"syntology":{"ran":3,"of":4,"n_ran_checked":0,"n_instrument":3,"unverified":1,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified","official":{"repos":["fabiopardo/tonic"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"critic-pi2-master-continuous-planning-via","title":"Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning","date":"2020-11-13","arxiv_id":"2011.06752","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-role-of-planning-in-model-based-deep-1","title":"On the role of planning in model-based deep reinforcement learning","date":"2020-11-08","arxiv_id":"2011.04021","n_code_links":0,"syntology":null},{"paper":null,"slug":"the-value-equivalence-principle-for-model","title":"The Value Equivalence Principle for Model-Based Reinforcement Learning","date":"2020-11-06","arxiv_id":"2011.03506","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":138,"with_a_code_link":61,"where_syntology_ran_a_sample":19,"not_listed_spam_title":0,"listed":138,"listed_where_code_ran":19,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":17,"every_run_a_failure_of_syntologys_instrument":2,"listed_with_a_run_with_no_instrument_failure":17,"listed_every_run_a_failure_of_syntologys_instrument":2,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":19,"papers_where_code_ran_first_page":"/method/prioritized-experience-replay/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":2,"url":"/method/prioritized-experience-replay/papers/2","json":"/method/prioritized-experience-replay/papers/2.json","rows":38},{"order":"ran","page":1,"of":1,"url":"/method/prioritized-experience-replay/papers/ran/1","json":"/method/prioritized-experience-replay/papers/ran/1.json","rows":19}],"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":72},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":68},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":60},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":41},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":20},{"task":"/task/atari-games","name":"Atari Games","papers":17},{"task":"/task/q-learning","name":"Q-Learning","papers":17},{"task":"/task/continuous-control","name":"Continuous Control","papers":15},{"task":"/task/continuous-control","name":"continuous-control","papers":14},{"task":"/task/decision-making","name":"Decision Making","papers":11},{"task":"/task/board-games","name":"Board Games","papers":8},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":8},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":7},{"task":"/task/game-of-go","name":"Game of Go","papers":6},{"task":"/task/mujoco","name":"MuJoCo","papers":5},{"task":"/task/offline-rl","name":"Offline RL","papers":5},{"task":"/task/starcraft","name":"Starcraft","papers":5},{"task":"/task/model","name":"model","papers":5},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":4},{"task":"/task/atari-games-100k","name":"Atari Games 100k","papers":3}],"tasks_shown":20,"n_tasks":82,"usage_by_year":[{"year":"2015","papers":1},{"year":"2017","papers":2},{"year":"2018","papers":8},{"year":"2019","papers":10},{"year":"2020","papers":22},{"year":"2021","papers":26},{"year":"2022","papers":21},{"year":"2023","papers":22},{"year":"2024","papers":17},{"year":"2025","papers":9}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/prioritized-experience-replay"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Prioritized%20Experience%20Replay","read_at":"2026-10-05T01:00:39+00:00","query":"Prioritized Experience Replay","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"PRIORITIZED EXPERIENCE REPLAY","origin_arxiv_id":"1511.05952"},"resolution":"case_insensitive","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"same_paper","archive_paper_arxiv_id":"1511.05952","repositories":{"listed":5,"in_graph":77,"list_truncated":true,"strongest_link":"mentioned_in_github"},"retrieve_with":null,"name_competition":{"verdict":"sole_bearer","may_be_wrong_sense":false,"bearers_found":1,"served":{"arxiv_id":"1511.05952","citations":4574},"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing else in this corpus is named 'Prioritized Experience Replay'. That is a statement about this corpus and not about the field: short names are reused, and a sense we never collected cannot show up here. Confirm `served.arxiv_id` if you had a specific paper in mind."},"name_competition_text":null,"near_matches":null,"code_nearby":[{"name":"Curious Replay","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Curious Replay')","origin_arxiv_id":"2306.15934"},{"name":"Dark Experience Replay","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Dark Experience Replay')","origin_arxiv_id":"2004.07211"},{"name":"a simple approach that learns a reward model from online experience","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='a simple approach that learns a reward model from online experience')","origin_arxiv_id":"2311.05067"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"1511.05952","page":"/paper/prioritized-experience-replay","syntology_url":"https://syntology.ai/paper/1511.05952","in_graph_layer":true,"samples":111,"ran":80,"n_ran_checked":72,"n_instrument":8,"unverified":31,"phrase":"80 ran (of which 62 constructed an object rather than computing a result; 72 with no instrument failure: 4 honoured, 0 violated, 68 with no contract checked; 8 where Syntology's instrument failed) · 31 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"repository known, no verified implementation (declared_code; repositories: 5 listed of 77 in the graph, strongest link: mentioned on GitHub). The lookup matched 'PRIORITIZED EXPERIENCE REPLAY' ignoring case. This answer is about arXiv:1511.05952, the paper the archive credits with this method.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"43cdfdfc1dec03bd51ef8081d60ec4491a0fb32a59630a60de4d2e5ebf35d25a","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}