Methods › Reinforcement Learning › Board Game Models › MuZero
MuZero
Introduced by Julian Schrittwieser et al. in Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
MuZero is a model-based reinforcement learning algorithm. It builds upon AlphaZero's search and search-based policy iteration algorithms, but incorporates a learned model into the training procedure.
The main idea of the algorithm is to predict those aspects of the future that are directly relevant for planning. The model receives the observation (e.g. an image of the Go board or the Atari screen) as an input and transforms it into a hidden state. The hidden state is then updated iteratively by a recurrent process that receives the previous hidden state and a hypothetical next action. At every one of these steps the model predicts the policy (e.g. the move to play), value function (e.g. the predicted winner), and immediate reward (e.g. the points scored by playing a move). The model is trained end-to-end, with the sole objective of accurately estimating these three important quantities, so as to match the improved estimates of policy and value generated by search as well as the observed reward.
There is no direct constraint or requirement for the hidden state to capture all information necessary to reconstruct the original observation, drastically reducing the amount of information the model has to maintain and predict; nor is there any requirement for the hidden state to match the unknown, true state of the environment; nor any other constraints on the semantics of state. Instead, the hidden states are free to represent state in whatever way is relevant to predicting current and future values and policies. Intuitively, the agent can invent, internally, the rules or dynamics that lead to most accurate planning.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 18 in the graph, strongest link: mentioned on GitHub). This answer is about arXiv:1911.08265, the paper the archive credits with this method.
Origin paper of this answer, arXiv:1911.08265 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 43 ran (of which 36 constructed an object rather than computing a result; 43 with no instrument failure: 4 honoured, 0 violated, 39 with no contract checked; 0 where Syntology's instrument failed) · 21 unverified (of 64 harvested samples) · 62 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 46 · with a code link: 23 · where Syntology ran a sample: 8 (7 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (8 of 46 tagged: 7 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 46 of 46, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Calibrated Value-Aware Model Learning with Stochastic Environment Models 28 May 2025 · 0 repositories · arXiv:2505.22772
-
OptionZero: Planning with Learned Options 23 Feb 2025 · 1 repository · arXiv:2502.16634Syntology official (archive's flag): 7 ran · 7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 7 samples that ran constructed an object rather than computing a result (of 8 harvested samples) · 8 pointer-only (licence)
-
Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations 14 Feb 2025 · 0 repositories · arXiv:2502.10303
-
Evaluating World Models with LLM for Decision Making 13 Nov 2024 · 0 repositories · arXiv:2411.08794
-
Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping 7 Nov 2024 · 0 repositories · arXiv:2411.04915
-
Interpreting the Learned Model in MuZero Planning 7 Nov 2024 · 0 repositories · arXiv:2411.04580
-
Combining AI Control Systems and Human Decision Support via Robustness and Criticality 3 Jul 2024 · 0 repositories · arXiv:2407.03210
-
Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction 2 Jun 2024 · 1 repository · arXiv:2406.00614
-
Efficient Multi-agent Reinforcement Learning by Planning 20 May 2024 · 1 repository · arXiv:2405.11778
-
ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze 25 Apr 2024 · 1 repository · arXiv:2404.16364
-
MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Games 17 Oct 2023 · 1 repository · arXiv:2310.11305Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)
-
Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction 10 Oct 2023 · 0 repositories · arXiv:2310.06513
-
Self-Predictive Universal AI 21 Sep 2023 · 0 repositories
-
AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning 7 Aug 2023 · 1 repository · arXiv:2308.03526Syntology 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples)
-
λ-models: Effective Decision-Aware Reinforcement Learning with Latent Models 30 Jun 2023 · 0 repositories · arXiv:2306.17366
-
What model does MuZero learn? 1 Jun 2023 · 0 repositories · arXiv:2306.00840
-
Model Predictive Control with Self-supervised Representation Learning 14 Apr 2023 · 1 repository · arXiv:2304.07219
-
Equivariant MuZero 9 Feb 2023 · 0 repositories · arXiv:2302.04798
-
Epistemic Monte Carlo Tree Search 21 Oct 2022 · 0 repositories · arXiv:2210.13455
-
Efficient Offline Policy Optimization with a Learned Model 12 Oct 2022 · 1 repository · arXiv:2210.05980
-
Human-level Atari 200x faster 15 Sep 2022 · 1 repository · arXiv:2209.07550Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples)
-
Transformers are Sample-Efficient World Models 1 Sep 2022 · 2 repositories · arXiv:2209.00588Syntology official (archive's flag): 17 ran · 17 ran (of which 13 constructed an object rather than computing a result; 16 with no instrument failure: 1 honoured, 0 violated, 15 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 26 harvested samples) · 26 pointer-only (licence)
-
An AlphaZero-Inspired Approach to Solving Search Problems 2 Jul 2022 · 0 repositories · arXiv:2207.00919
-
Impartial Games: A Challenge for Reinforcement Learning 25 May 2022 · 1 repository · arXiv:2205.12787
-
Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods 25 Apr 2022 · 1 repository · arXiv:2204.11464
-
Reward Reports for Reinforcement Learning 22 Apr 2022 · 1 repository · arXiv:2204.10817
-
MuZero with Self-competition for Rate Control in VP9 Video Compression 14 Feb 2022 · 0 repositories · arXiv:2202.06626
-
Procedural Generalization by Planning with Self-Supervised World Models 2 Nov 2021 · 0 repositories · arXiv:2111.01587
-
Mastering Atari Games with Limited Data 30 Oct 2021 · 3 repositories · arXiv:2111.00210Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Planning in Stochastic Environments with a Learned Model 29 Sep 2021 · 2 repositories
-
Policy improvement by planning with Gumbel 29 Sep 2021 · 2 repositories
-
Proper Value Equivalence 18 Jun 2021 · 1 repository · arXiv:2106.10316
-
Vector Quantized Models for Planning 8 Jun 2021 · 0 repositories · arXiv:2106.04615
-
Learning and Planning in Complex Action Spaces 13 Apr 2021 · 1 repository · arXiv:2104.06303
-
Online and Offline Reinforcement Learning by Planning with a Learned Model 13 Apr 2021 · 2 repositories · arXiv:2104.06294
-
Accelerating Online Reinforcement Learning via Model-Based Meta-Learning 15 Mar 2021 · 0 repositories
-
Visualizing MuZero Models 25 Feb 2021 · 1 repository · arXiv:2102.12924
-
Combining Off and On-Policy Training in Model-Based Reinforcement Learning 24 Feb 2021 · 0 repositories · arXiv:2102.12194
-
Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision 10 Feb 2021 · 2 repositories · arXiv:2102.05599
-
Playing Nondeterministic Games through Planning with a Learned Model 1 Jan 2021 · 0 repositories
-
Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning 13 Nov 2020 · 0 repositories · arXiv:2011.06752
-
On the role of planning in model-based deep reinforcement learning 8 Nov 2020 · 0 repositories · arXiv:2011.04021
-
The Value Equivalence Principle for Model-Based Reinforcement Learning 6 Nov 2020 · 0 repositories · arXiv:2011.03506
-
The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning 7 Jul 2020 · 2 repositories · arXiv:2007.03158Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result (of 2 harvested samples)
-
Continuous Control for Searching and Planning with a Learned Model 12 Jun 2020 · 0 repositories · arXiv:2006.07430
-
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model 19 Nov 2019 · 18 repositories · arXiv:1911.08265Syntology 43 ran (of which 36 constructed an object rather than computing a result; 43 with no instrument failure: 4 honoured, 0 violated, 39 with no contract checked; 0 where Syntology's instrument failed) · 21 unverified (of 64 harvested samples) · 62 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 38 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections