Papers › Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

28 Feb 2017ICML 2017 8arXiv:1702.08887archive 2025-07-28

Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H. S. Torr, Pushmeet Kohli, Shimon Whiteson

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

MUmarJaved/MultiAgent-Distributed-Reinforcement-Learning mentioned on GitHubtfSyntology: not harvested report
anonymous1234517/code mentioned on GitHubtfSyntology: not harvested report
cts198859/deeprl_dist mentioned on GitHubtfSyntology: not harvested report
cts198859/deeprl_network mentioned on GitHubtfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Multi-agent Reinforcement Learning for Networked System Control (arXiv:2004.01339; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 14 unverified); nothing checks that they implement this paper's method. report
dongchen06/macacc mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Multi-agent Reinforcement LearningQ-LearningReinforcement LearningReinforcement Learning (RL)Starcraftreinforcement-learning

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Experience ReplayQ-Learning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections