Papers › Training Graph Neural Networks with 1000 Layers

Training Graph Neural Networks with 1000 Layers

14 Jun 2021arXiv:2106.07476archive 2025-07-28

Guohao Li, Matthias Müller, Bernard Ghanem, Vladlen Koltun

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of 87.74 ±0.13 and 88.24 ±0.15 on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude. Please visit our project website https://www.deepgcns.org/arch/gnn1000 for more information.

PaperPDFCodeCode Syntology ran

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 11 code samples harvested from 1 repository linked to this paper: 11 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified. Where Syntology's instrument failed: 1 where our contract draft was wrong; failures of Syntology's instrument, not of the code.

By repository: official repository: 11 samples from 1 repository, 11 ran (10 with no instrument failure; 1 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

Levels written between 2026-09-22 and 2026-09-26 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed.

lightaime/deep_gcns_torch officialpytorchSyntology: 11 samples, 11 ran (1 instrument-failed) · 0 unverified report
lightaime/deep_gcns_torch mentioned on GitHubpytorchMITSyntology: 11 samples, 11 ran (1 instrument-failed) · 0 unverified report
lightaime/deep_gcns_torch mentioned on GitHubpytorchSyntology: 11 samples, 11 ran (1 instrument-failed) · 0 unverified report
lightaime/deep_gcns_torch mentioned on GitHubpytorchSyntology: 11 samples, 11 ran (1 instrument-failed) · 0 unverified report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

11 samples harvested: 11 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

1ran · our draft was wrong
10ran

Licence: 1 of the 11 samples is pointer only, meaning Syntology does not serve that copy's text. For that sample, this paper's copy is under a permissive licence and cleared, but the stored copy of the same code that Syntology serves is not; the licence label's tooltip gives Syntology's recorded reason. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from lightaime/deep_gcns_torch. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.

lightaime/deep_gcns_torch official repository 11 samples, 11 ran (1 instrument-failed) · 0 unverified
act_layer in 20 papers gcn_lib/dense/torch_nn.py ran MIT (permissive) · 81d31aad45664fa2 · report fixture: signature act:str/0/str, inplace:bool/0/bool, neg_slope:float/0/float, n_prelu:int/0/int · returned ReLU · level recorded 2026-09-25
act_layer in 20 papers gcn_lib/sparse/torch_nn.py ran MIT (permissive) · 6b65956f32ad8f13 · report fixture: signature act_type:str/0/str, inplace:bool/0/bool, neg_slope:float/0/float, n_prelu:int/0/int · returned ReLU · level recorded 2026-09-26
batched_index_select in 44 papers gcn_lib/dense/torch_nn.py ran MIT (permissive) · c52260211d1aa99f · report fixture: signature x:Tensor/4/float, idx:Tensor/3/int · returned Tensor · level recorded 2026-09-25
dense_knn_matrix gcn_lib/dense/torch_edge.py ran MIT (permissive) · 7bc52d942ecf4430 · report fixture: signature x:Tensor/4/float, k:int/0/int · returned Tensor · level recorded 2026-09-25
knn_graph_matrix gcn_lib/sparse/torch_edge.py ran MIT (permissive) · c69d40b71d37d2d2 · report fixture: signature x:Tensor/2/float, k:int/0/int, batch:Tensor/1/int · returned Tensor · level recorded 2026-09-25
knn_matrix in 5 papers gcn_lib/sparse/torch_edge.py ran fingerprinted MIT (permissive) · 2f0e8668c137d3aa · report fixture: signature x:Tensor/3/float, k:int/0/int, batch:NoneType/0/? · returned tuple · level recorded 2026-09-25
load_pretrained_models utils/ckpt_util.py ran MIT (permissive) · 3be91b595771f70b · report fixture: signature model:callable/0/?, pretrained_model:str/0/str, phase:str/0/str, ismax:bool/0/bool · returned tuple · level recorded 2026-09-25
load_pretrained_optimizer utils/ckpt_util.py ran MIT (permissive) · c845b58421ba1367 · report fixture: signature pretrained_model:str/0/str, optimizer:Adam/0/?, scheduler:StepLR/0/?, lr:float/0/float, use_ckpt_lr:bool/0/bool · returned tuple · level recorded 2026-09-25
norm_layer in 17 papers gcn_lib/dense/torch_nn.py ran · our draft was wrong MIT (permissive) here; the stored copy is pointer only · 09e6ddb17c0adf28 · report fixture: signature norm:str/0/str, nc:int/0/int · returned BatchNorm2d · level recorded 2026-09-22
norm_layer in 17 papers gcn_lib/sparse/torch_nn.py ran MIT (permissive) · 34c1fc07af3538f8 · report fixture: signature norm_type:str/0/str, nc:int/0/int · returned BatchNorm1d · level recorded 2026-09-25
pairwise_distance in 55 papers
census: 6 implementations from 7 papers, 5 distinct outputs across 3 buckets, one shared input per bucket
gcn_lib/dense/torch_edge.py ran fingerprinted MIT (permissive) · bc7cc66599be9207 · report fixture: signature x:Tensor/3/float · returned Tensor · level recorded 2026-09-25

Tasks

Graph SamplingNode Property Prediction

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Node Property Prediction ogbn-arxiv RevGAT+N.Adj+LabelReuse+SelfKD Ext. data No #22 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+N.Adj+LabelReuse+SelfKD Number of params 2098256 #22 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+N.Adj+LabelReuse+SelfKD Test Accuracy 0.7426 ± 0.0017 #22 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+N.Adj+LabelReuse+SelfKD Validation Accuracy 0.7497 ± 0.0008 #22 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+NormAdj+LabelReuse Ext. data No #29 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+NormAdj+LabelReuse Number of params 2098256 #29 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+NormAdj+LabelReuse Test Accuracy 0.7402 ± 0.0018 #29 of 86 Archive leaderboard report
Node Property Prediction ogbn-arxiv RevGAT+NormAdj+LabelReuse Validation Accuracy 0.7501 ± 0.0010 #29 of 86 Archive leaderboard report
Node Property Prediction ogbn-products RevGNN-112 Ext. data No #30 of 64 Archive leaderboard report
Node Property Prediction ogbn-products RevGNN-112 Number of params 2945007 #30 of 64 Archive leaderboard report
Node Property Prediction ogbn-products RevGNN-112 Test Accuracy 0.8307 ± 0.0030 #30 of 64 Archive leaderboard report
Node Property Prediction ogbn-products RevGNN-112 Validation Accuracy 0.9290 ± 0.0007 #30 of 64 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Wide Ext. data No #4 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Wide Number of params 68471608 #4 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Wide Test ROC-AUC 0.8824 ± 0.0015 #4 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Wide Validation ROC-AUC 0.9450 ± 0.0008 #4 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Deep Ext. data No #6 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Deep Number of params 20031384 #6 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Deep Test ROC-AUC 0.8774 ± 0.0013 #6 of 26 Archive leaderboard report
Node Property Prediction ogbn-proteins RevGNN-Deep Validation ROC-AUC 0.9326 ± 0.0006 #6 of 26 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections