Papers › Order Matters: Sequence to sequence for sets
Order Matters: Sequence to sequence for sets
19 Nov 2015arXiv:1511.06391archive 2025-07-28
Oriol Vinyals, Samy Bengio, Manjunath Kudlur
Sequences have become first class citizens in supervised learning thanks to
the resurgence of recurrent neural networks. Many complex tasks that require
mapping from or to a sequence of observations can now be formulated with the
sequence-to-sequence (seq2seq) framework which employs the chain rule to
efficiently represent the joint probability of sequences. In many cases,
however, variable sized inputs and/or outputs might not be naturally expressed
as sequences. For instance, it is not clear how to input a set of numbers into
a model where the task is to sort them; similarly, we do not know how to
organize outputs when they correspond to random variables and the task is to
model their unknown joint probability. In this paper, we first show using
various examples that the order in which we organize input and/or output data
matters significantly when learning an underlying model. We then discuss an
extension of the seq2seq framework that goes beyond sequences and handles input
sets in a principled way. In addition, we propose a loss which, by searching
over possible orders during training, deals with the lack of structure of
output sets. We show empirical evidence of our claims regarding ordering, and
on the modifications to the seq2seq framework on benchmark language modeling
and parsing tasks, as well as two artificial tasks -- sorting numbers and
estimating the joint probability of unknown graphical models.
In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.
Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
For agents, on Syntology's MCP service (how to connect):
get_citation_path(paper_1="1511.06391", paper_2="…") with another paper's arXiv id or titleget_concepts_for_paper(arxiv_id="1511.06391")
Code
davidtangGT/MEGNET
mentioned on GitHubtfBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Relational inductive biases, deep learning, and graph networks (arXiv:1806.01261; that paper's own run record: 39 ran (of which 5 constructed an object rather than computing a result; 38 with no instrument failure: 4 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 12 unverified); nothing checks that they implement this paper's method. report
materialsvirtuallab/megnet
mentioned on GitHubtfBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (arXiv:1812.05055; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
pyg-team/pytorch_geometric
mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis (arXiv:2606.17022; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report
teddykoker/mpnn-for-quantum-chem
mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Neural Message Passing for Quantum Chemistry (arXiv:1704.01212; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
dmlc/dgl
pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to node2vec: Scalable Feature Learning for Networks (arXiv:1607.00653; that paper's own run record: 15 ran (of which 2 constructed an object rather than computing a result; 12 with no instrument failure: 2 honoured, 0 violated, 10 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified), Interaction Networks for Learning about Objects, Relations and Physics (arXiv:1612.00222; that paper's own run record: 6 ran (of which 6 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; every one of the 6 samples that ran constructed an object rather than computing a result), Junction Tree Variational Autoencoder for Molecular Graph Generation (arXiv:1802.04364; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result) and 3 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
pyg-team/pytorch_geometric
pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis (arXiv:2606.17022; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections