Papers › Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions...

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

20 Nov 2017CVPR 2018 6arXiv:1711.07280archive 2025-07-28

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton Van Den Hengel

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matterport3D Simulator -- a large-scale reinforcement learning environment based on real imagery. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -- the Room-to-Room (R2R) dataset.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

peteanderson80/Matterport3DSimulator officialmentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with... (arXiv:2406.19236; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
MarSaKi/NvEM mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Neighbor-view Enhanced Model for Vision and Language Navigation (arXiv:2107.07201; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
YicongHong/Entity-Graph-VLN mentioned on GitHubpytorchSyntology: not harvested report
YicongHong/Recurrent-VLN-BERT mentioned on GitHubpytorchSyntology: not harvested report
batra-mlp-lab/vln-chasing-ghosts mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Chasing Ghosts: Instruction Following as Bayesian State Tracking (arXiv:1907.02022; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
batra-mlp-lab/vln-sim2real mentioned on GitHubpytorchSyntology: not harvested report
google-research-datasets/RxR mentioned on GitHubtfCC-BY-4.0Syntology: not harvested report
hlr/vln-trans mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Reinforcement LearningTranslationVision and Language NavigationVisual NavigationVisual Question AnsweringVisual Question Answering (VQA)

Datasets

Introduced by this paper, per the archive.

R2R

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Visual Navigation R2R Seq2Seq baseline spl 0.18 #11 of 11 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections