Browse State-of-the-Art › Multimodal Machine Translation
Multimodal Machine Translation
38 papers with code · 3 benchmarks · 5 datasets archive 2025-07-28
Multimodal machine translation is the task of doing machine translation with multiple data sources - for example, translating "a bird is flying over water" + an image of a bird over water to German text.
( Image credit: Findings of the Third Shared Task on Multimodal Machine Translation )
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| Multi30K (15 rows) | ERNIE-UniX2 | ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for... | — | — | Compare |
| Hindi Visual Genome (Test Set) (1 row) | ViTA | ViTA: Visual-Linguistic Translation by Aligning Object Tags | code | — | Compare |
| Hindi Visual Genome (Challenge Set) (1 row) | ViTA | ViTA: Visual-Linguistic Translation by Aligning Object Tags | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
5 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
2 subtasks in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 108 · with a code link: 38 · where Syntology ran a sample: 7 (6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (7 of 108 tagged: 6 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 108, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
12 Jun 2017 595 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified (of 946 harvested samples) · 451 pointer-only (licence)The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration.
-
18 Jul 2024 2 repositories listedCurrent multimodal machine translation (MMT) systems rely on fully supervised data (i. e models are trained on sentences with their translations and accompanying images).
-
20 Dec 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images.
-
17 Mar 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models.
-
2 May 2016 2 repositories listedWe introduce the Multi30K dataset to stimulate multilingual multimodal research.
-
29 Apr 2024 1 repository listedExperimental results show that MMT models trained on our dataset exhibit a greater ability to exploit visual information than those trained on other MMT datasets.
-
8 Dec 2023 1 repository listedIn this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion.
-
31 Oct 2023 1 repository listedIn addition to the extensive training set, EVA contains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guaranteed helpful for disambiguation.
-
26 Oct 2023 1 repository listedThis paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete.
-
20 Oct 2023 1 repository listedMultimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation.
-
29 Aug 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 7 pointer-only (licence)Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability.
-
28 May 2023 1 repository listedThis paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language.
-
23 May 2023 1 repository listedWe also introduce two deliberately designed test sets to verify the necessity of visual information: Ambiguous with the presence of ambiguous words, and Unambiguous in which the text context is self-contained for…
-
20 May 2023 1 repository listedIn this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup, inference-time image-free UMMT, where the model is trained with source-text image pairs, and tested with only…
-
20 Dec 2022 1 repository listedTherefore, this paper correspondingly establishes new methods and new datasets for MMT.
-
10 Oct 2022 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 2 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)Thus, in this work, we introduce IKD-MMT, a novel MMT framework to support the image-free inference phase via an inversion knowledge distillation scheme.
-
31 May 2022 1 repository listed Syntology 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 2 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples) · 3 pointer-only (licence)In particular, given a source sentence an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and the combined text and hallucinated representations are…
-
19 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them…
-
28 Feb 2022 1 repository listedIn this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context.
-
20 Jan 2022 1 repository listedExisting multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate…
-
8 Sep 2021 1 repository listedMultimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available.
-
7 Jun 2021 1 repository listedWe present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pretrained models VL-BERT and M-BERT, respectively.
-
1 Jun 2021 1 repository listedWe find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image…
-
1 Jun 2021 1 repository listedMultimodal Machine Translation (MMT) enriches the source text with visual information for translation.
-
25 Jan 2021 1 repository listedPre-trained language models have been shown to improve performance in many natural language tasks substantially.
-
4 Sep 2020 1 repository listedParticularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities.
-
1 Jul 2020 1 repository listedMultimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality.
-
22 Jun 2020 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 3 pointer-only (licence)Hence, it can be interpreted within a framework of knowledge distillation as a student becomes a teacher itself.
-
4 Jun 2020 1 repository listedWe present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training.
-
18 Jun 2019 1 repository listedPrevious work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.
-
1 Apr 2019 1 repository listed
-
1 Nov 2018 1 repository listed
-
15 Oct 2018 1 repository listed
-
1 Oct 2018 1 repository listed
-
24 Aug 2018 1 repository listed
-
1 May 2018 1 repository listed
-
1 Jun 2017 1 repository listed
-
30 May 2016 1 repository listed
-
CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation30 May 2025 0 repositories listed
-
Multimodal Machine Translation with Visual Scene Graph Pruning26 May 2025 0 repositories listed
-
TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries9 May 2025 0 repositories listed
-
Florenz: Scaling Laws for Systematic Generalization in Vision-Language Models12 Mar 2025 0 repositories listed
-
Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation17 Dec 2024 0 repositories listed
-
EMMeTT: Efficient Multimodal Machine Translation Training20 Sep 2024 0 repositories listed
-
Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets9 Apr 2024 0 repositories listed
-
Adding Multimodal Capabilities to a Text-only Translation Model5 Mar 2024 0 repositories listed
-
Detecting Concrete Visual Tokens for Multimodal Machine Translation5 Mar 2024 0 repositories listed
-
The Case for Evaluating Multimodal Translation Models on Text Datasets5 Mar 2024 0 repositories listed
-
A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation12 Jun 2023 0 repositories listed
-
Iterative Adversarial Attack on Image-guided Story Ending Generation16 May 2023 0 repositories listed
-
Generalization algorithm of multimodal pre-training model based on graph-text self-supervised training16 Feb 2023 0 repositories listed
-
9 Nov 2022 0 repositories listed
-
LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation19 Oct 2022 0 repositories listed
-
Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective16 Oct 2022 0 repositories listed
-
Supervised Visual Attention for Simultaneous Multimodal Machine Translation23 Jan 2022 0 repositories listed
-
On Vision Features in Multimodal Machine Translation16 Nov 2021 0 repositories listed
-
Experiences of Adapting Multimodal Machine Translation Techniques for Hindi1 Sep 2021 0 repositories listed
-
Low Resource Multimodal Neural Machine Translation of English-Hindi in News Domain1 Sep 2021 0 repositories listed
-
Multimodal Neural Machine Translation System for English to Bengali1 Sep 2021 0 repositories listed
-
Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation1 Aug 2021 0 repositories listed
-
Rakuten’s Participation in WAT 2021: Examining the Effectiveness of Pre-trained Models for Multilingual and Multimodal Machine Translation1 Aug 2021 0 repositories listed
-
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation30 May 2021 0 repositories listed
-
16 Mar 2021 0 repositories listed
-
Good for Misconceived Reasons: Revisiting Neural Multimodal Machine Translation1 Jan 2021 0 repositories listed
-
Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding18 Dec 2020 0 repositories listed
-
MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish13 Dec 2020 0 repositories listed
-
TMU Japanese-English Multimodal Machine Translation System for WAT 20201 Dec 2020 0 repositories listed
-
21 Sep 2020 0 repositories listed
-
A Visually-Grounded Parallel Corpus with Phrase-to-Region Linking1 May 2020 0 repositories listed
-
Investigating the Decoders of Maximum Likelihood Sequence Models: A Look-ahead Approach8 Mar 2020 0 repositories listed
-
28 Nov 2019 0 repositories listed
-
Adaptive Fusion Techniques for Multimodal Data10 Nov 2019 0 repositories listed
-
Understanding the Effect of Textual Adversaries in Multimodal Machine Translation1 Nov 2019 0 repositories listed
-
Transformer-based Cascaded Multimodal Speech Translation29 Oct 2019 0 repositories listed
-
On Leveraging the Visual Modality for Neural Machine Translation7 Oct 2019 0 repositories listed
-
Probing Representations Learned by Multimodal Recurrent and Transformer Models29 Aug 2019 0 repositories listed
-
Multilingual Multimodal Machine Translation for Dravidian Languages utilizing Phonetic Transcription1 Aug 2019 0 repositories listed
-
21 Jul 2019 0 repositories listed
-
Grounded Word Sense Translation1 Jun 2019 0 repositories listed
-
Debiasing Word Embeddings Improves Multimodal Machine Translation24 May 2019 0 repositories listed
-
Probing the Need for Visual Context in Multimodal Machine Translation20 Mar 2019 0 repositories listed
-
Adversarial Evaluation of Multimodal Machine Translation1 Oct 2018 0 repositories listed
-
CUNI System for the WMT18 Multimodal Translation Task1 Oct 2018 0 repositories listed
-
1 Oct 2018 0 repositories listed
-
Input Combination Strategies for Multi-Source Transformer Decoder1 Oct 2018 0 repositories listed
-
Sheffield Submissions for WMT18 Multimodal Translation Shared Task1 Oct 2018 0 repositories listed
-
The AFRL-Ohio State WMT18 Multimodal System: Combining Visual with Traditional1 Oct 2018 0 repositories listed
-
Ensemble Sequence Level Training for Multimodal MT: OSU-Baidu WMT18 Multimodal Machine Translation System Report31 Aug 2018 0 repositories listed
-
The MeMAD Submission to the WMT18 Multimodal Translation Task31 Aug 2018 0 repositories listed
-
Doubly Attentive Transformer Machine Translation30 Jul 2018 0 repositories listed
-
Multimodal Machine Translation with Reinforcement Learning7 May 2018 0 repositories listed
-
1 Mar 2018 0 repositories listed
-
MultiNews: A Web collection of an Aligned Multimodal and Multilingual Corpus1 Nov 2017 0 repositories listed
-
Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description19 Oct 2017 0 repositories listed
-
OSU Multimodal Machine Translation System Report7 Oct 2017 0 repositories listed
-
Findings of the 2017 Conference on Machine Translation (WMT17)1 Sep 2017 0 repositories listed
-
Generating Image Descriptions using Multilingual Data1 Sep 2017 0 repositories listed
-
NICT-NAIST System for WMT17 Multimodal Translation Task1 Sep 2017 0 repositories listed
-
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation1 Sep 2017 0 repositories listed
-
The AFRL-OSU WMT17 Multimodal Translation System: An Image Processing Approach1 Sep 2017 0 repositories listed
Syntology lines on 8 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections