Papers › Unsupervised Quality Estimation for Neural Machine Translation

Unsupervised Quality Estimation for Neural Machine Translation

21 May 2020arXiv:2005.10608archive 2025-07-28

Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia

Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By employing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

facebookresearch/mlqe officialmentioned in paperCC-BY-SA-4.0Syntology: not harvested report
pytorch/fairseq officialmentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (arXiv:2105.08206; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result), Cascaded Head-colliding Attention (arXiv:2105.14850; that paper's own run record: 9 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified) and Modeling Dual Read/Write Paths for Simultaneous Machine Translation (arXiv:2203.09163; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified); nothing checks that they implement this paper's method. report
ShiYaya/Awesome_Evaluation_Metrics_for_Text_Generation mentioned on GitHubSyntology: not harvested report
iinemo/lm-polygraph pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Mutual Information Alleviates Hallucinations in Abstractive Summarization (arXiv:2210.13210; that paper's own run record: 10 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Machine TranslationTranslationUncertainty Quantification

Datasets

Introduced by this paper, per the archive.

MLQE

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections