Papers › Does Summary Evaluation Survive Translation to Other Languages?

Does Summary Evaluation Survive Translation to Other Languages?

16 Sep 2021NAACL 2022 7arXiv:2109.08129archive 2025-07-28

Spencer Braun, Oleg Vasilyev, Neslihan Iskender, John Bohannon

The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.

PaperPDFConference PDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

primerai/primer-research mentioned on GitHubtfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Machine TranslationTranslation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections