Papers › NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist

15 May 2023arXiv:2305.08566archive 2025-07-28

Iftitahu Ni'mah, Meng Fang, Vlado Menkovski, Mykola Pechenizkiy

In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear. We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation. We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks. We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

inimah/metric-preference-checklist officialmentioned in papermentioned on GitHubpytorchSyntology: not harvested report
maszhongming/unieval mentioned in paperMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Towards a Unified Multi-Dimensional Evaluator for Text Generation (arXiv:2210.07197; that paper's own run record: 0 ran · 1 unverified) and RepEval: Effective Text Evaluation with LLM Representation (arXiv:2404.19563; that paper's own run record: 18 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 3 honoured, 0 violated, 6 with no contract checked; 9 where Syntology's instrument failed) · 9 unverified); nothing checks that they implement this paper's method. report
tanyuqian/ctc-gen-eval mentioned in paperpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural... (arXiv:2109.06379; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
thu-coai/ctrleval mentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (arXiv:2204.00862; that paper's own run record: 0 ran · 1 unverified); nothing checks that they implement this paper's method. report
huggingface/transformers pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
shikib/usr pytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RepEval: Effective Text Evaluation with LLM Representation (arXiv:2404.19563; that paper's own run record: 18 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 3 honoured, 0 violated, 6 with no contract checked; 9 where Syntology's instrument failed) · 9 unverified); nothing checks that they implement this paper's method. report
uber-research/PPLM pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Plug and Play Language Models: A Simple Approach to Controlled Text Generation (arXiv:1912.02164; that paper's own run record: 17 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 4 where Syntology's instrument failed) · 5 unverified) and Substance over Style: Document-Level Targeted Content Transfer (arXiv:2010.08618; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Controllable Language ModellingDialogue GenerationLanguage ModellingResponse GenerationText GenerationText Summarizationnlg evaluation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections