Papers › BERTScore: Evaluating Text Generation with BERT

BERTScore: Evaluating Text Generation with BERT

21 Apr 2019ICLR 2020 1arXiv:1904.09675archive 2025-07-28

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

PaperPDFConference PDFCodeCode Syntology ranMethod as code

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 76 code samples harvested from 8 repositories linked to this paper: 44 ran (of which 4 constructed an object rather than computing a result; 10 with no instrument failure: 3 honoured, 2 violated, 5 with no contract checked; 34 where Syntology's instrument failed) · 32 unverified. Where Syntology's instrument failed: 26 where our contract draft was wrong; 8 where the contract check could not run; failures of Syntology's instrument, not of the code.

By repository: official repository: 10 samples from 1 repository, 6 ran (0 with no instrument failure; 6 where Syntology's instrument failed); community (archive-listed): 66 samples from 7 repositories, 38 ran (10 with no instrument failure; 28 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

No sample is linked to this paper from 4 of its repositories that Syntology harvested for other papers:

Levels written on 2026-09-22 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed.

20 repositories listed; official and paper-mentioned ones first.

Tiiiger/bert_score officialmentioned in papermentioned on GitHubpytorchSyntology: 10 samples, 6 ran (6 instrument-failed) · 4 unverified report
ShiYaya/Awesome_Evaluation_Metrics_for_Text_Generation mentioned on GitHubSyntology: not harvested report
THUDM/KOBE mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to The Curious Case of Neural Text Degeneration (arXiv:1904.09751; that paper's own run record: 10 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report
THUcqb/KOBE mentioned on GitHubpytorchMITSyntology: not harvested report
allenai/mslr-shared-task mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to MS2: Multi-Document Summarization of Medical Studies (arXiv:2104.06486; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
ananyacoder/mee_wmt2021 mentioned on GitHubSyntology: not harvested report
cyr19/reproducibility mentioned on GitHubpytorchSyntology: 13 samples, 5 ran (5 instrument-failed) · 8 unverified report
danieldeutsch/bert_score_content_analysis mentioned on GitHubpytorchSyntology: 9 samples, 5 ran (5 instrument-failed) · 4 unverified report
facebookresearch/vizseq mentioned on GitHubSyntology: 10 samples, 7 ran (1 instrument-failed) · 3 unverified report
gotutiyan/gec-metrics mentioned on GitHubMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different... (arXiv:2502.09416; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
jonas-becker/text-generation mentioned on GitHubSyntology: not harvested report
lovit/KoBERTScore mentioned on GitHubpytorchSyntology: not harvested report
lovit/ko-BERTScore mentioned on GitHubpytorchSyntology: 13 samples, 9 ran (8 instrument-failed) · 4 unverified report
neulab/code-bert-score mentioned on GitHubpytorchSyntology: 14 samples, 7 ran (5 instrument-failed) · 7 unverified report
open-finance-lab/finlora mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasets (arXiv:2505.19819; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
qibinc/KOBE mentioned on GitHubpytorchMITSyntology: not harvested report
sebiancoder/bertscorevisualizer mentioned on GitHubSyntology: 1 sample, 1 ran (0 instrument-failed) · 0 unverified report
stair-lab/villm-eval mentioned on GitHubpytorchMITSyntology: not harvested report
yangletliu/finlora mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
yuhui-zh15/nlg_metrics mentioned on GitHubpytorchSyntology: 6 samples, 4 ran (4 instrument-failed) · 2 unverified report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

76 samples harvested: 44 ran (of which 4 constructed an object rather than computing a result; 10 with no instrument failure: 3 honoured, 2 violated, 5 with no contract checked; 34 where Syntology's instrument failed) · 32 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

3ran · honoured contract
2ran · violated contract
26ran · our draft was wrong
8ran; the contract check could not run
5ran
32unverified

Licence: 33 of the 76 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from 8 repositories linked to this paper, official or community; the list below is grouped by repository, each group headed by its name and what it is to the paper. 1 sample whose link to this paper records no repository is listed under one of this paper's own repositories, because identical code was harvested from it for another paper. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity.

Tiiiger/bert_score official repository 10 samples, 6 ran (6 instrument-failed) · 4 unverified
collate_idf bert_score/scorer.py ran · our draft was wrong MIT (permissive) · e1297e386354025c · report fixture: signature arr:list/1/str, tokenizer:MockTokenizer/0/?, idf_dict:dict/0/?, device:str/0/str · returned tuple · level recorded 2026-09-22
get_bert_embedding bert_score/scorer.py ran; the contract check could not run (RAISES) MIT (permissive) · 779e3927681cb63b · report fixture: signature all_sens:list/1/str, model:callable/0/?, tokenizer:MinimalTokenizer/0/?, idf_dict:dict/0/?, batch_size:int/0/int, device:str/0/str, all_layers:bool/0/bool · returned tuple · level recorded 2026-09-22
get_hash in 17 papers bert_score/scorer.py ran · our draft was wrong fingerprinted MIT (permissive) · 7a9d43b6fa2eafef · report fixture: signature model:str/0/str, num_layers:int/0/int, idf:bool/0/bool, rescale_with_baseline:bool/0/bool, use_custom_baseline:bool/0/bool, use_fast_tokenizer:bool/0/bool · returned str · level recorded 2026-09-22
get_tokenizer in 156 papers bert_score/scorer.py ran; the contract check could not run (TIMEOUT) MIT (permissive) · 0a3852d45d725a39 · report fixture: signature model_type:str/0/str, use_fast:bool/0/bool · returned MagicMock · level recorded 2026-09-22
process in 60 papers bert_score/scorer.py ran · our draft was wrong MIT (permissive) · 6fb229c8407fb6c8 · report fixture: signature a:str/0/str, tokenizer:MockTokenizer/0/? · returned set · level recorded 2026-09-22
sent_encode bert_score/scorer.py ran; the contract check could not run (TIMEOUT) MIT (permissive) · 37d4a40315d05e5e · report fixture: signature tokenizer:MockTokenizer/0/?, sent:str/0/str · returned list · level recorded 2026-09-22
BERTScorer bert_score/scorer.py unverified MIT (permissive) · 200a19a9f1229034 · report
bert_cos_score_idf bert_score/scorer.py unverified MIT (permissive) · 6a5eab3261e5500c · report
get_idf_dict bert_score/scorer.py unverified MIT (permissive) · edd1054d8a4511eb · report
get_model in 368 papers bert_score/scorer.py unverified MIT (permissive) · 271285be2338c8f6 · report
cyr19/reproducibility community (archive-listed) 13 samples, 5 ran (5 instrument-failed) · 8 unverified
bert_encode bert_score/scorer.py ran · our draft was wrong no licence file found · pointer only · e41bf854cd7b9a83 · report fixture: signature model:callable/0/?, x:Tensor/2/int, attention_mask:Tensor/2/int, all_layers:bool/0/bool · returned Tensor · level recorded 2026-09-22
cache_scibert bert_score/scorer.py ran · our draft was wrong fingerprinted no licence file found · pointer only · be27ce41f9ea0d1f · report fixture: signature model_type:str/0/str, cache_folder:str/0/str · returned str · level recorded 2026-09-22
collate_idf bert_score/scorer.py ran · our draft was wrong no licence file found · pointer only · 80aa6d93e1e6196f · report fixture: signature arr:list/1/str, tokenizer:MockTokenizer/0/?, idf_dict:dict/0/?, device:str/0/str · returned tuple · level recorded 2026-09-22
get_bert_embedding bert_score/scorer.py ran · our draft was wrong no licence file found · pointer only · e8c19548068c18c0 · report fixture: signature all_sens:list/1/str, model:callable/0/?, tokenizer:MinimalTokenizer/0/?, idf_dict:dict/0/?, batch_size:int/0/int, device:str/0/str, all_layers:bool/0/bool · returned tuple · level recorded 2026-09-22
process in 60 papers bert_score/scorer.py ran · our draft was wrong no licence file found · pointer only · 1d48904fd63d4a59 · report fixture: signature a:str/0/str, tokenizer:MockTokenizer/0/? · returned set · level recorded 2026-09-22
BERTScorer bert_score/scorer.py unverified no licence file found · pointer only · 0dbd601cded6f834 · report
bert_cos_score_idf bert_score/scorer.py unverified no licence file found · pointer only · e2471a1206f6d191 · report
get_hash in 17 papers bert_score/scorer.py unverified no licence file found · pointer only · c14c6317a3d158ea · report
get_idf_dict bert_score/scorer.py unverified no licence file found · pointer only · ab46123387327340 · report
get_model in 368 papers bert_score/scorer.py unverified no licence file found · pointer only · e655efd76f64a7aa · report
get_tokenizer in 156 papers bert_score/scorer.py unverified no licence file found · pointer only · 8d08178958f5f20d · report
greedy_cos_idf bert_score/scorer.py unverified no licence file found · pointer only · 227909002c4f2aba · report
sent_encode bert_score/scorer.py unverified no licence file found · pointer only · 7aa792f0b1258927 · report
danieldeutsch/bert_score_content_analysis community (archive-listed) 9 samples, 5 ran (5 instrument-failed) · 4 unverified
collate_idf bert_score/utils.py ran · our draft was wrong MIT (permissive) · 1d15c841be27109a · report fixture: signature arr:list/1/str, tokenizer:MinimalTokenizer/0/?, idf_dict:dict/0/?, device:str/0/str · returned tuple · level recorded 2026-09-22
get_bert_embedding bert_score/utils.py ran · our draft was wrong MIT (permissive) · 60a9029618bff133 · report fixture: signature all_sens:list/1/str, model:callable/0/?, tokenizer:MinimalTokenizer/0/?, idf_dict:dict/0/?, batch_size:int/0/int, device:str/0/str, all_layers:bool/0/bool · returned tuple · level recorded 2026-09-22
get_precision_token_alignments bert_score/utils.py ran · our draft was wrong MIT (permissive) · e93b135d3cd1c1ec · report fixture: signature ref_subwords_list:list/1/list, hyp_subwords_list:list/1/list, indices:Tensor/2/int, masks:Tensor/3/float, scores:Tensor/2/float, token_weights:Tensor/2/float · returned tuple · level recorded 2026-09-22
get_recall_token_alignments bert_score/utils.py ran · our draft was wrong MIT (permissive) · 0dc67c4f397a1eba · report fixture: signature ref_subwords_list:list/1/list, hyp_subwords_list:list/1/list, indices:Tensor/2/int, masks:Tensor/3/float, scores:Tensor/2/float, token_weights:Tensor/2/float · returned tuple · level recorded 2026-09-22
map_subword_index_to_token_index bert_score/utils.py ran · our draft was wrong MIT (permissive) · 47078bc5a6336435 · report fixture: signature subwords:list/1/str · returned dict · level recorded 2026-09-22
bert_cos_score_idf bert_score/utils.py unverified MIT (permissive) · c3818aacfecb9596 · report
bert_encode bert_score/utils.py unverified MIT (permissive) · 841ac1a3bf04a266 · report
greedy_cos_idf bert_score/utils.py unverified MIT (permissive) · 72ab4ee7aef7a2d7 · report
sent_encode bert_score/utils.py unverified MIT (permissive) · 5578268ced9f26ff · report
facebookresearch/vizseq community (archive-listed) 10 samples, 7 ran (1 instrument-failed) · 3 unverified
BERTScoreScorer vizseq/scorers/bert_score.py ran MIT (permissive) · c8edeba2ca8fea59 · report fixture: returned BERTScoreScorer · level recorded 2026-09-22
VizSeqScore vizseq/scorers/bert_score.py ran · metamorphic tier: deterministic MIT (permissive) · d7146a26ef592a40 · report fixture: returned VizSeqScore · level recorded 2026-09-22
VizSeqScorer vizseq/scorers/bert_score.py ran MIT (permissive) · 330236fc4b93abc7 · report fixture: returned VizSeqScorer · level recorded 2026-09-22
_available_cpu_count vizseq/scorers/bert_score.py ran · metamorphic tier: well formed MIT (permissive) · 034176029000712f · report fixture: returned int · level recorded 2026-09-22
_batch vizseq/scorers/bert_score.py ran · our draft was wrong MIT (permissive) · d05d990ceb06d2c9 · report fixture: signature a_list:list/1/int, n_batches:int/0/int · returned generator · level recorded 2026-09-22
_max_workers_for_platform vizseq/scorers/bert_score.py ran · honoured contract fingerprinted MIT (permissive) · ee17bb43374f3ef2 · report fixture: signature n_cpus:int/0/int, platform:str/0/str · returned int · level recorded 2026-09-22
map_optional vizseq/scorers/bert_score.py ran · honoured contract MIT (permissive) · e175c125be8d278b · report fixture: signature obj:int/0/int, map_fn:callable/0/? · returned int · level recorded 2026-09-22
_cgroup_cpu_quota vizseq/scorers/bert_score.py unverified MIT (permissive) · 2d6d94d1c61edf11 · report
_get_bert_scorer vizseq/scorers/bert_score.py unverified MIT (permissive) · c2dc3fa5dcf4e828 · report
register_scorer vizseq/scorers/bert_score.py unverified MIT (permissive) · b1d1743a9e193dd9 · report
lovit/ko-BERTScore community (archive-listed) 13 samples, 9 ran (8 instrument-failed) · 4 unverified
apply_idf KoBERTScore/score.py ran · our draft was wrong no licence file found · pointer only · 95e240faa9326986 · report fixture: signature ids:Tensor/2/int, idf_embed:callable/0/? · returned Tensor · level recorded 2026-09-22
bert_forwarding KoBERTScore/score.py ran · our draft was wrong no licence file found · pointer only · 369342a5449a0bfd · report fixture: signature bert_model:callable/0/?, input_ids:Tensor/2/int, attention_mask:Tensor/2/int, output_layer_index:int/0/int · returned Tensor · level recorded 2026-09-22
compute_RPF KoBERTScore/score.py ran; the contract check could not run (RAISES) · metamorphic tier: invariant no licence file found · pointer only · 9cc9523dfc879e70 · report fixture: signature refer_embeds:Tensor/3/float, candi_embeds:Tensor/3/float, refer_weight_mask:Tensor/2/float, candi_weight_mask:Tensor/2/float, refer_ids:Tensor/2/int, candi_ids:Tensor/2/int, idf:callable/0/?, rescale_base:float/0/float · returned tuple · level recorded 2026-09-22
compute_pairwise_cosine KoBERTScore/score.py ran; the contract check could not run (RAISES) · metamorphic tier: invariant fingerprinted no licence file found · pointer only · a6fb6a679479b806 · report fixture: signature refer_embeds:Tensor/3/float, candi_embeds:Tensor/3/float · returned Tensor · level recorded 2026-09-22
idf_numpy_to_embed KoBERTScore/score.py ran; the contract check could not run (RAISES) · metamorphic tier: deterministic no licence file found · pointer only · 9ccab64bf97fe100 · report fixture: signature idf_array:ndarray/1/float · returned Embedding · level recorded 2026-09-22
load_idf KoBERTScore/score.py ran · our draft was wrong no licence file found · pointer only · 89c6cff022037bf3 · report fixture: signature path:NoneType/0/?, tokenizer:MinimalTokenizer/0/? · returned Embedding · level recorded 2026-09-22
rescaling KoBERTScore/score.py ran · violated contract fingerprinted no licence file found · pointer only · 2b21ab2f6f331acc · report fixture: signature scores:Tensor/2/float, base:float/0/float · returned Tensor · level recorded 2026-09-22
sents_to_tensor KoBERTScore/score.py ran; the contract check could not run (TIMEOUT) · metamorphic tier: well formed no licence file found · pointer only · 58daeffb3ffb38df · report fixture: returned tuple · level recorded 2026-09-22
truncate_bert_layers KoBERTScore/score.py ran · our draft was wrong no licence file found · pointer only · 5bf3a9aaa9676734 · report fixture: signature encoder:callable/0/?, last_layer:int/0/int · returned BertModel · level recorded 2026-09-22
BERTScore KoBERTScore/score.py unverified no licence file found · pointer only · 573bfd9ac94c99ed · report
bert_score KoBERTScore/score.py unverified no licence file found · pointer only · 011256828cf4ffa5 · report
load_model in 479 papers KoBERTScore/score.py unverified no licence file found · pointer only · 3647b7dd8c5ac0d7 · report
train_idf KoBERTScore/score.py unverified no licence file found · pointer only · f61640ae3eacc0ce · report
neulab/code-bert-score community (archive-listed) 14 samples, 7 ran (5 instrument-failed) · 7 unverified
bert_encode code_bert_score/scorer.py ran · our draft was wrong MIT (permissive) · 949db13a96d3ef67 · report fixture: signature model:callable/0/?, x:Tensor/2/int, attention_mask:Tensor/2/int, tokenizer:MockTokenizer/0/?, all_layers:bool/0/bool, chunk_overlap:float/0/float · returned Tensor · level recorded 2026-09-22
collate_idf code_bert_score/scorer.py ran · our draft was wrong MIT (permissive) · 84a478e9730d32cf · report fixture: signature arr:list/1/str, tokenizer:MockTokenizer/0/?, idf_dict:dict/0/?, device:str/0/str · returned tuple · level recorded 2026-09-22
get_bert_embedding code_bert_score/scorer.py ran · our draft was wrong MIT (permissive) · 1eebdf77ab9a0ecd · report fixture: signature all_sens:list/1/str, model:callable/0/?, tokenizer:MockTokenizer/0/?, idf_dict:dict/0/?, batch_size:int/0/int, device:str/0/str, all_layers:bool/0/bool · returned tuple · level recorded 2026-09-22
get_hash in 17 papers code_bert_score/scorer.py ran · our draft was wrong MIT (permissive) · f982fdf447668cbb · report fixture: signature model:callable/0/?, num_layers:int/0/int, idf:bool/0/bool, rescale_with_baseline:bool/0/bool, use_custom_baseline:bool/0/bool · returned str · level recorded 2026-09-22
get_window_indices code_bert_score/scorer.py ran · honoured contract fingerprinted MIT (permissive) · ea4088dfeb479c9a · report fixture: signature total_seq_length:int/0/int, encoder_max_length:int/0/int, chunk_overlap:float/0/float · returned list · level recorded 2026-09-22
mark_all_punc_tokens code_bert_score/scorer.py ran · violated contract MIT (permissive) · aaf8f9d22087eb72 · report fixture: signature tokens:list/1/str · returned Tensor · level recorded 2026-09-22
process in 60 papers code_bert_score/scorer.py ran; the contract check could not run (TIMEOUT) MIT (permissive) · 5a7c2e48dc108411 · report fixture: signature a:str/0/str, tokenizer:MinimalTokenizer/0/? · returned set · level recorded 2026-09-22
BERTScorer code_bert_score/scorer.py unverified MIT (permissive) · 364dacc2df955455 · report
bert_cos_score_idf code_bert_score/scorer.py unverified MIT (permissive) · a6d2976f556bf53f · report
cache_scibert code_bert_score/scorer.py unverified MIT (permissive) · ffd1e746cb94818e · report
get_idf_dict code_bert_score/scorer.py unverified MIT (permissive) · a1d85069694206d2 · report
get_model in 368 papers code_bert_score/scorer.py unverified MIT (permissive) · 7e819d1873379c18 · report
greedy_cos_idf code_bert_score/scorer.py unverified MIT (permissive) · a85884cc1bdc6201 · report
sent_encode code_bert_score/scorer.py unverified MIT (permissive) · e5fcbbe27d124542 · report
sebiancoder/bertscorevisualizer community (archive-listed) 1 sample, 1 ran (0 instrument-failed) · 0 unverified
BERTScorer backend/BERTScorer/bertscorer.py ran licence not identified · pointer only · 41873629d485d64d · report fixture: returned BERTScorer · level recorded 2026-09-22
yuhui-zh15/nlg_metrics community (archive-listed) 6 samples, 4 ran (4 instrument-failed) · 2 unverified
bert_encode nlg_metrics/bertscore/bert_util.py ran · our draft was wrong no licence file found · pointer only · 29119b8712ceae26 · report fixture: signature model:callable/0/?, x:Tensor/2/int, attention_mask:Tensor/2/int, all_layers:bool/0/bool · returned Tensor · level recorded 2026-09-22
collate_idf nlg_metrics/bertscore/bert_util.py ran · our draft was wrong no licence file found · pointer only · a674ab62cada9f02 · report fixture: signature arr:list/1/str, tokenizer:MockTokenizer/0/?, idf_dict:dict/0/?, device:str/0/str · returned tuple · level recorded 2026-09-22
get_bert_embedding nlg_metrics/bertscore/bert_util.py ran · our draft was wrong no licence file found · pointer only · 466fbc555ce2478f · report fixture: signature all_sens:list/1/str, model:callable/0/?, tokenizer:MinimalTokenizer/0/?, idf_dict:dict/0/?, batch_size:int/0/int, device:str/0/str, all_layers:bool/0/bool · returned tuple · level recorded 2026-09-22
padding in 37 papers nlg_metrics/moverscore/moverscore.py identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own ran · our draft was wrong licence of this copy not recorded · 13ef8eeae8570519 · report fixture: signature arr:list/1/list, pad_token:int/0/int, dtype:dtype/0/? · returned tuple · level recorded 2026-09-22
bert_cos_score_idf nlg_metrics/bertscore/bert_util.py unverified no licence file found · pointer only · 74c9dc5efff4ab40 · report
greedy_cos_idf nlg_metrics/bertscore/bert_util.py unverified no licence file found · pointer only · 0b727a24b15c0d5e · report

Method as code Syntology

What Syntology's graph records as this paper's claimed method, lifted into code: each assembly names its bricks (functions and classes), each with its own role and its own verification level. An assembly is not a check that the code implements the method correctly or reproduces the paper's results. Levels are each brick's own verification level in Syntology's graph (level 0: no recorded run), listed brick by brick and never combined into one level; “ran” means the brick executed on a synthesized input, not that its output is correct, and the instrument figure counts failures of Syntology's instrument, not of the code. Where a model read the code against the paper's claim, each assembly's label verdict is one model's judgement, kept in this page's JSON twin; it is not calibrated, so no card prints it. Siblings are assemblies lifted from this paper in the same batch for the same method: alternative readings kept side by side; none replaces or supersedes another.

8 assemblies on this paper: 8 harvested, 0 generated, listed in 1 groups (siblings lifted in one batch form one group).

8 sibling assemblies for BERTSCORE (harvested), lifted in the same batch on 2026-09-09: alternative readings kept side by side; none replaces or supersedes another.

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 10 bricks; individually on a fixture: 7 ran (of which 3 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScoreScorer entry · level 1 · ran · c8edeba2ca8fea59
  2. VizSeqScore helper · level 1 · ran · d7146a26ef592a40
  3. VizSeqScorer helper · level 1 · ran · 330236fc4b93abc7
  4. _get_bert_scorer helper · level 0 · unverified · c2dc3fa5dcf4e828
  5. register_scorer helper · level 0 · unverified · b1d1743a9e193dd9
  6. map_optional helper · level 1 · ran · honoured contract · e175c125be8d278b
  7. _available_cpu_count helper · level 1 · ran · 034176029000712f
  8. _batch helper · level 1 · ran · our draft was wrong · d05d990ceb06d2c9
  9. _max_workers_for_platform helper · level 1 · ran · honoured contract · ee17bb43374f3ef2
  10. _cgroup_cpu_quota helper · level 0 · unverified · 2d6d94d1c61edf11

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 14 bricks; individually on a fixture: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 8 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScorer entry · level 0 · unverified · 0dbd601cded6f834
  2. bert_cos_score_idf metric · level 0 · unverified · e2471a1206f6d191
  3. get_bert_embedding helper · level 1 · ran · our draft was wrong · e8c19548068c18c0
  4. get_hash helper · level 0 · unverified · c14c6317a3d158ea
  5. get_idf_dict helper · level 0 · unverified · ab46123387327340
  6. get_model helper · level 0 · unverified · e655efd76f64a7aa
  7. get_tokenizer helper · level 0 · unverified · 8d08178958f5f20d
  8. sent_encode helper · level 0 · unverified · 7aa792f0b1258927
  9. greedy_cos_idf metric · level 0 · unverified · 227909002c4f2aba
  10. bert_encode helper · level 1 · ran · our draft was wrong · e41bf854cd7b9a83
  11. collate_idf helper · level 1 · ran · our draft was wrong · 80aa6d93e1e6196f
  12. process helper · level 1 · ran · our draft was wrong · 1d48904fd63d4a59
  13. cache_scibert helper · level 1 · ran · our draft was wrong · be27ce41f9ea0d1f
  14. padding helper · level 1 · ran · our draft was wrong · 13ef8eeae8570519

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 6 bricks; individually on a fixture: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. bert_cos_score_idf entry · level 0 · unverified · 74c9dc5efff4ab40
  2. get_bert_embedding helper · level 1 · ran · our draft was wrong · 466fbc555ce2478f
  3. greedy_cos_idf helper · level 0 · unverified · 0b727a24b15c0d5e
  4. bert_encode helper · level 1 · ran · our draft was wrong · 29119b8712ceae26
  5. collate_idf helper · level 1 · ran · our draft was wrong · a674ab62cada9f02
  6. padding helper · level 1 · ran · our draft was wrong · 13ef8eeae8570519

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 13 bricks; individually on a fixture: 9 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 4 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScore entry · level 0 · unverified · 573bfd9ac94c99ed
  2. bert_score metric · level 0 · unverified · 011256828cf4ffa5
  3. idf_numpy_to_embed helper · level 1 · ran; the contract check could not run (RAISES) · 9ccab64bf97fe100
  4. load_idf helper · level 1 · ran · our draft was wrong · 89c6cff022037bf3
  5. load_model helper · level 0 · unverified · 3647b7dd8c5ac0d7
  6. train_idf helper · level 0 · unverified · f61640ae3eacc0ce
  7. bert_forwarding helper · level 1 · ran · our draft was wrong · 369342a5449a0bfd
  8. compute_RPF metric · level 1 · ran; the contract check could not run (RAISES) · 9cc9523dfc879e70
  9. sents_to_tensor helper · level 1 · ran; the contract check could not run (TIMEOUT) · 58daeffb3ffb38df
  10. truncate_bert_layers helper · level 1 · ran · our draft was wrong · 5bf3a9aaa9676734
  11. apply_idf metric · level 1 · ran · our draft was wrong · 95e240faa9326986
  12. compute_pairwise_cosine metric · level 1 · ran; the contract check could not run (RAISES) · a6fb6a679479b806
  13. rescaling metric · level 1 · ran · violated contract · 2b21ab2f6f331acc

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 10 bricks; individually on a fixture: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. bert_cos_score_idf entry · level 0 · unverified · c3818aacfecb9596
  2. get_bert_embedding helper · level 1 · ran · our draft was wrong · 60a9029618bff133
  3. greedy_cos_idf metric · level 0 · unverified · 72ab4ee7aef7a2d7
  4. bert_encode helper · level 0 · unverified · 841ac1a3bf04a266
  5. collate_idf helper · level 1 · ran · our draft was wrong · 1d15c841be27109a
  6. get_precision_token_alignments helper · level 1 · ran · our draft was wrong · e93b135d3cd1c1ec
  7. get_recall_token_alignments helper · level 1 · ran · our draft was wrong · 0dc67c4f397a1eba
  8. padding helper · level 1 · ran · our draft was wrong · 13ef8eeae8570519
  9. sent_encode helper · level 0 · unverified · 5578268ced9f26ff
  10. map_subword_index_to_token_index helper · level 1 · ran · our draft was wrong · 47078bc5a6336435

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 15 bricks; individually on a fixture: 8 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 7 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScorer entry · level 0 · unverified · 364dacc2df955455
  2. bert_cos_score_idf metric · level 0 · unverified · a6d2976f556bf53f
  3. cache_scibert helper · level 0 · unverified · ffd1e746cb94818e
  4. get_bert_embedding helper · level 1 · ran · our draft was wrong · 1eebdf77ab9a0ecd
  5. get_hash helper · level 1 · ran · our draft was wrong · f982fdf447668cbb
  6. get_idf_dict helper · level 0 · unverified · a1d85069694206d2
  7. get_model helper · level 0 · unverified · 7e819d1873379c18
  8. sent_encode helper · level 0 · unverified · e5fcbbe27d124542
  9. greedy_cos_idf metric · level 0 · unverified · a85884cc1bdc6201
  10. mark_all_punc_tokens helper · level 1 · ran · violated contract · aaf8f9d22087eb72
  11. bert_encode helper · level 1 · ran · our draft was wrong · 949db13a96d3ef67
  12. collate_idf helper · level 1 · ran · our draft was wrong · 84a478e9730d32cf
  13. process helper · level 1 · ran; the contract check could not run (TIMEOUT) · 5a7c2e48dc108411
  14. get_window_indices helper · level 1 · ran · honoured contract · ea4088dfeb479c9a
  15. padding helper · level 1 · ran · our draft was wrong · 13ef8eeae8570519

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 1 brick; individually on a fixture: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScorer entry · level 1 · ran · 41873629d485d64d

harvested Assembly for BERTSCORE (graph method: BERTSCORE). 14 bricks; individually on a fixture: 9 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 9 where Syntology's instrument failed) · 5 unverified. The assembly has not been run end-to-end.

one of 8 siblings

  1. BERTScorer entry · level 0 · unverified · 200a19a9f1229034
  2. bert_cos_score_idf metric · level 0 · unverified · 6a5eab3261e5500c
  3. get_bert_embedding helper · level 1 · ran; the contract check could not run (RAISES) · 779e3927681cb63b
  4. get_hash helper · level 1 · ran · our draft was wrong · 7a9d43b6fa2eafef
  5. get_idf_dict helper · level 0 · unverified · edd1054d8a4511eb
  6. get_model helper · level 0 · unverified · 271285be2338c8f6
  7. get_tokenizer helper · level 1 · ran; the contract check could not run (TIMEOUT) · 0a3852d45d725a39
  8. sent_encode helper · level 1 · ran; the contract check could not run (TIMEOUT) · 37d4a40315d05e5e
  9. greedy_cos_idf metric · level 0 · unverified · 227909002c4f2aba
  10. bert_encode helper · level 1 · ran · our draft was wrong · e41bf854cd7b9a83
  11. collate_idf helper · level 1 · ran · our draft was wrong · e1297e386354025c
  12. process helper · level 1 · ran · our draft was wrong · 6fb229c8407fb6c8
  13. cache_scibert helper · level 1 · ran · our draft was wrong · be27ce41f9ea0d1f
  14. padding helper · level 1 · ran · our draft was wrong · 13ef8eeae8570519

Each brick ends with its code_sha256, Syntology's identity for that exact code; click one to copy the get_code call that returns it. For agents, on Syntology's MCP service: assembly(arxiv_id="1904.09675") returns these assemblies with each brick's role and wiring (how to connect).

Tasks

Image CaptioningMachine TranslationModel SelectionSentenceText GenerationTranslation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections