{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/code/compute-f1","entry":"compute_f1","source":"Syntology graph, per-sample; not an archive number","read_at":"2026-10-05T00:36:11+00:00","claim":"Names are grouped by exact entry-name string. Same-named routines are NOT asserted to be equivalent; 'ran' means executed on a synthesized fixture, not correctness. n_samples_ran = sum of by_status over every status except 'unverified' (ran_draft_wrong and ran_fixture are failures of Syntology's instrument, not of the code); n_papers_ran = papers with at least one such sample = n_papers_ran_no_instrument_failure (papers with a run with no instrument failure) + n_papers_ran_instrument_only (papers where every run was a failure of Syntology's instrument); n_samples_ran = n_samples_ran_checked (honoured, violated, no contract) + n_samples_instrument (failures of Syntology's instrument, not of the code).","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"},"n_papers":40,"n_papers_ran":36,"n_papers_ran_no_instrument_failure":33,"n_papers_ran_instrument_only":3,"units":"n_samples, n_samples_ran, n_samples_fingerprinted and by_status count distinct code bodies (code_sha256); n_places and n_places_pointer_only count places, one per (paper, code body) pair, which is also the unit of the samples list","n_samples":37,"n_samples_ran":33,"n_samples_constructed":0,"n_samples_ran_checked":30,"n_samples_instrument":3,"n_samples_unverified":4,"n_samples_fingerprinted":18,"n_places":40,"n_places_pointer_only":8,"n_places_pointer_only_link_open":1,"inline_ok_is":"inline_ok is true only when both licence gates allow Syntology to serve this code's text: this paper's link (inline_ok_link, from the licence of this paper's copy) and the stored copy of the same code that Syntology serves, which may have been first harvested from another repository. withheld_clause is Syntology's own wording of which gate keeps the text a pointer and why, and is null when inline_ok is true. This site shows no code text either way; n_places_pointer_only_link_open counts the places where this paper's link allows the text and the stored copy does not","by_status":{"ran_honours":5,"ran_violates":0,"ran_draft_wrong":2,"ran_fixture":1,"ran":25,"unverified":4},"key_notes":{"n_samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","n_samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"syntology":{"atlas_url":null,"mcp":null,"mcp_per_sample":{"tool":"get_code","arguments_in":"samples[].mcp_get_code"},"developers":"https://syntology.ai/developers"},"samples":[{"arxiv_id":"2608.11775","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2608-11775","title":"The Sleeping Agent: What Gist-Based Context Compression Loses and Why","date":null,"month_inferred_from_arxiv_id":"2026-08","title_source":"syntology","repo":"kyrkewood/sleeping-agent","path":"src/scoring.py","file_url":"https://github.com/kyrkewood/sleeping-agent/blob/HEAD/src/scoring.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"b2bc687a414410e6","code_sha256_prefix":"b2bc687a414410e6","mcp_get_code":{"code_sha256":"b2bc687a414410e6"}},{"arxiv_id":"2607.20833","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2607-20833","title":"ReFact: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning","date":null,"month_inferred_from_arxiv_id":"2026-07","title_source":"syntology","repo":"NEUIR/REFACT","path":"verl/verl/utils/reward_score/evdience_reward.py","file_url":"https://github.com/NEUIR/REFACT/blob/HEAD/verl/verl/utils/reward_score/evdience_reward.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"ecc47b6b4d41b5db","code_sha256_prefix":"ecc47b6b4d41b5db","mcp_get_code":{"code_sha256":"ecc47b6b4d41b5db"}},{"arxiv_id":"2605.31500","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2605-31500","title":"On Efficient Scaling of GNNs via IO-Aware Layers Implementations","date":null,"month_inferred_from_arxiv_id":"2026-05","title_source":"syntology","repo":"yandex-research/On-Efficient-Scaling-Of-GNNs","path":"src/training/metrics.py","file_url":"https://github.com/yandex-research/On-Efficient-Scaling-Of-GNNs/blob/HEAD/src/training/metrics.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"5c551f60830ac131","code_sha256_prefix":"5c551f60830ac131","mcp_get_code":{"code_sha256":"5c551f60830ac131"}},{"arxiv_id":"2605.16198","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2605-16198","title":"Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems","date":null,"month_inferred_from_arxiv_id":"2026-05","title_source":"syntology","repo":"praal/llm-monitoring","path":"LLMFailure/forest_evaluate.py","file_url":"https://github.com/praal/llm-monitoring/blob/HEAD/LLMFailure/forest_evaluate.py","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"86da62288e7880c4","code_sha256_prefix":"86da62288e7880c4","mcp_get_code":{"code_sha256":"86da62288e7880c4"}},{"arxiv_id":"2601.21159","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2601-21159","title":"Spatial-Regularization-Aware Dual-Branch Collaborative Inference for Training-Free OVSS in Remote Sensing Imagery","date":null,"month_inferred_from_arxiv_id":"2026-01","title_source":"syntology","repo":"yu-ni1989/SDCI","path":"util/tools.py","file_url":"https://github.com/yu-ni1989/SDCI/blob/HEAD/util/tools.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"c3562972f2ab9e25","code_sha256_prefix":"c3562972f2ab9e25","mcp_get_code":{"code_sha256":"c3562972f2ab9e25"}},{"arxiv_id":"2503.01785","arxiv_id_is_arxiv":true,"paper":"/paper/visual-rft-visual-reinforcement-fine-tuning","title":"Visual-RFT: Visual Reinforcement Fine-Tuning","date":"2025-03-03","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"liuziyu77/visual-rft","path":"Visual-ARFT/evaluation_coding/evaluation_mat_coding_visual_arft.py","file_url":"https://github.com/liuziyu77/visual-rft/blob/HEAD/Visual-ARFT/evaluation_coding/evaluation_mat_coding_visual_arft.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"fa20176e2b95a3d7","code_sha256_prefix":"fa20176e2b95a3d7","mcp_get_code":{"code_sha256":"fa20176e2b95a3d7"}},{"arxiv_id":"2410.08821","arxiv_id_is_arxiv":true,"paper":"/paper/retriever-and-memory-towards-adaptive-note","title":"Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation","date":"2024-10-11","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"thunlp/adaptive-note","path":"src/metrics_questeval_crud.py","file_url":"https://github.com/thunlp/adaptive-note/blob/HEAD/src/metrics_questeval_crud.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, thunlp/adaptive-note when it was checked","code_sha256":"84280cf42e5cf118","code_sha256_prefix":"84280cf42e5cf118","mcp_get_code":{"code_sha256":"84280cf42e5cf118"}},{"arxiv_id":"2401.17043","arxiv_id_is_arxiv":true,"paper":"/paper/crud-rag-a-comprehensive-chinese-benchmark","title":"CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models","date":"2024-01-30","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"iaar-shanghai/crud_rag","path":"src/metric/quest_eval.py","file_url":"https://github.com/iaar-shanghai/crud_rag/blob/HEAD/src/metric/quest_eval.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, iaar-shanghai/crud_rag when it was checked","code_sha256":"33efd8e471b6cb65","code_sha256_prefix":"33efd8e471b6cb65","mcp_get_code":{"code_sha256":"33efd8e471b6cb65"}},{"arxiv_id":"2311.17154","arxiv_id_is_arxiv":true,"paper":"/paper/pragmatic-radiology-report-generation","title":"Pragmatic Radiology Report Generation","date":"2023-11-28","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"chicagohai/llm_radiology","path":"CXRMetric/radgraph_evaluate_model.py","file_url":"https://github.com/chicagohai/llm_radiology/blob/HEAD/CXRMetric/radgraph_evaluate_model.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, chicagohai/llm_radiology when it was checked","code_sha256":"b59ac1362f3babba","code_sha256_prefix":"b59ac1362f3babba","mcp_get_code":{"code_sha256":"b59ac1362f3babba"}},{"arxiv_id":"2310.15047","arxiv_id_is_arxiv":true,"paper":"/paper/meta-out-of-context-learning-in-neural","title":"Implicit meta-learning may lead language models to trust more reliable sources","date":"2023-10-23","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"krasheninnikov/internalization","path":"src/metrics.py","file_url":"https://github.com/krasheninnikov/internalization/blob/HEAD/src/metrics.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, krasheninnikov/internalization when it was checked","code_sha256":"e0d035eca2e2c3c5","code_sha256_prefix":"e0d035eca2e2c3c5","mcp_get_code":{"code_sha256":"e0d035eca2e2c3c5"}},{"arxiv_id":"2310.12836","arxiv_id_is_arxiv":true,"paper":"/paper/knowledge-augmented-language-model","title":"Knowledge-Augmented Language Model Verification","date":"2023-10-19","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"JinheonBaek/KALMV","path":"models/verifiers.py","file_url":"https://github.com/JinheonBaek/KALMV/blob/HEAD/models/verifiers.py","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, JinheonBaek/KALMV when it was checked","code_sha256":"b004a421a9a1ba54","code_sha256_prefix":"b004a421a9a1ba54","mcp_get_code":{"code_sha256":"b004a421a9a1ba54"}},{"arxiv_id":"2305.12295","arxiv_id_is_arxiv":true,"paper":"/paper/logic-lm-empowering-large-language-models","title":"Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning","date":"2023-05-20","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"teacherpeterpan/logic-llm","path":"models/evaluation.py","file_url":"https://github.com/teacherpeterpan/logic-llm/blob/HEAD/models/evaluation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"f5e5c9b9aa79ab16","code_sha256_prefix":"f5e5c9b9aa79ab16","mcp_get_code":{"code_sha256":"f5e5c9b9aa79ab16"}},{"arxiv_id":"2305.11435","arxiv_id_is_arxiv":true,"paper":"/paper/syllable-discovery-and-cross-lingual","title":"Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model","date":"2023-05-19","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"jasonppy/syllable-discovery","path":"estonian.py","file_url":"https://github.com/jasonppy/syllable-discovery/blob/HEAD/estonian.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"6b7104858152a9ea","code_sha256_prefix":"6b7104858152a9ea","mcp_get_code":{"code_sha256":"6b7104858152a9ea"}},{"arxiv_id":"2304.13007","arxiv_id_is_arxiv":true,"paper":"/paper/answering-questions-by-meta-reasoning-over","title":"Answering Questions by Meta-Reasoning over Multiple Chains of Thought","date":"2023-04-25","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"oriyor/reasoning-on-cots","path":"src/pred_evaluators/evaluation.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/pred_evaluators/evaluation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"9118271336e0821a","code_sha256_prefix":"9118271336e0821a","mcp_get_code":{"code_sha256":"9118271336e0821a"}},{"arxiv_id":"2304.11633","arxiv_id_is_arxiv":true,"paper":"/paper/evaluating-chatgpt-s-information-extraction","title":"Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness","date":"2023-04-23","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"pkuserc/chatgpt_for_ie","path":"Code/EAE/score_EAE_E+.py","file_url":"https://github.com/pkuserc/chatgpt_for_ie/blob/HEAD/Code/EAE/score_EAE_E%2B.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"fada67a6d1459131","code_sha256_prefix":"fada67a6d1459131","mcp_get_code":{"code_sha256":"fada67a6d1459131"}},{"arxiv_id":"2303.08964","arxiv_id_is_arxiv":true,"paper":"/paper/cs-tgn-community-search-via-temporal-graph","title":"CS-TGN: Community Search via Temporal Graph Neural Networks","date":"2023-03-15","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"joint-em/cs-tgn","path":"f1score.py","file_url":"https://github.com/joint-em/cs-tgn/blob/HEAD/f1score.py","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"09a1f144a23eae42","code_sha256_prefix":"09a1f144a23eae42","mcp_get_code":{"code_sha256":"09a1f144a23eae42"}},{"arxiv_id":"2303.05977","arxiv_id_is_arxiv":true,"paper":"/paper/open-ended-medical-visual-question-answering","title":"Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models","date":"2023-03-10","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"tjvsonsbeek/open-ended-medical-vqa","path":"predict.py","file_url":"https://github.com/tjvsonsbeek/open-ended-medical-vqa/blob/HEAD/predict.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, tjvsonsbeek/open-ended-medical-vqa when it was checked","code_sha256":"1a3cb58ee8e001d4","code_sha256_prefix":"1a3cb58ee8e001d4","mcp_get_code":{"code_sha256":"1a3cb58ee8e001d4"}},{"arxiv_id":"2210.12582","arxiv_id_is_arxiv":true,"paper":"/paper/language-model-pre-training-with-sparse","title":"Language Model Pre-Training with Sparse Latent Typing","date":"2022-10-23","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"renll/sparselt","path":"evaluations/supervised-ie/scorer.py","file_url":"https://github.com/renll/sparselt/blob/HEAD/evaluations/supervised-ie/scorer.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"fada67a6d1459131","code_sha256_prefix":"fada67a6d1459131","mcp_get_code":{"code_sha256":"fada67a6d1459131"}},{"arxiv_id":"2210.10678","arxiv_id_is_arxiv":true,"paper":"/paper/towards-realistic-low-resource-relation","title":"Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study","date":"2022-10-19","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"zjunlp/KnowPrompt","path":"lit_models/util.py","file_url":"https://github.com/zjunlp/KnowPrompt/blob/HEAD/lit_models/util.py","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"526185d23747e06f","code_sha256_prefix":"526185d23747e06f","mcp_get_code":{"code_sha256":"526185d23747e06f"}},{"arxiv_id":"2205.02593","arxiv_id_is_arxiv":true,"paper":"/paper/metgen-a-module-based-entailment-tree","title":"METGEN: A Module-Based Entailment Tree Generation Framework for Answer Explanation","date":"2022-05-05","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"tencent-ailab/metalogic","path":"code/evaluate_metric.py","file_url":"https://github.com/tencent-ailab/metalogic/blob/HEAD/code/evaluate_metric.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"6724ecbf0decf84e","code_sha256_prefix":"6724ecbf0decf84e","mcp_get_code":{"code_sha256":"6724ecbf0decf84e"}},{"arxiv_id":"2112.01640","arxiv_id_is_arxiv":true,"paper":"/paper/longchecker-improving-scientific-claim","title":"MultiVerS: Improving scientific claim verification with weak supervision and full-document context","date":"2021-12-02","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"dwadden/longchecker","path":"multivers/metrics.py","file_url":"https://github.com/dwadden/longchecker/blob/HEAD/multivers/metrics.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"7e52643dedb5d9d4","code_sha256_prefix":"7e52643dedb5d9d4","mcp_get_code":{"code_sha256":"7e52643dedb5d9d4"}},{"arxiv_id":"2106.15838","arxiv_id_is_arxiv":true,"paper":"/paper/hyspa-hybrid-span-generation-for-scalable","title":"HySPA: Hybrid Span Generation for Scalable Text-to-Graph Extraction","date":"2021-06-30","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"dwadden/dygiepp","path":"dygie/training/f1.py","file_url":"https://github.com/dwadden/dygiepp/blob/HEAD/dygie/training/f1.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"fada67a6d1459131","code_sha256_prefix":"fada67a6d1459131","mcp_get_code":{"code_sha256":"fada67a6d1459131"}},{"arxiv_id":"2106.14463","arxiv_id_is_arxiv":true,"paper":"/paper/radgraph-extracting-clinical-entities-and","title":"RadGraph: Extracting Clinical Entities and Relations from Radiology Reports","date":"2021-06-28","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"rajpurkarlab/cxr-report-metric","path":"CXRMetric/radgraph_evaluate_model.py","file_url":"https://github.com/rajpurkarlab/cxr-report-metric/blob/HEAD/CXRMetric/radgraph_evaluate_model.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"inline_ok_link":true,"withheld_clause":"this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from chicagohai/llm_radiology, and GitHub detected no licence in that repository when it was checked","code_sha256":"b59ac1362f3babba","code_sha256_prefix":"b59ac1362f3babba","mcp_get_code":{"code_sha256":"b59ac1362f3babba"}},{"arxiv_id":"2104.08737","arxiv_id_is_arxiv":true,"paper":"/paper/low-rank-subspaces-for-unsupervised-entity","title":"Low-Rank Subspaces for Unsupervised Entity Linking","date":"2021-04-18","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"lephong/dl4el","path":"jrk/el_statstest.py","file_url":"https://github.com/lephong/dl4el/blob/HEAD/jrk/el_statstest.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"8bad2058cd892964","code_sha256_prefix":"8bad2058cd892964","mcp_get_code":{"code_sha256":"8bad2058cd892964"}},{"arxiv_id":"2010.12812","arxiv_id_is_arxiv":true,"paper":"/paper/a-frustratingly-easy-approach-for-joint","title":"A Frustratingly Easy Approach for Entity and Relation Extraction","date":"2020-10-24","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"princeton-nlp/PURE","path":"run_relation.py","file_url":"https://github.com/princeton-nlp/PURE/blob/HEAD/run_relation.py","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"f39c75c8e38877a5","code_sha256_prefix":"f39c75c8e38877a5","mcp_get_code":{"code_sha256":"f39c75c8e38877a5"}},{"arxiv_id":"2010.03656","arxiv_id_is_arxiv":true,"paper":"/paper/exposing-shallow-heuristics-of-relation","title":"Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data","date":"2020-10-07","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"shacharosn/CRE","path":"evaluate_challenge_set_output.py","file_url":"https://github.com/shacharosn/CRE/blob/HEAD/evaluate_challenge_set_output.py","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, shacharosn/CRE when it was checked","code_sha256":"39081ff7aefd4fcb","code_sha256_prefix":"39081ff7aefd4fcb","mcp_get_code":{"code_sha256":"39081ff7aefd4fcb"}},{"arxiv_id":"2005.00669","arxiv_id_is_arxiv":true,"paper":"/paper/contrastive-self-supervised-learning-for","title":"Contrastive Self-Supervised Learning for Commonsense Reasoning","date":"2020-05-02","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"SAP-samples/acl2020-commonsense","path":"scorer.py","file_url":"https://github.com/SAP-samples/acl2020-commonsense/blob/HEAD/scorer.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"0f32d4ee5a8aae9f","code_sha256_prefix":"0f32d4ee5a8aae9f","mcp_get_code":{"code_sha256":"0f32d4ee5a8aae9f"}},{"arxiv_id":"1911.02984","arxiv_id_is_arxiv":true,"paper":"/paper/the-techqa-dataset","title":"The TechQA Dataset","date":"2019-11-08","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"ibm/techqa","path":"techqa_evaluation.py","file_url":"https://github.com/ibm/techqa/blob/HEAD/techqa_evaluation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"192f44f7e75dcf94","code_sha256_prefix":"192f44f7e75dcf94","mcp_get_code":{"code_sha256":"192f44f7e75dcf94"}},{"arxiv_id":"1906.02780","arxiv_id_is_arxiv":true,"paper":"/paper/syntactically-supervised-transformers-for","title":"Syntactically Supervised Transformers for Faster Neural Machine Translation","date":"2019-06-06","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"dojoteef/synst","path":"analysis.py","file_url":"https://github.com/dojoteef/synst/blob/HEAD/analysis.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"315cda1182f75c8e","code_sha256_prefix":"315cda1182f75c8e","mcp_get_code":{"code_sha256":"315cda1182f75c8e"}},{"arxiv_id":"1904.07399","arxiv_id_is_arxiv":true,"paper":"/paper/adaptive-wing-loss-for-robust-face-alignment","title":"Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression","date":"2019-04-16","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"affromero/SMILE","path":"metrics/f1_score.py","file_url":"https://github.com/affromero/SMILE/blob/HEAD/metrics/f1_score.py","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"cf6388d3dfe650b4","code_sha256_prefix":"cf6388d3dfe650b4","mcp_get_code":{"code_sha256":"cf6388d3dfe650b4"}},{"arxiv_id":"1803.09578","arxiv_id_is_arxiv":true,"paper":"/paper/why-comparing-single-performance-scores-does","title":"Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches","date":"2018-03-26","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"UKPLab/emnlp2017-bilstm-cnn-crf","path":"util/BIOF1Validation.py","file_url":"https://github.com/UKPLab/emnlp2017-bilstm-cnn-crf/blob/HEAD/util/BIOF1Validation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"2561978b0fd1e5d3","code_sha256_prefix":"2561978b0fd1e5d3","mcp_get_code":{"code_sha256":"2561978b0fd1e5d3"}},{"arxiv_id":"1802.05365","arxiv_id_is_arxiv":true,"paper":"/paper/deep-contextualized-word-representations","title":"Deep contextualized word representations","date":"2018-02-15","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"UKPLab/elmo-bilstm-cnn-crf","path":"util/BIOF1Validation.py","file_url":"https://github.com/UKPLab/elmo-bilstm-cnn-crf/blob/HEAD/util/BIOF1Validation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"a2735b43c037f7c3","code_sha256_prefix":"a2735b43c037f7c3","mcp_get_code":{"code_sha256":"a2735b43c037f7c3"}},{"arxiv_id":"1511.08308","arxiv_id_is_arxiv":true,"paper":"/paper/named-entity-recognition-with-bidirectional","title":"Named Entity Recognition with Bidirectional LSTM-CNNs","date":"2015-11-26","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"mxhofer/Named-Entity-Recognition-BidirectionalLSTM-CNN-CoNLL","path":"validation.py","file_url":"https://github.com/mxhofer/Named-Entity-Recognition-BidirectionalLSTM-CNN-CoNLL/blob/HEAD/validation.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"e7fa1b4511321cb3","code_sha256_prefix":"e7fa1b4511321cb3","mcp_get_code":{"code_sha256":"e7fa1b4511321cb3"}},{"arxiv_id":"openreview_krRRnq3E1w","arxiv_id_is_arxiv":false,"paper":null,"title":"OpenReview id openreview_krRRnq3E1w","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"zhangSchnee/BoG","path":"bog/actions.py","file_url":"https://github.com/zhangSchnee/BoG/blob/HEAD/bog/actions.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"32758d0edc2f464a","code_sha256_prefix":"32758d0edc2f464a","mcp_get_code":{"code_sha256":"32758d0edc2f464a"}},{"arxiv_id":"aaai_29863","arxiv_id_is_arxiv":false,"paper":null,"title":"AAAI id aaai_29863","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"facebookresearch/EditEval","path":"src/metrics/easse_sari.py","file_url":"https://github.com/facebookresearch/EditEval/blob/HEAD/src/metrics/easse_sari.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"ba0b4ade2e899b0c","code_sha256_prefix":"ba0b4ade2e899b0c","mcp_get_code":{"code_sha256":"ba0b4ade2e899b0c"}},{"arxiv_id":"2024.findings-emnlp.286","arxiv_id_is_arxiv":false,"paper":null,"title":"ACL Anthology id 2024.findings-emnlp.286","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"kgarg8/Stanceformer","path":"utils/model_eval2.py","file_url":"https://github.com/kgarg8/Stanceformer/blob/HEAD/utils/model_eval2.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"f9d4e2c7ec69b923","code_sha256_prefix":"f9d4e2c7ec69b923","mcp_get_code":{"code_sha256":"f9d4e2c7ec69b923"}},{"arxiv_id":"2023.findings-acl.888","arxiv_id_is_arxiv":false,"paper":null,"title":"ACL Anthology id 2023.findings-acl.888","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"posuer/Check-COVID","path":"fact-checking-system/evaluate/lib/metrics.py","file_url":"https://github.com/posuer/Check-COVID/blob/HEAD/fact-checking-system/evaluate/lib/metrics.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"c2a07902a914fdd4","code_sha256_prefix":"c2a07902a914fdd4","mcp_get_code":{"code_sha256":"c2a07902a914fdd4"}},{"arxiv_id":"2023.acl-short.31","arxiv_id_is_arxiv":false,"paper":null,"title":"ACL Anthology id 2023.acl-short.31","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"JFChi/PLUE","path":"finetuning/qa/privacyqa/scorer.py","file_url":"https://github.com/JFChi/PLUE/blob/HEAD/finetuning/qa/privacyqa/scorer.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"8ad94f241bcc8e90","code_sha256_prefix":"8ad94f241bcc8e90","mcp_get_code":{"code_sha256":"8ad94f241bcc8e90"}},{"arxiv_id":"2021.findings-acl.208","arxiv_id_is_arxiv":false,"paper":null,"title":"ACL Anthology id 2021.findings-acl.208","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"chuchun8/PStance","path":"source/utils/model_eval.py","file_url":"https://github.com/chuchun8/PStance/blob/HEAD/source/utils/model_eval.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"4c85e690058b984a","code_sha256_prefix":"4c85e690058b984a","mcp_get_code":{"code_sha256":"4c85e690058b984a"}},{"arxiv_id":"2021.findings-acl.204","arxiv_id_is_arxiv":false,"paper":null,"title":"ACL Anthology id 2021.findings-acl.204","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"chuchun8/MTSD","path":"src/utils/model_eval.py","file_url":"https://github.com/chuchun8/MTSD/blob/HEAD/src/utils/model_eval.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"c697744301d5ea90","code_sha256_prefix":"c697744301d5ea90","mcp_get_code":{"code_sha256":"c697744301d5ea90"}}],"record_sha256":"0db2089bfa33c11d9f766feeb96639b53c2b62683a04032e01bd2d91a842c34e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}