{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/code/extract-score","entry":"extract_score","source":"Syntology graph, per-sample; not an archive number","read_at":"2026-10-05T00:36:11+00:00","claim":"Names are grouped by exact entry-name string. Same-named routines are NOT asserted to be equivalent; 'ran' means executed on a synthesized fixture, not correctness. n_samples_ran = sum of by_status over every status except 'unverified' (ran_draft_wrong and ran_fixture are failures of Syntology's instrument, not of the code); n_papers_ran = papers with at least one such sample = n_papers_ran_no_instrument_failure (papers with a run with no instrument failure) + n_papers_ran_instrument_only (papers where every run was a failure of Syntology's instrument); n_samples_ran = n_samples_ran_checked (honoured, violated, no contract) + n_samples_instrument (failures of Syntology's instrument, not of the code).","status_vocabulary":{"ran_honours":"ran, honoured the contract we drafted","ran_violates":"ran, violated the contract we drafted","ran_draft_wrong":"ran; our contract draft was wrong, not the code","ran_fixture":"ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)","ran":"ran on a synthesized input","unverified":"unverified (harvested, no recorded run)"},"n_papers":18,"n_papers_ran":18,"n_papers_ran_no_instrument_failure":17,"n_papers_ran_instrument_only":1,"units":"n_samples, n_samples_ran, n_samples_fingerprinted and by_status count distinct code bodies (code_sha256); n_places and n_places_pointer_only count places, one per (paper, code body) pair, which is also the unit of the samples list","n_samples":20,"n_samples_ran":20,"n_samples_constructed":0,"n_samples_ran_checked":19,"n_samples_instrument":1,"n_samples_unverified":0,"n_samples_fingerprinted":13,"n_places":20,"n_places_pointer_only":8,"n_places_pointer_only_link_open":0,"inline_ok_is":"inline_ok is true only when both licence gates allow Syntology to serve this code's text: this paper's link (inline_ok_link, from the licence of this paper's copy) and the stored copy of the same code that Syntology serves, which may have been first harvested from another repository. withheld_clause is Syntology's own wording of which gate keeps the text a pointer and why, and is null when inline_ok is true. This site shows no code text either way; n_places_pointer_only_link_open counts the places where this paper's link allows the text and the stored copy does not","by_status":{"ran_honours":12,"ran_violates":0,"ran_draft_wrong":1,"ran_fixture":0,"ran":7,"unverified":0},"key_notes":{"n_samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","n_samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"syntology":{"atlas_url":null,"mcp":null,"mcp_per_sample":{"tool":"get_code","arguments_in":"samples[].mcp_get_code"},"developers":"https://syntology.ai/developers"},"samples":[{"arxiv_id":"2606.25152","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2606-25152","title":"Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift","date":null,"month_inferred_from_arxiv_id":"2026-06","title_source":"syntology","repo":"kkr36/llm_detection","path":"arxiv/inference_set_rewrite/iterative_prompt_rewrite_scale/llm_judge.py","file_url":"https://github.com/kkr36/llm_detection/blob/HEAD/arxiv/inference_set_rewrite/iterative_prompt_rewrite_scale/llm_judge.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, kkr36/llm_detection when it was checked","code_sha256":"a2706b98edd8c58b","code_sha256_prefix":"a2706b98edd8c58b","mcp_get_code":{"code_sha256":"a2706b98edd8c58b"}},{"arxiv_id":"2606.07970","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2606-07970","title":"Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks","date":null,"month_inferred_from_arxiv_id":"2026-06","title_source":"syntology","repo":"haomingwen/patcher","path":"evaluate/gpt_evaluate.py","file_url":"https://github.com/haomingwen/patcher/blob/HEAD/evaluate/gpt_evaluate.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, haomingwen/patcher when it was checked","code_sha256":"1e9b2b7a8f925a69","code_sha256_prefix":"1e9b2b7a8f925a69","mcp_get_code":{"code_sha256":"1e9b2b7a8f925a69"}},{"arxiv_id":"2604.27453","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2604-27453","title":"From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks","date":null,"month_inferred_from_arxiv_id":"2026-04","title_source":"syntology","repo":"Rainier-rq1/From_Coarse_to_Fine","path":"WEval/WritingBench-Critic.py","file_url":"https://github.com/Rainier-rq1/From_Coarse_to_Fine/blob/HEAD/WEval/WritingBench-Critic.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, Rainier-rq1/From_Coarse_to_Fine when it was checked","code_sha256":"7da4354e2c6f6c2c","code_sha256_prefix":"7da4354e2c6f6c2c","mcp_get_code":{"code_sha256":"7da4354e2c6f6c2c"}},{"arxiv_id":"2604.27453","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2604-27453","title":"From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks","date":null,"month_inferred_from_arxiv_id":"2026-04","title_source":"syntology","repo":"Rainier-rq1/From_Coarse_to_Fine","path":"WEval/llm_judge.py","file_url":"https://github.com/Rainier-rq1/From_Coarse_to_Fine/blob/HEAD/WEval/llm_judge.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, Rainier-rq1/From_Coarse_to_Fine when it was checked","code_sha256":"a2fd3492b9408427","code_sha256_prefix":"a2fd3492b9408427","mcp_get_code":{"code_sha256":"a2fd3492b9408427"}},{"arxiv_id":"2602.03548","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2602-03548","title":"SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue","date":null,"month_inferred_from_arxiv_id":"2026-02","title_source":"syntology","repo":"Da1yuqin/SEAD","path":"utils/analyze_chatbot_mistakes.py","file_url":"https://github.com/Da1yuqin/SEAD/blob/HEAD/utils/analyze_chatbot_mistakes.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"fb8290dd4e4e685b","code_sha256_prefix":"fb8290dd4e4e685b","mcp_get_code":{"code_sha256":"fb8290dd4e4e685b"}},{"arxiv_id":"2601.09270","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2601-09270","title":"MCGA: A Multi-task Classical Chinese Literary Genre Audio Corpus","date":null,"month_inferred_from_arxiv_id":"2026-01","title_source":"syntology","repo":"yxduir/MCGA","path":"eval/eval_model.py","file_url":"https://github.com/yxduir/MCGA/blob/HEAD/eval/eval_model.py","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, yxduir/MCGA when it was checked","code_sha256":"b5f599588a4adc4a","code_sha256_prefix":"b5f599588a4adc4a","mcp_get_code":{"code_sha256":"b5f599588a4adc4a"}},{"arxiv_id":"2510.20956","arxiv_id_is_arxiv":true,"paper":"/paper/arxiv-2510-20956","title":"Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training","date":null,"month_inferred_from_arxiv_id":"2025-10","title_source":"syntology","repo":"BatsResearch/self-jailbreaking","path":"scripts/eval/strongreject_eval.py","file_url":"https://github.com/BatsResearch/self-jailbreaking/blob/HEAD/scripts/eval/strongreject_eval.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, BatsResearch/self-jailbreaking when it was checked","code_sha256":"e3b7df9a5aa407fe","code_sha256_prefix":"e3b7df9a5aa407fe","mcp_get_code":{"code_sha256":"e3b7df9a5aa407fe"}},{"arxiv_id":"2505.17018","arxiv_id_is_arxiv":true,"paper":"/paper/sophiavl-r1-reinforcing-mllms-reasoning-with","title":"SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward","date":"2025-05-22","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"kxfan2002/sophiavl-r1","path":"verl/workers/reward/custom.py","file_url":"https://github.com/kxfan2002/sophiavl-r1/blob/HEAD/verl/workers/reward/custom.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"41ec901c077b9860","code_sha256_prefix":"41ec901c077b9860","mcp_get_code":{"code_sha256":"41ec901c077b9860"}},{"arxiv_id":"2505.16211","arxiv_id_is_arxiv":true,"paper":"/paper/audiotrust-benchmarking-the-multifaceted","title":"AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models","date":"2025-05-22","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"jusperlee/audiotrust","path":"audio_evals/eval_task.py","file_url":"https://github.com/jusperlee/audiotrust/blob/HEAD/audio_evals/eval_task.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"67daa768dfd59db2","code_sha256_prefix":"67daa768dfd59db2","mcp_get_code":{"code_sha256":"67daa768dfd59db2"}},{"arxiv_id":"2505.02370","arxiv_id_is_arxiv":true,"paper":"/paper/superedit-rectifying-and-facilitating","title":"SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing","date":"2025-05-05","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"bytedance/superedit","path":"eval/eval_instructpix2pix.py","file_url":"https://github.com/bytedance/superedit/blob/HEAD/eval/eval_instructpix2pix.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"ea7b78776c588d7f","code_sha256_prefix":"ea7b78776c588d7f","mcp_get_code":{"code_sha256":"ea7b78776c588d7f"}},{"arxiv_id":"2503.08741","arxiv_id_is_arxiv":true,"paper":"/paper/oasis-one-image-is-all-you-need-for","title":"Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis","date":"2025-03-11","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"Letian2003/MM_INF","path":"pp_func/oasis.py","file_url":"https://github.com/Letian2003/MM_INF/blob/HEAD/pp_func/oasis.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, Letian2003/MM_INF when it was checked","code_sha256":"c39c295401d2dd49","code_sha256_prefix":"c39c295401d2dd49","mcp_get_code":{"code_sha256":"c39c295401d2dd49"}},{"arxiv_id":"2502.16182","arxiv_id_is_arxiv":true,"paper":"/paper/ipo-your-language-model-is-secretly-a","title":"IPO: Your Language Model is Secretly a Preference Classifier","date":"2025-02-22","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"shivank21/Implicit_Preference_Optimization","path":"Preference_Comparison/RM_Bench/self_reward.py","file_url":"https://github.com/shivank21/Implicit_Preference_Optimization/blob/HEAD/Preference_Comparison/RM_Bench/self_reward.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"7204c024b35e7517","code_sha256_prefix":"7204c024b35e7517","mcp_get_code":{"code_sha256":"7204c024b35e7517"}},{"arxiv_id":"2502.03052","arxiv_id_is_arxiv":true,"paper":"/paper/understanding-and-enhancing-the-1","title":"Understanding and Enhancing the Transferability of Jailbreaking Attacks","date":"2025-02-05","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"tmllab/2025_ICLR_PiF","path":"attack_clm.py","file_url":"https://github.com/tmllab/2025_ICLR_PiF/blob/HEAD/attack_clm.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"21b2a816defb053b","code_sha256_prefix":"21b2a816defb053b","mcp_get_code":{"code_sha256":"21b2a816defb053b"}},{"arxiv_id":"2407.19633","arxiv_id_is_arxiv":true,"paper":"/paper/optimus-0-3-using-large-language-models-to","title":"OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale","date":"2024-07-29","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"teshnizi/optimus","path":"parameters.py","file_url":"https://github.com/teshnizi/optimus/blob/HEAD/parameters.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"edbd5ba229bb9dfc","code_sha256_prefix":"edbd5ba229bb9dfc","mcp_get_code":{"code_sha256":"edbd5ba229bb9dfc"}},{"arxiv_id":"2407.19633","arxiv_id_is_arxiv":true,"paper":"/paper/optimus-0-3-using-large-language-models-to","title":"OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale","date":"2024-07-29","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"teshnizi/optimus","path":"variables.py","file_url":"https://github.com/teshnizi/optimus/blob/HEAD/variables.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"90629c8ff26c53f1","code_sha256_prefix":"90629c8ff26c53f1","mcp_get_code":{"code_sha256":"90629c8ff26c53f1"}},{"arxiv_id":"2407.02099","arxiv_id_is_arxiv":true,"paper":"/paper/helpful-assistant-or-fruitful-facilitator","title":"Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior","date":"2024-07-02","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"peluz/persona-behavior","path":"compute_results.py","file_url":"https://github.com/peluz/persona-behavior/blob/HEAD/compute_results.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"0dbfb54acfddc8c8","code_sha256_prefix":"0dbfb54acfddc8c8","mcp_get_code":{"code_sha256":"0dbfb54acfddc8c8"}},{"arxiv_id":"2406.18522","arxiv_id_is_arxiv":true,"paper":"/paper/chronomagic-bench-a-benchmark-for-metamorphic","title":"ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation","date":"2024-06-26","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"PKU-YuanGroup/ChronoMagic-Bench","path":"UMT/step3_get_merge_umt_scores.py","file_url":"https://github.com/PKU-YuanGroup/ChronoMagic-Bench/blob/HEAD/UMT/step3_get_merge_umt_scores.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"690f98d4b6d870df","code_sha256_prefix":"690f98d4b6d870df","mcp_get_code":{"code_sha256":"690f98d4b6d870df"}},{"arxiv_id":"2405.19744","arxiv_id_is_arxiv":true,"paper":"/paper/x-instruction-aligning-language-model-in-low","title":"X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions","date":"2024-05-30","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"znlp/x-instruction","path":"utils/gpt4_eval.py","file_url":"https://github.com/znlp/x-instruction/blob/HEAD/utils/gpt4_eval.py","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"c70b233c8023d8d3","code_sha256_prefix":"c70b233c8023d8d3","mcp_get_code":{"code_sha256":"c70b233c8023d8d3"}},{"arxiv_id":"2401.10019","arxiv_id_is_arxiv":true,"paper":"/paper/r-judge-benchmarking-safety-risk-awareness","title":"R-Judge: Benchmarking Safety Risk Awareness for LLM Agents","date":"2024-01-18","month_inferred_from_arxiv_id":null,"title_source":"archive","repo":"lordog/r-judge","path":"eval/risk_identification.py","file_url":"https://github.com/lordog/r-judge/blob/HEAD/eval/risk_identification.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"inline_ok_link":false,"withheld_clause":"GitHub detected no licence in this paper's repository, lordog/r-judge when it was checked","code_sha256":"38ff8148457f4eff","code_sha256_prefix":"38ff8148457f4eff","mcp_get_code":{"code_sha256":"38ff8148457f4eff"}},{"arxiv_id":"openreview_JjILY9i6Wi","arxiv_id_is_arxiv":false,"paper":null,"title":"OpenReview id openreview_JjILY9i6Wi","date":null,"month_inferred_from_arxiv_id":null,"title_source":null,"repo":"nwlt/IPMark","path":"baseline/baseline_attack_detect.py","file_url":"https://github.com/nwlt/IPMark/blob/HEAD/baseline/baseline_attack_detect.py","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"inline_ok_link":true,"withheld_clause":null,"code_sha256":"6ce1677e395105a2","code_sha256_prefix":"6ce1677e395105a2","mcp_get_code":{"code_sha256":"6ce1677e395105a2"}}],"record_sha256":"4dc0bd9089138c8e77768f4889c2a2d758ded07d5421cff278f0038a467ed19e","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}