{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/not-shown","type":"not_shown","what":"the build's exclusion ledger: what the archive and Syntology's graph hold that no page renders, with the number and the reason","rule":"the same dict the page renders (static_pages.not_shown_context); every number on the page is in this twin","built_at":"2026-10-05T01:46:13Z","snapshot":"2025-07-28","graph_read_at":"2026-10-05T00:36:11+00:00","pages":[{"what":"papers in the archive catalogue","count":625187,"why":"abstracts table plus links-only rows"},{"what":"paper pages rendered","count":225441,"why":"papers with a code link, a benchmark row, an introduced dataset or method, or Syntology samples"},{"what":"papers without a page","count":399746,"why":"abstract-only: nothing runnable or ranked to show; the search index does not list them"},{"what":"method entries removed as spam (name pattern)","count":12604,"why":"customer-service phrasing in the name, counted over the methods table and the paper catalogue's embedded method list; the pattern is public in the build script"},{"what":"duplicate leaderboard tables collapsed","count":10354,"why":"the same benchmark reached from several task nodes"},{"what":"method entries removed as spam (dummy-paper clusters)","count":6785,"why":"entries hung off five dummy paper records, counted over the methods table and the paper catalogue's embedded method list; see the attribution page"},{"what":"dataset rows removed as spam","count":2794,"why":"contact-centre spam in the dataset's name or full name; the pattern is public in the build script"},{"what":"papers excluded for having no title","count":636,"why":"a page needs a title"},{"what":"per-paper method tags dropped as spam","count":37766,"why":"the method-name pattern, applied to the tags the paper catalogue carries"},{"what":"dataset descriptions blanked as vandalised","count":201,"why":"row kept, text removed"},{"what":"method-collection descriptions blanked as vandalised","count":65,"why":"text removed"},{"what":"method descriptions blanked as vandalised","count":22,"why":"row kept, text removed"},{"what":"task descriptions blanked as vandalised","count":5,"why":"row kept, text removed"},{"what":"per-paper method tags with no method page","count":4230,"why":"shown as plain text, not linked"},{"what":"per-paper task tags with no task page","count":29815,"why":"not shown; the archive tagged papers with labels that never had a task page"},{"what":"benchmark dataset nodes with no table","count":190,"why":"no metrics and no rows"},{"what":"leaderboard rows with no paper page","count":3740,"why":"row has no paper URL, or a URL that matches no catalogue paper"},{"what":"benchmarks whose dataset has no dataset page","count":3096,"why":"name matched nothing in the dataset catalogue"},{"what":"task slugs derived rather than archived","count":1324,"why":"the archive's task URLs cover only tasks that a dataset lists"},{"what":"Syntology: paper pages whose arXiv id is not in the Atlas id list","count":108712,"why":"checked against the Atlas id list (549,449 ids, as this build read them; the read time is in the build record); these pages link to no Syntology page and do not say the graph lacks the paper, because the list is not the whole graph (7,185 papers in this site's copy of the graph have an arXiv id the list lacks); Syntology's feed shows papers as they enter the graph"},{"what":"Syntology: paper pages that cannot be joined (no arXiv id)","count":12898,"why":"this site joins papers to Syntology's graph by arXiv id; these pages do not say whether the graph holds the paper"}],"papers_without_a_page":{"papers":2441,"non_arxiv":1954,"non_arxiv_by_family":{"acl_anthology":731,"cvf":489,"aaai":439,"ijcai":117,"openreview":96,"bare_number":82},"pre_snapshot":487,"other":0,"links":18061,"samples_distinct":16187,"papers_with_a_ran_sample":2343,"counted_in_totals":true,"rule":"papers in Syntology's layer with harvested samples and no page here; counted in the headline sample and paper totals they already inflate; never given a page or a syntology.ai link (a non-arXiv id is never read as an arXiv id); the ids are repaired upstream in export_graph_layer.py, not here","sentence":"Papers in Syntology's layer with harvested samples and no page here: 2,441. Of these, 1,954 carry an id in the export that is not an arXiv id (731 ACL Anthology ids, 489 CVF open access ids, 439 AAAI ids, 117 IJCAI ids, 96 OpenReview ids, 82 bare-number ids), and 487 have an arXiv id from before the snapshot that the archive catalogue does not hold. Their 18,061 sample links, and the 2,343 of these papers with a sample that ran, are counted in the site's sample and paper totals","causes":"Of these, 1,954 carry an id in the export that is not an arXiv id (731 ACL Anthology ids, 489 CVF open access ids, 439 AAAI ids, 117 IJCAI ids, 96 OpenReview ids, 82 bare-number ids), and 487 have an arXiv id from before the snapshot that the archive catalogue does not hold. Their 18,061 sample links, and the 2,343 of these papers with a sample that ran, are counted in the site's sample and paper totals","family_label":{"acl_anthology":"ACL Anthology","cvf":"CVF open access","aaai":"AAAI","ijcai":"IJCAI","openreview":"OpenReview","bare_number":"bare-number"}},"content_rules":{"rows":[{"what":"dataset rows withheld, all reasons","count":40},{"what":"dataset rows withheld: affiliate or product advertising","count":24},{"what":"dataset rows withheld: adult","count":4},{"what":"dataset rows withheld: travel-booking or airline-support spam","count":4},{"what":"dataset rows withheld: gambling advertising","count":4},{"what":"dataset rows withheld: payment or crypto support","count":2},{"what":"dataset rows withheld: tech support","count":1},{"what":"dataset rows withheld: contact-centre spam","count":1},{"what":"method entries withheld, all reasons","count":32},{"what":"method entries withheld: contact-centre spam","count":10},{"what":"method entries withheld: travel-booking or airline-support spam","count":7},{"what":"method entries withheld: affiliate or product advertising","count":6},{"what":"method entries withheld: gambling advertising","count":3},{"what":"method entries withheld: pharmacy advertising","count":2},{"what":"method entries withheld: a test entry","count":2},{"what":"method entries withheld: essay mill","count":1},{"what":"method entries withheld: name and text replaced by an advert (affiliate or product advertising)","count":1},{"what":"task entries withheld, all reasons","count":2},{"what":"task entries withheld: contact-centre spam","count":2},{"what":"leaderboard tables withheld, all reasons","count":27},{"what":"leaderboard tables withheld: its only row was a test entry","count":23},{"what":"leaderboard tables withheld: every row was withheld as spam","count":3},{"what":"leaderboard tables withheld: its task was withheld","count":1},{"what":"leaderboard rows withheld, all reasons","count":26},{"what":"leaderboard rows withheld: a single row credited to a dummy paper record","count":23},{"what":"leaderboard rows withheld: contact-centre spam","count":2},{"what":"leaderboard rows withheld: travel-booking or airline-support spam","count":1},{"what":"dataset rows: description blanked, row kept: llm paste","count":35},{"what":"method entries: description blanked, row kept: one junk word as the whole text","count":2},{"what":"dataset rows: licence link blanked, row kept: it pointed at a spam or darknet-market host","count":1},{"what":"dataset rows: description blanked, row kept: darknet advertising","count":1},{"what":"dataset rows: licence name blanked, row kept: it repeated the spam name that replaced the dataset's own name in the same edit (the name was restored from the dataset's address; the licence link was removed with it)","count":1},{"what":"method entries: description blanked, row kept: a phone number","count":1},{"what":"task entries: description blanked, row kept: a phone number","count":1},{"what":"dataset names changed in the same edit as a spam description, restored from the dataset's address","count":2},{"what":"distinct method tag names on papers withheld as spam","count":255},{"what":"distinct task tag names on papers withheld as spam","count":1},{"what":"method \"see code\" links shown as plain text: not a code host","count":26},{"what":"method source links shown as plain text: not a paper host","count":12}],"rules":"cwp/cwp_lib/data.py PROSE_RULES / NAME_RULES (NFKC-folded text); applied at build time to TEMP copies of the catalog tables, never to the catalog file"},"extracted_results":{"coverage":{"population_definition":"papers in graph_layer.jsonl marked post_freeze that carry a title (the graph's, else the catalog's): the population lb_extract.py --run --population site processes","population_source":"graph_layer.jsonl as read by this build","M_papers_newer_than_archive":18637,"N_papers_checked":8886,"papers_checked_no_candidate_board":3717,"R_papers_read_nothing_to_place":733,"papers_read_nothing_to_place_by_status":{"no_html":260,"no_tables":473},"F_papers_could_not_be_read":41,"papers_could_not_be_read_by_status":{"unparseable":41},"E_papers_extracted_not_verified":0,"U_boards_without_verdict":3,"boards_without_verdict":{"on_papers_not_yet_verified":0,"check_did_not_answer_or_unparseable":3},"papers_verified_with_a_board_awaiting_verdict":2,"papers_not_yet_extracted":8975,"papers_outside_population":{"extracted":0,"verified":0},"extract_file":"cwp/data/lb_rows_run.jsonl","extract_file_present":true,"extract_file_sha256_as_read":"62e8b7e1d5aa85213b0b588f865a8f949b4b8f73d51cc528dce1de67465aa3cf","extract_file_bytes_as_read":29018600,"extract_file_lines_unreadable":0,"extract_file_papers":9662,"extract_run_papers_examined_per_sidecar":9665,"complete":false,"sentence":"Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.","short":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","post_freeze_pages_on_site":18637,"population_without_page_on_site":0},"present":true,"source":"lb_extract (leaderboard-shaped extractor), verified","file":"cwp/data/lb_rows_run_verified.jsonl","file_sha256_as_read":"e64d2b4c9755cec39adae45b8f3820ada1e499898dbff0b77441e2be8ed71f51","file_bytes_as_read":29998752,"file_lines":{"lines":9662},"papers_in_file":9662,"status":{"no_html":260,"no_candidate_board":3717,"answered":5171,"no_tables":473,"unparseable":41},"boards_proposed":4465,"refused_by_rule":{"metric_not_in_column_header":3803,"cell_text_mismatch":1224,"dataset_not_in_table":849,"scale_mismatch":749,"no_quoted_value":648,"label_mismatch":628,"ablation_label":547,"one_configuration_two_values":273,"split_mismatch":265,"header_row":206,"several_configurations_no_direction":157,"quoted_value_ambiguous_in_row":141,"metric_qualifier_in_header":70,"board_column_not_on_board":36,"bad_coordinates":34,"header_names_two_metrics":27,"not_a_configuration_row":13},"refused_by_rule_cells":9670,"verifier":{"reject":969,"accept":132,"unparseable":3},"declined_by_site":{},"entries_accepted":132,"cells_accepted":160,"boards_with_entries":95,"papers_with_entries":93,"configuration_not_shown":{"names_the_metric":8,"names_the_dataset":9,"generic_header_word":2},"measurement":{"test_papers":883,"papers_with_output":881,"papers_with_output_from":null,"accepted_entries":110,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"adjudicator":"Claude Opus 5.5","adjudicator_from":"groundwork/59_code_with_papers/LB_EXTRACTOR_PREREG.md (not recorded in lb_measurement.json; the file's frozen_commit, measured_on and counts equal the run the pre-registration records)","adjudicator_sensitivity":[69,70],"adjudicator_specificity":[101,103],"passed":true,"frozen_commit":"0e3de0df94","result_commit":null,"measured_on":"2026-09-24"},"measurement_sha256_as_read":"f33f040865aa9b360a85c26e097b3f2d66cb34149574b9366039d37791554b75","rows":[{"what":"papers on this site newer than the archive","count":18637,"why":"the extractor reads every one, and the independent check follows wherever it proposes a leaderboard; counted from this build's graph layer"},{"what":"  checked","count":8886,"why":"the extractor read the paper's tables and compared them with the archive's leaderboards, and the independent check gave a verdict (accept or reject) on every leaderboard the extractor proposed; for 3,717 of these papers no archive leaderboard matched the tables, so none was proposed; a paper checked in part can also show a result, on the leaderboards the check accepted"},{"what":"  read, nothing to place","count":733,"why":"arXiv has no HTML version of the paper (260), or that version has no tables (473); there is no table to place a result from, so these are not counted as checked"},{"what":"  could not be read","count":41,"why":"the extractor's reply could not be parsed; not counted as checked, and nothing from these papers is shown"},{"what":"  read by the extractor, not yet checked","count":0,"why":"nothing from these papers is shown until the check has run"},{"what":"  checked in part","count":2,"why":"the check gave no verdict on at least one of the paper's leaderboards; nothing unchecked is shown"},{"what":"  not yet read by the extractor","count":8975,"why":"the extractor has not reached the paper yet, or its last attempt stopped before writing a result and the paper is read again on the next run"},{"what":"leaderboards awaiting a verdict","count":3,"why":"(paper, leaderboard) pairs with rule-passing cells the independent check has not yet accepted or rejected; none is shown"},{"what":"papers with a line in the verified file","count":9662,"why":"one line per paper the extractor finished with, whatever the outcome, read once at build time (29,998,752 bytes, sha256 e64d2b4c9755…); by outcome below"},{"what":"papers: answered","count":5171,"why":"the extractor answered for the paper's tables"},{"what":"papers: no candidate board","count":3717,"why":"no archive leaderboard plausibly matches the paper's tables"},{"what":"papers: no tables","count":473,"why":"the paper's HTML rendering has no tables"},{"what":"papers: no html","count":260,"why":"arXiv holds no HTML rendering of the paper, so it has no readable tables"},{"what":"papers: unparseable","count":41,"why":"the extractor's reply could not be parsed; counted as could not be read, never as checked"},{"what":"leaderboards proposed","count":4465,"why":"(paper, leaderboard) pairs the model said report the paper's own method on that leaderboard"},{"what":"refusals by a rule","count":9670,"why":"each cell, row or configuration the model pointed at that failed a deterministic check, counted once under the first check it failed (below); none is shown"},{"what":"  rule: metric not in column header","count":3803,"why":"the table's column header does not name the leaderboard's metric"},{"what":"  rule: cell text mismatch","count":1224,"why":"the number the model quoted is not the number printed in the cell it pointed at"},{"what":"  rule: dataset not in table","count":849,"why":"the table does not name the leaderboard's dataset"},{"what":"  rule: scale mismatch","count":749,"why":"the value is not on the scale of the leaderboard column's archive values"},{"what":"  rule: no quoted value","count":648,"why":"the model gave no value to check against the cell"},{"what":"  rule: label mismatch","count":628,"why":"the row label the model quoted is not the row's label"},{"what":"  rule: ablation label","count":547,"why":"the row is an ablation variant, not the method itself"},{"what":"  rule: one configuration two values","count":273,"why":"one configuration gave two values for one column"},{"what":"  rule: split mismatch","count":265,"why":"the table reports a different split or setting than the leaderboard"},{"what":"  rule: header row","count":206,"why":"the model pointed at a header row"},{"what":"  rule: several configurations no direction","count":157,"why":"several configurations of the method, and no metric direction to pick one"},{"what":"  rule: quoted value ambiguous in row","count":141,"why":"the quoted value appears more than once in the row"},{"what":"  rule: metric qualifier in header","count":70,"why":"the column header qualifies the metric in a way the leaderboard does not"},{"what":"  rule: board column not on board","count":36,"why":"the column named is not one of the leaderboard's columns"},{"what":"  rule: bad coordinates","count":34,"why":"the table, row or column pointed at does not exist"},{"what":"  rule: header names two metrics","count":27,"why":"the column header names two metrics"},{"what":"  rule: not a configuration row","count":13,"why":"the row is not a configuration of the method"},{"what":"independent check: reject","count":969,"why":"rejected by the independent check"},{"what":"independent check: accept","count":132,"why":"accepted by the independent check; placed on the leaderboard"},{"what":"independent check: unparseable","count":3,"why":"the independent check's reply could not be parsed; not placed"},{"what":"entries placed on leaderboards","count":132,"why":"one per paper per leaderboard; only these render"},{"what":"cells placed","count":160,"why":"the numbers those entries print, one per leaderboard column"},{"what":"leaderboards carrying an entry","count":95,"why":"listed below the archive rows, never ranked among them"},{"what":"papers with an entry","count":93,"why":"each has a paper page that says where"}],"measurement_full":{"what":"Blind adjudication of the leaderboard extractor's accepted entries on a held-out test split of archive papers, against a bar committed before the test (groundwork/59_code_with_papers/LB_EXTRACTOR_PREREG.md).","test_papers":883,"papers_with_output":881,"accepted_entries":110,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"adjudicator_sensitivity":[69,70],"adjudicator_specificity":[101,103],"bar":{"point":0.95,"wilson_lower":0.9,"min_n":60},"passed":true,"frozen_commit":"0e3de0df94","measured_on":"2026-09-24","note":"883 attempted; 2 excluded before any call (no title); 881 produced output. The earlier round-one statement of 883 covered only 678 because of a threading crash, corrected the same day.","_provenance":{"schema":"syntology-provenance/1","created_at":"2026-09-24T16:07:39+00:00","generator":{"script":"-","argv":[],"git_commit":"f4dd223087e6307cd61e30159101c42ace439949","git_dirty":true},"inputs":[{"path":"cwp/data/lb_test_combined_score.json","sha256":"251b0be1cc7f13af045ea4da378f40aed26d1d8b61d357700e9051b9e0188340","bytes":3513,"mtime":"2026-09-24T16:07:13+00:00"}]},"adjudicator":"Claude Opus 5.5","adjudicator_from":"groundwork/59_code_with_papers/LB_EXTRACTOR_PREREG.md (not recorded in lb_measurement.json; the file's frozen_commit, measured_on and counts equal the run the pre-registration records)"},"measurement_sentence":"Measured before publishing, on held-out archive papers: the extractor was given 883 of them and produced output for 881 (the other 2 had no title and were excluded before any model call, so they are not part of this measurement). A model adjudicator (Claude Opus 5.5), blind to which entries were real, judged 108 of 110 accepted entries correct on those held-out archive papers; its own controls: 69 of 70 archive-confirmed entries judged true, 101 of 103 corrupted copies judged false. It was not measured on the newer papers whose results are placed on this site."},"post_freeze_pages":{"pages":18637,"with_arxiv_metadata":18637,"without_arxiv_metadata":0,"claims_only":0,"meta_present":true},"reader_reports":{"present":false,"error":null,"rows":[],"removed":[],"applied":0,"not_applied":0,"shown_on_page":true},"syntology_layer":[{"what":"papers in Syntology's layer","count":63466,"unit":"papers"},{"what":"papers with harvested code samples","count":51887,"unit":"papers"},{"what":"papers newer than the archive with a repository link in the graph","count":18637,"unit":"papers"},{"what":"distinct code samples","count":294260,"unit":"samples"},{"what":"samples that ran (any status except unverified)","count":211169,"unit":"samples"},{"what":"  of which checked (honoured, violated, or ran with no contract)","count":147384,"unit":"samples"},{"what":"  of which Syntology's instrument failed (our draft was wrong, or the contract check could not run): failures of Syntology's instrument, not of the code","count":63785,"unit":"samples"},{"what":"ran, honoured the contract we drafted","count":15745,"unit":"samples"},{"what":"ran, violated the contract we drafted","count":5703,"unit":"samples"},{"what":"ran; our contract draft was wrong, not the code","count":50332,"unit":"samples"},{"what":"ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)","count":13453,"unit":"samples"},{"what":"ran on a synthesized input","count":125936,"unit":"samples"},{"what":"unverified (harvested, no recorded run)","count":83091,"unit":"samples"},{"what":"samples with a behaviour fingerprint","count":58064,"unit":"samples"},{"what":"links whose sample has no function or class name","count":0,"unit":"links"},{"what":"paper–sample links","count":361576,"unit":"links"},{"what":"links that are pointer only (Syntology does not serve that copy's text)","count":142682,"unit":"links"},{"what":"  of which this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not","count":17699,"unit":"links"},{"what":"links where this paper's copy is the one Syntology first harvested","count":243453,"unit":"links"},{"what":"links whose repository and path the harvester recorded for this paper","count":100333,"unit":"links"},{"what":"links whose repository and path come from a harvest plan row for this paper","count":12948,"unit":"links"},{"what":"links to identical code first harvested for another paper (this paper's copy not located)","count":4842,"unit":"links"}],"unverified_is_largest_status":false,"graph_summary_counts":{"distinct_samples_written":294260,"link_basis:first_harvest_node":243453,"link_basis:harvester_set":100333,"link_basis:identical_code_first_harvested_elsewhere":4842,"link_basis:plan_row":12948,"links_link_open_stored_copy_withheld":17699,"links_written":361576,"papers_link_open_stored_copy_withheld":6853,"papers_total":63466,"papers_with_harvested_samples":51887,"papers_written":63466,"post_freeze_repo_linked":18637,"post_freeze_without_date":17639,"samples_link_open_stored_copy_withheld":5897,"samples_pointer_only":142682,"samples_pointer_only_link_gate":124983,"seconds":512.4,"status_ran":138300,"status_ran_draft_wrong":74974,"status_ran_fixture":20126,"status_ran_honours":23637,"status_ran_violates":10238,"status_unverified":94301},"record_sha256":"fc8b6daaa7133062a5a76ab495a1165467c090eb830714c0de18ef2e93857bc3","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}