About Code with Papers
Code with Papers is Syntology's community app: a free site, with no account, built on the archived Papers with Code data (last public snapshot 2025-07-28): its papers, benchmarks, datasets and methods, plus one thing the archive never held: for the papers whose code Syntology harvested, each code sample is marked with whether it ran, and what happened when it did. Syntology builds a graph of AI research papers, their code and their citations; this app puts one part of it, the harvested code and whether it ran, next to the archive for anyone to browse, and links into the rest.
Papers with Code told you where the code is. We ran it. In this build, Syntology harvested 294,260 distinct code samples from the repositories of 51,887 papers: 211,169 ran (of which 22,388 constructed an object rather than computing a result; 147,384 with no instrument failure: 15,745 honoured, 5,703 violated, 125,936 with no contract checked; 63,785 where Syntology's instrument failed) · 83,091 unverified. Each sample is one function or class, executed on a synthesized input; the 63,785 where Syntology's instrument failed are failures of Syntology's instrument, not of the code, and the 83,091 unverified were harvested and not run. A sample that ran is not a sample shown to be correct, and a paper with a sample that ran is not a reproduced paper. The per-sample counts are on Code, where the unverified samples are shown beside those that ran, status by status.
paperswithcode.com now redirects to Hugging Face Papers, which lists new papers and links to their code. This site is different in two ways: it serves the archive's leaderboards, datasets and methods as the archive left them (the changes Syntology made are listed on Attribution), and it records, sample by sample, whether the linked code ran.
Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.
Where the data comes from
- Papers, code links, benchmarks, datasets, methods: the archived Papers with Code dataset, as published by the pwc-archive community mirror on Hugging Face (five tables, byte-identical to the mirror; licence CC BY-SA 4.0; snapshot 2025-07-28). Full attribution and what we changed: Attribution & licence.
- Title, abstract, authors and date of papers newer than the snapshot: arXiv's own metadata, CC0 1.0 (arXiv licence page), where held.
- Everything marked Syntology: Syntology's own graph. Repository links found in a paper's own text, code harvested from those repositories, and per-sample execution records. When this page's record last changed is under How current this is.
How current this is. Archive snapshot 2025-07-28. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. The pages that used to print when a build read Syntology's graph now print when their record last changed, and each page's JSON twin carries that date as record_changed_at. A page's record is its JSON twin without the about block every twin opens with, or, for a page with no twin, the page's main text without the site's header and footer. Its date moves when that record changes, not when a build only reads the graph or the Atlas id list again (the build record lists the per-build clocks left out of that comparison); /index/changes.json lists what each build changed and the version of this site's code that built it, so a change a code release made can be told from a change in the data. Nothing on the site changes between builds; the build record also gives the build time and every input.
Which papers have a page. An archive paper has one when the archive gives it a code link, a leaderboard row, or an introduced dataset or method, or when Syntology's graph links it to code; archive papers with only an abstract have none. A paper newer than the snapshot has a page only when Syntology's graph links it to a code repository, or when one of its results was placed on an archive leaderboard, so a widely discussed recent paper is missing until that link exists. Every archive paper left out is counted on what is not shown. A newer paper that Syntology's graph does not link to code is not held by the site and is not counted. Papers in Syntology's layer with harvested samples and no page here: 2,441. Of these, 1,954 carry an id in the export that is not an arXiv id (731 ACL Anthology ids, 489 CVF open access ids, 439 AAAI ids, 117 IJCAI ids, 96 OpenReview ids, 82 bare-number ids), and 487 have an arXiv id from before the snapshot that the archive catalogue does not hold. Their 18,061 sample links, and the 2,343 of these papers with a sample that ran, are counted in the site's sample and paper totals (the ledger).
What "ran" means, exactly, and every label the pages print
Each harvested sample is one function or class. Syntology harvests Python functions and classes from .py files; notebooks and other languages are not read, so a paper whose code is only in notebooks or in another language has no harvested samples here, which does not mean it has no code. Syntology executes each sample on a synthesized fixture and records what happened. Every sample carries exactly one execution status; a sample that ran with no contract checked, or whose contract check could not run, may also carry a metamorphic tier. Licence and repository labels are separate axes: they say where a copy came from and under what licence, never whether it ran.
Execution status (one per sample)
| Label | What it licenses you to conclude |
|---|---|
| ran · honoured contract | The sample executed, and its behaviour on the fixture matched the contract Syntology drafted from the code itself. Not a claim about the paper's results. |
| ran · violated contract | The sample executed, and its behaviour contradicted that drafted contract. Often a real defect; sometimes a wrong contract. |
| ran · our draft was wrong | The sample executed, and the contract Syntology drafted for it did not describe it correctly (its parameters or its output). A failure of Syntology's instrument, not a statement about the code. |
| ran; the contract check could not run | The sample executed on Syntology's synthesized input, but the contract check could not run on it: the check raised an error when it called the sample with inputs shaped by the drafted contract (RAISES), its environment lacked a dependency the sample needs (DEP_MISSING), or it ran past its time limit (TIMEOUT). The row prints which, and its tooltip says what it means. Such a sample can carry a metamorphic tier and a behaviour fingerprint, shown as on a plain “ran” row. A failure of Syntology's instrument, not a statement about the code. |
| ran | The sample executed on the fixture, with no contract checked. Nothing about correctness. |
| unverified | Harvested, not executed: 83,091 of the 294,260 samples in this build. |
| constructed an object | Not a status: a part of the samples that ran, counted beside them. An executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result. This is the rule and the definition of Syntology's own run record for the paper. It says only that the run returned an instance of the sample's own class; it is not a failure, and it says nothing about whether that object is correct. |
“Ran”, wherever a count uses it, means every status above except unverified. A count that includes the constructed figure prints it split this way: “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”. The brick counts on a paper page's Method-as-code cards include the constructed figure only when every brick that ran is among the harvested samples this build reads. Where one is not, the card prints the same split without its “of which C” clause and the section says so. Such a count has not counted the constructed samples; it does not say there were none. “Constructed” (the last row above) is a part of N, never taken away from it. “With no instrument failure” is honoured, violated and ran with no contract checked; “where Syntology's instrument failed” is our draft was wrong and the contract check could not run, failures of Syntology's instrument, not of the code. A count whose every sample that ran constructed an object says so at its end. A paper page whose every sample that ran is a failure of Syntology's instrument says so in words: “every run on this page is a failure of Syntology's instrument, not of the code”.
Earlier builds of this site printed the same count as “N ran (K checked: H honoured, V violated, P no contract; I where Syntology's instrument failed) · U unverified”. The part once labelled “checked” is now labelled “with no instrument failure”, the words Syntology's run record uses to define that part (the record's own key for it is still named checked), because it includes every sample that ran with no contract checked: in this build, 125,936 of its 147,384 samples had no contract checked. The rename changed no number; the constructed count was added beside it.
Metamorphic tier (on a sample labelled “ran”, or “ran; the contract check could not run”)
Syntology's metamorphic check repeats or perturbs the input and records the highest tier the sample's outputs passed: evidence of the output's shape, not that the sample implements the method. The tier is printed after the status on the row (for example “ran · metamorphic tier: invariant”), and each row's tier carries that caption as its tooltip. A sample with no recorded tier shows its status alone.
| Label | What the outputs passed |
|---|---|
| metamorphic tier: deterministic | The same input gives the same output. |
| metamorphic tier: well formed | Outputs are finite, or non-negative where expected. |
| metamorphic tier: invariant | A relation such as scaling or permutation survived a perturbation of the input. |
Behaviour fingerprint
| Label | What it means |
|---|---|
| fingerprinted | A behaviour fingerprint exists: outputs on a shared input battery, comparable with other implementations that accepted the same battery. Nothing is asserted about other inputs. The function-name census compares these fingerprints across papers. |
Repository labels
On a sample, where its repository stands relative to the paper. None of these asserts that the code is the authors' implementation of the paper's method.
| Label on a sample | What it means |
|---|---|
| official repository | The archive marks this repository official for the paper. |
| named in the paper | The archive records that the paper mentions this repository; it is not marked official. |
| community (archive-listed) | In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. |
| found in paper text by Syntology | Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. |
| community | Not in the archive's code links for this paper; a community repository Syntology harvested. |
On a paper's repository list:
| Label on a repository | What it means |
|---|---|
| official | The Papers with Code community marked this repository official for the paper (archive flag). |
| mentioned in paper | The archive records that the paper mentions this repository. |
| mentioned on GitHub | The archive records that the repository mentions the paper. |
| framework | The framework the archive records for the repository, such as pytorch or tf. It is also printed beside the first repository on a home-feed card. |
| found in paper text by Syntology | Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. |
| not reachable when probed date | Syntology's probe of the repository returned an HTTP error, such as not found; hover the label for the code. Repositories for recent papers often appear after camera-ready. |
| empty when probed dates | The repository answered and held no files when probed. |
| licence | The licence the repository's licence file named when probed. |
Licence labels, and what “pointer only” means
No page on this site shows code text. Every sample is linked: its row points at the file in its repository. “Pointer only” means the sample is linked, not reproduced: Syntology does not serve that copy's text, because of the licence records behind it. Two records decide it, and both must allow the text: this paper's copy, and the stored copy of the same code that Syntology serves, which may have been harvested first from another repository. It is a licence property of the code's copies, recorded for samples that ran and samples that did not alike. It says nothing about whether the code runs or is correct.
| Label as printed | When | What it means |
|---|---|---|
| <licence> (permissive) | The licence is on the permissive list, this copy's record is marked cleared, and so is the stored copy of the same code that Syntology serves. | Repository licence at harvest: <licence>. Syntology may serve this code's text; this site links to the file. |
| <licence> (permissive) here; the stored copy is pointer only | This paper's copy is under a licence on the permissive list and its record is marked cleared, but the stored copy of the same code that Syntology serves is not cleared (the example is a stored copy first harvested from a repository with no licence file; the reason printed names that repository and its licence record). | Syntology does not serve this code's text: this paper's link allows the body inline (<licence>), but get_code serves the stored copy, first harvested from another repository, and GitHub detected no licence in that repository when it was checked. This site links to the file in this paper's repository. |
| <licence> recorded; this copy not marked cleared · pointer only | The licence is on the permissive list, but this copy's record is not marked cleared. | The repository's licence is recorded as <licence>, but this copy's record is not marked cleared for Syntology to serve its text, and the record does not say why. Syntology does not serve its text; this record links to the file. |
| <licence> (copyleft) · pointer only | A copyleft licence (GPL, AGPL, LGPL, MPL, EPL, EUPL, CC BY-SA, OSL, CDDL). | <licence> is a copyleft licence: it permits redistribution under its conditions. Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| <licence> · pointer only | Any other licence recorded at harvest, including licences that permit redistribution but are not on the permissive list, such as WTFPL. | Repository licence at harvest: <licence>. Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| no licence file found · pointer only | No licence file was found at harvest. | No licence file was found in the repository at harvest time. Without a licence, Syntology does not serve the code's text; this record links to the file. |
| licence not identified · pointer only | A licence file whose licence was not identified. | The repository has a licence file whose licence was not identified (NOASSERTION). Syntology serves code text only under MIT, Apache-2.0, BSD and similar permissive licences, so this record links to the file instead. |
| licence of this copy not recorded | Identical code first harvested from another repository; this paper's copy was not located. | Identical code Syntology first harvested from another repository; this paper's own copy was not located, so its licence is not recorded. |
<licence> stands for the licence name recorded at harvest, such as MIT or GPL-3.0. The permissive list is MIT, Apache-2.0, BSD and similar licences. In this build, 17,699 of the 142,682 pointer-only paper–sample links (5,897 distinct samples, on 6,853 papers) are the second case: this paper's copy is cleared and the stored copy is not. Each such row's licence tooltip gives Syntology's recorded reason, which is about the stored copy's licence record, never about the paper's own repository.
Where a record comes from
| Label | What it means |
|---|---|
| archive 2025-07-28 | From the archived Papers with Code data, snapshot 2025-07-28, as the archive left it. |
| links table only | The paper is in the archive's code-links table, but its catalogue row (abstract, authors, date) was never published. |
| Syntology | Syntology's own record or measurement, not part of the archive. |
| added by Syntology | A paper not in the archive (frozen 2025-07-28), added from Syntology's graph. |
| In Syntology | Links from the page into Syntology's own services, such as the paper's place on the Atlas (below). |
Function-name census labels
| Label | What it means |
|---|---|
| split | Implementations of one name fall into more than one cluster of identical output on the same shared input: more than one computation goes by that name. |
| one output per bucket | Every compared implementation of the name returned the same output digest on each shared input. Agreement on those inputs, not equivalence. |
| class | A class-bearing member: the class was constructed with seeded weights and then called. |
| non-finite | The recorded output holds a non-finite value (NaN or infinity). |
| same digest as torch: name | The harness also ran torch's own computation under a named convention, and its output digest is the same as this cluster's on this input. |
Served verification levels (Syntology's, printed on method pages)
A method page's In Syntology block prints the level Syntology serves for its answer, such as “level V3”. What each level means is Syntology's own definition, quoted here from its MCP server card as this build read it (the time of that read is in the build record); the card defines 4 levels, V0 to V3. Where the card separates clauses with a double hyphen, each side is quoted on its own.
| Level | Syntology's definition, quoted |
|---|---|
| V0 | “imports” |
| V1 | “executes against a synthesized fixture and returns a value of the recorded type” |
| V2 | “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)” |
| V3 | “paper-derived property tests pass on both of those same two generations.” |
The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.”
Counts on this site are stated per paper and per sample. There is no site-wide score, and the unverified count is always printed beside what ran: what is not shown lists every exclusion with its number.
What this site deliberately does not do
- No GitHub stars, no "trending". The original ranked its feed by stars per hour. That needs constant polling and measures attention, not whether code works. The home feed here is ordered by recency and by whether code ran.
- No new rows inside the archive's tables. Benchmarks are served exactly as the archive left them on 2025-07-28, in the archive's own row order, labelled as such. Results Syntology placed from newer papers (below) are listed in a separate section under a table's archive rows, labelled Syntology, dated, and never ranked among the archive rows. Every leaderboard the extractor proposed and did not place is counted on what is not shown.
- No accounts, no comments, no tracking, and no direct edits. Readers can report errors and propose changes (below), and a person checks each one against the source before anything changes. See the data policy.
- No code text. Samples are linked to their files, never reproduced here (above).
- No claim that code works. "Ran" is not "reproduces the paper". Every label above says what it says and nothing more.
Results placed on leaderboards since the archive Syntology
The archive's leaderboards stopped at 2025-07-28. Syntology places newer papers' results for their own method on those leaderboards this way: a model is shown the paper's own tables (arXiv's HTML rendering) and the archive leaderboards they plausibly belong to, and points at the cell that holds the paper's result; it never writes a number. The code reads the number from that cell and checks it against the leaderboard's metric, dataset, split and scale, and refuses ablation rows and ambiguous configurations. An independent check, which sees the leaderboard's other rows and every other leaderboard on the same dataset, then accepts or rejects each entry. Only accepted entries are shown: 132 on 95 leaderboards in this build, each with the table and row it was read from. None is reviewed by the paper's authors or by the archive's editors. Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.
Measured before publishing, on held-out archive papers: the extractor was given 883 of them and produced output for 881 (the other 2 had no title and were excluded before any model call, so they are not part of this measurement). A model adjudicator (Claude Opus 5.5), blind to which entries were real, judged 108 of 110 accepted entries correct on those held-out archive papers; its own controls: 69 of 70 archive-confirmed entries judged true, 101 of 103 corrupted copies judged false. It was not measured on the newer papers whose results are placed on this site. The rules and the pass bar were frozen before that test was run.
Go deeper in Syntology
Code with Papers shows the archive and Syntology's per-sample run records. Syntology's own services hold more, and reach further than this app does.
The graph
Syntology's graph of AI and machine-learning research: papers, authors, citations, methods and code, with every relation tagged with where it came from. Ask it a question in plain English and the sources come back under the answer.
The Atlas
Papers and the citations between them, drawn as one navigable map. https://app.syntology.ai/?focus=<arXiv id> opens the map on one paper, for a paper the Atlas holds; for any other id the map opens with nothing selected.
API and MCP for agents
get_harvested_code_for_paper(arxiv_id) lists every function and class harvested from a paper, with whether each one ran; it does not verify the paper. get_code(code_sha256) fetches one stored sample. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed.
The way in, for an agent
The same block is in this site's llms.txt and, as about.syntology, on every JSON twin.
- MCP server:
https://syntology.ai/mcp(transport: streamable-http). Server card:/.well-known/mcp/server-card.json, listing 17 tools when this build read it (2026-10-05). - Auth: a trial token, no account and no card, minted by
POST https://syntology.ai/api/oauth/trial/token; the gate and its limits are stated on syntology.ai/developers. - Free coverage check, no token:
https://syntology.ai/api/graph/have?x=<method, arXiv id or title>(free, answers coverage only). - A paper in the graph:
https://syntology.ai/paper/<arXiv id>; its place on the Atlas:https://app.syntology.ai/?focus=<arXiv id>. A paper page here links both only when this build confirmed them. - Syntology's own llms.txt.
syntology.ai has a page for each paper in its graph, at https://syntology.ai/paper/<arXiv id>, linked from that paper's page here; it has no web page per sample, and a sample's code is reached through the API.
For agents
Every paper page has a JSON twin at the same path plus .json, a sitemap index at /sitemap.xml, and the search index, starting at /search/manifest.json. For the graph itself and the stored samples, use Syntology's API and MCP service (syntology.ai/developers).
Stay updated. Atom feeds, rebuilt with the site, holding at most the 100 newest items: /feeds/papers.xml, the newest papers with at least one harvested code sample Syntology ran; /feeds/results.xml, the newest results from newer papers that Syntology placed on archive leaderboards, each accepted by an independent check; not archive rows. A feed entry links to its page here, and a placed result to its leaderboard's “Since the archive” section.
Found something wrong?
If a code link, a leaderboard value, a title, or one of Syntology's own records is wrong or missing, use the small “report” link next to it, or “Report a problem or propose a change” at the foot of any paper, benchmark, task, dataset, method or code page (or send a report directly). A report is a claim, not a correction: nothing changes until a person has checked it against the paper or source. Accepted corrections are made at the next build and marked on the page (“corrected after a report on <date>”, with the archive's original value kept in the marker); removed items are counted on what is not shown; every decision, accepted or not, is listed on /corrections. What a report collects: data policy.
Contact, rights and removal
- Copyright, licence questions, or a repository or sample you want delisted: legal@syntology.ai. The formal copyright-complaint procedure is in section 11 of Syntology's terms.
- Personal data, such as an author name you want corrected or removed: privacy@syntology.ai (Syntology's privacy policy).
- An error on a page: send a report, or use the report link on the page; a person checks it against the source before anything changes.
- Anything else: media@syntology.ai.