Browse › Methodology › Multi-task Language Understanding › MML

Multi-task Language Understanding archive 2025-07-28

MML Benchmark (Multi-task Language Understanding)

44 rows 35 with code listed 1 metric Dataset page

Syntology Since the archive (Syntology): none placed; 59 newer papers judged to report on this board so far.

The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. https://arxiv.org/pdf/2009.03300.pdf

The archive carries no text for this table; the description above is the archive's text for the task Multi-task Language Understanding. archive 2025-07-28

Over time archive 2025-07-28

Not inferred: Average (%). Points are placed at the row's paper date, not an evaluation date; 40 of 44 rows carry one.

The chart needs JavaScript; the table below carries every value.

Results archive 2025-07-28

Archive rows end at the archive snapshot, 2025-07-28: no result published after that date is in this table. Rank is the archive's row order at that snapshot; not re-ranked here. Metric values are the archive's strings. Column headers sort the table in your browser; each row keeps its archive rank.

Paper Code Ran Syntology Report
1 GPT-4 o1(300b) 87 ✓ Paper – 2024 no code linked report
2 Llama 3.1 (405B) 86.6 ✓ Paper Code 2024 linked, not harvested report
3 Llama 3.1 (70B) 86.0 ✓ Paper Code 2024 linked, not harvested report
4 Gemini Ultra (5-shot) 83.7 – – – not matched report
5 Claude 3 Sonnet (5-shot) 79 – Paper – 2024 no code linked report
6 Qwen1.5 72B (5-shot) 77.5 – – – not matched report
7 Claude 3 Haiku (5-shot) 75.2 – Paper – 2024 no code linked report
8 DBRX Instruct 132B (5-shot) 73.7 – Paper Code 2024 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
9 llama 2(65b) 73.5 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
10 Llama 3.1 8B (CoT) 73.0 ✓ Paper Code 2024 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
11 Mixtral 8x7B (5-shot) 70.6 – Paper Code 2024 5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result report
12 GPT-3.5 Turbo 70.0 ✓ Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
13 LLaMA 65B (fine-tuned) 68.9 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
14 chatgpt/gpt3.5(20B) 67.5 – Paper Code 2022 8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified report
15 LLaMA 65B (5-shot) 63.4 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
16 LLaMA 2 34B (5-shot) 62.6 – Paper Code 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified report
17 Mistral 7B (5-shot) 62.5 ✓ Paper Code 2024 5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result report
18 Mistral 7B (5-shot) 60.1 – Paper Code 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
19 GPT-3 Davinci 175B (CoT) 59.5 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
20 LLaMA 33B (5-shot) 57.8 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
21 Falcon 40B 57.0 – Paper – 2023 no code linked report
22 Qwen 7B (5-shot) 56.7 – – – not matched report
23 LLaMA 2 13B (5-shot) 54.8 – Paper Code 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified report
24 Branch-Train-MiX 4x7B (sampling top-1 experts) 53.2 – Paper Code 2024 linked, not harvested report
25 GAL 120B (zero-shot) 52.6 – Paper Code 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
26 Atlas (5-shot) 47.9 – Paper Code 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
27 Flan-T5-XL 3B (CoT) 45.5 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
28 LLaMA 2 7B (5-shot) 45.3 – Paper Code 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified report
29 Flan-T5-Large 780M 45.1 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
30 GLM-130B 44.8 – Paper Code 2022 official (archive's flag): 9 ran · 15 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified report
31 Flan-T5-Large 780M (CoT) 40.5 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
32 GPT-3 Davinci 175B (5-shot) 39.7 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
33 Bloomberg GPT 50B (5-shot) 39.2 – Paper Code 2023 linked, not harvested report
34 UL2 20B (5-shot) 39.2 – Paper Code 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
35 BLOOM 176B (5-shot) 39.1 – Paper Code 2023 linked, not harvested report
36 phi-1.5-web 1.3B 37.9 – Paper Code 2023 linked, not harvested report
37 OPT 66B (5-shot) 36 – Paper Code 2023 linked, not harvested report
38 Flan-T5-Base 250M 35.9 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
39 Flan-T5-Base 250M (CoT) 33.7 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
40 GPT-NeoX 20B (5-shot) 33.6 – Paper Code 2022 community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
41 RWKV v5 Eagle 7B 31 – – – not matched report
42 LLaMA7B-MiLe-Loss(5-shot) 29.68 – Paper Code 2023 linked, not harvested report
43 Flan-T5-Small 80M 28.7 – Paper Code 2022 8 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 9 unverified report
44 Falcon 7B (5-shot) 28.0 – Paper – 2023 no code linked report

All 44 rows shown. 40 link to a paper page on this site; 6 are marked as using additional training data in the archive. No GitHub stars are tracked; "Code" is the first repository the archive lists for the row. The archive carries no row tags, review links or community-submitted rows for this table; none are shown. archive 2025-07-28

Syntology Ran reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”: of the code samples Syntology harvested from repositories linked to that row's paper (joined by arXiv id), N executed on a synthesized input and U are unverified (harvested, no recorded run). Of the N, C constructed an object rather than computing a result (a part of N, never taken away from it), K ran with no instrument failure (honoured or violated the contract Syntology drafted, or ran with no contract checked) and I are failures of Syntology's instrument, not of the code (a wrong contract draft, or a contract check that could not run). It counts code from repositories linked to that row's paper, not this result: the row's number was not reproduced and nothing here is a correctness claim. The other cell texts mean no graph line for the row: "linked, not harvested" (the archive links code, Syntology has not harvested it), "no code linked" (no code link in the archive), "not matched" (the row's paper URL matched no paper on this site). 27 rows have a graph line, from 13 distinct papers. Rows whose paper has a sample that ran: 27 (27 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument). Papers: 13 with a sample that ran: 13 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument. Whose code ran. When the archive marks a repository official for the row's paper, the cell starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code): “official (archive's flag): N ran” counts the samples from it that ran, instrument failures included; “official: harvested, nothing ran”; when it has no sample for this paper and other samples ran, “community repositories only” if every one came from a repository the archive lists for the paper but neither marks official nor records as mentioned in it, or one Syntology harvested outside the archive's links, and “official: no sample here; runs from other or unrecorded repositories” if any came from a repository named in the paper or found in its text by Syntology, or has no repository recorded; “official: harvested for another paper” (no sample from it is linked to this paper) or “official: not harvested”. No state is printed when the archive marks none official. The cell's tooltip names the repositories the samples that ran came from. Of the 27 rows with a graph line, 13 have an archive-official repository: on 4 a sample from it ran (4 with a run with no instrument failure), on 6 the samples that ran came from community repositories only, and on 3 from other or unrecorded repositories. The filters above hide rows and never renumber them; the first two count, by default, only a run with no instrument failure, and offer any run, instrument failures included. Counting each paper once, of 212 samples: 151 ran (of which 25 constructed an object rather than computing a result; 113 with no instrument failure: 7 honoured, 4 violated, 102 with no contract checked; 38 where Syntology's instrument failed) · 61 unverified. Separately, 35 of those 212 are pointer-only (licence): the site points at that code rather than redistributing it, a licence property recorded for ran and unverified samples alike; each cell's tooltip carries the row's own pointer-only count. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. Per-sample status is on the paper page; each Ran cell links to that paper's sample list. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Since the archive Syntology

Of the 8,888 newer papers whose tables Syntology has read (8,886 checked in full and 2 with a leaderboard still awaiting a verdict), 179 had this board among the leaderboards the extractor considered for their tables (it keeps at most 10 boards per paper and 4 per dataset; the lists of 1,483 of the 8,888 were already full without this board, so if it was crowded out of them it is not counted), and 59 proposed a result; of those 59, 57 gave no cell that passed the extractor's own checks, 2 were rejected by the independent check, 0 await a verdict and 0 were placed. Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.

Newer papers whose tables Claude Sonnet 4.5 judged to report on this benchmark (a model's judgement, not a result; newest first by arXiv id)

Paper arXiv date Ran Syntology Outcome
Towards Efficient Reasoning: Learning Causal Shortcuts for Diffusion Language Models arXiv:2609.28272 23 Sep 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation arXiv:2609.21677 18 Sep 2026 repository linked, no samples harvested refused by a rule: the value is not on the scale of the leaderboard column's archive values
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction arXiv:2609.18063 16 Sep 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Instella-MoE Technical Report arXiv:2609.00791 1 Sep 2026 5 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs arXiv:2609.00575 1 Sep 2026 16 harvested: 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 11 where Syntology's instrument failed) · 2 unverified refused by a rule: the table's column header does not name the leaderboard's metric
SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature arXiv:2608.30214 31 Aug 2026 8 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified refused by a rule: the row is an ablation variant, not the method itself
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 arXiv:2608.27370 27 Aug 2026 8 harvested: 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 3 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy arXiv:2608.15602 16 Aug 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Training-Free Hashing-Based Attention via Binary Principal Components arXiv:2608.04405 5 Aug 2026 3 harvested: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; the one sample that ran constructed an object rather than computing a result refused by a rule: the table's column header does not name the leaderboard's metric
AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion... arXiv:2608.02602 3 Aug 2026 repository linked, no samples harvested rejected by the independent check
CBD: API-Only LLM Black-Box Unlearning through Controlled Behavioral Divergence arXiv:2606.27683 26 Jun 2026 11 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization arXiv:2606.00079 22 May 2026 5 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified refused by a rule: the table's column header does not name the leaderboard's metric
ExpGraph: Model-Agnostic Experience Learning with Graph-Structured Memory for LLM Agents arXiv:2605.30712 29 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Curriculum Learning for Safety Alignment arXiv:2605.26315 25 May 2026 19 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 9 unverified refused by a rule: several configurations of the method, and no metric direction to pick one
Hide to Guide: Learning via Semantic Masking arXiv:2605.25198 24 May 2026 10 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified refused by a rule: the table's column header does not name the leaderboard's metric
NITP: Next Implicit Token Prediction for LLM Pre-training arXiv:2605.24956 24 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Lngram: N-gram Conditional Memory in Latent Space arXiv:2605.24869 24 May 2026 9 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified refused by a rule: the table's column header does not name the leaderboard's metric
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs arXiv:2605.23078 21 May 2026 1 harvested: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result refused by a rule: several configurations of the method, and no metric direction to pick one
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation arXiv:2605.21699 20 May 2026 repository linked, no samples harvested refused by rules: the row label the model quoted is not the row's label; the table's column header does not name the leaderboard's metric
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining arXiv:2605.20296 19 May 2026 10 harvested: 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified refused by a rule: the model gave no value to check against the cell
Prune, Update and Trim: Robust Structured Pruning for Large Language Models arXiv:2605.18331 18 May 2026 5 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified refused by a rule: several configurations of the method, and no metric direction to pick one
Adaptive Steering and Remasking for Safe Generation in Diffusion Language Models arXiv:2605.13043 13 May 2026 9 harvested: 9 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 0 unverified refused by a rule: the row is an ablation variant, not the method itself
From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuning arXiv:2605.12944 13 May 2026 8 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Macro: Enhancing Multilingual Counterfactual Explanations through... arXiv:2605.11632 12 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Mixture of Heterogeneous Grouped Experts for Language Modeling arXiv:2604.23108 25 Apr 2026 2 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual... arXiv:2603.28858 30 Mar 2026 repository linked, no samples harvested refused by rules: the table's column header does not name the leaderboard's metric; several configurations of the method, and no metric direction to pick one
SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection arXiv:2603.22213 23 Mar 2026 11 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified refused by a rule: one configuration gave two values for one column
IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring arXiv:2603.13792 14 Mar 2026 held, no recorded run rejected by the independent check
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR arXiv:2603.10101 10 Mar 2026 10 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified refused by a rule: several configurations of the method, and no metric direction to pick one
SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization arXiv:2603.08185 9 Mar 2026 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified refused by a rule: one configuration gave two values for one column
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought arXiv:2603.05488 5 Mar 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
CRISP: Compressed Reasoning via Iterative Self-Policy Distillation arXiv:2603.05433 5 Mar 2026 6 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning arXiv:2602.17835 19 Feb 2026 7 harvested: 5 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified refused by a rule: the row label the model quoted is not the row's label
Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling arXiv:2602.10623 11 Feb 2026 repository linked, no samples harvested refused by rules: the row label the model quoted is not the row's label; the number the model quoted is not the number printed in the cell it pointed at
Your Language Model Secretly Contains Personality Subnetworks arXiv:2602.07164 6 Feb 2026 4 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified refused by a rule: the table's column header does not name the leaderboard's metric
CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs arXiv:2602.05258 5 Feb 2026 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Models arXiv:2602.04448 4 Feb 2026 13 harvested: 10 ran (of which 1 constructed an object rather than computing a result; 6 with no instrument failure: 5 honoured, 0 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified refused by a rule: the table's column header does not name the leaderboard's metric
How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning arXiv:2602.02103 2 Feb 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Who Transfers Safety? Identifying and Targeting Cross-Lingual Shared Safety Neurons arXiv:2602.01283 1 Feb 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for... arXiv:2601.18533 26 Jan 2026 5 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 4 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified refused by rules: the table's column header does not name the leaderboard's metric; the quoted value appears more than once in the row
OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts... arXiv:2601.12996 19 Jan 2026 4 harvested: 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 4 samples that ran constructed an object rather than computing a result refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts arXiv:2601.10272 15 Jan 2026 2 harvested: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result refused by a rule: the table's column header does not name the leaderboard's metric
StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided... arXiv:2601.04740 8 Jan 2026 4 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
FOAM: Blocked State Folding for Memory-Efficient LLM Training arXiv:2512.07112 8 Dec 2025 6 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified refused by a rule: several configurations of the method, and no metric direction to pick one
Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning arXiv:2511.12344 15 Nov 2025 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling arXiv:2511.00056 28 Oct 2025 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic arXiv:2510.24061 28 Oct 2025 15 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 10 where Syntology's instrument failed) · 3 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning arXiv:2510.16882 19 Oct 2025 repository linked, no samples harvested refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs arXiv:2510.10467 12 Oct 2025 27 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 15 unverified refused by a rule: one configuration gave two values for one column
The Alignment Waltz: Jointly Training Agents to Collaborate for Safety arXiv:2510.08240 9 Oct 2025 repository linked, no samples harvested refused by a rule: the row is an ablation variant, not the method itself
TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA arXiv:2510.04682 6 Oct 2025 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified refused by a rule: the row label the model quoted is not the row's label
Cache-to-Cache: Direct Semantic Communication Between Large Language Models arXiv:2510.03215 3 Oct 2025 14 harvested: 13 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 3 honoured, 1 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Stochastic Self-Organization in Multi-Agent Systems arXiv:2510.00685 1 Oct 2025 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified refused by a rule: one configuration gave two values for one column
MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open... arXiv:2509.24945 29 Sep 2025 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data... arXiv:2509.23371 27 Sep 2025 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection arXiv:2509.16060 19 Sep 2025 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in... arXiv:2509.14651 18 Sep 2025 repository linked, no samples harvested refused by rules: the row label the model quoted is not the row's label; the table's column header does not name the leaderboard's metric
AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models arXiv:2509.12019 15 Sep 2025 6 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified refused by rules: the row label the model quoted is not the row's label; the table's column header does not name the leaderboard's metric
Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models arXiv:2508.00410 1 Aug 2025 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified refused by a rule: the row is an ablation variant, not the method itself

Syntology 59 newer papers judged to report on this board: the papers counted above as having proposed a result (0 placed); every one is listed here and in the JSON twin. The order is the arXiv identifier's, newest first, not a rank. The judgement is the model's, read from the paper's own tables; the outcome is what Syntology's checks did with it, and no value from a paper that was not placed is printed. “refused by a rule” names the extractor's rule; “no cell passed the extractor's checks” means the extractor's record holds no cell for this board, passed or refused. The 57 counted above as giving no cell that passed the extractor's own checks are the 57 marked “refused by a rule”. Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections