| 1 |
Gemini 2.0 Flash Experimental |
89.7 | | – |
– |
– |
|
not matched |
report |
| 2 |
Qwen2.5-Math-72B-Instruct(TIR,Greedy) |
88.1 | 72 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 3 |
GPT-4 Turbo (MACM, w/code, voting) |
87.920 | | – |
Paper |
Code |
2024 |
official: harvested, nothing ran · 0 ran · 1 unverified |
report |
| 4 |
Qwen2.5-Math-72B-Instruct(COT,Greedy) |
85.9 | 72 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 5 |
Qwen2.5-Math-7B-Instruct(TIR,Greedy) |
85.2 | 7 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 6 |
GPT-4-code model (CSV, w/ code, SC, k=16) |
84.3 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 7 |
Qwen2-Math-72B-Instruct(greedy) |
84.0 | 72 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 8 |
Qwen2.5-Math-7B-Instruct(COT,Greedy) |
83.6 | 7 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 9 |
Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) |
79.9 | 1.5 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 10 |
OpenMath2-Llama3.1-70B (majority@256) |
79.6 | | ✓ |
Paper |
Code |
2024 |
12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified |
report |
| 11 |
OpenMath2-Llama3.1-8B (majority@256) |
76.1 | | ✓ |
Paper |
Code |
2024 |
12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified |
report |
| 12 |
Qwen2.5-Math-1.5B-Instruct(COT,Greedy) |
75.8 | 1.5 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 13 |
GPT-4-code model (CSV, w/ code) |
73.5 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 14 |
CR (GPT-4-turbo model, w/ code) |
72.2 | | – |
Paper |
Code |
2023 |
official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified |
report |
| 15 |
OpenMath2-Llama3.1-70B |
71.9 | | ✓ |
Paper |
Code |
2024 |
12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified |
report |
| 16 |
LogicNet (with code interpreter) |
71.2 | | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 17 |
Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) |
70.8 | | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 18 |
GPT-4-code model (w/ code) |
69.7 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 19 |
OpenMath2-Llama3.1-8B |
67.8 | | ✓ |
Paper |
Code |
2024 |
12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified |
report |
| 20 |
AlphaMath-7B-SBS@3 |
66.3 | | – |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 21 |
Minerva 62B (maj5@256) |
64.9 | 62 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 22 |
DAMOMath-7B |
64.5 | 7 | ✓ |
– |
– |
|
not matched |
report |
| 23 |
MMOS-DeepSeekMath-7B(0-shot,k=50) |
63.7 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified |
report |
| 24 |
GPT-4-code model (w/o code) |
60.8 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 25 |
OpenMath-CodeLlama-70B (w/ code, SC, k=50) |
60.4 | 70 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 26 |
OpenMath-CodeLlama-34B (w/ code, SC, k=50) |
60.2 | 34 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 27 |
ToRA-Code 34B model (w/ code, SC, k=50) |
60.0 | 34 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 28 |
DeepSeekMATH-RL-7B (w/ code, greedy decoding) |
58.8 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 8 ran · 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 11 where Syntology's instrument failed) · 10 unverified |
report |
| 29 |
OpenMath-Llama2-70B (w/ code, SC, k=50) |
58.3 | 70 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 30 |
CR (GPT-4 model, w/o code) |
58.0 | | – |
Paper |
Code |
2023 |
official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified |
report |
| 31 |
OpenMath-CodeLlama-13B (w/ code, SC, k=50) |
57.6 | 13 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 32 |
OpenMath-Mistral-7B (w/ code, SC, k=50) |
57.2 | 7 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 33 |
ToRA 70B (w/ code, SC, k=50) |
56.9 | 70 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 34 |
SKiC (GPT-4 model) |
56.4 | | – |
Paper |
– |
2023 |
no code linked |
report |
| 35 |
DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) |
56.1 | 70 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 36 |
OpenMath-CodeLlama-7B (w/ code, SC, k=50) |
55.6 | 7 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 37 |
MMOS-DeepSeekMath-7B(0-shot) |
55.0 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified |
report |
| 38 |
DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) |
54.9 | 70 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 39 |
PHP (GPT-4 model) |
53.9 | | – |
Paper |
Code |
2023 |
official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified |
report |
| 40 |
DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) |
53.6 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 41 |
Gemini Ultra (4-shot) |
53.2 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 42 |
DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) |
52.9 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 43 |
GPT-4 model (w/ code, PAL) |
51.8 | | – |
Paper |
Code |
2022 |
3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 44 |
DeepSeekMATH-RL-7B (greedy decoding) |
51.7 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 8 ran · 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 11 where Syntology's instrument failed) · 10 unverified |
report |
| 45 |
AlphaLLM (with MCTS) |
51 | | – |
Paper |
Code |
2024 |
official (archive's flag): 14 ran · 14 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 9 where Syntology's instrument failed) · 4 unverified |
report |
| 46 |
ToRA-Code 34B (w/ code) |
50.8 | 34 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 47 |
OpenMath-CodeLlama-70B (w/ code) |
50.7 | 70 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 48 |
Minerva 540B (maj1@k, k=64) |
50.3 | | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 49 |
ToRA 70B (w/ code) |
49.7 | 70 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 50 |
MMOS-CODE-34B(0-shot) |
49.5 | 34 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified |
report |
| 51 |
DeepSeekMath-7B-KPMath-Plus |
48.8 | 7 | – |
Paper |
– |
2024 |
no code linked |
report |
| 52 |
PaLM 2 (few-shot, k=4, SC) |
48.8 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 53 |
Llemma-34B-KPMath-Plus |
48.6 | 34 | – |
Paper |
– |
2024 |
no code linked |
report |
| 54 |
OpenMath-CodeLlama-34B (w/ code) |
48.3 | 34 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 55 |
Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) |
48.1 | 67 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 56 |
ToRA-Code 13B (w/ code) |
48.1 | 13 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 57 |
Minerva 8B (maj5@256) |
47.6 | 8 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 58 |
Mistral-7B-KPMath-Plus |
46.8 | 7 | ✓ |
Paper |
– |
2024 |
no code linked |
report |
| 59 |
DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) |
46.6 | 8 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 60 |
OpenMath-Llama2-70B (w/ code) |
46.3 | 70 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 61 |
OpenMath-CodeLlama-13B (w/ code) |
45.5 | 13 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 62 |
DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) |
45.5 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 63 |
DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) |
45.3 | 8 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 64 |
MathCoder-CL-34B |
45.2 | 34 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 65 |
MathCoder-L-34B |
45.1 | 34 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 66 |
MMIQC-72B |
45.0 | 72 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 67 |
ToRA-Code 7B (w/ code) |
44.6 | 7 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 68 |
OpenMath-Mistral-7B (w/ code) |
44.5 | 7 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 69 |
MMOS-CODE-7B(0-shot) |
44.3 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified |
report |
| 70 |
OpenMath-CodeLlama-7B (w/ code) |
43.6 | 7 | ✓ |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 71 |
Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) |
43.5 | 7 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 72 |
DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) |
43.5 | 7 | ✓ |
Paper |
Code |
2024 |
official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified |
report |
| 73 |
Minerva 62B (maj1@k, k=64) |
43.4 | 62 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 74 |
ToRA 13B (w/ code) |
43.0 | 13 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 75 |
GPT-4 |
42.5 | | – |
Paper |
Code |
2023 |
9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified |
report |
| 76 |
SFT-Mistral-7B |
41.8 | 7 | ✓ |
– |
– |
|
not matched |
report |
| 77 |
Llama2-13B-KPMath-Plus |
41 | 13 | – |
Paper |
– |
2024 |
no code linked |
report |
| 78 |
ToRA 7B (w/ code) |
40.1 | 7 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified |
report |
| 79 |
MathCoder-CL-13B |
35.9 | 13 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 80 |
MuggleMATH-70B |
35.6 | 70 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 81 |
PaLM 2 (few-shot, k=4, CoT) |
34.3 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 82 |
Minerva 540B |
33.6 | 540 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 83 |
Minerva 540B (5-shot) mCoT |
33.6 | 540 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 84 |
Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) |
33.0 | 7 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 85 |
WizardMath-7B-V1.1 |
33.0 | 7 | ✓ |
Paper |
Code |
2023 |
10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified |
report |
| 86 |
Gemini Pro (4-shot) |
32.6 | | – |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 87 |
MuggleMATH-13B |
30.7 | 13 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 88 |
MathCoder-CL-7B |
30.2 | 7 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 89 |
MathCoder-L-13B |
29.9 | 13 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 90 |
Qwen2idae-16x14B (4-shot) |
29.9 | | – |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 91 |
OpenChat-3.5-1210 7B |
28.9 | 7 | – |
Paper |
Code |
2023 |
official: harvested, nothing ran · 0 ran · 1 unverified |
report |
| 92 |
OpenChat-3.5 7B |
28.6 | 7 | – |
Paper |
Code |
2023 |
official: harvested, nothing ran · 0 ran · 1 unverified |
report |
| 93 |
Mixtral 8x7B (maj@4) |
28.4 | | – |
Paper |
Code |
2024 |
5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result |
report |
| 94 |
Minerva 62B (4-shot) |
27.6 | 62 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 95 |
MetaMath 70B |
26.0 | 70 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified |
report |
| 96 |
MuggleMATH 7B |
25.8 | 7 | ✓ |
Paper |
Code |
2023 |
linked, not harvested |
report |
| 97 |
Minerva 8B (maj1@k, k=64) |
25.4 | 8 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 98 |
MathCoder-L-7B |
23.3 | 7 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified |
report |
| 99 |
WizardMath-70B-V1.0 |
22.7 | 70 | ✓ |
Paper |
Code |
2023 |
10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified |
report |
| 100 |
Camelidae-8×34B (4-shot) |
22.6 | | – |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 101 |
MetaMath 13B |
22.5 | 13 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified |
report |
| 102 |
LLaMA 65B (maj1@k) |
20.5 | 65 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 103 |
GAL 120B (5-shot) mCoT |
20.4 | 120 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 104 |
MetaMath 7B |
19.4 | 7 | ✓ |
Paper |
Code |
2023 |
official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified |
report |
| 105 |
davinci-002 175B |
19.1 | 175 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 106 |
Branch-Train-MiX 4x7B (sampling top-2 experts) |
17.8 | | – |
Paper |
Code |
2024 |
linked, not harvested |
report |
| 107 |
GAL 120B <work> |
16.6 | 120 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 108 |
LLaMA 33B-maj1@k |
15.2 | 33 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 109 |
Minerva 8B |
14.1 | 8 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 110 |
WizardMath-13B-V1.0 |
14.0 | 13 | ✓ |
Paper |
Code |
2023 |
10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified |
report |
| 111 |
Mistral 7B (maj@4) |
13.1 | 7 | – |
Paper |
Code |
2023 |
official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified |
report |
| 112 |
GAL 30B (5-shot) mCoT |
12.7 | 30 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 113 |
Mistral 7B (maj@4) |
12.7 | 7 | – |
Paper |
Code |
2024 |
5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result |
report |
| 114 |
GAL 30B <work> |
11.4 | 30 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 115 |
WizardMath-7B-V1.0 |
10.7 | 7 | ✓ |
Paper |
Code |
2023 |
10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified |
report |
| 116 |
LLaMA 65B |
10.6 | 65 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 117 |
PaLM 540B |
8.8 | 540 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 118 |
PaLM 540B (5-shot) mCoT |
8.8 | 540 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 119 |
LLaMA 13B-maj1@k |
8.8 | 13 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 120 |
LLaMA 33B |
7.1 | 33 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 121 |
LLaMA 7B-maj1@k |
6.9 | 7 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 122 |
GPT-2 (1.5B) |
6.9 | 1.5 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 123 |
GPT-2 (0.7B) |
6.4 | 0.7 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 124 |
GPT-2 (0.3B) |
6.2 | 0.3 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 125 |
GPT-3 13B |
5.6 | 13 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 126 |
PaLM 8B (fine-tuned) |
5.6 | 8 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 127 |
GPT-2 (0.1B) |
5.4 | 0.1 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 128 |
GPT-3-175B (few-shot) |
5.2 | 175 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 129 |
GPT-3 175B (8-shot) |
5.2 | 175 | – |
Paper |
Code |
2022 |
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified |
report |
| 130 |
PaLM 62B |
4.4 | 62 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |
| 131 |
LLaMA 13B |
3.9 | 13 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 132 |
GPT-3-13B (few-shot) |
3.0 | 13 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 133 |
LLaMA 7B |
2.9 | 7 | – |
Paper |
Code |
2023 |
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified |
report |
| 134 |
GPT-3 2.7B |
2.9 | 2.7 | – |
Paper |
Code |
2021 |
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified |
report |
| 135 |
PaLM 8B |
1.5 | 8 | – |
Paper |
Code |
2022 |
linked, not harvested |
report |