The report
What the benchmarks measure
The census as data: how the 64 benchmarks were built and what they found, recomputed from the records of round 2026-11_claude whenever the census changes. Select a bar to see the benchmarks behind it.
The working state, not a release. These charts are recomputed live from the census as it stands, without text. The report is published as editions: once the census is closed, its data are frozen, written up and approved by the editors.
I. The map
When
52 of 64 appeared in 2025 or later.
What the model is shown
All 64 use text; 23 also show images.
Which disciplines
Among the 51 benchmarks with a focus of their own; 13 general-purpose benchmarks span many subjects and are counted apart.
What they ask
How large
- fewer than 10069%
- 100 – 9992336%
- 1,000 – 9,9992133%
- 10,000 – 99,999813%
- 100,000 or more23%
- not stated46%
How the models are asked
- zero-shot4063%
- few-shot58%
- mix1828%
- fine-tuned12%
Which metric the papers headline
- accuracy3352%
- other1523%
- bleu23%
- f123%
- win rate23%
- cer12%
- none914%
Where the humanities sit
II. Who decides what is correct
Where the reference answers come from
Knowledge beyond the stimulus
44 of 64 need knowledge absent from what the model is shown — and so from what the scorer can check against it.
III. How many answers are accepted
Answers accepted per item
43 of 64 accept a single answer.
How disagreement is treated
32 of 64 do not say how they treat divergence between annotators or between defensible readings.
- noise2742%
- signal58%
- not stated3250%
Answers accepted, against knowledge beyond the stimulus
| answers accepted ↓ · needs knowledge beyond the stimulus → | yes | no | not stated |
|---|---|---|---|
| single answer | 30 | 9 | 4 |
| answer set | 6 | 1 | 1 |
| graded human judgment | 8 | 4 | |
| not stated | 1 |
IV. When the instrument changes
Same material, measured two ways
45 of 64 report a score that moves when only the instrument changes.
Do they measure humans too?
15 of 64 report a human baseline, expert score or inter-annotator ceiling.
- yes1523%
- no4977%
How much the instrument moves the score
Same material, same models, a different measuring rule (kinds: scoring, format); the dark tick is each test’s noise floor. Gap kinds reported across the corpus: input 37 · scoring 15 · difficulty 9 · format 3 · depth 2.
Differences across cultures
28 report a systematic difference across cultures, languages or traditions.
- yes2844%
- no813%
- not stated2844%
How small a difference each test can see
Smallest difference between two models each test can resolve, in points (≈ 100/√n).
V. What the field does not ask
The genesis of artefacts
Date, author, place of origin, provenance — the questions a catalogue asks of an object — are the target of 3 of 64. Only 7 treat the correct answer as varying by community.
- sequence prediction23%
The task types of the schema used by three benchmarks or fewer.
VI. How far these numbers reach
How often the two readers agreed
- Analytical tasks56
- Kinds of gap reported77
- Source of the reference answers84
- Disagreement treated as84
- Humanities role86
- Evaluation setting88
- Answer needs knowledge beyond the stimulus88
- Metric of the best score88
- Answers accepted per item91
- Score changes with the instrument91
- Cultural asymmetry reported91
- Human baseline reported94
- Correctness varies by community95
- Targets how the artefact came to be98
- Benchmark modalities100
Share of papers on which the two readings give the same value, in %.
Where the editors are still needed
0 questions the reviewer could not settle from the text, by field.
Values of low confidence
137 values the reviewer marked as uncertain; the fields with the most.
- Kind of that gap1712%
- Largest gap (points)1612%
- Answer needs knowledge beyond the stimulus139%
- Best reported score129%
- Evaluation setting129%
- Source of the reference answers97%
- Disagreement treated as97%
- Cultural asymmetry reported97%
- Answers accepted per item86%
- Items scored86%
- Score changes with the instrument64%
- Correctness varies by community54%
Earlier editions and the working state
- No earlier edition.
- The working state — the charts recomputed live from the census as it stands, without text: not a release.