The report

What the benchmarks measure

The census as data: how the 64 benchmarks were built and what they found, recomputed from the records of round 2026-11_claude whenever the census changes. Select a bar to see the benchmarks behind it.

The working state, not a release. These charts are recomputed live from the census as it stands, without text. The report is published as editions: once the census is closed, its data are frozen, written up and approved by the editors.

I. The map

When

52 of 64 appeared in 2025 or later.

What the model is shown

All 64 use text; 23 also show images.

Which disciplines

Among the 51 benchmarks with a focus of their own; 13 general-purpose benchmarks span many subjects and are counted apart.

A benchmark can belong to more than one discipline. A general-purpose benchmark lists every discipline its questions touch, which says what it contains, not what it is about: counting it under each would make an exam suite with a few music questions a music benchmark.

What they ask

A benchmark can ask more than one kind of thing: the seven families group the 22 task types of the schema.

How large

  • fewer than 10069%
  • 100 – 9992336%
  • 1,000 – 9,9992133%
  • 10,000 – 99,999813%
  • 100,000 or more23%
  • not stated46%
Items scored in the evaluation set the paper reports, not the size of the whole dataset.

How the models are asked

  • zero-shot4063%
  • few-shot58%
  • mix1828%
  • fine-tuned12%

Which metric the papers headline

  • accuracy3352%
  • other1523%
  • bleu23%
  • f123%
  • win rate23%
  • cer12%
  • none914%
The metric of the headline result, a property of how the benchmark scores. The scores themselves are not compared: each belongs to its own task.

Where the humanities sit

Content: humanities material supplies the items, but the competence tested is general. Domain: the competence is the discipline’s own.

II. Who decides what is correct

Where the reference answers come from

Who produced the answers the models are scored against.

Knowledge beyond the stimulus

44 of 64 need knowledge absent from what the model is shown — and so from what the scorer can check against it.

III. How many answers are accepted

Answers accepted per item

43 of 64 accept a single answer.

How disagreement is treated

32 of 64 do not say how they treat divergence between annotators or between defensible readings.

  • noise2742%
  • signal58%
  • not stated3250%

Answers accepted, against knowledge beyond the stimulus

answers accepted ↓ · needs knowledge beyond the stimulus →yesnonot stated
single answer3094
answer set611
graded human judgment84
not stated1
The top-left cell: benchmarks that demand inference, and admit one result.

IV. When the instrument changes

Same material, measured two ways

45 of 64 report a score that moves when only the instrument changes.

Kinds of change: the scoring rule, the answer format, the input shown, a harder subset, a deeper layer of the task.

Do they measure humans too?

15 of 64 report a human baseline, expert score or inter-annotator ceiling.

  • yes1523%
  • no4977%
Whether the benchmark gives the models a human reference. How far the models fall from it is not compared here: each gap is on its own task and metric.

How much the instrument moves the score

Ancient Greek synset-selection evaluation
88 pts
LongFact
80 pts
Appear2Meaning
74 pts
CulturalBench
53 pts
Temporal Anachronism Benchmark for Vision-Language Models
37 pts
CzechTopic
31 pts
NORMVIZ-BENCH
31 pts
History Seshat Test for LLMs
24 pts
LLMs4Subjects
24 pts
DeepSearchQA
22 pts
SimpleQA
17 pts
FMG-Bench v1
12 pts

Same material, same models, a different measuring rule (kinds: scoring, format); the dark tick is each test’s noise floor. Gap kinds reported across the corpus: input 37 · scoring 15 · difficulty 9 · format 3 · depth 2.

Each bar is a difference inside one benchmark — the same models on the same items, measured two ways — in that benchmark's own points. It describes the instrument, not the models; the bars are not a ranking of benchmarks.

Differences across cultures

28 report a systematic difference across cultures, languages or traditions.

  • yes2844%
  • no813%
  • not stated2844%

How small a difference each test can see

AdMIRe
n = 15
±25.8
Ancient Greek synset-selection evaluation
n = 16
±25
Changing grammatical-use detection evaluation
n = 36
±16.7
ATR4CH
n = 45
±14.9
Does Bigger Mean Funnier? Qwen3 evaluation
n = 50
±14.1
RISE Humanities Data Benchmark
n = 50
±14.1
Exploring the Limits of Prompting LLMs with Speaker-Specific Rhetorical Fingerprints
n = 100
±10
StoicLLM evaluation set
n = 100
±10
LLM-based Machine Translation and Summarization for Latin
n = 121
±9.1
OfficeQA Pro
n = 133
±8.7
AllFaith Religious Representation Benchmark
n = 150
±8.2
FMG-Bench v1
n = 157
±8
FActScore
n = 183
±7.4
Apartheid witness-report summarization/NER evaluation
n = 200
±7.1
Finding the Plea
n = 200
±7.1
LongFact
n = 250
±6.3
QuestBench
n = 256
±6.3
VULCA-BENCH tri-tier evaluation framework
n = 294
±5.8
Cultural Moment Benchmark
n = 306
±5.7
VULCA-BENCH
n = 336
±5.5
HistBench
n = 414
±4.9
METATR v1.0
n = 453
±4.7
Historical Germanic POS-tagging evaluation
n = 511
±4.4
Temporal Anachronism Benchmark for Vision-Language Models
n = 600
±4.1
Appear2Meaning
n = 750
±3.7
Text Is Not All You Need
n = 810
±3.5
IRLBench
n = 850
±3.4
EduArt
n = 871
±3.4
DeepSearchQA
n = 900
±3.3

Smallest difference between two models each test can resolve, in points (≈ 100/√n).

For the tests under a thousand items: a gap smaller than the bar is within noise.

V. What the field does not ask

The genesis of artefacts

Date, author, place of origin, provenance — the questions a catalogue asks of an object — are the target of 3 of 64. Only 7 treat the correct answer as varying by community.

  • sequence prediction23%

The task types of the schema used by three benchmarks or fewer.

VI. How far these numbers reach

How often the two readers agreed

  • Analytical tasks56
  • Kinds of gap reported77
  • Source of the reference answers84
  • Disagreement treated as84
  • Humanities role86
  • Evaluation setting88
  • Answer needs knowledge beyond the stimulus88
  • Metric of the best score88
  • Answers accepted per item91
  • Score changes with the instrument91
  • Cultural asymmetry reported91
  • Human baseline reported94
  • Correctness varies by community95
  • Targets how the artefact came to be98
  • Benchmark modalities100

Share of papers on which the two readings give the same value, in %.

Ciop and Cip, after the supervisor's check and before the reviewer; closed-vocabulary fields only. Least agreement first.

Where the editors are still needed

0 questions the reviewer could not settle from the text, by field.

    Each is answered by both editors before the first complete report. How they are settled.

    Values of low confidence

    137 values the reviewer marked as uncertain; the fields with the most.

    • Kind of that gap1712%
    • Largest gap (points)1612%
    • Answer needs knowledge beyond the stimulus139%
    • Best reported score129%
    • Evaluation setting129%
    • Source of the reference answers97%
    • Disagreement treated as97%
    • Cultural asymmetry reported97%
    • Answers accepted per item86%
    • Items scored86%
    • Score changes with the instrument64%
    • Correctness varies by community54%
    A random sample of 20 low-confidence values was checked by hand against the papers: 3 had to be corrected (15%, 95% interval 5–36%).
    Earlier editions and the working state
    • No earlier edition.
    • The working state — the charts recomputed live from the census as it stands, without text: not a release.