The census is a pipeline: sources, screening, two readings, a check, a collation, the editors. It is drawn here the way a philologist draws the descent of a text — a stemma codicum, with its sigla. Each step below says what it does and where the census stands at that step today. As you read, the diagram lights the step you are reading about; select a step in it to jump there. Throughout the site, marks the work of an AI agent and that of an editor.
I. Recensio
finding the witnesses
F1
By handthe editors
The editors searched where benchmarks with humanities material are announced: the evaluation suites in the system cards of frontier models, the programmes of digital-humanities and computational-linguistics venues (LT4HALA, NLP4DH, CLEF/JOKER, SemEval, CLRel), shared tasks and benchmark aggregators. Who found each record, and where, is kept with it.
now34 records
F2
From the poolAI and the editors
Topic queries pairing benchmark and evaluation terms with humanities and cultural-heritage terms built a pool of 61,295 records from OpenAlex, Semantic Scholar, the ACL Anthology, arXiv, CEUR and Hugging Face. ASReview ordered it by active learning: every label retrains a classifier that brings the next most likely record to the top.
The classifier decides the order, never the outcome: no record was excluded by a machine. Screening stopped before the pool was exhausted, so recall is imperfect — HSSBench meets every criterion and is missing. That is reported as a result, not hidden in a caveat.
now12 records, screened 22 May – 16 June 2026
F3
By deep researchAI agent
The first search closed on 22 May 2026. To look past it, an agentic deep-research run is given a written prompt — kept, with its hash, beside the results — and asked to report the benchmarks it finds even when they are already in the census. The overlap between routes is the point: what two independent routes both find, and what only one does, gives an estimate of what all of them miss.
now30 candidates, searched 1 May 2026 – 9 November 2026
Σ
The census table
Every candidate enters one table, with an identifier assigned once and never changed, and its provenance: the route, the run, the reference inside the run, the day it was catalogued. A candidate found again is matched against the table — by DOI, arXiv identifier and normalised title — and recorded as found again, not added twice.
now76 catalogued · 0 awaiting screening
RGGS
Two editors, independentlythe editors
Each editor decides on each candidate alone, against four criteria. It is a benchmark: a fixed evaluation set with a scoring procedure that an independent party could re-run for a comparable number. It evaluates at least one generative language model. Its material belongs to the humanities — the domain, not the methods, reading of digital humanities, with the Arts and Humanities area of the ASJC as vocabulary. It evaluates in English, alone or among other languages.
A paper enters only when both editors accept it; a disagreement is discussed and closed with a joint record. Recency (after March 2023) is applied with one documented exception, MMLU, without which the general-purpose benchmarks cannot be described.
now64 included
II. Collatio
reading the paper twice, and checking the readings
ω
The paper
Each included paper is fetched as a PDF — a landing page that only looks like one is refused — fingerprinted with its sha256 and turned into text. Every reading descends from that one file: two readers who read different files are flagged.
now64 papers, one file each
α
Ciop, the humanist readerAI agent
◆ Reads as a humanist would: what material the benchmark is made of, who decided the correct answers, whether correctness could reasonably be disputed. It fills every field of the schema — 35 of them — and answers the census’s research questions.
now64 readings
β
Cip, the technical readerAI agent
● Reads as an NLP researcher would: the evaluation set and its size, the metric and how it is computed, the models and the scores. Same fields, independently: neither reader sees the other’s reading.
For the 18 fields that carry the census’s argument, every value must come with a verbatim quote from the paper. A script, not a model, finds each quote in the text, word by word, after undoing line breaks and hyphenation: a paraphrase does not match. A value that records an absence — no human baseline reported — needs no quote; whether the paper is really silent is for the supervisor to check.
now64 readings
γ
The supervisorAI agent
Checks each reading on its own, against the paper rather than against the other reading: values the text does not support, quotes that do not say what they are cited for, absences that are not absences. A reading with problems is sent back once, with the notes; the redo is kept beside the original, never written over it. What is still wrong after the redo goes to the reviewer as a flag.
now116 of 128 readings sent back once · 60 passed on with open flags
δ
The reviewerAI agent
A larger model (claude-opus-5-5) collates the two supervised readings against the paper, field by field: where they agree it confirms, where they differ it chooses and says why, and it may set a value neither reader gave. Every value of the record carries its evidence and a confidence. What cannot be settled from the text is handed to the editors as a question, never guessed.
now64 records · 137 values of low confidence · 0 questions for the editors
III. Constitutio textus
settling the record
RGGS
The editors settlethe editors
Each question reaches the editors as a card: the definition of the field, the reviewer’s value and reason, the two readings side by side with their quotes, and the page of the paper. An editor can confirm the reviewer, adopt either reader, set the value by hand, or declare it undecidable from the paper, with a note. Every verdict keeps its author and its date, and none is written over the machines’ record.
now0 questions open
T
The record — textus constitutus
What is published is, field by field, the editors’ verdict where there is one and otherwise the reviewer’s value — each with its source: which round, which agent, which verdict. Releases will be frozen snapshots with a version number and checksums. Until the first one, the benchmarks show the latest round as it stands, marked as such.
The round
All the machine readings of one pass belong to a round, whose protocol is frozen before the first paper is read: the model of each agent, and the hash of its prompt and of everything rendered into it — schema, vocabularies, research questions, evidence rules. A reading made with another model or another prompt is refused. Changing anything means opening a new round; the old one stays as it was. Select a prompt to read it in full.
Round 2026-11_claude, frozen 9 November 2026 by RG
Before the rounds there was a first extraction, made in August 2026 with Gemini models on 42 papers. It is kept as round legacy-2026-08_gemini: the article’s draft still reports it, and each new round is compared with it.
Colophon
The census is edited by Remo Grillo and Gianmarco Spinaci. The argument is set out in What Counts as Correct: A Census of Digital Humanities Benchmarks for Large Language Models, by Remo Grillo, Gianmarco Spinaci, Lukas Klic and Giovanni Colavizza (in preparation). The data behind it are in the report.
Set in Fraunces, Newsreader and IBM Plex Mono. Built with Next.js and SQLite; the readers, the supervisor and the reviewer are Claude models, run on the editors’ subscription. Source code.