Recensio

A census of the benchmarks that test large language models on humanities material: what they ask, where their answers come from, and how far a score depends on the instrument.

The review keeps to the principles of the digital humanities: provenance, human supervision, source criticism. It is human-in-the-loop, but the loop begins and ends with people — the editors choose what enters, the models read and collate, the editors settle what the models cannot. The aim is to explore the dimensions along which an AI system can be called correct.

An article, What counts as correct?, by Remo Grillo, Gianmarco Spinaci, Lukas Klic and Giovanni Colavizza, is coming soon.

Show the benchmarks →Read the reportRead the article coming soon

recensio, -ōnis f. — the census and collation of the witnesses of a text, before any reading is preferred.

The census pipelineThree discovery routes bring in candidates; two editors screen each one independently; every included paper is read by two independent readers, Ciop and Cip; a supervisor checks each reading, a reviewer collates them, and the editors settle what remains — the record.F1curated34 by handF2asreview12 from the poolF3agentic30 by deep researchΣcandidates76 cataloguedRGscreensGSscreensωthe paper64 included, each read twiceα◆ Ciop64 readingsβ● Cip64 readingsγsupervisor116 readings sent back onceδreviewer64 records collatedRGsettlesGSsettlestextus constitutus — the record
The pipeline — each step explainededitor the editorsAI agent AI agents