Reports that lost the right to numbers

On 21 September eight requests for gaps in the collection went out, and the answers began to arrive one by one: long reports from an outside research model, each with a structured block of candidates giving a date, sources, figures, a confidence, disagreements and the status of every address. The working rule was the same for all of them: every candidate, every address and every figure is checked again, in the version of the document the record is dated to. Thirteen such passes in two days. This entry is about what they found and the decision it led to.

What was being closed

The gaps were measured on the data, not guessed. Images had forty empty years between 1963 and AlexNet; there are sixteen records there now, from Marr and Hildreth to PTAM. Evaluation had the twenty years in which the tests themselves were made, and none of them: BLEU, SQuAD, GLUE, HELM with zero mentions, MMLU with one, in passing. The layer that nearly every lawsuit of 2023–2026 is actually about: where the corpora came from and who labelled them — Mechanical Turk, Common Crawl, C4, The Pile, LAION, the Kenyan labellers. The line of harm began with a 2026 court ruling about a chatbot; it now begins with the 2018 face-recognition audits and the investigation of a recidivism risk algorithm. 2024 and 2025 had been recorded almost entirely as product launches; now they hold publications after which something became known. And then: who prints the accelerators and what the export rules actually forbid, the power and water of data centres, China’s algorithm rules, the courts and the elections, a symbolic line that stopped in 1990, three decades of speech before Baum and Welch, and money that reached a regulator’s filing.

What did not survive the check

The cheapest way to describe what was found is to list the defect classes, because each one recurred across several reports.

A later version’s number under the first version’s date. The report dates a record to a preprint’s first version, which is right, and describes the work from the arXiv abstract page, which shows the latest. Every number is real and none belongs to that date. MMLU’s 89.8 per cent human bar is absent from v1 and called an “educated guess” outright in v3; SuperGLUE was described with eight tasks where the v1 table has six.

A link to a different work. The title is right, the address opens, the status is 200, the date on the page matches, and the page is about something else. Both sources for the ROOTS corpus led to the BLOOM paper and to DataComp-LM from 2024. “Compute Trends Across Three Eras” turned out to be a paper on open-sourcing capable models. Under AfriBERTa lay MasakhaNER, and the report had carried its date over to AfriBERTa. More than once the answer was in the file name: “Wikidata” sat in freebase_to_wikidata.pdf, and Intel’s annual report ended in def14a.htm, a proxy statement.

A claim absent from the source altogether. “Sampling efficiency correlates with output entropy”, in a paper where the word entropy occurs zero times. A project cost of 1.6 billion dollars not in the press release it supposedly comes from; 25,000 robocalls where the regulator says 9,581. And its quieter relative, a claim turned the other way round: in Snell, sequential revision suits easier problems and parallel search harder ones, and the report had the reverse. The numbers are right, the relation is flipped, and no search for a number reaches it.

A rule read through a summary. The export rule of October 2022 is conjunctive: 600 Gbyte/s of interconnect AND a product of 4800. The report had OR, and that AND is what explains the next record, since NVIDIA’s A800 keeps the compute and lowers the link. Two articles of China’s deep synthesis provisions were swapped in the report along with both duties. A rule has several effective dates rather than one, and a rule names no products: the chip list everyone quotes belongs to the company’s own filing.

A paper credited with a concept it lacks. DeepSeekMath was presented as the paper that introduced reinforcement learning with verifiable rewards; the word verifiable is not in it, and its RL phase trains a reward model. This is the most expensive defect, because it is about what a record is for rather than about a number.

One price in the press release and another in the same buyer’s annual report. Intel announced Habana Labs at about 2 billion and booked it at 1.7; Mobileye at 15.3, booked at 14.5; Microsoft announced Nuance at 19.7 and booked 18.8. Both documents are primary, both are the company’s own, and the figure that went round the world is not the one that passed audit. DeepMind’s price is in no filing at all, and EDGAR full-text search is exactly how that gets established.

And invented actors. The report credited a paper on feature flow to “Feature Flow Research”, which appears nowhere in the paper. Its authors are in fact at T-Tech, the technology arm of the former Tinkoff, and the Moscow Institute of Physics and Technology, with correspondence at tbank.ru. It did not enter, under the project’s rule on Russian organisations. The rule had to be applied not to the name in the report but to what the name was hiding.

Each class went into the handoff for the next session on the day it was found, with the routes to primary texts: the Federal Register serves full text at a different address from the document page; CourtListener itself answers 403 while its storage and search API do not, so seven complaints were read whole and each filing date confirmed twice; www.sec.gov refuses the command line while data.sec.gov and efts.sec.gov answer it. And one rule that came from a loss rather than a check: a candidate’s verdict is written into the package the moment it is formed. The pass of 21 September held over 400 thousand tokens of context and was cut by the five-hour limit mid-integration; compaction loses the same thing more quietly, folding “status 200, date checked against the page” into “verified”.

Asterisks that reached the reader

One defect was not in a report but in us. Record texts are rendered as plain text, so markdown emphasis reaches the reader as asterisks. In one record the asterisks carried a restriction rather than stress: 27.33 per cent is the best result among open-source approaches, not the best result. The restriction is a word now, and a check refuses a pair of asterisks in any stored text, in any field, and breaks not only data validation but the build itself.

Budgets that grew faster than predicted

Two pages grow with the collection: the list of records and the list of sources. The slope was first estimated at 130 bytes a record and then measured between two points, and it came out twice that, because each pass wrote the whole verification verdict into the source’s note and what a record does not claim into its details. A bound was raised only for a pass that actually broke it, as that pass’s first act and with the arithmetic in the comment, not at the end to make things fit. Nobody shortened a note: it is the only place the checking survives the pass.

Three internal passes and parallel branches

On the evening of the 22nd three passes ran with no outside report. A second independent source under forty-five figures from the 2020s: eleven confirmed, seven diverged and now carry both numbers, the rest kept one source because no independent measurement exists or the source is itself a court record. A primary source in place of Wikipedia under ten records that rested on it alone; six figures in them were not in the new source and were corrected. And twenty-five links for twenty-six isolated records, each on a source that names the connection itself.

The passes ran in parallel and each numbered its new sources from the same place. Merging meant renumbering, one branch at a time in the order they finished, and merging data field by field rather than file by file.

A control pass and a new scheme

Late on the 22nd a new brief of twelve requests went out, measured at 641 records, and in the night its first request, robotics 1960–1999, was run with no report at all. Before opening a single source the session wrote down its hypotheses about dates and figures, and after reading, the verdict. Sixteen records in thirty-one minutes: of fourteen dates nine matched fully, four by year only, one was a year off; of thirty-seven figures twenty-nine were found, three were not and did not enter, five were wrong. One’s own memory errs just as a report does, but these errors were caught before they became text.

The tally of the thirteen report passes: 155 of 163 candidates accepted, about 165 claims in them cut or rewritten, roughly one a record, and every fifth date moved. The candidates were good. Everything else had to be checked again, and even the fields meant to help could not be trusted: confidence was “high” on all sixteen candidates of the first report, and disagreements empty where the report’s own prose describes two disputes about a date.

Hence the scheme the 6-in-1 approves that same night: requests 1–10 are run by sessions themselves, and for 11 and 12 a report carries names only — an identifier, open it, check the title, and then as with no report. A report no longer carries dates, numbers, confidence, disagreements or statuses.

The first record accepted by a person

And the last thing in these three days. Until now every record in the atlas had been checked by a machine, and the badge on the card said so honestly. The record of Dudley’s vocoder, October 1939, is the first to pass human review: an unsupported “first” was removed, Bell Telephone Laboratories named, and the Ukrainian and English versions brought into step. A defect turned up at once: the editor’s form did not know the field that records this review, and the next save would have quietly erased it. The form carries it now, and a test saves a record twice untouched and checks that the human acceptance is still there.

The acceptance did not end there. After it, the 6-in-1 sent two pages that no machine pass had found, neither ours nor the Gemini pass used to check the text: Dudley’s full paper in PNAS, open in PubMed Central, and the abstract of the October paper in the Bell System Technical Journal. The October paper sits behind a $40 paywall, and the record said honestly that the ten channels and the 25-cycle filters were not in the accessible abstract. But this was the same work, printed earlier and in the open. It was read before the Academy on 24 April 1939, and everything is there: ten bands from 0 to 2950 cycles, a 25-cycle filter in each channel, a separate pitch channel, eleven slow signals and a lag of 17 milliseconds. No intelligibility percentage and no word vocoder appear there either, so those figures from the report stayed out of the record.

The record was rewritten from the full text and dated to the day of the reading. Its human acceptance was taken off until the new text had been read, and put back once it had. A footnote in the same paper mentions a demonstration of an earlier stage of the apparatus on 11 September 1936; that is now a separate question, and if a source confirms it, the earliest record in the atlas moves again.

The lesson is simple and a little uncomfortable: a paywall on one paper is not a paywall on the work. Authors of that period printed the same apparatus in several places, and the PNAS and BSTJ archives are open.