The first double-blind test of a closed model
On 27 August 2026 Google DeepMind, with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons, published a technical report on an evaluation in which the evaluator could not see the model's weights and the owner could not see the questions. Gemini 2.5 Flash Lite was tested against a selection of the private AILuminate corpus inside a secure enclave on an NVIDIA H100.
Why it matters
Until then an external check of a closed model required somebody to give up a secret: either the evaluator handed the questions to the owner, or the owner handed over the weights. The pilot showed a third way, in which each side verifies in hardware that the other cannot see its own — making a check possible where a no-logging contract is no substitute for trust.
The protection rests on an enclave that encrypts data in memory: an Intel TDX host, an NVIDIA H100 with encrypted video memory, all inside Confidential Space on Google Cloud, with OpenMined's PySyft keeping the data and the model private. Weights stream into video memory, prompts into RAM, and only bounded metrics come back out. The report is signed by authors from five organisations: AVERI, Google, Singapore AISI, OpenMined and ML Commons. It gives three reasons why benchmarks are not trusted: per Singh et al. (2025), one frontier lab tested 27 private model variants on Chatbot Arena and published the best scorer; per Xu et al. (2024), signs of benchmark leakage appeared in roughly half of 31 models tested; per Schaeffer et al. (2026), test-set contamination inflates the measurement, and the inflation grows with both the amount of contamination and model size. What the record does not claim. That this is the world's first double-blind evaluation: that is the company's own claim, and the record offers no independent confirmation of the precedence. What the results were: this record is about the method, not about a score. That the method is available to others: a pilot is described, not a service. The company's post names the model only as 'a Gemini Flash Lite model'; the exact name is taken from the technical report.