Illegal material found in an open set, distribution halted
On 20 December 2023 David Thiel of the Stanford Internet Observatory published an audit of LAION-5B: 3,226 entries suspected of being child sexual abuse material. LAION pulled both LAION-5B and LAION-400M from distribution and did not return the set until August 2024.
Why it matters
Until then the openness of a large set was treated as a virtue needing no argument: anyone can look inside. This audit showed the other face of the same property - someone outside really could look, and what they found required distribution to stop. It is the first case of an open training set of world scale being withdrawn from circulation.
The method: PhotoDNA perceptual hashes, MD5 cryptographic hashes, and a nearest-neighbour search over the set's own embeddings. 32,138,129 entries above a chosen threshold were evaluated. PhotoDNA returned 1,679 matches; of those still live and reviewed by Canada's C3P, 746 were classified as abuse material or possible abuse material. In total the report names 3,226 suspected entries. The record deliberately does not attribute the figure 1,008 to the report. That number came later and belongs to LAION: in August 2024 the organisation said it had removed 2,236 links against hash lists from the Internet Watch Foundation and C3P, and that those 2,236 subsume 1,008 links from the Stanford report. LAION itself adds that a substantial share of such links are most likely already dead, so the number is an upper bound. What is at stake are links, not images: LAION-5B consists of URLs and captions and holds no image files.