A transparency index: nobody says what the model is made of
On 19 October 2023 Stanford's CRFM scored the ten largest foundation model developers against a hundred transparency indicators. The mean came to 37 out of 100, with Meta highest at 54 and Amazon lowest at 12.
Why it matters
Until then the closedness of training data was discussed as an impression. Here it was measured for the first time on one ruler for everyone, and what turned out worst disclosed was the start of the chain - data, data labour and compute - which is exactly what every copyright argument depends on.
The hundred indicators divide into the upstream of the chain (data, labour, compute, methods, code), the model itself, and downstream use. Each developer was scored on its flagship model: GPT-4 for OpenAI, PaLM 2 for Google, Llama 2 for Meta. Subdomain averages upstream: Data 20 per cent, Data Labor 17 per cent, Compute 17 per cent. Three developers - AI21 Labs, Inflection and Amazon - scored none of the 32 upstream indicators; the best there was Hugging Face with 21 of 32. The sharpest statement in the report is not about averages. Not one of the ten companies scored any point on the indicators about data creators, about the copyright and licence status of the data, or about copyright-related mitigations. Not "few" - zero out of ten.