Open weights out of Abu Dhabi
On 25 May 2023 the Technology Innovation Institute in Abu Dhabi opened Falcon 40B, a model of 40 billion parameters trained on a trillion tokens of the RefinedWeb corpus. Training took two months on 384 A100 40GB accelerators in the AWS cloud. On 31 May the institute announced the model royalty-free under the Apache 2.0 licence.
Why it matters
This is the first time a state research institute outside the United States and Europe put a model of this class into the open, and put it out whole, without the research-only restriction that then came with the LLaMA weights. For countries that could neither buy such weights nor train them, it changed what there was to start from.
The corpus was gathered from public web crawls: after filtering machine-generated text and duplicates a pool of nearly five trillion tokens remained, of which a trillion went into training, extended with curated sources such as research papers and conversations from social media. The architecture is a causal decoder with multiquery attention and FlashAttention. Training ran from December 2022 on AWS P4d instances under a 3D parallelism scheme combined with ZeRO. On the dates. The Hugging Face repository was created on 24 May at 12:08 UTC, the day before the announcement. On 29 May the institute reported that Falcon 40B had taken first place on the Hugging Face leaderboard; on 31 May, that the model was royalty-free. The report placed the licence change on 7 June: under that date the institute's page carries a different item, about a call for proposals for compute time. What the record does not claim. The first place on the leaderboard is taken from the institute's own announcement: the archived copy of the leaderboard itself for 2 June 2023 is empty because the page is drawn by script, so no independent record of that ranking could be obtained. The comparison against GPT-3 and the claim to be the best open model are likewise the maker's words about its own product. Confidence is therefore medium.