LLaMA

Meta released a family of models from 7 to 65 billion parameters, trained only on public data, and gave the weights to researchers.

Why it matters

A smaller model beat one ten times its size, and weights that leaked within a week spawned a whole open-model industry.

Weights were issued on application, but within a week they appeared in the open without Meta's permission. The company did not pursue the distribution, and Alpaca, Vicuna and hundreds of fine-tuned variants grew on LLaMA within months. That broke the assumption that frontier models would stay inside a few companies: fine-tuning someone else's base turned out to cost hundreds of dollars.

Event record

Event date

February 24, 2023

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

Openness

ID

evt-0271

The models were announced on 24 February 2023; the preprint is dated 27 February.

Sources

primary LLaMA: Open and Efficient Foundation Language Models
arXiv · Published February 2023
secondary Llama (language model)
Wikipedia

Related events

Earlier

Related Stable Diffusion 2022
The second event of the period to put a capable model outside its developer's walls.
Enables Chinchilla: compute-optimal training 2022links to this record
LLaMA cites the recommendation of Hoffmann et al. (2022) to train a 10B model on 200B tokens and departs from it: a 7B model keeps improving even after a trillion tokens.
- LLaMA: Open and Efficient Foundation Language Models · arXiv
Enables Rotary position embedding (RoPE) 2021links to this record
LLaMA removes the absolute positional embeddings and adds rotary positional embeddings (RoPE), 'introduced by Su et al. (2021)', at each layer.
- LLaMA: Open and Efficient Foundation Language Models · arXiv
Related GPT-3 2020
A 13-billion-parameter model beats the 175-billion GPT-3 on most measures.

Later

Builds on Alpaca 2023links to this record
The model is a fine-tune of the 7-billion-parameter LLaMA.
- Alpaca: A Strong, Replicable Instruction-Following Model · Stanford CRFM
Builds on Three per cent of a training budget, spent on one language 2023links to this record
Less than two months after the LLaMA weights appeared, a Brazilian group further pretrained them in Portuguese.
- Sabia: Portuguese Large Language Models (arXiv:2304.07880v1) · arXiv
Related QLoRA 2023links to this record
The method makes fine-tuning open models possible on a single graphics card.
Contrasts Open weights out of Abu Dhabi 2023links to this record
Three months later weights of the same class went out not under a research licence but royalty-free, and from Abu Dhabi rather than California.
- Falcon LLM, Technology Innovation Institute (archived 8 June 2023) · Technology Innovation Institute, via Internet Archive
Extends Llama 2 2023links to this record
The weights are now released openly and commercial use is permitted.
- Meta and Microsoft Introduce the Next Generation of Llama · Meta