Llama by Meta · The Techno Optimist

Milestones

No announced next step

Chinese open models caught upDeepSeek, Qwen and Kimi releases in 2025 overtook Llama on many open-model rankings.
Benchmark trustThe Llama 4 version that was tested differed from the one released, raising doubts about the scores.

Shrinking numbers loses informationRunning on smaller hardware means storing weights in 4 to 8 bits instead of 16. Below about 4 bits accuracy drops quickly, so model size sets a floor on memory needed.
Running out of human-written textFrontier models already train on 15 to 40 trillion tokens. Estimates put the usable stock of public human text at a few hundred trillion, so data, not chips, starts to cap plain scaling.
Trained for likely text, not true textThe model learns to produce plausible words. Facts seen only once or twice in training can't be recalled reliably, so some rate of confident errors is built into the method.

3 parts

Weights

Anyone can run Llama on their own hardware, retrain it on private data or shrink it to fit a laptop or phone.

MoE

Llama 4 Maverick has about 400B parameters but uses only 17B per word, sending each word to the right experts.

Fusion

Pictures and text enter the same model from the first layer, instead of through a separate vision add-on.

2 items

Jul 2024paper
The Llama 3 Herd of ModelsExplained how a 405B dense model was trained on about 15 trillion tokens using 16,000 H100 chips.
Feb 2023paper
LLaMA: Open and Efficient Foundation Language ModelsShowed smaller models trained on more data can match bigger ones. It started the open-weight wave.

6 updates

The team behind Llama