Scaling laws
Patterns showing that AI models improve predictably as they get more compute, data and parameters.
Scaling laws are regular patterns found by measuring many AI models of different sizes. They show that, when a model is trained with more compute, more data and more parameters, its error on a basic task falls in a smooth and fairly predictable way. Researchers at OpenAI published an influential early set in 2020, in a paper called “Scaling Laws for Neural Language Models”.
The practical value is planning. If the pattern holds, a lab can train a small model, measure how it behaves, and estimate how a much larger model will perform before spending the money to build it. Later work refined the balance between model size and the amount of training data. A 2022 study from DeepMind, known by the name of its model, Chinchilla, found that the two should grow in step, and that the large models of the time had been trained on too little data for their size.
Scaling laws are descriptions of past trends, not laws of nature. People disagree about how long they will hold. Some expect the pattern to continue and drive further progress. Others point to limits such as running short of good training data, or to gains that show up in the error measure but not in usefulness. The laws also say little about specific abilities.