Pre-training
The first, very large stage of building a model, where it learns general patterns from huge amounts of data.
Pre-training is the first and most expensive stage of building a modern AI model. The model is shown a vast amount of text, and sometimes images or code, and learns to predict what comes next. By doing this billions of times, it picks up grammar, facts, writing styles and patterns of reasoning.
The result is a general-purpose base model. It is not yet a helpful assistant: it may continue a question with more questions rather than answering it. Later stages, such as fine-tuning and training with human feedback, shape it into something more useful and better behaved.
Pre-training matters because most of a model’s underlying knowledge and ability comes from it, and because it uses most of the compute. It is also where questions about training data arise, such as which sources were used, whether they were permitted, and whether test questions leaked in. Researchers debate how much further this stage can be pushed, since good data is finite. Some expect more of the future gains to come from later stages, and from extra thinking time when the model is answering, rather than from pre-training alone.