Definition

Scaling Laws

Scaling laws are the empirical finding that model performance improves predictably, as a power law, with more parameters, more data, and more compute. Kaplan's 2020 work made training runs an engineering estimate rather than a gamble. The Chinchilla refinement showed data matters as much as size: a 70B model trained on enough tokens beats an undertrained 175B model. Practical takeaway: expect today's models to be obsolete within eighteen months.