Definition

Tokenization

Tokenization is the step that turns text into numbers a model can process. Instead of whole words, text is split into sub-word pieces called tokens, each mapped to an integer ID. Common words get one token; rare words are broken into familiar fragments. An average English word costs about 1.3 tokens, which matters because pricing and context windows are both measured in tokens.