◆ Definition
Tokenization
Chopping raw text (or images, or audio) into the discrete units a model actually processes. Explains many odd model behaviors — why counting letters is hard, and why some languages cost more tokens than others.
Chopping raw text (or images, or audio) into the discrete units a model actually processes. Explains many odd model behaviors — why counting letters is hard, and why some languages cost more tokens than others.