The AI Learning Hub
◆ Definition

Tokenization

Chopping raw text (or images, or audio) into the discrete units a model actually processes. Explains many odd model behaviors — why counting letters is hard, and why some languages cost more tokens than others.

Related terms