The AI Learning Hub
◆ Definition

Token

Unit of input or output processed by an AI model. For text, typically 3-4 characters of English. Images are tokenized as patches, video as frames, audio as discrete sound units. Pricing, throughput, and context limits are all measured in tokens.

Learn more in these free lessons