← Back to Tech Radars

Tokenization Explorer

LLMs don’t read words — they read tokens. Type anything and watch your text get split. Hover a token to inspect its bytes.

Approximate GPT-4 tokenization — pre-tokenization + known BPE splits.
Try:
Each colored block is one token. Spaces are shown as ·
Token visualization
Counts
Tokens —
Characters —
Tokens / word —
Token types
Words
0
Numbers
0
Punctuation
0
Whitespace
0
Other
0
Token detail
Hover a token to inspect it