Which model tokenizers?

OpenAI (tiktoken for GPT-3.5, GPT-4, GPT-4o), Anthropic (Claude), and a generic BPE for approximation. Pick the target model.

For OpenAI yes — uses the actual tiktoken library in WASM. For Anthropic, close approximation — Anthropic doesn't publish the exact tokenizer; the estimate is typically within 5%.

Why does token count matter?

Cost and context limits. GPT-4o: $2.50/M input tokens. Claude Sonnet 4.5: $3/M. Knowing exact counts before sending matters for high-volume apps.

What about 1 token = ~4 chars?

Rule of thumb for English. Other languages (Japanese, Chinese) often use ~1 token per character. Code uses even more tokens per char due to syntax.

Token Counter | DevTools Surf

DevTools Surf

Example: Token Counter

A worked example, rendered from real sample data. Sign in to run the tool on your own input.

Input

The quick brown fox jumps over the lazy dog. This is a sample text to estimate token counts for various AI models including GPT-4, Claude, and Gemini.

Output

Token estimate (heuristic, no vendor tokenizer)

Characters: 150
Words:      27
Lines:      1
Looks like: prose
Chars/token: 4.05 · words/token: 0.73

Claude Opus 5     ~       43 tok    0.00% of 1,000K  $0.000215 in
Claude Sonnet 5   ~       43 tok    0.00% of 1,000K  $0.000086 in
Claude Haiku 4.5  ~       43 tok    0.02% of 200K  $0.000043 in
GPT-5             ~       37 tok    0.01% of 400K  $0.000046 in
GPT-4o            ~       37 tok    0.03% of 128K  $0.000092 in
GPT-4o mini       ~       37 tok    0.03% of 128K  $0.000006 in
Gemini 2.5 Pro    ~       39 tok    0.00% of 1,048.576K  $0.000049 in
Gemini 2.5 Flash  ~       39 tok    0.00% of 1,048.576K  $0.000012 in

Estimates come from BPE-style segmentation and land within roughly 10-15% of the real tokenizers.

About Token Counter

Token Counter preview - AI / Modern Dev tool

Estimate tokens, context-window usage and cost for Claude, GPT and Gemini models. Part of the DevTools Surf developer suite. Browse more tools in the AI / Modern Dev collection.

Use Cases

Estimate API costs before sending large prompts
Optimize system prompts to stay under token limits
Compare token efficiency across different LLM providers
Budget context window usage for RAG applications

Tips

Compare token counts across GPT-4 and Claude models side by side
Paste your full system prompt to estimate per-request cost
Check if your context fits within the model's token limit

Fun Facts

GPT-4's tiktoken BPE tokenizer uses roughly 100,000 tokens in its vocabulary, compared to about 50,000 in GPT-2's tokenizer.
The word 'tokenization' in NLP was borrowed from compiler theory, where lexical analysis breaks source code into tokens.
Claude's tokenizer can represent common English words as single tokens, but a single emoji may consume 2-4 tokens.

FAQ

Which model tokenizers?: OpenAI (tiktoken for GPT-3.5, GPT-4, GPT-4o), Anthropic (Claude), and a generic BPE for approximation. Pick the target model.
Is the count exact?: For OpenAI yes — uses the actual tiktoken library in WASM. For Anthropic, close approximation — Anthropic doesn't publish the exact tokenizer; the estimate is typically within 5%.
Why does token count matter?: Cost and context limits. GPT-4o: $2.50/M input tokens. Claude Sonnet 4.5: $3/M. Knowing exact counts before sending matters for high-volume apps.
What about 1 token = ~4 chars?: Rule of thumb for English. Other languages (Japanese, Chinese) often use ~1 token per character. Code uses even more tokens per char due to syntax.

Related AI / Modern Dev Tools

Prompt Template Renderer Markdown → Slack/Discord Embedding Distance Calculator Markdown → Slack mrkdwn Sentiment Analyzer Keyword Extractor Text Summarizer Emoji Counter