What is a token? How AI models count text and what it costs

What is a token? The short answer

A token is the unit of text an AI language model reads and writes. Before a model sees your message, a tokenizer cuts it into pieces from a fixed vocabulary: whole common words, parts of longer words, punctuation, spaces and single characters. Each piece becomes a number, and the model works with those numbers.

Tokens matter to you for two reasons. Models have a limit on how many tokens they can handle at once, and API providers charge per token. In English a token averages roughly four characters, or about three quarters of a word, but the real count depends on the text and the model.

How text is split into tokens

Most tokenizers use a method called byte pair encoding. It starts from individual bytes and, during training, repeatedly merges the pairs that appear together most often into new vocabulary entries. Frequent words end up as a single token; rare words are built from several pieces.

With OpenAI's o200k_base tokenizer, used by GPT-4o and later models:

The name “tiktoken” isn't in the vocabulary, so it takes three pieces, while common words like “is” and “great” are one each. Notice also that the space before a word belongs to that word's token. That is why “world” at the start of a line and “ world” in the middle of a sentence can be different tokens. Capital letters, numbers and unusual spellings also change how text is split.

Rules of thumb from the providers

Each provider publishes its own rough conversion for English text:

These are averages for ordinary English prose. Use them for a quick estimate of a long document, not for budgeting a large job.

Why the same text gives different counts

Different vocabularies. Every model family has its own tokenizer. OpenAI moved from cl100k_base (about 100,000 entries, used by GPT-4 and GPT-3.5) to o200k_base (about 200,000 entries) with GPT-4o. A bigger vocabulary has room for more whole words, including words in more languages, so it usually needs fewer tokens. Anthropic changed tokenizer with Claude 4.7, and says the newer one produces about 30% more tokens for the same text.

Languages. Tokenizers are trained mostly on English, so other languages, especially those not written in Latin letters, often take more tokens for the same meaning. Chinese, Japanese, Hindi or Arabic text can have far fewer characters per token than English, so the four-characters rule badly undercounts it.

Code and data. Code, JSON, long numbers, URLs and unusual identifiers split into many small pieces. Indentation counts too.

Tokens and context windows

A model's context window is the number of tokens it can consider at once: your instructions, any documents you attach, the conversation so far, and the answer it is writing. When a chat outgrows the window, the app has to drop or summarise older messages, which is when a model seems to forget what you said earlier.

Because the window is measured in tokens, not words or pages, the same window holds less text in a language or format that tokenizes poorly. A long spreadsheet pasted as text fills it much faster than the same number of characters of prose.

What tokens cost

APIs charge per million tokens, with separate prices for input (what you send) and output (what the model writes). Output is usually several times more expensive than input.

cost = (input tokens × input price + output tokens × output price) ÷ 1,000,000

Say you summarise a 10,000-token report into an 800-token summary with Claude Sonnet 5.5, listed at $2 per million input tokens and $10 per million output tokens (as of October 2026):

• Input: 10,000 × 2 ÷ 1,000,000 = $0.02
• Output: 800 × 10 ÷ 1,000,000 = $0.008
• Total: $0.028 per report, or $14 for 500 reports

Prices change often and differ a lot between models, so check the provider's pricing page. The cost calculator lists current prices side by side and lets you edit them.

Where tokens add up without you noticing

• Conversation history. Most chat APIs don't remember earlier turns, so each request sends the whole conversation again and you pay for it as input. A ten-turn chat sends the first message ten times.
• System prompts and tool definitions. Instructions and tool descriptions are input on every request, so a long system prompt multiplies across thousands of calls.
• Reasoning. Models that think before answering bill that hidden reasoning as output tokens, at the output price.
• Images and files. Pictures and PDFs are converted to tokens too, by their own rules, separately from the text.
• Long prompts. Some models charge a higher rate once a request passes a size threshold, such as 200,000 input tokens for Gemini 2.5 Pro.

Prompt caching and batch processing reduce the price of repeated or non-urgent work. Each provider documents its own discounts.

How to count tokens exactly

OpenAI publishes its tokenizers as an open-source library called tiktoken, so GPT counts can be made exactly anywhere, including in a browser. The token counter does this on your device, so the text you paste isn't sent to anyone.

Anthropic and Google don't publish their tokenizers for offline use. For exact Claude or Gemini counts, use their token-counting API endpoints, which means sending the text to them. For a private estimate, the counter applies their published rules of thumb and labels the results as estimates. To see how much text your own chats contain, the export viewer opens your ChatGPT or Claude history so you can copy any chat into the counter.

Sources

• OpenAI Help Center: What are tokens and how to count them?. 4 characters, ¾ of a word
• OpenAI tiktoken on GitHub. o200k_base and cl100k_base
• Claude Docs: Glossary, Tokens. 3.5 characters; newer tokenizer about 30% more tokens
• Claude Docs: Pricing. Claude Sonnet 5.5 prices
• Gemini API: Understand and count tokens. about 4 characters per token
• Gemini API: Pricing. prompts over 200,000 tokens