Tokens: The Currency of AI

When you ask a chatbot for a recipe, it does not see words. It sees tokens. Everything you type gets chopped into small, numbered pieces. Those pieces are what you pay for, what fills the model's working memory, and what the model actually reads and writes.

The same text can come out as a different number of tokens in different models. One very long English word, "pneumonoultramicroscopicsilicovolcanoconiosis", is 17 tokens for GPT-4's tokenizer and 15 for GPT-4o's. Those differences add up in your bill and in how much fits into one conversation. This piece shows what a token is, where tokens come from, and how to think in them.

What exactly is a token?

A token is the smallest unit a large language model reads. You hand it text, and a tokenizer turns that text into a list of whole numbers. Each number points to an entry in a fixed vocabulary of word pieces, single characters and a few special markers.

Think of tokens as the currency of AI. You do not pay for a conversation in words or paragraphs. You pay in tokens. The model's inputs and outputs are token numbers, not the letters you typed, and when that stays invisible it is easy to misjudge cost and how much the model can take in.

The vocabulary is built once, before the model is trained, from a huge pile of text. The tokenizers in this piece hold from about 32,000 entries (Llama 2) to about 200,000 (GPT-4o). A token can be a whole word like "the", a common piece like "ing", a punctuation mark, or a special marker, such as one that tells the model where a message ends. OpenAI shares the tokenizers behind its models in a free library called tiktoken, so you can see this for yourself.

Why can "strawberry" be one token or three?

You met the idea in Breaking Language Into Pieces: instead of storing every possible word, a tokenizer keeps a set of reusable pieces. Common words stay whole. Rarer words are built from smaller parts. That way the model can read anything you type, even typos and made up words.

The most common way to build those pieces is called byte pair encoding, or BPE (Neural Machine Translation of Rare Words with Subword Units). While the tokenizer is being built, BPE starts from single characters (in modern tokenizers, the raw bytes of the text) and looks through a huge pile of text for the pair of neighbors that appears most often. It glues that pair into a new piece, then looks for the most frequent pair again, and repeats until the vocabulary is full. When you later give it a word, it splits the word into single bytes and applies the merges it learned, in the order it learned them, until none apply (Hugging Face's course on BPE).

The result depends on the tokenizer and on the space in front of the word. Here is what OpenAI's tiktoken gives for GPT-4o's tokenizer, called o200k_base, which also serves the GPT-5 models:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")    # the o200k_base tokenizer
tokens = enc.encode("strawberry")
print(tokens)                                  # [302, 1618, 19772]
for t in tokens:
    print(enc.decode_single_token_bytes(t))    # b'st', then b'raw', then b'berry'

print(len(enc.encode(" strawberry")))          # 1: with a space in front, one token

On its own, "strawberry" is three tokens: "st", "raw" and "berry". GPT-4's older tokenizer, cl100k_base, splits it differently: "str", "aw" and "berry". But in the middle of a sentence the word carries a space in front of it, and " strawberry" is a single token in both. Ask "How many r's are in strawberry?" and the model gets the whole word as one block.

Either way, the model never sees the ten separate letters, and the letters inside each block are hidden from it. That is why counting the r's in "strawberry" has tripped up so many chatbots: the question is about letters, and the model reads pieces.

How does a tokenizer turn my recipe request into numbers?

Every message follows the same fixed steps. Text goes in, a list of ID numbers comes out, and those IDs are all the model receives.

Take "Give me a recipe for chocolate chip cookies." First the tokenizer splits it into chunks: eight words, each carrying the space in front of it, plus the full stop. Then the merges run inside each chunk. Every word here is common enough to be a single token, so in both GPT-4's and GPT-4o's tokenizers the sentence becomes nine tokens: "Give", " me", " a", " recipe", " for", " chocolate", " chip", " cookies" and ".". A rarer word comes out in pieces: in the middle of a sentence, " summarizing" becomes " summar" and "izing". Starting from raw bytes is also what guarantees that any character, emoji or typo can still be written in tokens (Hugging Face's course on BPE).

In What an LLM Actually Is (and Why Predicting Words Works) you saw that an LLM predicts the next word. Now you can see what "word" really means there: the next token in this list of IDs. Everything the model learns about grammar, recipes and style is tied to these pieces, not to the letters you typed.

Why is pricing per token, not per word?

Pricing is per token because the model's work is per token. Every token you send is processed by every layer of the model, and every token you get back takes one more full pass through it. More tokens means more work, so the bill follows the token count. Providers list their prices per million tokens, with input and output priced separately (OpenAI's API pricing).

A "word" is a fuzzy idea that changes from language to language. Token counts are not: run the same tokenizer on the same text and you always get the same number. That makes tokens a fair billing unit across languages, code and anything else you send.

Here is the arithmetic for the recipe request. With GPT-4o's tokenizer, "Give me a recipe for chocolate chip cookies" is 8 tokens. Say the reply is 200 tokens, for 208 in all. Take a price of $2 per million input tokens and $10 per million output tokens, which is what OpenAI's GPT-6.1 Sol and Anthropic's Claude Sonnet 5.5 list as of October 2026 (Anthropic's models overview). The prompt costs $0.000016 and the reply $0.002, about a fifth of a cent in total. Each model counts with its own tokenizer, so its exact counts differ a little. Notice that the reply cost far more than the question: both companies charge five times as much per output token as per input token across their current lineups.

Short does not always mean cheap. A single rare word can cost about as much as a whole short sentence, because rare words break into many pieces. These counts come from running tiktoken with GPT-4o's tokenizer:

The token count, not the character count, is what you pay for.

How many tokens fit at once?

The context window is the most tokens a model can take in at once, and everything counts toward it: the instructions, the conversation so far, your new message and the reply. As of October 2026, the newest models from OpenAI, Anthropic and Google each take about a million tokens. The more tokens you actually send, the more each request costs and the slower it gets. Context Windows: The Model's Working Memory explains what the limit really means, and why a model can miss things long before it is full.

How can a repeated prompt cost less?

Many apps send the same long block of instructions with every request. Say your cooking assistant starts every request with the same 2,000 token block of rules and examples. With prompt caching, the provider keeps the work it already did on that block, so a later request that starts with the identical block runs faster and pays a reduced rate for those tokens. The new message and the reply are billed as normal.

There are two catches. A block must be long enough to be cached: at least 1,024 tokens on OpenAI's newer models and 512 on Anthropic's current Claude models, as of October 2026. And storing it can cost a little extra the first time: on those models, both companies charge 1.25 times the normal input price for a standard cache write. After that, reads cost a tenth of the normal input price or less (OpenAI's prompt caching guide, Anthropic's pricing).

The practical rule is simple. Put the fixed part of your prompt first and keep it identical, word for word, and put anything that changes after it. Change one token near the start and the provider has to redo the work from that point on. Training vs Inference: Why Building Costs Millions and Asking Costs Cents and the Context Windows piece show what the provider is saving inside the model.

Quick reference

Frequently Asked Questions

Q: Why does my text sometimes cost more tokens than I expected? Rare words, technical terms, names and many languages other than English break into more pieces than everyday English words. Two sentences of the same length can come out quite differently. Always measure with the model's own tokenizer or counting tool instead of estimating from characters or words.

Q: Do all models use the same tokenizer? No. Each model family usually builds its own. GPT-4 uses cl100k_base with about 100,000 entries, and GPT-4o and the GPT-5 models use o200k_base with about 200,000. Llama 3 uses a tokenizer built on tiktoken with 128,000 entries, 100,000 of them taken from tiktoken plus 28,000 added for other languages (The Llama 3 Herd of Models); Llama 2 used SentencePiece with 32,000 (Llama 2: Open Foundation and Fine-Tuned Chat Models). Tokenizers change within a family too: Anthropic says its Claude 4.7 and later models produce about 30 percent more tokens for the same text than earlier ones (Anthropic's token counting guide).

Q: Can I count tokens myself before sending a request? Yes. For OpenAI models and open models such as Llama, the published tokenizers (tiktoken, or the Hugging Face tokenizers library) give the exact count. For Claude, use Anthropic's free token counting endpoint, and for Gemini, the countTokens method (Gemini docs on tokens). Every response also reports the tokens it actually used.

Q: What happens if I go over the context window? Through an API (the connection programs use to send text to a model directly), a request whose input is longer than the window is refused with an error; Anthropic's API, for example, answers "prompt is too long" (Anthropic's guide to context windows). Chat apps manage this for you, and the same guide notes that chat apps such as claude.ai can drop the oldest parts of a long conversation first. That is how early instructions can quietly disappear. Context Windows: The Model's Working Memory covers what to do about it.

Q: Is a smaller vocabulary better or worse? It is a trade. A smaller vocabulary keeps the tokenizer simple but splits text into more tokens, so every message takes more work. A larger one covers more words and languages whole, so the same text takes fewer tokens. Each model team settles the balance when it builds the tokenizer.

Test yourself

You are building a tool that summarizes articles people paste in. A user submits a 2,500 word news story and wants a summary of about 150 words. Before you write any code, which token questions should you answer, and how would you check them?

Answer: Estimate first. At about 0.75 words per token, which is about 1.33 tokens per word, 2,500 words is roughly 3,300 tokens, plus a little more for your instructions. That is far below any current model's window, so the story fits easily. A 150 word summary is about 200 tokens, and output tokens cost several times more than input, so set a sensible length limit. Then check instead of trusting the estimate: run the model's own tokenizer, or the provider's counting endpoint, on a few real articles. Names, jargon and other languages split into more pieces, and tokenizers differ, so the true count can be noticeably higher. If the same long instructions go with every request, put them first and keep them identical so prompt caching can discount them. Finally, read the token counts the API returns on real requests to confirm your costs.

Next: Attention, Gently: How Models Decide What Matters. It explains how each token looks back at the others and decides which ones matter, with no math needed.

If you want this kind of breakdown every week, the real internals behind the tools you use, subscribe to Internals Decoded at internalsdecoded.com.

Sources

• tiktoken (OpenAI), used for every token count in this piece
• Neural Machine Translation of Rare Words with Subword Units (Sennrich et al. 2016)
• Byte-Pair Encoding tokenization, Hugging Face LLM course
• Hugging Face tokenizers library
• API pricing (OpenAI, read October 2026)
• Models overview (Anthropic, read October 2026)
• Prompt caching guide (OpenAI, read October 2026)
• Pricing (Anthropic, read October 2026)
• Token counting (Anthropic)
• Context windows (Anthropic)
• Understand and count tokens (Google)
• The Llama 3 Herd of Models (Meta 2024)
• Llama 2: Open Foundation and Fine-Tuned Chat Models (Touvron et al. 2023)