Breaking Language Into Pieces

A quick note on how we teach here. You won't find equations or numbers in this series, and that is on purpose. The goal is understanding, not arithmetic, so every idea is explained in plain words and simple pictures. When you want the math later, the rest of Internals Decoded is ready for you.

Welcome back. In the last piece we gave a name to the bundle of learned patterns a machine carries around after training. We called it a model. Today we look at the very first thing that happens when that model meets a sentence. Before it can even begin to understand, it has to break the sentence into small, standard pieces.

The idea is simple but powerful. A model does not read words the way you do. It chops language into tiny chunks, and it works with those chunks instead of whole words or individual letters. This changes everything about how it sees language.

A pile of bricks, not a lump of clay

Imagine you are building a toy castle out of interlocking bricks. You do not sit down and mould a brand new custom shape for every tower, every window, every door. Instead you reach into a big bucket of standard bricks. Some bricks are shaped for very common parts. A single brick might be a whole door. Another might be a tiny flag. For something rare, like a fancy curved turret, you snap together several smaller bricks to make the shape you need.

A language model works the same way. It does not look at a sentence as a smooth stream of letters. It has a fixed set of building blocks for language. Some blocks are for whole common words. A word like "the" appears so often that it gets its own single block. A long or unusual word might not have its own block. Instead the model snaps together a few smaller blocks to build it. So the sentence "The cat sat on the mat" becomes a sequence of these snap together pieces. The model never sees the raw letters. It only sees the blocks.

Why not just whole words?

You might wonder why the model does not simply give every word its own block. The answer has two parts. First, there are far too many words. New words appear all the time, and a model cannot carry a block for every single one. Second, words share parts. Think about the words "walk", "walking", and "walked". They all share the same root. If the model had a completely separate block for each of these, it would miss the connection between them. It would not know that "walking" is related to "walk".

A smart chopping tool notices these shared parts. It creates blocks that are common enough to be useful, but small enough to combine in flexible ways. So a very frequent word like "the" stays as one block. A word like "unbelievable" might be split into three blocks: "un", "believ", and "able". The model can then reuse "able" in "comfortable" or "lovable". It can reuse "un" in "unhappy" or "unkind". This is how the model handles words it has never seen before. If it knows the blocks, it can still work with the new word.

The surprise about spelling

Earlier we talked about a model guessing what comes next. That trick was all about predicting the next word. But now we know the truth. The model is really predicting the next piece, the next building block. Every time it adds to a sentence, it chooses one block from its bucket and snaps it into place.

Because the model sees blocks and not letters, it does not have a direct sense of spelling. It never looked at a word letter by letter the way you do. If you ask it "how many times does the letter 'r' appear in 'strawberry'?", it might stumble. The model probably saw "strawberry" as two blocks, maybe "straw" and "berry". It never saw the individual letters inside those blocks as separate things. So it cannot easily count them. This is not a flaw in thinking. It is a natural result of the building block approach. The model trades a letter by letter view for a system that handles meaning and patterns much more efficiently.

A name for the pieces

In the world of AI, each little building block is called a token. The tool that does the chopping is called a tokenizer. When you hear people say that a model processes text as tokens, this is exactly what they mean. The model breaks every sentence into tokens before it does anything else. And when it generates new text, it is picking one token after another, not one letter or one whole word at a time.

Where this is heading

So before a model reads, it tokenizes text into tokens. That is the first quiet step that happens behind the scenes. Once we have these tokens, the next step feels familiar. We turn each token into a position on a map of meaning, just like we did with whole words when we first explored turning meaning into places. That step gives every token a spot where similar meanings sit close together. We will give that step its proper name next time.