What an LLM Actually Is (and Why Predicting Words Works)
It predicts the next word. That one trick, at scale, is doing everything you've seen.
An LLM is a gigantic probability calculator. It takes a sequence of words (tokens), runs them through a stack of transformer layers, and outputs a distribution over the next possible word. Pick one, feed it back, and repeat. That cycle, trained on trillions of tokens, is the entire secret behind chatbots that generate recipes, draft emails, and summarize articles.
The surprising part? There is no database of facts, no planner, and no grammar checker. Every impressive behavior, from solving math problems to mimicking a therapist, emerges from the singular pressure to guess the next word correctly.
Start With the Autocomplete on Your Phone
Before we open the hood, think about the word suggestions that appear above your smartphone keyboard. Type “I’m going to bake” and it might suggest “cookies,” “a cake,” or “bread.” That little strip is a miniature language model, trained on your typing history. It looks at the last few words and ranks the next most probable words. An LLM does the same thing, but with three critical differences: it has seen an enormous fraction of the internet, it can attend to thousands of previous words instead of three, and it uses a far deeper architecture to capture complex patterns.
When you ask a chatbot for a chocolate chip cookie recipe, the model does not search a recipe database. It starts with your prompt as a sequence of word fragments, then repeatedly predicts the next fragment. “Ingredients:” leads to “1 cup” leads to “butter” and so on. The whole recipe rolls out one token at a time, driven only by a massive pattern matcher that has learned from every recipe, blog, and cooking forum in its training data. That is the autocomplete mental model. Now we will see why, at scale, that simple trick produces such rich behavior.
What Does an LLM Actually Compute?
An LLM is a function (f_\theta) with billions of parameters (\theta). It accepts a sequence of discrete symbols (tokens) and returns, for each position, a probability distribution over all possible next tokens Vaswani et al. 2017. The function has two main parts: an embedding layer that turns token IDs into vectors, and a stack of transformer blocks that refine those vectors. At the final layer, a linear projection plus a softmax gives the probabilities.
Tokens: The Model’s Native Vocabulary
Before any neural math happens, raw text is chopped into tokens by a tokenizer such as Byte Pair Encoding (BPE) Sennrich et al. 2016. A token is usually a short subword. For example, “chocolate” might become one token, while “summarizing” might split into “summar” and “izing.” The tokenizer builds a fixed vocabulary (often 50,000 to 100,000 tokens) and maps every text snippet to a sequence of integer IDs. The model never sees characters or words directly; it only processes those IDs.
When you type “write a birthday message for my sister,” the tokenizer converts it to something like [1024, 347, 8912, 15008, 305, 623, 411, 8902]. Those integers are the only input the model receives. The model learns statistical structure at the token granularity, which is why odd spacing or a rare spelling can throw it off SentencePiece. Once you understand that, you see why latency is measured in tokens per second and why the same prompt can behave differently if you add a stray space.
From Integers to Vectors: The Embedding Table
Each token ID is mapped to a dense vector via an embedding table (E \in \mathbb{R}^{V \times d_{\text{model}}}) Vaswani et al. 2017. If the vocabulary size (V) is 50,000 and the hidden dimension (d_{\text{model}}) is 4,096, the table holds 50,000 rows of 4,096 numbers. The embedding for token 1024 is simply its row, a point in a high-dimensional space. At first these rows are random, but during training they shift so that tokens with similar roles end up close together. The distance between the vectors for “cat” and “dog” becomes smaller than the distance between “cat” and “car.”
These raw embeddings get better after they pass through the transformer layers. Each layer enriches a token’s representation by mixing in information from every other token in the sequence. That is how “bank” in “river bank” produces a different final vector than “bank” in “I went to the bank.” The embedding table is the starting point; the stack does the contextualization.
The Autoregressive Distribution
The model’s job is to estimate (p_\theta(x_t \mid x_1, \dots, x_{t-1})) for every position (t). Over a full sequence, the joint probability factorizes as
[
p_\theta(x_1, \dots, x_T) = \prod_{t=1}^T p_\theta(x_t \mid x_{<t}).
]
This is the autoregressive assumption: each token depends only on its prefix. At the final layer, a linear “language model head” projects each position’s hidden vector to a vector of logits, one per vocabulary token. A softmax turns those logits into probabilities that sum to one Vaswani et al. 2017.
During training the model sees the true next token and is penalized with cross-entropy loss: (-\log p_\theta(\text{true token})). The optimizer reduces this loss across billions of examples. At inference, the model picks a token (by greedy argmax, sampling, or top-p) and feeds the extended sequence back into itself. The cycle repeats, generating new text one token at a time.
The whole pipeline looks like this:
For efficiency, most inference engines cache the keys and values from previous positions so each new token only requires one forward pass through the stack rather than recomputing the entire prefix.
Why Does Predicting the Next Word Work So Well?
The secret lies in compression. Minimizing next-token cross-entropy is equivalent to compressing the training corpus as tightly as possible Kaplan et al. 2020. To assign high probability to the correct continuation, the model must internally model syntax, facts, causal relationships, and even the intent behind a prompt. That pressure forces the discovery of reusable patterns, from subject-verb agreement to the steps in a recipe.
Scaling laws show that as we increase parameters, data, and compute, the loss drops predictably Kaplan et al. 2020. The Chinchilla paper later clarified that for a given compute budget, the optimal balance is roughly 20 training tokens per parameter Hoffmann et al. 2022. So bigger models trained on more data simply compress better, and that improved compression delivers abilities that smaller models do not have: translation, coding, basic arithmetic, and in-context learning.
In-context learning is a spectacular example. If you give an LLM a few examples of a new task inside the prompt, and then a new input, it often produces the correct output, even though its weights never changed Brown et al. 2020. Mechanistic interpretability work reveals that circuits called “induction heads” allow the model to detect repeated patterns and copy them forward Olsson et al. 2022. These heads emerge purely from next-token training. So the simple objective, at scale, sculpts general-purpose machinery.
Thus, predicting the next word is not a trivial trick. It is a universal training signal that forces the model to approximate the data-generating distribution, building internal shortcuts for grammar, logic, and pattern recognition.
How Does a Transformer Do This?
The transformer architecture, introduced in “Attention Is All You Need,” processes all tokens in parallel using self-attention Vaswani et al. 2017. A decoder-only LLM stacks many identical blocks. Each block has two main sublayers: a multi-head self-attention layer and a position-wise feed-forward network. Residual connections and layer normalization wrap both.
Self-Attention: Mixing Information Across Positions
For each token, self-attention computes three vectors: query ((Q)), key ((K)), and value ((V)) by multiplying the token’s hidden representation by learned weight matrices. For every pair of positions, the dot product between a query and a key measures compatibility. After scaling by (1/\sqrt{d_k}) and applying a causal mask (so a token cannot see future tokens), a softmax converts those scores into attention weights. The output for a token is a weighted sum of all other tokens’ value vectors, where the weights capture how much each position should influence the current one Vaswani et al. 2017.
Multi-head attention repeats this process with (h) independent (Q,K,V) sets in parallel, then concatenates the results. Different heads can attend to different phenomena: one head might track previous occurrences of a word, another might link a pronoun to its antecedent, a third might focus on delimiters in code.
Feed-Forward Network: Per-Token Nonlinear Transformation
After attention mixes information across positions, each token’s vector passes through a two-layer MLP with an activation like GELU: (\text{MLP}(x) = W_2 , \text{GELU}(W_1 x + b_1) + b_2). This transformation is applied independently to every position. While attention handles cross-token communication, the MLP introduces nonlinear capacity and can store factual associations in its weights. Studies often find that specific neurons in these MLP layers activate for interpretable concepts Elhage et al. 2021.
Residual Connections and Layer Normalization: The Training Scaffold
Without residual connections, gradients would shrink or explode across dozens of layers. A residual connection simply adds the input of a sublayer to its output: (x + F(x)). This gives gradients a direct path during backpropagation He et al. 2016. Layer normalization normalizes each token’s feature vector to zero mean and unit variance, then applies learned scale and shift. It stabilizes activations and accelerates training Ba et al. 2016. Modern LLMs usually place layer norm before each sublayer (pre-norm), which further improves gradient flow Xiong et al. 2020.
Together, residuals and layer norm make it possible to train networks with 96, 120, or more layers, scaling the model’s capacity to capture long-range dependencies.
Positional Encodings: Telling the Model About Order
Self-attention is permutation invariant. To inject order, the model adds positional information to the input embeddings. Many modern LLMs use Rotary Position Embedding (RoPE), which encodes positions as rotations of the query and key vectors in 2D subspaces Su et al. 2021. The dot product between a rotated query and a rotated key naturally captures relative distance, decaying with separation. RoPE also lets the model extrapolate to sequence lengths longer than those seen during training.
Without this step, the model would treat “dog bites man” and “man bites dog” identically. Positional encodings make the model sensitive to word order, syntax, and temporal flow.
Quick Reference
| Concept | Typical Value or Description |
|---|---|
| Tokenizer | BPE or SentencePiece; vocabulary size 50k-100k subword tokens |
| Embedding dimension ((d_{\text{model}})) | 4096 for a 7B model, up to 8192+ for larger models |
| Number of layers ((L)) | 32 for a 7B model, often 80-96 for 70B+ models |
| Attention heads ((h)) | 32 per layer (128-dimensional per head when (d_{\text{model}}=4096)) |
| Feed-forward intermediate size ((d_{\text{ff}})) | Typically 4× (d_{\text{model}}) (e.g., 11008 or 14336) |
| Position encoding | Rotary Position Embedding (RoPE) in most current LLMs |
| Activation function | GELU in most transformers (SwigLU in newer variants) |
| Training objective | Next-token cross-entropy loss |
| Typical training tokens | Order of 1-2 trillion tokens for a 7B Chinchilla-optimal model |
| Inference strategy | Autoregressive sampling with KV-caching, top-(p) and temperature |
Frequently Asked Questions
Q: Does the model actually understand language, or is it just statistical mimicry? It builds a compressed, predictive model of token sequences. That internal model captures syntax, semantics, and factual relationships well enough to produce coherent output. Whether that constitutes “understanding” depends on your definition, but the model certainly abstracts patterns far beyond surface-level statistics.
Q: How can it do arithmetic if it only predicts the next word? Long training on text that includes calculations forces the model to approximate algorithmic reasoning. It learns to attend to digits and mimic the step-by-step process, though it can fail on large numbers because it does not execute true symbolic operations.
Q: Why do bigger models suddenly acquire new abilities? Scaling laws show a continuous drop in loss, but certain capabilities appear abruptly when the model reaches sufficient capacity to compress the required pattern. This emergent behavior arises because the optimization landscape changes qualitatively at certain scales Wei et al. 2022.
Q: What are tokens, and why not use whole words? Tokens are the atomic pieces the model sees. Subword tokenization handles rare words and morphology gracefully and keeps the vocabulary size manageable, avoiding the explosion that would come from a full-word vocabulary.
Q: Can an LLM memorize its training data? Yes. Large models can memorize verbatim passages, especially when data is duplicated. Researchers study this as a privacy and copyright concern, and it is one reason deduplication of training data is important Carlini et al. 2022.
Test yourself
You are building an internal tool that uses an LLM to generate SQL queries from natural language questions. Users report that the model occasionally produces syntactically correct queries that nevertheless reference nonexistent table names. Why might this happen, given that the model never executed a line of SQL during training?
Answer: The model never accesses a live database schema. During training it saw millions of text examples that included SQL snippets and natural language descriptions, often paired with plausible table and column names. It learned to map question patterns to likely SQL structures. When a user asks about “customer revenue,” the model generates a query that looks statistically correct based on training, but it invents table names like customer_revenue or sales_data because it has no ground truth about the actual database. It is fulfilling the prompt under the next-token objective, not executing a grounded lookup. To fix this, you must supply the schema explicitly in the prompt, letting the model’s in-context learning bind to concrete identifiers.
If you want this kind of breakdown every week, the real internals behind the tools you use, subscribe to Internals Decoded at internalsdecoded.com. Next time we will open the black box of training and see how billions of weights actually learn from raw text.
Sources
- Vaswani et al. 2017, Attention Is All You Need
- Sennrich et al. 2016, Neural Machine Translation of Rare Words with Subword Units
- SentencePiece tokenizer
- Kaplan et al. 2020, Scaling Laws for Neural Language Models
- Hoffmann et al. 2022, Training Compute-Optimal Large Language Models (Chinchilla)
- Brown et al. 2020, Language Models are Few-Shot Learners (GPT-3)
- Olsson et al. 2022, In-context Learning and Induction Heads
- Su et al. 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding
- He et al. 2016, Deep Residual Learning for Image Recognition
- Ba et al. 2016, Layer Normalization
- Xiong et al. 2020, On Layer Normalization in the Transformer Architecture
- Elhage et al. 2021, A Mathematical Framework for Transformer Circuits
- Wei et al. 2022, Emergent Abilities of Large Language Models
- Carlini et al. 2022, Quantifying Memorization Across Neural Language Models