How CLIP Taught AI to See and Read at Once
Contrastive learning on 400 million image-text pairs: the foundation under modern visual AI.
CLIP (Contrastive Language-Image Pre-training) trains an image encoder and a text encoder together on 400 million image and caption pairs collected from the web. A symmetric contrastive objective pulls matching pairs together and pushes mismatched pairs apart in a shared embedding space. Cosine similarity in that space then gives zero-shot classification, search by description, and a surprising ability to match printed text inside images.
The whole design rests on one measured result. In the CLIP paper (Radford et al. 2021, Learning Transferable Visual Models From Natural Language Supervision), a model that only had to answer "which caption goes with which image?" reached the same zero-shot accuracy on ImageNet after seeing about a quarter as many images as a model that had to predict which words appear in each caption. That word predicting model was itself three times faster than one that generated the caption word by word, so the matching game beat caption writing by roughly twelve times. That is why your photo app can find "red car at dusk" without a single labeled example.
Imagine you have a deck of cards. Each card shows a photo on one side and a caption on the other. You want a learner to figure out which captions describe which photos. You don't ask it to write new captions. Instead you deal out a few hundred cards face up, mix up the text sides, and ask: for this photo, which caption belongs to it? If the learner gets it right, you reward it. If it picks a caption from a different card, you penalize it. Over millions of cards, it builds an internal mapping between visual patterns and words. That mapping lets it later look at a brand new photo and tell you whether "a sunset over the ocean" or "a crowded subway" fits better.
CLIP does exactly this at very large scale. Its training loop turns every batch of image and caption pairs into a giant matching game. The score it is graded on is the contrastive loss, and the mapping it builds becomes a joint embedding space where closeness means the picture and the words agree. For a photo app that wants to search your camera roll by description, CLIP is the piece that puts photos and words on the same map. The rest of this series adds the app's other jobs: answering questions about a photo (Part 2) and drawing or editing images (Part 3).
How does CLIP's contrastive training objective work internally?
CLIP computes a similarity matrix between every image and every caption in a batch, then applies a symmetric version of the InfoNCE loss (a standard contrastive loss that scores each correct pair against all the wrong ones) to pull correct pairs together and push all other pairs apart. A learned temperature controls how sharp the softmax becomes.
Each training batch holds image and caption pairs. The image encoder produces a vector for each image and the text encoder produces a vector for each caption. Both are scaled to length one (L2 normalized), so the dot product of two vectors is their cosine similarity. Stack the image vectors as the rows of a matrix and the caption vectors as the rows of a matrix . The model forms the similarity matrix , where each entry is the cosine similarity between image and caption , multiplied by a scale . Here is the temperature. CLIP learns the scale directly, stored as its logarithm (the variable logit_scale in its code).
The correct matches lie on the diagonal (). CLIP treats each row as a classification problem whose correct answer is column , and each column as a classification problem whose correct answer is row . The loss is the average of two cross-entropy terms:
In words: for each photo (a row), how well does the model pick its own caption out of the whole batch, and for each caption (a column), how well does it pick its own photo. The loss is the average of the two. Here is a tiny batch of three pairs, with made-up scores after scaling:
| "a dog in snow" | "a sunny beach" | "a birthday cake" | |
|---|---|---|---|
| Dog photo | 9.1 | 1.2 | 0.8 |
| Beach photo | 1.5 | 8.4 | 0.9 |
| Cake photo | 0.7 | 1.1 | 9.6 |
Training raises the bold diagonal and lowers everything else, along every row and down every column.
All off-diagonal entries act as negatives. With a batch of 32,768 pairs, the matrix holds 32,768 × 32,768 scores, a little over a billion pairings per step, and all but 32,768 of them are mismatches. The model gets that flood of negatives for the cost of one matrix multiplication, so no separate search for hard negatives is needed. The temperature starts at 0.07, so the scale starts near 14.3. Training adjusts it, and the scale is capped at 100 because larger values made training unstable, as the Learning Transferable Visual Models From Natural Language Supervision paper reports.
For the photo app developer, this means the model builds an embedding space where "golden retriever in snow" and a photo of exactly that are neighbors. The app can later measure that closeness with a single dot product.
How does a computer turn an image into numbers?
To a computer, a photo is a grid of numbers. A 224 by 224 pixel color image is 224 × 224 × 3 values: a red, a green and a blue brightness for every pixel. An image encoder is a network that turns that grid into one vector that describes what is in the picture. CLIP tried two kinds.
The older kind is a convolutional neural network (CNN). It slides small filters, each a few pixels wide, across the image. Early filters respond to edges and changes of color, and deeper layers combine those into textures, parts and whole objects. A ResNet is a deep CNN with shortcut connections that let each block add to what came before, which makes very deep networks trainable (He et al. 2015, Deep Residual Learning for Image Recognition). The newer kind is the Vision Transformer, or ViT (Dosovitskiy et al. 2020, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale). It cuts the image into small squares called patches (CLIP's versions use squares of 32, 16 or 14 pixels), turns each patch into a vector the way a language model turns a word into a token vector, adds a position so the model knows where each patch came from, and runs the same attention layers you met in Attention, Gently: How Models Decide What Matters. One extra summary token, often called [CLS], joins the sequence, and after the last layer its vector stands for the whole image.
How does CLIP's architecture support this joint embedding?
CLIP uses a dual-encoder design: an image encoder and a text encoder that each produce a single vector. Each vector passes through a linear projection into the same number of dimensions and is scaled to length one, so cosine similarity becomes a plain dot product. Both encoders are trained from scratch on the contrastive objective, without ImageNet weights or a pretrained language model.
The image encoder is a modified ResNet or a ViT. In the ResNets, CLIP replaces the usual final averaging step with attention pooling: a single attention layer whose query is the average of all the spatial features, so it can pick out what matters. In the ViTs, the output at the class token becomes the image vector.
The text encoder is a Transformer with masked self-attention, the left to right design of GPT-2. Its base size has 63 million parameters, 12 layers, a width of 512 and 8 attention heads. It splits captions into lowercased byte pair encoding (BPE) tokens from a vocabulary of about 49,000 and reads at most 76 tokens. Start and end tokens mark the caption's edges, and the last layer's output at the end token is projected into the shared space. That single vector summarizes the caption for every similarity score.
The shared space has 512 to 1,024 dimensions depending on the model (768 for the largest ViT). All parameters of both encoders, including the text encoder's attention layers, are updated by the contrastive loss. That pushes the text encoder toward visual meaning: words whose photos look alike, such as "crimson" and "red", tend to end up close together.
For the photo app, this means the same table of image vectors answers any text query. The text encoder maps "sunsets at the beach" to a point in the same space, and the nearest image vectors are the best matches. For a large library you keep those vectors in a vector index, the kind of store covered in Embeddings and Vector Search, Demystified.
How does the image encoder learn to "read" text in images?
Many captions in the dataset refer to words that appear in the image. Memes, screenshots, street signs and web pages all contain visible text that the caption mentions. The contrastive loss gives the model a strong reason to encode that visible text into the image vector, because it must match captions that mention it.
Consider a photo of a protester holding a sign that reads "JUSTICE NOW". A web caption might be "person holding a justice now sign at a rally". If the image encoder ignores the words on the sign, the caption will not match well and the loss stays high. Over millions of such examples, the image encoder learns to pick up printed words without any dedicated OCR (optical character recognition) module. The paper describes a high quality semantic OCR representation that works well on digitally rendered text, such as its Rendered SST2 test (movie review sentences drawn as images). The limits are just as clear. CLIP reached only 88% accuracy on the handwritten digits of MNIST, below a simple logistic regression on the raw pixels, and it cannot transcribe text or say where the words are. That job belongs to document models, covered in Document AI: How Models Read PDFs, Tables, and Receipts.
This skill also creates a weakness. Goh et al. showed a CLIP model a photo of an apple, which it called a Granny Smith with 85.6% confidence, then stuck a label reading "iPod" on the fruit: the model switched to "iPod" with 99.7% confidence (2021, Multimodal Neurons in Artificial Neural Networks). This "typographic attack" shows how tightly the model binds what it sees to the words it reads.
How does CLIP achieve zero-shot classification from text prompts?
Zero-shot classification builds a classifier from text descriptions alone. For a new set of categories, you embed a prompt like "a photo of a {label}" for each label. The resulting text vectors act as class prototypes. Each test image is embedded, and the class whose prototype has the highest cosine similarity to the image vector is the prediction.
This pipeline means the photo app can add new album categories without any retraining. If the user types "puppies wearing hats", the app embeds that string, computes its similarity to every camera roll image vector, and returns the top results. The class prototype is just a text vector, computed on the fly.
Prompt wording matters. CLIP was trained on internet captions, not single words, so "a photo of a cat" works better than just "cat". The paper found that averaging the text vectors of 80 different prompt templates added 3.5 points of ImageNet accuracy over a single default prompt, and prompt engineering and ensembling together added almost 5 points. For fine-grained tasks, naming the kind of thing helped: "A photo of a {label}, a type of pet." for pet breeds, or "a satellite photo of a {label}." for satellite images. In your photo app, you might register a handful of prompts for common searches and average their vectors to improve retrieval.
What makes the training pipeline work at web scale?
Training on 400 million pairs took a careful mix of data, model and optimization choices.
Variety in the data replaces heavy augmentation. A random square crop from each resized image is the only augmentation; there is no color jitter or other distortion. Captions are lowercased, split into BPE tokens and capped at 76 tokens.
The batch size is pushed to 32,768 to flood the loss with in-batch negatives. To fit that in memory, the authors used mixed precision, gradient checkpointing (recomputing some activations during backpropagation instead of storing them) and half precision optimizer statistics, and they split the similarity calculation across GPUs so each one computes only the part it needs. Every model trains for 32 epochs with the Adam optimizer, decoupled weight decay and a cosine learning rate schedule. All weights start random, so the representations are shaped entirely by the image and caption matching signal. Even so, the largest ViT took 12 days on 256 V100 GPUs.
The recipe still needs a lot of compute, but the efficiency result from the start of this article is what made that scale affordable: matching captions instead of predicting their words cut the training needed for the same zero-shot accuracy by about four times, and by roughly twelve times compared with writing captions word by word. If you want to train or run this recipe yourself, the open source OpenCLIP project reimplements it, and the open dataset in LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (Schuhmann et al. 2021) matches the original's size.
Quick Reference
| Property | Value |
|---|---|
| Dataset size | 400M image-text pairs collected from the web |
| Batch size | 32,768 |
| Image encoder variants | Modified ResNet (with attention pooling) or ViT |
| Text encoder | Transformer with masked self-attention; base size 63M parameters, 12 layers, 512 wide, 8 heads |
| Tokenization | Lowercased BPE, vocabulary of about 49,000 |
| Max text length | 76 tokens |
| Shared embedding dim | 512 to 1,024 depending on the model (768 for ViT-L/14) |
| Temperature | Starts at 0.07 (logit scale about 14.3), learned, scale capped at 100 |
| Training epochs | 32 |
| Optimizer | Adam with decoupled weight decay, cosine learning rate schedule |
| Augmentation | Random square crop only |
| Zero-shot prompt convention | "A photo of a {label}.", often ensembled |
Frequently Asked Questions
Q: Why not train CLIP to generate captions instead of using a contrastive loss? Writing a caption forces the model to get every word and its order right, which is much harder than deciding which caption matches an image. In the paper's comparison, the contrastive model reached the same zero-shot accuracy with about a quarter of the training images a word predicting model needed, and roughly a twelfth of what a word by word caption generator needed.
Q: How does CLIP handle images with multiple objects? The training captions often mention several objects, so a photo of a cat and a dog on a sofa does land near text that lists them, and a query like "cat and dog on a sofa" usually finds such photos. What CLIP handles poorly is who does what to whom. On Winoground, where two captions use exactly the same words in a different order, the models tested did barely better than chance (Thrush et al. 2022, Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality). Yuksekgonul et al. found that such models often attach attributes to the wrong object and ignore word order (2022, When and why vision-language models behave like bags-of-words, and what to do about it?), and a 2025 benchmark found them near chance on negation (Alhamoud et al. 2025, Vision-Language Models Do Not Understand Negation). So "a dog chasing a cat" and "a cat chasing a dog" can return almost the same photos.
Q: What are the main limitations of CLIP's zero-shot abilities? CLIP can fail on fine-grained distinctions where language is ambiguous, on systematic tasks such as counting the objects in an image, and on images unlike anything in its training data, such as the handwritten digits above. It also inherits biases from web captions, which can lead to stereotyped associations.
Q: Can I fine-tune CLIP for a custom domain? Yes. You can fine-tune the full model or train just a linear probe (a simple classifier) on top of the frozen embeddings. Fine-tuning the image encoder often improves accuracy in your domain, but the model may lose some of its zero-shot generality. Many production systems keep the zero-shot backbone and add a small classification head for the categories that matter most.
Q: How does the temperature affect training? The temperature divides the similarity scores before the softmax. A high temperature (a small scale) flattens the softmax, so every caption looks about as likely as the others and the training signal is weak. A low temperature (a large scale) sharpens it, which concentrates learning on the closest wrong matches but can make training unstable. CLIP starts the temperature at 0.07, lets training adjust it, and caps the scale at 100.
Q: Is CLIP still used? Its idea is everywhere. A popular successor, SigLIP, trains the same kind of image and text encoders with a simpler sigmoid loss that scores each image and caption pair on its own instead of normalizing across the whole batch (Zhai et al. 2023, Sigmoid Loss for Language Image Pre-Training). The LLaVA models in Part 2 began with a CLIP image encoder and later moved to a SigLIP one.
Test yourself
You are building the search feature of a camera roll app with CLIP. A user types "wooden boat on a foggy lake". The camera roll holds two untagged photos: a small wooden rowboat on a misty lake, and a sailboat on clear blue water. Which photo ranks higher, and why? Then the user types "a boat that is not on water". What might go wrong?
Answer: The rowboat photo ranks higher. When each photo was added, the app ran it through the image encoder and stored its vector; no captions are involved. At search time the text encoder turns the query into a vector in the same space, and the app ranks photos by the cosine similarity between that query vector and each image vector. During training CLIP saw millions of web captions about boats, fog and mist next to matching pictures, so the query lands closer to the misty rowboat than to the bright sailboat, even though no words were ever attached to either photo. The second query is a trap. Contrastive models largely record which words are present, and tests find them close to chance on negation, so "not on water" may still pull up boats on water. A practical fix is to search for what the user does want, such as "boat on a beach" or "boat on a trailer".
Next: Inside LLaVA: Bolting Eyes Onto a Language Model. It connects a CLIP image encoder to a language model so the app can answer questions about a photo.
If you want a breakdown like this every week, how real systems like CLIP actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Learning Transferable Visual Models From Natural Language Supervision (Radford et al. 2021)
- CLIP code and model card (OpenAI)
- Deep Residual Learning for Image Recognition (He et al. 2015)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Dosovitskiy et al. 2020)
- Multimodal Neurons in Artificial Neural Networks (Goh et al. 2021)
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (Thrush et al. 2022)
- When and why vision-language models behave like bags-of-words, and what to do about it? (Yuksekgonul et al. 2022)
- Vision-Language Models Do Not Understand Negation (Alhamoud et al. 2025)
- Sigmoid Loss for Language Image Pre-Training (Zhai et al. 2023)
- OpenCLIP (open source implementation)
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (Schuhmann et al. 2021)
