Inside LLaVA: Bolting Eyes Onto a Language Model
How open vision-language models connect an image encoder to an LLM, layer by layer.
In Part 1 we saw how CLIP (Contrastive Language-Image Pre-training) embeds images and text in a shared space. LLaVA builds on that: it takes a frozen CLIP vision encoder, attaches a small projector that maps image features into the token space of an LLM (large language model), and then trains the system on multimodal instruction dialogues. The LLM does all the cross-modal reasoning through its own self-attention. There are no cross-attention modules and no Q-Former (a small transformer with learned queries that squeezes image features into a fixed number of tokens). Just a projector and a pretrained transformer.
The surprising part is how small the projector is. In the original LLaVA it is a single linear layer of about 4 to 5 million parameters; in LLaVA-1.5 it is a two-layer MLP of about 21 million with a 7 billion parameter LLM, or about 31 million with a 13 billion one. It learns to translate CLIP's visual features into something the LLM can treat like text. The rest (answering questions, describing scenes, even reading text in images) comes from the LLM's language skills, tuned on instruction data. In our photo app, this is the engine that answers questions about a photo you have picked. Finding photos across the whole camera roll is still CLIP's job from Part 1.
How does LLaVA connect a vision encoder to an LLM?
LLaVA has three pieces: a frozen vision encoder, a small trainable projector, and a pretrained LLM that writes its answer one token at a time. The vision encoder is CLIP's ViT-L/14 (at 224 pixels in the original LLaVA, the 336 pixel version in LLaVA-1.5). The LLM was Vicuna, an instruction-tuned derivative of LLaMA. Later versions swap in other parts: LLaVA-OneVision uses a SigLIP encoder with Qwen2 language models. The projector sits between the encoder and the LLM, mapping each image feature vector to the LLM's embedding size.
The vision encoder turns an image into a sequence of patch features. A 336 × 336 image cut into 14 × 14 pixel patches gives a 24 × 24 grid, so 576 patch features. LLaVA takes those 576 features from the layer before CLIP's last one and drops the class token. The original paper found the earlier layer slightly better, and suggested that the last layer leans toward global, abstract properties while the one before it keeps the local details that answering questions needs. The projector maps each feature, one by one, to the LLM's hidden size. In LLaVA-1.5 the projector is a two-layer MLP with a GELU (a smooth activation function) between the layers. After projection you have a stack of visual embeddings with the same shape as word embeddings, and the LLM can interleave them with text tokens in its usual forward pass.
This design is deliberate. The LLM is never told which tokens came from an image. It just sees a long sequence of embeddings, and all cross-modal interaction happens in the self-attention layers it already has. That keeps the parts swappable. For a photo app you could plug in a better image encoder, but then you retrain the projector and repeat the instruction tuning, because the LLM was tuned on the old encoder's features.
How does the visual information flow into the LLM's token stream?
The flow has four stages: encode, project, merge, generate. Let's follow a request from the photo app about one photo the user has open: "Is there a dog in the snow in this photo? Describe it."
First, the image is resized to the encoder's resolution and fed into the frozen CLIP vision transformer. The output is a tensor of shape (N, C), where N is the number of patch features (576) and C is 1024. The projector maps this to (N, d_model), where d_model is the size of each token vector inside the LLM: 4096 for Vicuna-7B and 5120 for Vicuna-13B. This step is two small matrix multiplications, tiny next to the LLM's own work.
At the same time, the text prompt is tokenized. The prompt includes a special <image> marker that tells the system where to insert the visual tokens, for example "USER: <image> Is there a dog in the snow in this photo? Describe it." The tokenizer splits this into token IDs, and the system replaces the <image> placeholder with the whole block of visual embeddings. If the image contributes 576 tokens, the placeholder expands to 576 slots. The merged sequence now holds visual and text embeddings, all in the same d_model-dimensional space.
This merged sequence is fed into the LLM. The transformer processes it layer by layer, applying self-attention across the entire sequence. The visual tokens act like a prefix: they stay in the attention window while the model writes its answer, token by token. In our photo app, the answer is a description: yes, a golden retriever is digging in fresh snow. To search the whole camera roll for such photos you would use CLIP embeddings from Part 1; LLaVA answers questions about photos you have already picked.
The key detail is that the LLM's token embedding matrix is never used for the visual tokens. The visual embeddings are injected directly where the embedding layer's output would go. This means the LLM's vocabulary is unchanged. The model learns to attend to the visual tokens just as it would attend to any other prefix.
How does LLaVA handle bigger images, several images and video?
Later versions extend the single-image pipeline without new fusion machinery. The first problem they tackled was detail. A 336 pixel image loses small print and small objects, so LLaVA-NeXT (January 2024) introduced AnyRes. It splits a large image into a grid of crops, each at the encoder's resolution, encodes every crop separately, and adds one downscaled copy of the whole image for context. That raised the input to four times more pixels, up to 672 × 672 or long shapes like 336 × 1344, as the LLaVA-NeXT: Improved reasoning, OCR, and world knowledge post describes. It also multiplies the tokens: the LLaVA-OneVision paper writes the total as (crops + 1) × tokens per crop, so a 2 × 2 grid of CLIP crops costs (4 + 1) × 576 = 2,880 visual tokens instead of 576.
LLaVA-OneVision (August 2024) then put a ceiling on that growth. Its Higher AnyRes counts the total, and if it passes a threshold, it shrinks each crop's features with bilinear interpolation (Li et al. 2024, LLaVA-OneVision: Easy Visual Task Transfer). With its SigLIP encoder at 729 tokens per 384 pixel crop, a single image can still reach (1 + 9) × 729 = 7,290 tokens.
Several images work the same way. Each image is encoded on its own, projected, and placed into the sequence, and the prompt gets one <image> marker per image, each expanded to the right number of tokens. OneVision trains on up to 12 images per example. The model can then compare images, spot differences, or follow an object from one to the next.
Video is treated as a sequence of frames. Each frame is encoded like a still image, then pooled to save room: OneVision shrinks each frame from 729 tokens to 196 with 2 × 2 bilinear interpolation and samples up to 32 frames, so a clip costs at most 32 × 196 = 6,272 tokens. Without pooling, the same 32 frames would cost more than 23,000. Video-focused descendants push further: SlowFast-LLaVA-1.5 combines two streams of frames into a token-efficient design for long videos (Xu et al. 2025, SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding).
No special video adapter is needed. The same projector and LLM handle single images, several images and video frames, which is a big win for maintenance: you deploy one model and feed it different token sequences.
How is LLaVA trained to follow multimodal instructions?
Training happens in stages, each updating a different set of parameters. The first stage trains the projector alone, keeping both the vision encoder and the LLM frozen. Its goal is to align visual features with the LLM's embedding space. The original LLaVA used 595K image and caption pairs filtered from the CC3M dataset (Liu et al. 2023, Visual Instruction Tuning); LLaVA-1.5 used 558K. Each pair becomes a one-turn conversation: the projected image tokens plus a short instruction such as "Describe the image briefly", with the real caption as the answer. The training signal is ordinary next-token prediction: the frozen LLM must produce the caption, and only the projector's weights change. The paper calls this training a compatible visual tokenizer for the frozen LLM.
Next comes multimodal instruction tuning. Now the projector and the LLM are both trained, while the original LLaVA keeps the vision encoder frozen. The original instruction data was 158K examples written by a text-only GPT-4 from each image's captions and object boxes: 58K conversations, 23K detailed descriptions and 77K reasoning questions. LLaVA-1.5 added academic VQA (visual question answering) datasets, OCR (optical character recognition) data and region-level tasks, 665K examples in all (Liu et al. 2023, Improved Baselines with Visual Instruction Tuning). The loss is cross-entropy on the response tokens, given the image and the prompt. LLaVA-1.5's whole training uses 1.2 million publicly available examples and finishes in about a day on one machine with 8 A100 GPUs.
LLaVA-OneVision adds a middle stage, Stage 1.5, for "high-quality knowledge": about 4 million examples, mostly detailed captions written by an earlier LLaVA model, plus document and OCR data. From Stage 1.5 on, OneVision trains the full model, vision encoder included, with the vision encoder's learning rate five times smaller than the LLM's.
For a photo app, you could fine-tune a LLaVA checkpoint on your own usage data: a few thousand examples of the instructions your users give. To keep costs down you can freeze most of the LLM and train only the projector plus a few LoRA (low-rank adaptation) adapters. Test on general questions before and after, since tuning on a narrow set can make the model worse at everything else.
Why does the simple projector work so well?
The projector's power comes from the quality of the pretrained parts. CLIP already encodes rich semantic and spatial information, and the LLM already understands language and follows instructions. The projector only has to translate one representation into the other, and a linear layer or a shallow MLP turns out to be enough.
Compare models that add a dedicated resampler. BLIP-2 and InstructBLIP place a Q-Former between the image encoder and the LLM, and the original Qwen-VL (2023) used a single cross-attention adapter that squeezes every image into 256 tokens. These were pretrained on hundreds of millions to billions of image and text pairs: 129 million for InstructBLIP and 1.4 billion for Qwen-VL, as the LLaVA-1.5 report's comparison table lists. LLaVA-1.5's MLP connector, aligned on 558K pairs, reported state-of-the-art results across 11 benchmarks. The gain came from the full recipe (the data mix, the higher resolution, tuning the LLM), not from the connector alone, but it showed that a heavy fusion module is not required: the LLM's own self-attention can do the work.
The projector is also a lever for efficiency. TokenPacker replaces the MLP with a coarse-to-fine projector that compresses the visual tokens by 75% to 89% while keeping performance comparable or better across diverse benchmarks (Li et al. 2024, TokenPacker: Efficient Visual Projector for Multimodal LLM). For a photo app on a phone, a projector like that sends fewer tokens to the LLM, which lowers latency and memory use. The rest of the architecture stays the same.
Where do LLaVA-style models fail?
The best known failure is object hallucination: describing things that are not in the image. A study of several such models found that most suffer from severe object hallucination, and that objects which often appear in the training instructions, or often appear next to the objects really in the picture, are the most likely to be invented (Li et al. 2023, Evaluating Object Hallucination in Large Vision-Language Models). A photo of a dinner table may get a fork it does not have, because forks usually sit beside plates. That is the language prior overriding the image.
Fine detail and small text suffer at low resolution. The LLaVA-1.5 authors found that hallucination in detailed descriptions dropped sharply when the input grew to 448 pixels, and suggested that a model learns to guess when its training captions describe details its input is too small to show. Simple visual questions can also go wrong: on image pairs that CLIP sees as nearly identical, models built on CLIP struggle with basic patterns such as counting, direction and position (Tong et al. 2024, Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs). For the photo app, treat counts, small text and "is there an X?" answers as claims to check, not facts.
Where does LLaVA sit in 2026?
The recipe LLaVA made popular, a vision transformer, a small MLP connector and an LLM, is now the standard layout of open vision-language models, usually with images processed at their native resolution instead of one fixed size. The Qwen3-VL Technical Report (Bai et al. 2025, Qwen3-VL Technical Report) describes a vision encoder, an MLP-based vision and language merger that compresses each 2 × 2 group of visual features into one token, and an LLM. InternVL3.5 keeps the same layout of vision transformer, MLP and LLM as earlier InternVL versions (Wang et al. 2025, InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency). The LLaVA line itself continues: LLaVA-OneVision-2 (May 2026) still joins its vision encoder to a Qwen3-8B language model through a lightweight two-layer MLP (An et al. 2026, LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence).
Quick Reference
| Property | Value |
|---|---|
| Default vision encoder | CLIP ViT-L/14 at 224 pixels (LLaVA) or 336 pixels (LLaVA-1.5); SigLIP in OneVision |
| Vision encoder status | Frozen in LLaVA and LLaVA-1.5; trained from Stage 1.5 on in OneVision, at a 5 times smaller learning rate |
| Language model | Vicuna-7B or 13B (LLaVA, LLaVA-1.5); Qwen2 (OneVision) |
| Projector | Linear layer (LLaVA) or 2-layer MLP with GELU (LLaVA-1.5 and later) |
| Projector parameters | About 4M (linear, 7B) to about 31M (MLP, 13B) |
| Visual tokens per image (336 pixels) | 576 patch features from the layer before last, no class token |
| Projector output dimension | Matches the LLM embedding size (4096 for 7B, 5120 for 13B) |
| Alignment data | 595K CC3M pairs (LLaVA), 558K pairs (LLaVA-1.5) |
| Instruction tuning data | 158K GPT-4 written dialogues (LLaVA); 665K mixed examples (LLaVA-1.5) |
| Training stages | 1. Projector alignment (frozen LLM and vision); 1.5. High-quality knowledge (OneVision only); 2. Instruction tuning (LLM trained) |
| Key design principle | No cross-attention; all fusion via LLM self-attention |
The stat card below sizes each part for a typical 7B setup. Parameter counts are worked out from each part's published dimensions: the ViT-Large size in An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, and the MLP's layers (1024 to 4096 to 4096) in the LLaVA code.
Frequently Asked Questions
Q: Can I use a different vision encoder, like SigLIP or DINOv2, instead of CLIP? Yes. Any encoder that produces a sequence of feature vectors can feed the projector. You retrain the projector for the new feature size and repeat the instruction tuning, since the LLM learned to read the old encoder's features. LLaVA-OneVision, for example, uses SigLIP. Just be aware that the model can only be as good as its visual features, and you may need to adjust the projector's capacity.
Q: What happens if my prompt does not include an <image> token?
The pipeline inserts no visual tokens. The LLM sees only the text tokens and behaves like a pure text model. The same model can handle both text-only and multimodal requests without any mode switch.
Q: How many visual tokens are too many? Every visual token takes a place in the LLM's context window, alongside the prompt and the answer. With LLaVA-1.5, one 336 pixel image uses 576 tokens; AnyRes multiplies that by the number of crops plus one, and OneVision allows up to 7,290 for a single image. You must keep the total within the model's context length, which is why video models pool each frame's tokens and why OneVision caps the total.
Q: Why does the projector need to be trained at all? Couldn't you just use CLIP's features directly? CLIP's features live in CLIP's space, not in the LLM's token embedding space. The LLM's token embeddings have their own distribution and scale, learned for language modeling. The projector learns a mapping that turns CLIP features into vectors the LLM can use. Without that training, the visual tokens would be out of distribution and the LLM would treat them as noise.
Q: Is it possible to fine-tune only the projector and keep the LLM frozen for a new visual task? Yes, and it is cheap. If you have a small set of task-specific instruction examples, you can freeze the LLM and train only the projector, possibly with a few LoRA adapters. The model learns to read the visual tokens in the context of your task without changing much of its language behavior. Expect a smaller gain than full instruction tuning, and test it on your own examples.
Test yourself
You are building a photo app. A user asks: "Show me all photos taken in Paris with the Eiffel Tower in the background." The gallery holds 10,000 photos, and running LLaVA's full generation on every one would be far too slow. How would you combine the models from Parts 1 and 2 to answer quickly and well?
Answer: Use two stages. If the photos carry location data, filter by it first. Stage 1 is retrieval with full CLIP. LLaVA keeps only CLIP's vision tower, so load CLIP's text encoder as well. When photos are added, store each one's CLIP image vector. At query time, embed the request with the CLIP text encoder, compare it with all 10,000 stored image vectors, and keep the top K, say 50. That costs one text encoding and a fast vector search. Stage 2 is verification with LLaVA: for each candidate, ask " Is the Eiffel Tower in the background of this photo?" and keep the ones it confirms, perhaps with a short reason to show the user. Do not compare the projected visual tokens with the query's word embeddings: those spaces were never trained for retrieval, and a handful of token embeddings is not a sentence vector. CLIP was built for the first job and LLaVA for the second.
Next: Diffusion: How Image Generators Actually Draw. It turns from reading images to making them, from noise to a finished picture.
If you want this kind of breakdown every week, how real vision-language models actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- Visual Instruction Tuning (Liu et al. 2023, the original LLaVA)
- Improved Baselines with Visual Instruction Tuning (Liu et al. 2023, LLaVA-1.5)
- LLaVA-NeXT: Improved reasoning, OCR, and world knowledge (LLaVA team, January 2024)
- LLaVA-OneVision: Easy Visual Task Transfer (Li et al. 2024)
- TokenPacker: Efficient Visual Projector for Multimodal LLM (Li et al. 2024)
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding (Xu et al. 2025)
- Evaluating Object Hallucination in Large Vision-Language Models (Li et al. 2023)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (Tong et al. 2024)
- Qwen3-VL Technical Report (Bai et al. 2025)
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (Wang et al. 2025)
- LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence (An et al. 2026)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Dosovitskiy et al. 2020)
- LLaVA code and LLaVA-NeXT repository
- Hugging Face LLaVA documentation
- Learning Transferable Visual Models From Natural Language Supervision (Radford et al. 2021, CLIP)
