Inside LLaVA: Bolting Eyes Onto a Language Model
How open vision-language models connect an image encoder to an LLM, layer by layer.
In Part 1 we saw how CLIP (Contrastive Language-Image Pre-training) embeds images and text in a shared space. LLaVA builds on that: it takes a frozen CLIP vision encoder, attaches a tiny projector to map image features into the LLM (large language model)’s token space, and then trains the whole system on multimodal instruction dialogues. The LLM does all cross-modal reasoning through its own self-attention. No cross-attention modules, no Q-formers. Just a small MLP and a pretrained transformer.
The surprising part is that the projector is often a single linear layer, or a two-layer MLP with a few million parameters. It learns to translate CLIP’s visual tokens into something the LLM can treat like text. The rest, answering questions, describing scenes, even reading text in images, emerges from the LLM’s existing language abilities. If you built a photo app that can search, caption, and edit your camera roll, this is the engine you would wire up under the hood.
How does LLaVA connect a vision encoder to an LLM?
LLaVA has three pieces: a frozen vision encoder, a small trainable projector, and a pretrained autoregressive LLM. The vision encoder is CLIP’s ViT-L/14 (or 336-resolution variant). The LLM is Vicuna, an instruction-tuned LLaMA derivative. The projector sits between them, mapping CLIP’s feature vectors into the LLM’s embedding dimension.
The vision encoder processes an image and outputs a sequence of patch embeddings. For a 336×336 input, you get 576 patch tokens plus a class token. The projector takes all these tokens, one by one, and projects them to the LLM’s hidden size. In LLaVA-1.5, the projector is a two-layer MLP with a GELU activation. After projection, you have a stack of visual embeddings that look exactly like word embeddings. The LLM can then interleave them with text tokens and run its usual forward pass.
This design is deliberate. The LLM is never told which tokens came from an image. It just sees a long sequence of embeddings. All cross-modal interaction is handled by the self-attention layers that already exist. This means you can swap in a different LLM or vision encoder without changing the fusion logic. For a photo app, you could plug in a better CLIP variant and retrain only the projector, or even fine-tune the whole stack with a small set of user-specific instructions.
How does the visual information flow into the LLM’s token stream?
The flow is a pipeline of four stages: encode, project, merge, generate. Let’s follow a typical request from the photo app: “Find all photos with a dog in the snow.”
First, the image is preprocessed to the expected resolution and fed into the frozen CLIP vision transformer. The output is a tensor of shape (N, C), where N is the number of tokens (for example, 576) and C is 1024. The projector then maps this to (N, d_model). For Vicuna-13B, d_model is 5120. This step is a simple matrix multiplication, or a small MLP, and it runs in a few milliseconds.
At the same time, the text prompt is tokenized. The prompt includes a special <image> marker that tells the system where to insert the visual tokens. For example: "USER: <image> Find all photos with a dog in the snow." The tokenizer splits this into a sequence of token IDs. The system then replaces every <image> placeholder with the whole block of visual embeddings. If the image contributes 576 tokens, the placeholder is expanded to 576 slots. The merged sequence now contains both visual and text embeddings, all in the same d_model-dimensional space.
This merged sequence is fed into the LLM. The transformer processes it layer by layer, applying self-attention across the entire sequence. The visual tokens act like a prefix: they are always present in the attention window while the model generates text. The LLM then autoregressively produces the answer, token by token, using the visual context. In our photo app, the answer might be a list of image IDs or a description.
The key detail is that the LLM’s token embedding matrix is never used for the visual tokens. The visual embeddings are injected directly into the embedding layer output. This means the LLM’s vocabulary is unchanged. The model learns to attend to the visual tokens just as it would attend to any other prefix.
How does LLaVA handle multiple images and video?
LLaVA-NeXT and OneVision extend the single-image pipeline without adding new architectures. Multiple images? Each image is encoded independently, projected, and then concatenated into one long visual token sequence. The prompt gets multiple <image> markers, each expanded to the right number of tokens. The LLM sees a sequence of visual tokens from all images, followed by the textual instruction. It can compare images, spot differences, or track objects across them.
Video is treated as a sequence of frames. Each frame is processed like a still image. The visual tokens from all frames are concatenated in temporal order. For a 10-second video at 1 fps, that’s 10 frames, each potentially 576 tokens, for a total of 5760 visual tokens. That’s a lot of context. So LLaVA-NeXT introduces AnyRes token allocation to keep the token count manageable.
AnyRes splits an image into a grid of crops, each processed separately. A threshold limits the total token count across all crops and frames. If the raw token count would exceed the threshold, the system bilinearly downsamples the embeddings per crop. This lets the model handle high-resolution images and long videos without blowing up the context window. For a photo search app, you could ingest a burst of 20 photos and describe them all without truncation.
The LLM’s attention mechanism discovers temporal structure on its own, because the visual tokens are in order. No special “video” adapter is needed. The same projector and LLM handle single images, multi-image comparisons, and video frames. That’s a huge win for maintainability: you have one model to deploy, and you feed it different token sequences.
How is LLaVA trained to follow multimodal instructions?
Training happens in stages, each fine-tuning only a subset of parameters. The first stage trains the projector alone, keeping both the vision encoder and the LLM frozen. The goal is to align visual features with the LLM’s embedding space. The original LLaVA used a filtered subset of CC3M (about 600K image-caption pairs). The projector learns to map an image’s CLIP features to the embedding of its caption. The loss is a simple L2 distance between the projected visual tokens and the caption’s language embedding. Or, alternatively, the projector is trained to make the LLM generate the caption, with the LLM’s weights frozen. Either way, the projector is the only part that learns.
After alignment, the projector is warm-started. The next stage is multimodal instruction tuning. Here, both the projector and the LLM are unfrozen, while the vision encoder usually stays frozen or is trained with a tiny learning rate. The training data is a mix of GPT-4-generated multimodal dialogues, academic VQA datasets, and OCR-rich corpora. The LLM sees prompt-image-response triples and learns to follow instructions. The loss is standard cross-entropy on the response tokens, conditioned on the image and the prompt. This stage is computationally heavy but still far cheaper than training from scratch.
LLaVA-1.5 and NeXT add an intermediate “high-quality knowledge” stage before the full instruction tuning. They train on millions of samples that include self-captioned images, OCR (optical character recognition) data, and plain language SFT (supervised fine-tuning). This extra step improves factual visual knowledge and text-reading ability. The vision encoder remains frozen, while the projector and LLM are updated.
For a photo app, you could fine-tune a LLaVA checkpoint on your own usage data: a few thousand examples of the kinds of instructions your users give (search, caption, edit). Because the projector is so small, you can even keep the LLM frozen and only train the projector and a few LoRA (low-rank adaptation) adapters. The model quickly adapts to your domain without forgetting general visual skills.
Why does the simple projector work so well?
The projector’s power comes from the quality of the pretrained backbones. CLIP already encodes rich semantic and spatial information. Vicuna already understands language and can follow complex instructions. The projector just needs to translate one representation into the other. A linear layer or a shallow MLP is enough because the two spaces are already well-structured.
This is the opposite of models like InstructBLIP or Qwen-VL. Those models introduce Q-formers or cross-attention modules that require billions of image-text pairs to train. LLaVA’s projector, trained on just a few hundred thousand pairs, matches or beats them on many benchmarks. The LLaVA-1.5 report shows that a simple MLP connector, trained on 558K pairs, outperforms the complex resampler used in InstructBLIP, which was trained on 129M pairs. The difference is that the LLM’s self-attention is doing the heavy lifting, not an external fusion module.
This also means the projector is a lever for efficiency. TokenPacker, for instance, replaces the MLP with a coarse-to-fine downsampling projector that cuts visual token count by 75-89% while keeping accuracy high. For a photo app on a mobile device, you could use a TokenPacker-style projector to reduce the number of tokens sent to the LLM, lowering latency and memory use. The architecture stays the same; only the projector changes.
Quick Reference
| Property | Value |
|---|---|
| Default vision encoder | CLIP ViT-L/14 (224×224) or ViT-L/14-336 (336×336) |
| Vision encoder status | Frozen during all training stages (or fine-tuned with very low LR) |
| Language model | Vicuna-7B/13B (LLaMA-based, instruction-tuned) |
| Projector | Linear layer (original) or 2-layer MLP with GELU (LLaVA-1.5+) |
| Projector parameters | ~2M-20M depending on dimensions |
| Visual tokens per image (336²) | 576 (patch tokens, no class token) |
| Projector output dimension | Matches LLM embedding size (e.g., 4096 for 7B, 5120 for 13B) |
| Training data for projector alignment | ~600K image-caption pairs (CC3M subset) |
| Instruction tuning data | GPT-4-generated multimodal dialogues + VQA/OCR benchmarks |
| Training stages | 1. Projector alignment (frozen LLM & vision); 2. (optional) High-quality knowledge; 3. Instruction tuning (LLM unfrozen) |
| Key design principle | No cross-attention; all fusion via LLM self-attention |
Test yourself
You’re building a photo app that uses LLaVA to answer natural language queries like “Show me all photos taken in Paris with the Eiffel Tower in the background.” At inference time, you have a gallery of 10,000 images. You need to produce a ranked list of matches. You don’t want to run the full autoregressive generation for every image. How can you leverage LLaVA’s architecture to efficiently filter the gallery before generating the final answer?
Answer: You can use the visual tokens as a pre-filtering step. Run each image through the frozen CLIP encoder and the projector to get the projected visual tokens. Then, take the user’s query and embed it into the same space by feeding it through the LLM’s embedding layer (or a separate text encoder). You now have a dense vector representation for each image and a vector for the query. Compute cosine similarity between the query embedding and the image embeddings, and keep the top-K images. This is fast because it avoids the LLM’s transformer layers. For the top-K candidates, you then construct the full prompt with the image tokens and the query, and run the LLM to generate the final answer. This two-stage approach gives you the power of full multimodal reasoning only on the most promising images, keeping latency manageable for large galleries. You can improve the query embedding by using a caption-based model or by fine-tuning the projector to also produce a pooled “CLS” token that is optimized for retrieval.
Frequently Asked Questions
Q: Can I use a different vision encoder, like SigLIP or DINOv2, instead of CLIP? Yes. The projector is the only component that depends on the vision encoder’s output dimension. You can swap in any encoder that produces a sequence of tokens, retrain the projector, and the rest of the pipeline stays unchanged. Many recent LLaVA variants already experiment with alternative encoders. Just be aware that the LLM’s performance will depend on the quality of the visual features, and you may need to adjust the projector’s capacity.
Q: What happens if my prompt does not include an <image> token?
The pipeline will not insert any visual tokens. The LLM sees only the text tokens and behaves like a pure text model. This is actually how LLaVA preserves its text-only capabilities. The same model can handle both text-only and multimodal requests without any mode switching.
Q: How many visual tokens are too many? Does the LLM’s context window limit the number of images? Yes. For a 4096-token context window, a single 336×336 image uses 576 tokens, leaving 3520 tokens for text and other images. With AnyRes, that number can multiply by the grid size. You must ensure the total token count (visual + text) fits within the LLM’s maximum context length. LLaVA-NeXT’s token budget management helps, but for very long videos or many high-res images, you may need to downsample or use a model with a larger context window.
Q: Why does the projector need to be trained at all? Couldn’t you just use CLIP’s embedding space directly? CLIP’s embedding space is a contrastive joint space, not the LLM’s token embedding space. The LLM’s token embeddings have a specific distribution and scale optimized for language modeling. The projector learns a linear (or nonlinear) transformation that maps CLIP features into that space, aligning them with the LLM’s internal representations. Without this training, the visual tokens would be out-of-distribution and the LLM would ignore them.
Q: Is it possible to fine-tune only the projector and keep the LLM frozen for a new visual task? Yes, and this is a common and efficient strategy. If you have a small set of domain-specific instruction examples, you can freeze the LLM and train only the projector (and possibly a few LoRA adapters). The model will learn to interpret the visual tokens in the context of your task without changing its language understanding. This is how you can quickly adapt a general vision-language model to a specialized photo-editing or search interface.
If you want this kind of breakdown every week, how real vision-language models actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Sources
- LLaVA: Visual Instruction Tuning (original paper)
- LLaVA-1.5: Improved Baselines with Visual Instruction Tuning
- LLaVA-OneVision: Easy Visual Task Transfer
- TokenPacker: Efficient Visual Projector for MLLM
- Hugging Face LLaVA documentation
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video LLMs
- LLaVA-NeXT GitHub repository
- CLIP: Learning Transferable Visual Models