LLaVA

Liu 2023 · Paper

One-line summary — LLaVA (Large Language and Vision Assistant) connects a frozen CLIP vision encoder to the Vicuna LLM through a single linear projection and instruction-tunes on GPT-4-generated visual instruction data, showing that data quality — not architectural complexity — is the key to a strong open-source conversational VLM.

Problem

Instruction tuning LLMs on machine-generated instruction-following data had dramatically improved zero-shot capabilities on new language tasks, but the idea was largely unexplored in the multimodal field. VLMs of the time either demanded massive compute (Flamingo), used intricate bridging architectures (BLIP-2’s Q-Former, Flamingo’s gated cross-attention), or were closed (GPT-4V) — and manually labeling multimodal conversation data at scale is expensive and ill-defined. LLaVA asks whether a minimal architecture plus machine-generated multimodal instruction data can yield a competitive, reproducible open VLM.

Method & architecture

• GPT-assisted data generation: language-only GPT-4 is prompted with two symbolic representations of a COCO image — its captions and object bounding-box coordinates — and asked to produce three response types: multi-turn conversation (58K), detailed description (23K), and complex reasoning (77K), totaling 158K language-image instruction-following samples. A few manually designed seed examples for in-context learning are the only human annotations.
• Architecture: input image $\mathbf{X}_v$ passes through the pre-trained CLIP ViT-L/14 encoder $g(\cdot)$ (grid features before the last Transformer layer), and one trainable projection matrix $\mathbf{W}$ maps the features into the word-embedding space of the Vicuna LLM $f_{\boldsymbol{\phi}}$:

$$\mathbf{H}_v = \mathbf{W} \cdot \mathbf{Z}_v, \quad \text{with } \mathbf{Z}_v = g(\mathbf{X}_v)$$

The visual tokens $\mathbf{H}_v$ are simply placed in the LLM’s context alongside text — no Q-Former, no cross-attention module.

• Training objective: multi-turn data $(\mathbf{X}_q^1, \mathbf{X}_a^1, \dots, \mathbf{X}_q^T, \mathbf{X}_a^T)$ is serialized into a single sequence and the model is trained auto-regressively on the assistant tokens only:

$$p(\mathbf{X}_a \mid \mathbf{X}_v, \mathbf{X}_{\text{instruct}}) = \prod_{i=1}^{L} p_{\boldsymbol{\theta}}\big(x_i \mid \mathbf{X}_v, \mathbf{X}_{\text{instruct},<i}, \mathbf{X}_{a,<i}\big)$$

where $\mathbf{X}_{\text{instruct},<i}$ and $\mathbf{X}_{a,<i}$ are instruction and answer tokens before the current prediction token $x_i$.

• Two-stage training: Stage 1 (feature alignment) — CC3M filtered to 595K image-text pairs, treated as naive single-turn instructions; both backbones frozen, only $\boldsymbol{\theta} = \mathbf{W}$ trained (1 epoch, lr 2e-3, batch 128) — effectively learning a visual tokenizer for the frozen LLM. Stage 2 (end-to-end fine-tuning) — vision encoder stays frozen; $\boldsymbol{\theta} = \{\mathbf{W}, \boldsymbol{\phi}\}$ trained on the 158K instruction set (3 epochs, lr 2e-5, batch 32) on 8×A100.
• Evaluation protocol: GPT-4 judges candidate answers against reference answers produced by text-only GPT-4 given ground-truth descriptions, yielding a relative score — the basis of the LLaVA-Bench (COCO) and LLaVA-Bench (In-the-Wild) benchmarks introduced by the paper.

Results

• LLaVA-Bench (COCO) (90 questions): full-data LLaVA scores 85.1% relative to text-only GPT-4; removing instruction tuning collapses the score to 21.5 (−63.6), and conversation-only training gives 73.8 — data mix matters.
• LLaVA-Bench (In-the-Wild) (24 images, 60 questions): LLaVA reaches 67.3 ± 2.0 overall vs BLIP-2’s 38.1 ± 1.0 (+29%) and OpenFlamingo’s 19.1 ± 0.4 (+48%), with 81.7 on complex reasoning.
• ScienceQA: LLaVA alone achieves 90.92% accuracy (close to the then-SoTA MM-CoT-Large at 91.68%); using GPT-4 as a judge to arbitrate disagreements yields a new SoTA of 92.53%. Two-shot text-only GPT-4 alone gets 82.69%.
• Ablations: training from scratch without Stage 1 drops accuracy by 5.11 points to 85.81%; a 7B model scores 89.84% vs 90.92% for 13B; last-layer CLIP features score 0.96 lower than penultimate-layer features.
• Qualitatively, LLaVA follows instructions on out-of-domain images (e.g., the “Extreme Ironing” GPT-4 demo) where BLIP-2 and OpenFlamingo merely describe the scene.

Why it matters for SLAM

LLaVA democratized conversational vision-language models: the “encoder + projection + LLM” recipe underlies most robotics VLMs and vision-language-action systems that followed (including navigation VLAs such as NaVILA, whose VILA backbone follows the same pattern, and OpenVLA’s Prismatic backbone, trained on the LLaVA-1.5 data mixture). For SLAM, a LLaVA-style VLM attached to rendered views of a map provides scene understanding, object identification, and spatial question answering — one of the most direct integration paths from a geometric SLAM map to open-vocabulary semantic reasoning.

Related

• CLIP
• BLIP-2
• SigLIP
• VLM vs VLA
• OpenVLA