BLIP-2

Li (Salesforce) 2023 · Paper

One-line summary — BLIP-2 bridges a frozen pre-trained image encoder and a frozen large language model with a lightweight Querying Transformer (Q-Former), achieving strong vision-language capabilities at a fraction of the training cost of end-to-end multimodal models.

Problem

The cost of vision-and-language pre-training had become prohibitive because it meant end-to-end training of large-scale models — every new VLM re-trained billions of visual and language parameters. Yet powerful ingredients already existed off the shelf: large pre-trained image encoders and LLMs, each excellent in its own modality. The open question was whether a small trainable module could bridge the modality gap — feeding visual information into an LLM that has never seen an image — without unfreezing either giant, and without the catastrophic forgetting that unfreezing causes.

Method & architecture

Q-Former. The only trainable module (188M parameters, initialized from BERT-base) sits between the frozen image encoder and the frozen LLM. It has two transformer submodules sharing the same self-attention layers: an image transformer whose input is a fixed set of 32 learnable query embeddings (dim 768) that cross-attend to the frozen image encoder’s output features (cross-attention inserted every other block), and a text transformer that acts as text encoder or decoder. Whatever the input resolution, the queries compress the image to 32 output vectors $Z$ — an information bottleneck forced to extract the visual content most relevant to language.

Stage 1 — vision-language representation learning (frozen image encoder + Q-Former, on image-text pairs), jointly optimizing three objectives that differ only in their attention masks:

• Image-Text Contrastive (ITC): align $Z$ with the [CLS] text embedding $t$; the image-text similarity is the max over queries, $s(I,T) = \max_k \langle z_k, t \rangle$, contrasted against in-batch negatives with a unimodal mask (queries and text cannot see each other).
• Image-grounded Text Generation (ITG): generate the caption with a multimodal causal mask — text tokens attend to all queries, so all caption-relevant information must flow through the queries.
• Image-Text Matching (ITM): binary matched/unmatched classification with a bidirectional mask and hard negative mining.

Stage 2 — generative learning from a frozen LLM. A single fully-connected layer projects $Z$ to the LLM’s embedding dimension; the projected queries are prepended to the text as soft visual prompts. Decoder LLMs (OPT 2.7B/6.7B) are trained with a language-modeling loss; encoder-decoder LLMs (FlanT5-XL/XXL) with a prefix-LM loss. Because stage 1 already made $Z$ language-informative, the LLM’s burden of learning alignment is small, mitigating catastrophic forgetting.

Training. Frozen encoders: CLIP ViT-L/14 or EVA-CLIP ViT-g/14 (second-to-last layer features). Data: 129M images (COCO, Visual Genome, CC3M, CC12M, SBU, 115M from LAION-400M) with CapFilt synthetic captions. 250k steps (stage 1) + 80k steps (stage 2); the largest model (ViT-g + FlanT5-XXL) pre-trains in under 6 + 3 days on a single 16×A100(40G) machine.

Results

Zero-shot overview (BLIP-2 has 188M trainable parameters):

• Headline: BLIP-2 (ViT-g, FlanT5-XXL) outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54× fewer trainable parameters (65.0 vs 56.3 test-dev); it also leads GQA (44.7) and comes second only to Flamingo80B on knowledge-heavy OK-VQA (45.9 vs 50.6).
• Scaling ablation: a stronger image encoder (ViT-g > ViT-L) or a stronger/instruction-tuned LLM (FlanT5 > OPT, larger > smaller) both improve performance — validating BLIP-2 as a generic recipe that harvests progress in both communities.
• Removing stage-1 representation learning collapses zero-shot VQA for both LLM types, with OPT showing catastrophic forgetting — evidence the Q-Former bottleneck, not the LLM, does the alignment.
• With an instruction-tuned FlanT5, BLIP-2 exhibits emerging instructed zero-shot image-to-text generation: visual conversation, visual knowledge/commonsense reasoning, storytelling.

Why it matters for SLAM

BLIP-2’s frozen-encoder + lightweight-adapter + frozen-LLM pattern became the standard recipe for building vision-language systems cheaply (InstructBLIP, MiniGPT-4, many robotics VLMs), and it defines the practical path for adding language-based reasoning to spatial systems: rather than retraining a giant model, a small bridge can connect a SLAM system’s visual (or rendered-map) features to an off-the-shelf LLM. In the VLM-to-VLA lineage that runs through this level, BLIP-2 is the efficiency milestone between CLIP’s contrastive embeddings and LLaVA/OpenVLA’s instruction-following stacks.

Related

• CLIP
• LLaVA
• SigLIP
• VLM vs VLA
• Foundation models