landmark

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi · Salesforce Research · 2023-01 · arXiv:2301.12597

Why it matters

Bridges a frozen image encoder to a frozen language model with a lightweight Q-Former whose learnable queries cross-attend to image features. The other major answer to the same question Flamingo asked, and arguably the more widely copied one.

What this paper does

read: full text

Before this

Vision-language pre-training before BLIP-2 was dominated by end-to-end training of large multimodal models, either training the whole stack from scratch or, as in Flamingo, freezing the language model but inserting new gated cross-attention layers directly into it and training those plus a Perceiver resampler on hundreds of millions of image-text pairs. Both routes required large compute budgets and large trainable parameter counts, and unfreezing or heavily modifying the LLM risked catastrophic forgetting of its language ability.

The problem

How to connect an already-strong frozen image encoder to an already-strong frozen language model, getting vision-language competence without paying to train either backbone and without eroding the LLM's original capabilities.

The idea

Insert a small trainable Querying Transformer (Q-Former) as an information bottleneck between a frozen image encoder and a frozen LLM. A fixed set of learnable query embeddings cross-attend to the frozen image features, are first pre-trained to extract text-relevant visual representations, and are then projected and prepended as soft prompts to the frozen LLM's input.

How it works

The Q-Former has 32 learnable queries of dimension 768, cross-attending to frozen image encoder features every other transformer block; it is initialized from BERT-base weights (cross-attention layers randomly initialized), giving it 188M parameters. Training has two frozen-backbone stages. Stage 1 (representation learning) jointly optimizes image-text contrastive learning, image-grounded text generation, and image-text matching, using different attention masks so queries and text interact in task-specific ways. Stage 2 (generative learning) attaches the Q-Former plus frozen image encoder to a frozen LLM: the 32 output query embeddings are linearly projected into the LLM's embedding dimension and prepended to the input text embeddings as soft visual prompts, trained with a language-modeling (or prefix-LM) loss. Only the Q-Former and this projection layer are trained; the image encoder and LLM are frozen throughout.

Evidence

Zero-shot VQAv2: BLIP-2 (ViT-g + FlanT5-XXL) reaches 65.0% accuracy versus Flamingo80B's 56.3%, an 8.7-point gain with 54x fewer trainable parameters (188M vs Flamingo's 10.2B). On NoCaps captioning (validation), BLIP-2 (ViT-g + OPT-2.7B) gets CIDEr 123.0 / SPICE 15.8, above prior BLIP's 113.2 / 14.8. Zero-shot Flickr30K retrieval: image-to-text R@1 97.6% and text-to-image R@1 89.7%, both above BLIP's 96.7% / 86.7%.

Limitations

The paper's own limitations section states that in-context few-shot VQA examples did not improve performance, which it attributes to the pretraining data containing only single image-text pairs per sample rather than multi-pair sequences. It concedes BLIP-2 inherits LLM risks such as offensive language, social bias, and leaked private information. It also reports generation errors from inaccurate LLM knowledge, incorrect reasoning paths, or lack of up-to-date information about image content.

Why it matters

It showed that near-Flamingo-level vision-language performance is reachable with orders of magnitude fewer trainable parameters by never touching the LLM's own weights, only feeding it a short, learned visual prefix. That prefix-injection pattern (compress image into a handful of tokens, prepend, keep the LLM frozen or later fine-tune it cheaply) became one of the two dominant templates for connecting vision to LLMs, alongside Flamingo's in-LLM cross-attention.

Abstract, in the authors' own words

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

Research line

Cross-attention into a frozen language modelcontested

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.