Research line · contested
Cross-attention into a frozen language model
Inject visual information through dedicated gated cross-attention layers interleaved into a frozen language model.
What defines membership
Vision should enter through its own pathway rather than competing for the token sequence, so the language model's own capabilities are provably untouched.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
The idea
Flamingo · 2022-04landmark
Gated cross-attention layers interleaved into a frozen language model, with a perceiver resampler fixing the visual token budget.
BLIP-2 · 2023-01landmark
The other answer to the same question: a lightweight Q-Former whose learnable queries cross-attend to frozen image features.
Evidence
NVLM · 2024-09core
Compares cross-attention against decoder-only head to head, marking roughly where the pattern stopped being the default.
Improvement
Otter · 2023-05strong-followup
Builds on Flamingo's design with in-context instruction tuning.
What it gets right
- Visual tokens do not consume the language model's context window
- The frozen backbone is mathematically guaranteed intact at initialization
- Handles interleaved image-text sequences naturally
Where it is weak
- Adds parameters and complexity the projector approach avoids
- Largely displaced by encoder-plus-projector in open work, which is simpler and performs comparably
- The resampler's fixed token budget is a bottleneck on dense detail