Research line · dominant
Encoder plus projector
Bolt a frozen vision encoder onto a language model with a small trained projection, and treat the projected patches as prompt tokens.
What defines membership
The language model's embedding space is a good enough interface that a linear map into it is sufficient; nothing about the language model needs to change.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
The idea
LLaVA · 2023-04landmark
A single linear projection from frozen CLIP into a language model, trained on generated instruction data.
At scale
LLaVA-OneVision · 2024-08core
Extends the same interface to multi-image and video with resolution-adaptive tiling, without architectural change.
Combination
MetaQuery · 2025-04core
Runs the pattern in reverse for generation: learnable queries read a frozen multimodal model and drive a diffusion decoder.
What it gets right
- Cheapest possible bridge between vision and language, and easy to train
- The frozen language model keeps its abilities intact
- The same interface can be reversed to condition a generator, which is how frozen-model generation works
Where it is weak
- A linear map is a narrow channel for everything an image contains
- Visual tokens consume context length the language model needs elsewhere
- Fine spatial grounding is limited by what the frozen encoder chose to keep