Vision-language models

Transfusion · 2024-08Continuous-token autoregressionUnified understanding and generation

Runs next-token prediction for text and a diffusion objective for continuous image latents inside one transformer, with one set of weights and two losses. The cleanest demonstration that unifying understanding and generation does …

LLaVA · 2023-04Encoder plus projector

Connects a frozen CLIP vision encoder to a language model through a single linear projection, treating the projected patches as soft prompt tokens, and trains on generated instruction data. The cheapest possible bridge between the…

BLIP-2 · 2023-01Cross-attention into a frozen language model

Bridges a frozen image encoder to a frozen language model with a lightweight Q-Former whose learnable queries cross-attend to image features. The other major answer to the same question Flamingo asked, and arguably the more widely…

Flamingo · 2022-04Cross-attention into a frozen language model

Injects visual information into a frozen language model through gated cross-attention layers interleaved between its existing blocks, with a perceiver resampler compressing images to a fixed token budget first. The gates initializ…

CLIP · 2021-02Contrastive language-image pretraining

Trains an image encoder and a text encoder jointly with a contrastive objective on 400M (image, text) pairs. Predates this atlas's 2023-2026 window and is not a diffusion paper, but its text tower became the first widely used prom…

InternVL3 · 2025-04Native-resolution vision-language models

Breaks with the usual recipe of adapting a finished text language model, and instead pretrains on multimodal and text data jointly from the start, with variable visual position encoding.

MetaQuery · 2025-04Encoder plus projectorUnified understanding and generation

Bridges a frozen, understanding-only multimodal LLM to a diffusion image decoder with a small set of learnable query tokens, so the MLLM's weights (and its reasoning/understanding ability) never have to change to gain generation. …

Qwen2.5-VL · 2025-02Native-resolution vision-language models

Adds windowed attention to the native-resolution vision transformer and absolute time encoding for video. This is the model Qwen-Image uses to encode prompts and Step1X-Edit uses to parse editing instructions, so it sits underneat…

SigLIP 2 · 2025-02Contrastive language-image pretraining

Extends SigLIP's sigmoid contrastive language-image objective with captioning, self-distillation and masked-prediction losses in one recipe, improving dense/localization features specifically; the frozen encoder Scale-RAE uses for…

Janus · 2024-10Unified understanding and generation

Diagnoses the conflict in single-encoder unified models, where understanding wants semantic abstraction and generation wants reconstructable detail, and resolves it with two separate visual encoders feeding one shared language bac…

NVLM · 2024-09Cross-attention into a frozen language model

Compares decoder-only and cross-attention multimodal designs head to head and ships a hybrid. Useful to this atlas as evidence for when the cross-attention pattern stopped being the default rather than as a claim that it failed.

Qwen2-VL · 2024-09Native-resolution vision-language models

Processes images at their native resolution and aspect ratio rather than resizing to a fixed square, and introduces a rotary position encoding decomposed across time, height and width. Directly load-bearing for this atlas: its suc…

LLaVA-OneVision · 2024-08Encoder plus projector

Unifies single-image, multi-image and video handling under one projector with resolution-adaptive tiling, showing the encoder-plus-projector pattern generalizes without architectural change.

Otter · 2023-05Cross-attention into a frozen language model

Builds directly on Flamingo's perceiver resampler and gated cross-attention design and adds in-context instruction tuning. Evidence the mechanism had genuine uptake beyond its originating lab.

Qwen3-VL · 2025-11Native-resolution vision-language models

Interleaved multimodal rotary embeddings, multi-level vision feature fusion, and text-timestamp alignment. The current generation of the family most open generators depend on.