core
Generative Multimodal Models are In-Context Learners
Quan Sun, Yufeng Cui, Xiaosong Zhang · BAAI, Tsinghua University, Peking University · 2023-12 · arXiv:2312.13286 · code
Why it matters
A 37B multimodal model that autoregressively predicts the next CLIP visual embedding alongside text tokens, then decodes embeddings back to pixels with a diffusion decoder — an autoregressive, MLLM-scale continuation of unCLIP's semantic-latent idea, and a direct ancestor of MetaQuery/BLIP3-o's frozen-MLLM-plus-diffusion-decoder pattern.
What this paper does
read: full textBefore this
Prior multimodal systems such as CLIP, Flamingo, the Kosmos series, BLIP and LLaVA, plus the smaller 14B Emu1 predecessor, used task-specific architectures or discrete visual tokens, and their few-shot in-context learning across interleaved text-and-image tasks lagged far behind what large language models achieve with text alone.
The problem
Multimodal systems had largely failed to imitate the human ability to solve a task from only a handful of demonstrations or a simple instruction, the in-context learning behavior that scaling had already unlocked for text-only language models.
The idea
Unify text and image prediction under one autoregressive objective that predicts the next multimodal element, either a visual embedding or a text token, over interleaved sequences, and scale the whole system to language-model size to see whether in-context learning emerges the same way it does for text.
How it works
A frozen-during-stage-2 EVA-02-CLIP-E-plus vision encoder turns each image into 64 fixed-dimension continuous visual embeddings, via mean pooling over 8 by 8 patches followed by a linear projection, and these embeddings are interleaved with text tokens as input to a LLaMA-33B autoregressive backbone, for a total of 37 billion parameters across encoder, backbone and decoder. Stage one trains with a captioning loss on text tokens only; stage two adds an image regression loss that predicts continuous visual embeddings directly, with no vector-quantization step, alongside the text classification loss. A separately and fully trained SDXL-base model serves as the visual decoder, converting predicted embeddings back into 1024 by 1024 pixel images, trained with classifier-free guidance using 10 percent conditioning dropout. Instruction-tuned variants follow the base model, Emu2-Chat for conversational question answering and Emu2-Gen for controllable generation from mixed text, location and image conditions.
Evidence
In few-shot in-context evaluation, Emu2 reaches 67.8 percent (8-shot) and 68.8 percent (16-shot) on VQAv2 against Flamingo-80B's 65.6 and 66.8, and 49.3 and 50.3 percent on TextVQA against Flamingo-80B's 37.3 and 37.6. Instruction-tuned Emu2-Chat reaches 84.9 percent VQAv2, 65.1 percent GQA, 66.6 percent TextVQA and an MM-Vet score of 48.5. For generation, on 30,000 MS-COCO samples Emu2 reaches CLIP-I 0.686, ahead of SDXL's 0.674 and SEED's 0.682, though CLIP-T trails DALL-E 3, 0.297 versus 0.320. On DreamBench subject-driven generation Emu2 reaches a DINO score of 0.766 and CLIP-I of 0.850, ahead of Kosmos-G's 0.694 DINO score.
Limitations
The authors concede in-context learning can still fail on complex scenes, giving counting objects in a crowded image as an example, and they concede a remaining gap between Emu2-Chat's question-answering ability and closed multimodal systems, citing GPT-4V's MM-Vet score of 67.7 against Emu2-Chat's 48.5.
Why it matters
Emu2 is an early, MLLM-scale demonstration that predicting continuous semantic visual embeddings autoregressively, then handing them to a separately trained diffusion model to decode into pixels, produces strong in-context learning at 37B parameters. It is a direct ancestor of the frozen-backbone, diffusion-decoder pattern that later work such as MetaQuery and BLIP3-o refines by freezing the multimodal backbone entirely and adding only a lightweight bridge, rather than training the whole multimodal model end to end as Emu2 does.
Abstract, in the authors' own words
The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.
Research lines
Design-axis choices
Representation
Method note — the shared flow-matching interpolation
Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:
Builds on
- builds_on Hierarchical Text-Conditional Image Generation with CLIP Latents — Emu2 extends unCLIP's CLIP-embedding generation into an autoregressive multimodal LLM that predicts the next visual embedding alongside text.
Built on by
- BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset builds_on this — BLIP3-o's diffusion-over-CLIP-features approach continues the semantic-latent generation lineage Emu2 established with autoregressive CLIP-feature regression.