Generative Vision Atlas

core

Emu3: Next-Token Prediction is All You Need

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo · BAAI · 2024-09 · arXiv:2409.18869 · code

Why it matters

Tokenizes text, images, and video into one discrete vocabulary and trains a single decoder-only transformer purely on next-token prediction — no diffusion, no compositional modules — arguing this single objective is enough to match or beat specialized diffusion (SDXL) and vision-language (LLaVA-1.6) systems on their own tasks.

What this paper does

read: full text

Before this

Vision generation was dominated by diffusion models such as Stable Diffusion, and vision-language understanding was dominated by compositional systems that bolt a CLIP-style encoder onto an LLM, such as LLaVA. Earlier attempts at unification, including Emu and Chameleon, either still connected an LLM to a separate diffusion model or fell short of task-specific systems on their own benchmarks.

The problem

Whether a single next-token-prediction objective, with no diffusion component and no compositional encoder-plus-LLM assembly, can match specialized systems at their own tasks.

The idea

Tokenize text, images, and video into one discrete vocabulary and train a single decoder-only transformer from scratch purely by next-token prediction over that vocabulary.

How it works

A single SBER-MoVQGAN-based visual tokenizer, codebook size 32,768, compressing 4 by 8 by 8 in time and space, converts both images and video into discrete tokens. This is the only visual encoder in the system, used both for what the model reads and what it generates, so there is no separate continuous encoder for understanding. One 8B-parameter, 32-layer transformer processes a single combined vocabulary of 184,622 text and vision tokens, trained with one cross-entropy loss over the whole sequence, with vision-token loss weighted 0.5 relative to text. There is no diffusion head, no adapter module, and no mixture of experts; understanding and generation are the same forward pass through the same weights, differing only in which side of the sequence is being predicted. Training runs two pretraining stages (text and image only, then adding video with context extended to 131,072 tokens) followed by quality fine-tuning, instruction tuning, and DPO.

Evidence

On GenEval, Emu3 scores 0.66 overall against SDXL's 0.55; on T2I-CompBench color, 0.7913 against 0.6369; on DPG-Bench, 80.60 against 74.65. Against LLaVA-1.6, Emu3 leads on SEEDBench (68.2 vs 64.7) and OCRBench (687 vs 532) but trails on MMBench (58.5 vs 67.4) and MMVet (37.2 vs 43.9). On video, VBench total score is 80.96, ahead of most open video diffusion models but behind proprietary Kling (81.85) and Gen-3 (82.32).

Limitations

The paper concedes a mixed picture against LLaVA-1.6 rather than a clean win, trailing on MMBench and MMVet even as it leads on OCR and perception benchmarks. It also concedes that applying DPO caused "a slight decline in the evaluation results," attributed to a mismatch between the DPO preference data's emphasis on aesthetics and the focus of the automated evaluation models. Video generation is conceded to fall short of the strongest proprietary systems.

Why it matters

Emu3 is the field's cleanest existence proof that one vocabulary, one tokenizer, one backbone, and one training objective can reach or beat specialized diffusion and compositional-VLM baselines on their own turf, with no diffusion component anywhere in the system. It sets up the contrast with later unified models such as Janus that keep next-token prediction but split the visual encoder in two.

Abstract, in the authors' own words

While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

Research line

Unified understanding and generationascendant

Design-axis choices

Conditioning & control

Native autoregressive token conditioning

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

xt=(1−t) x0+t x1,vθ(xt,t)≈x1−x0x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.

Challenges / competes with