landmark

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang · ByteDance, Peking University · 2024-04 · arXiv:2404.02905 · code

Why it matters

Redefines autoregressive image generation as coarse-to-fine next-*scale* prediction instead of raster-order next-token prediction; for the first time an AR image model beats diffusion transformers on ImageNet FID while also showing GPT-style scaling laws. NeurIPS 2024 best paper.

What this paper does

read: full text

Before this

Autoregressive image generators flattened a VQ tokenizer's 2D grid of discrete tokens into a 1D sequence with a raster scan, spiral, or z-curve order, then predicted tokens left to right with causal attention as language models do. Generating an n by n grid this way costs O(n^2) autoregressive steps and O(n^6) total computation, and diffusion transformers such as DiT-XL/2 had overtaken these raster-order AR models on ImageNet, reaching FID 2.27 where prior AR baselines scored around 18.65.

The problem

Raster-order next-token prediction is mismatched to images. A VQVAE's token grid carries bidirectional spatial correlations that a strictly left-to-right causal order cannot respect, and flattening destroys the 2D locality between neighboring tokens, both of which the authors argue hold autoregressive image generation back from matching diffusion transformers.

The idea

Redefine autoregression over images as coarse-to-fine next-scale prediction instead of next-token prediction. Generate an entire low-resolution token map first, then predict each next, higher-resolution token map conditioned on every coarser map already generated, mirroring how a human first perceives global structure before local detail.

How it works

Space is discrete tokens. A multi-scale VQVAE encodes an image into K token maps r_1 through r_K at increasing resolutions, sharing one codebook of size 4096, using a residual-style design in which each map is refined by extra convolution layers to recover information lost during upscaling. A GPT-2-style decoder-only transformer with adaptive layer norm predicts, at each of the K autoregressive steps, every token in that entire scale's map in parallel, conditioned on all previously generated coarser scales via a block-wise causal attention mask that lets a scale attend only to itself and everything coarser than it. The objective is standard cross-entropy over the shared codebook's 4096-way vocabulary at every token, factorizing the joint distribution over all K maps as a chain rule product of per-scale conditionals; nothing about the objective is diffusion-based. At inference, key-value caching removes the need for the attention mask, and generation proceeds scale by scale.

Evidence

On ImageNet 256x256, VAR-d30 (2.0B params) reaches FID 1.92 and Inception Score 323.1, and VAR-d30-re, the same model with classifier-free guidance at scale 2.0 plus a rejection-sampling step added at inference, reaches FID 1.73 and IS 350.2, against DiT-XL/2's FID 2.27 (675M params) and L-DiT-3B's FID 2.10 (3.0B params). VAR models are trained for 200 to 350 epochs depending on model size, and the paper contrasts this against DiT-XL/2's far longer 1400-epoch training budget. VAR-d30 generates an image roughly 45 times faster than DiT-XL/2 by the paper's relative-speed comparison. Across 12 model sizes from 18.5M to 2.0B parameters, test loss and token error rate both follow power laws in parameter count and in training compute, with Pearson correlation coefficients near -0.998 across six orders of magnitude of compute, mirroring the scaling laws reported for language models.

Limitations

The authors concede text-to-image generation is not covered and is left as an ongoing, high-priority direction for future work. They state the multi-scale VQVAE tokenizer's own architecture and training are kept unchanged from the baseline VQGAN design, and describe improving the tokenizer itself as an orthogonal direction they did not pursue. Video generation is also explicitly not implemented, though the authors suggest a 3D extension of next-scale prediction as future work. The zero-shot inpainting and outpainting results are shown only qualitatively, with no quantitative metrics reported for them.

Why it matters

Reformulating autoregression as next-scale rather than next-token prediction let a GPT-style image model surpass diffusion transformers on ImageNet FID for the first time while training on fewer epochs and running an order of magnitude faster at inference. The clean power-law scaling and zero-shot task transfer argue that image autoregression, not only language modeling, can inherit the scaling behavior that made large language models predictable to train.

Abstract, in the authors' own words

We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes GPT-like AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.

Research line

Discrete-token autoregressioncontested

Reported results

Benchmark	Value	Guidance	Budget	Source
ImageNet 256x256 gFID	1.73	guidance ratio 2.0 + rejection sampling	350 epochs	Table 1 (v2)

arXiv v1 reported 1.80 for this configuration and the current version reports 1.73; cite which version you mean. Uses rejection sampling, which the diffusion baselines do not.

Design-axis choices

Representation

Multi-scale (next-scale) tokens

Objective

Next-scale autoregression

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.

Challenges / competes with

competes_with Scaling Rectified Flow Transformers for High-Resolution Image Synthesis — VAR proposes next-scale autoregression as an alternative generative paradigm to flow-matching diffusion transformers, matching or beating their ImageNet FID.
challenges Autoregressive Image Generation without Vector Quantization — MAR shows continuous per-token diffusion loss removes the vector-quantization step VAR still depends on for its discrete multi-scale tokens.