core

Inductive Moment Matching

Linqi Zhou, Stefano Ermon, Jiaming Song · Stanford University, Luma AI · 2025-03 · arXiv:2503.07565

Why it matters

A single-stage few-step objective that matches distributions between marginals of a stochastic interpolant, giving distribution-level convergence guarantees that consistency models lack, with no pretraining or distillation. Reaches 1.99 FID on ImageNet 256 in 8 steps.

What this paper does

read: full text

Before this

Diffusion models and flow matching produce high-quality samples only after many sequential denoising steps. Existing ways to cut that down, either distilling a pretrained diffusion model or training Consistency Models, are described by the authors as prone to training collapse without careful tuning and regularization, such as pre-generated data-noise pairs and early stopping, and Consistency Models typically need a separately maintained teacher or EMA target network alongside the trained student.

The problem

The paper frames few-step generative modeling as a trilemma among high-fidelity outputs, efficient inference and stable training, and sets out to train stable few-step models from scratch, without distilling from a pretrained teacher and without the tuning burden Consistency Models carry.

The idea

Learn a mapping between the marginal distribution of a stochastic interpolant at any time t and its marginal at any earlier time s, and train it by matching distributions, via maximum mean discrepancy across many sampled particles at s, rather than enforcing the pointwise self-consistency constraint Consistency Models use. Because the target distribution at each step is built inductively from the model's own previous-iteration output rather than from a frozen teacher, single-stage training with one network suffices.

How it works

IMM trains a randomly initialized network with no pretrained initialization and no separate teacher or EMA target network, which the paper places in a few-step-from-scratch category distinct from distillation methods. For ImageNet 256, it uses the DiT-XL/2 architecture at 675M parameters. The objective is not a per-sample regression onto noise, velocity or clean data; it minimizes an MMD-based moment-matching loss between two independently sampled mappings from a shared start time t to a shared target time s, so training pushes the model's output distribution at s toward the true marginal distribution at s in kernel-embedding moments rather than toward one target value per sample. The paper proves this loss reduces to the Consistency Model loss for a particular kernel and step schedule, making Consistency Models a special case of the IMM family. At sampling time, classifier-free guidance is applied with weight w; conditioning is randomly dropped with some probability during training, and guidance is applied only at inference.

Evidence

On ImageNet 256x256 with classifier-free guidance at w=1.5, IMM reaches 1.99 FID at 8 inference steps, 2.51 FID at 4 steps, 3.99 FID at 2 steps and 8.05 FID at 1 step. On CIFAR-10 trained from scratch, it reaches a 2-step FID of 1.98 and a 1-step FID of 3.20, which the paper reports as state of the art for 2-step generation among models trained from scratch, ahead of iCT's 2.46 and sCT's 2.06 at 2 steps and matching or exceeding CTM with a GAN loss at 1.87.

Limitations

The authors concede that a pushforward objective combined with restart samplers can achieve superior results, and they leave that combination to future work rather than incorporating it. They also report that training with very small particle counts, for example two particles, causes training instability and performance degradation, while larger particle counts mean fewer diffusion times are sampled per step, making particle count a tuning trade-off rather than a free win. The accessible text does not state whether the ImageNet-256 experiments model images in raw pixel space or in a pretrained autoencoder's latent space, and it does not disclose the training compute budget.

Why it matters

By replacing pointwise self-consistency with distribution-level moment matching, IMM gives few-step generative training a provable convergence guarantee, the model distribution converges to the true marginal at every time pair, which Consistency Models lack. It also removes the two-network distillation setup that both diffusion distillation and Consistency Models rely on.

Abstract, in the authors' own words

Diffusion models and Flow Matching generate high-quality samples but are slow at inference, and distilling them into few-step models often leads to instability and extensive tuning. To resolve these trade-offs, we propose Inductive Moment Matching (IMM), a new class of generative models for one- or few-step sampling with a single-stage training procedure. Unlike distillation, IMM does not require pre-training initialization and optimization of two networks; and unlike Consistency Models, IMM guarantees distribution-level convergence and remains stable under various hyperparameters and standard model architectures. IMM surpasses diffusion models on ImageNet-256x256 with 1.99 FID using only 8 inference steps and achieves state-of-the-art 2-step FID of 1.98 on CIFAR-10 for a model trained from scratch.

Research line

Natively few-step objectivesemerging

Reported results

Benchmark	Value	Guidance	Budget	Source
ImageNet 256x256 gFID (NFE=8)	1.99	not stated	not stated	abstract

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.