landmark
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Muller, Harry Saini · Stability AI · 2024-03 · arXiv:2403.03206
Why it matters
Takes rectified flow to production scale with a new dual-stream 'MM-DiT' architecture (separate weights for text/image tokens, joined by shared attention) plus a logit-normal timestep-sampling schedule; establishes the flow-matching + MM-DiT recipe that FLUX, Qwen-Image, Seedream and most 2025 open T2I systems are variations of.
What this paper does
read: full textBefore this
Rectified flow, which connects data and noise along a straight line, had better theoretical properties than standard diffusion and should in principle allow low-error few-step sampling, but it had only been validated in small and medium-scale, mostly class-conditional experiments and had not displaced diffusion as standard practice for text-to-image generation.
The problem
Uniform timestep sampling during rectified-flow training wastes capacity, since the velocity-prediction target is hardest to learn for timesteps in the middle of the interval and easiest near the boundaries, so treating all timesteps equally does not use training compute well. No architecture had been shown to let a text-to-image rectified-flow model scale predictably to production size either.
The idea
Reweight rectified-flow training toward the timesteps that matter most using a logit-normal timestep distribution, and replace single-stream conditioning with an MM-DiT architecture that gives text and image tokens separate weights joined by shared joint attention, so information flows bidirectionally between the two modalities instead of only from text into image.
How it works
Training samples timesteps t from a logit-normal distribution with location m and scale s rather than uniformly. Empirically m=0, s=1 outperformed uniform rectified-flow sampling and matched established diffusion schedules such as LDM-Linear. MM-DiT gives image and text tokens separate weight sets, each with hidden dimension scaled by model depth, joined at every block by one joint attention operation so both streams can attend to each other. Text conditioning combines three frozen encoders, CLIP-L/14, CLIP-G/14 and T5-XXL, each independently dropped out 46.3% of the time during training so the model tolerates missing encoders at inference. QK-normalization, RMSNorm with a learnable scale applied to queries and keys before attention, was added to stabilize high-resolution fine-tuning from 256x256 up to 1024px, since attention entropy would otherwise collapse under bf16 mixed precision. Models from 264M (depth 15) to 8B parameters (depth 38) were trained at 256x256 for 500k steps at batch size 4096 to study scaling.
Evidence
Validation loss decreases smoothly with model size from 264M to 8B parameters with no observed saturation, and correlates with GenEval, human preference, and T2I-CompBench scores. The 8B model reaches GenEval overall 0.74, with objects 0.99, counting 0.94 and position 0.89, against DALL-E 3's overall 0.67, SDXL's 0.55 and PixArt-alpha's 0.48. Increasing the autoencoder's latent channels from 8 to 16 improves reconstruction FID from 1.56 to 1.06. Larger models tolerate fewer sampling steps better, since at 5 steps the 8B model's CLIP score drops only 2.71% from its 50-step value against a 4.30% drop for the 264M model, consistent with the 8B model's shorter, straighter ODE path length of 185.96 versus 191.13.
Limitations
The authors concede that T5-XXL is critical specifically for complex prompts with detailed scene descriptions or rendered text. Dropping it at inference costs roughly half the aesthetic-quality parity and drives typography win rate down to about 38%. Training the 8B model required roughly 5x10^22 FLOPs, and QK-normalization was needed to prevent divergence at the largest depth, which suggests even larger models may need further stabilization. The authors explicitly frame the scaling trend as unresolved, stating that results show no saturation for image or video models, so the ceiling of the approach is not established by this paper.
Why it matters
SD3 is the point at which rectified flow and joint dual-stream text-image attention move from small-scale demonstrations to a production-scale, predictably scaling recipe. MM-DiT plus rectified flow becomes the architecture that FLUX, Qwen-Image, Seedream and most 2025 open text-to-image systems build on.
Abstract, in the authors' own words
Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.
Research lines
Reported results
| Benchmark | Value | Guidance | Budget | Source |
|---|---|---|---|---|
| GenEval | 0.74 | — | — | Table 5 |
SD3's own paper reports neither DPG-Bench nor MJHQ-30K; any such number attributed to SD3 is a third-party re-measurement.
Design-axis choices
Objective
Architecture
Conditioning & control
Inference
Representation
Method note — the shared flow-matching interpolation
Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:
Builds on
- uses_architecture_from Scalable Diffusion Models with Transformers — SD3's MM-DiT is a dual-stream extension of the plain DiT block.
- uses_objective_from Flow Matching for Generative Modeling — SD3 trains with a rectified-flow / flow-matching objective and a logit-normal timestep sampler.
- uses_objective_from Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow — SD3 explicitly builds on the rectified-flow reflow formulation for straight paths.
Built on by
Nothing recorded yet.
Challenges / competes with
- challenges Cross-attention conditioning — SD3's own ablation (Sec. 4 / Figure 4 of the paper) reports that its dual-stream MM-DiT ("two separate sets of weights for the two modalities," joined only through the shared attention operation) "significantly outperforms the cross-attention and vanilla variants" it was compared against — a direct, paper-internal head-to-head of conditioning mechanisms.
- competes_with Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction — VAR proposes next-scale autoregression as an alternative generative paradigm to flow-matching diffusion transformers, matching or beating their ImageNet FID.