core

Taming Rectified Flow for Inversion and Editing

· 2024-11 · arXiv:2411.04746

Why it matters

Attacks per-step numerical truncation error with an exact rectified-flow formulation and a high-order Taylor expansion in place of naive Euler steps. Its editing variant also shares self-attention features between the inversion and editing passes.

What this paper does

read: full text

Before this

Rectified-flow diffusion transformers such as FLUX and OpenSora generate well but invert poorly. Vanilla first-order Euler discretization of the rectified-flow ODE accumulates truncation error at every step, so reconstructions drift from the source, and any editing built on that inversion inherits the drift. Prior fixes such as RF-Inversion used optimization-based corrections that only handled simple edits.

The problem

Get accurate, training-free inversion for rectified-flow models, so DiT-based image and video editing can be built on top of it without retraining the base model.

The idea

Derive the exact solution of the rectified-flow ODE by variation of constants and approximate its nonlinear term with a high-order Taylor expansion instead of a first-order Euler step, then reuse the self-attention features recorded during inversion to anchor source structure during editing.

How it works

RF-Edit operates entirely in attention-and-feature space; it writes no new latent. During the final N inversion timesteps, self-attention Value features from the last M transformer blocks are cached; during the corresponding early denoising timesteps, those cached Value matrices replace the freshly computed ones, so attention becomes a function of the current queries and keys but the inverted image's stored values. This value-injection is the paper's protection mechanism, and it acts as a structural anchor that preserves the source image's or video's unedited content while the query and key computation still lets the instruction steer the targeted region. There is no mask, no latent blending, and no explicit locality loss; protection is purely an attention-level substitution, tunable by how many steps and blocks share features.

Evidence

On image editing, RF-Edit reaches CLIP Score 33.66 against Prompt-to-Prompt's 30.70, DiffEdit's 32.68, and RF-Inversion's 33.02; its LPIPS of 0.149 is not the best in the comparison, Plug-and-Play scores 0.080 but the paper reports it fails on complex edits. On reconstruction, RF-Solver cuts image MSE to 0.0094 from the rectified-flow baseline's 0.0268 and video LPIPS to 0.3299 from 0.4159. On video editing it reaches subject consistency 0.9501 against TokenFlow's 0.9439 and motion smoothness 0.9712 against COVE's 0.9697.

Limitations

The authors report that pushing the Taylor expansion beyond second order gives no further gain, because the extra function evaluations it costs reduce the number of timesteps available under a fixed inference budget. They note the number of feature-sharing steps needs case-by-case tuning, with their own ablation showing the optimal count differs across examples. The paper does not discuss the roughly doubled per-timestep forward-pass cost the high-order correction adds relative to a plain Euler step.

Why it matters

It showed a numerically exact ODE solver, not a learned component, was the missing piece for rectified-flow inversion, and it established attention-feature sharing between inversion and denoising as a training-free way to protect source structure during editing, without ever writing a new latent.

Abstract, in the authors' own words

Rectified-flow-based diffusion transformers like FLUX and OpenSora have demonstrated outstanding performance in the field of image and video generation. Despite their robust generative capabilities, these models often struggle with inversion inaccuracies, which could further limit their effectiveness in downstream tasks such as image and video editing. To address this issue, we propose RF-Solver, a novel training-free sampler that effectively enhances inversion precision by mitigating the errors in the ODE-solving process of rectified flow. Specifically, we derive the exact formulation of the rectified flow ODE and apply the high-order Taylor expansion to estimate its nonlinear components, significantly enhancing the precision of ODE solutions at each timestep. Building upon RF-Solver, we further propose RF-Edit, a general feature-sharing-based framework for image and video editing. By incorporating self-attention features from the inversion process into the editing process, RF-Edit effectively preserves the structural information of the source image or video while achieving high-quality editing results. Our approach is compatible with any pre-trained rectified-flow-based models for image and video tasks, requiring no additional training or optimization. Extensive experiments across generation, inversion, and editing tasks in both image and video modalities demonstrate the superiority and versatility of our method. The source code is available at https://github.com/wangjiangshan0725/RF-Solver-Edit.

Research line

Inversion for flow modelsascendant

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.