landmark
InstructPix2Pix: Learning to Follow Image Editing Instructions
Tim Brooks, Aleksander Holynski, Alexei A. Efros · UC Berkeley · 2022-11 · arXiv:2211.09800
Why it matters
Creates the instruction-editing task by manufacturing its training data: a language model writes edit instructions and a prompt-to-prompt pipeline renders the before and after images, giving supervised triplets that never existed. Edits in one forward pass with separate guidance scales for the image and the instruction. Nothing constrains what stays fixed, which is the failure every later paper measures against.
What this paper does
read: full textBefore this
Prompt-to-Prompt could keep an edited image consistent with a source image by reusing cross-attention maps, but only within its own generation process and only by re-describing the whole target image, not by giving an instruction. SDEdit edited by adding noise to an input image and denoising it toward a new full target caption, but needed that full re-description and struggled to preserve identity under substantial changes. Text2Live produced a color-and-opacity overlay layer, which restricted it to additive edits.
The problem
Let a user edit an image by writing a plain instruction ("turn him into a cyborg") rather than a full description of the desired result, and do it in a single forward pass with no per-image optimization or inversion.
The idea
Manufacture a paired training set that never existed by combining a language model that writes edit instructions with a text-to-image model that renders consistent before-and-after image pairs from those instructions, then train an ordinary conditional diffusion model to imitate that mapping directly, which lets it generalize to real images and freely written instructions at inference.
How it works
Space: VAE latent space, built on Stable Diffusion v1.5. This is a single forward-pass conditional diffusion model: the noisy latent z_t is denoised conditioned on the encoded input image E(c_I), which is concatenated channel-wise with z_t at the first convolutional layer (the new channels are zero-initialized), plus the text instruction c_T fed through the existing text-conditioning pathway. Training data comes from two stages: GPT-3 Davinci is fine-tuned on 700 human-written (input caption, instruction, output caption) triples to generate 454,445 instruction triples from LAION-Aesthetics V2 6.5+ captions; each caption pair is then rendered into 100 candidate image pairs with Stable Diffusion plus Prompt-to-Prompt, sweeping the cross-attention injection strength p ~ U(0.1, 0.9), and filtered by CLIP thresholds (image-image similarity >= 0.75, image-caption similarity >= 0.2, directional CLIP similarity >= 0.2), keeping up to 4 pairs per caption pair. At inference the model uses two independent classifier-free guidance scales, s_I for the input image and s_T for the instruction, so the two forms of adherence can be traded off. Protection: none. There is no mask, no attention injection, and no locality loss anywhere in the model; edits are global over the whole latent, and the only lever affecting how much of the source image survives is the image guidance scale s_I, a scalar that trades off consistency against instruction-following rather than protecting any specific region.
Evidence
The editing model trained for 10,000 steps on 8x40GB A100 GPUs over 25.5 hours at 256x256, batch size 1024, and generalizes at inference to 512x512 with 100 Euler-ancestral denoising steps, taking roughly 99 seconds per image on an A100. Against SDEdit, the paper reports InstructPix2Pix achieves higher CLIP image similarity (input consistency) at matched directional CLIP similarity (instruction adherence) across a sweep of both methods' guidance parameters; no absolute CLIP or LPIPS numbers are reported, the comparison is a tradeoff curve. Ablations show that training on a smaller generated dataset reduces the model's ability to perform larger edits, and removing the CLIP-based filtering from dataset generation reduces image consistency.
Limitations
The authors concede the model is limited by the visual quality of its generated dataset and therefore by the underlying Stable Diffusion model used to build it. They concede it struggles with counting objects and with spatial reasoning, including instructions like moving an object left or swapping the positions of two objects. They concede it inherits well-documented biases from the data and pretrained models, citing gender-profession correlations in outputs. They also concede the approach is bounded by the human-written instructions used to fine-tune GPT-3, by GPT-3's own ability to generate good instructions, and by Prompt-to-Prompt's ability to render a consistent edited counterpart image.
Why it matters
It established that an instruction-following image editor could be trained entirely from synthetic paired data distilled out of a language model and a generative model, with no real edited-image pairs at all. Because it has no mechanism at all for protecting unedited regions, it also became the reference case of uncontrolled global editing that essentially every later editing paper defines its own locality mechanism against.
Abstract, in the authors' own words
We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.
Research line
Method note — the shared flow-matching interpolation
Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.