strong-followup
ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling
Chaojie Mao, Jingfeng Zhang, Yulin Pan · Alibaba (Tongyi Lab) · 2025-01 · arXiv:2501.02487
Why it matters
Unifies creation and editing under a long-context conditioning unit, implemented as a LoRA on FLUX.1 and therefore cheap to adapt. Worth noting that the paper itself reports no quantitative benchmark results, so its standing rests on adoption rather than on published numbers.
What this paper does
read: full textBefore this
The original ACE model's Long-context Condition Unit concatenated the conditioning image and mask with the noise sequence along the token dimension. The paper says this "disrupts the established context perception framework honed during text-to-image generation training" and lengthens the attention sequence, raising adaptation cost. FLUX.1-Fill-dev used channel-wise concatenation but only for its own inpainting task, not generalized to other editing or generation tasks.
The problem
Adapt a powerful pretrained text-to-image model, FLUX.1-dev, to a broad set of editing and generation tasks under one conditioning scheme, without the extra attention cost and disrupted pretraining prior that sequence-dimension conditioning causes.
The idea
Concatenate the input image, mask, and noisy latent along the channel dimension rather than the sequence dimension, generalizing FLUX.1-Fill-dev's inpainting-style conditioning into a Long-context Condition Unit usable for arbitrary editing and reference-generation tasks, trained in two stages.
How it works
Editing happens in FLUX's VAE latent space: the noisy latent, reference image, and mask are stacked as extra channels feeding the diffusion transformer, written as [input image; mask; noisy latent] per condition unit, with multiple units concatenated sequentially for multi-reference tasks. There is no explicit mask-based latent blending or post-hoc compositing step described; protection of un-edited regions is implicit, learned end-to-end through the channel-stacked mask conditioning and the training loss (a target-prediction loss plus a reference-reconstruction loss that is zero for 0-reference tasks) rather than any inference-time attention injection or blending. Training runs in two stages: stage one pretrains on 0-reference tasks (no reference image, e.g. inpainting, controllable generation) initialized from FLUX.1-dev or FLUX.1-Fill-dev; stage two finetunes on the full ACE task set including reference-conditioned tasks. The model is released as both a full finetune and lightweight LoRA variants for portrait consistency, subject consistency, local editing, and repainting.
Evidence
The paper reports no quantitative benchmark numbers. Its results section presents only qualitative visualizations across five task categories, subject-driven generation, portrait consistency, local editing, flexible instructions, and local reference editing, and asserts superiority in image quality and prompt-following without a metric to support the claim.
Limitations
The paper concedes no CLIP score, FID, human evaluation, or comparison to ACE, FLUX.1-Fill-dev, or any other baseline is reported; the claimed reduction in adaptation cost from channel-dimension conditioning is asserted but never measured, and no dedicated limitations section is given.
Why it matters
Channel-dimension conditioning is a cheap way to reuse a pretrained model's own attention prior instead of disturbing it with extra sequence tokens, which plausibly explains why ACE++, released as a LoRA on FLUX, became a widely reused editing backbone even though the paper itself offers no quantitative evidence for its claims.
Abstract, in the authors' own words
We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage. For example, FLUX.1-Fill-dev deals primarily with painting tasks and can be used as an initialization to accelerate the training process. In the second stage, we finetune the above model to support the general instructions using all tasks defined in ACE. To promote the widespread application of ACE++ in different scenarios, we provide a comprehensive set of models that cover both full finetuning and lightweight finetuning, while considering general applicability and applicability in vertical scenarios. The qualitative analysis showcases the superiority of ACE++ in terms of generating image quality and prompt following ability. Code and models will be available on the project page: https://ali-vilab. github.io/ACE_plus_page/.
Research line
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
This is a strong-followup entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.