core

RISEBench: Evaluating Reasoning-Informed Visual Editing

· Shanghai AI Laboratory, SJTU, Princeton · 2025-04 · arXiv:2504.02826

Why it matters

The first benchmark for edits that require reasoning rather than recognition, across temporal, causal, spatial and logical cases. Its headline result is how far the field has to go: the strongest model tested reached only 35.9 percent.

What this paper does

read: full text

Before this

Existing image-editing benchmarks tested edits that a model could execute from recognition alone, following an explicit instruction such as changing a color or removing an object. None tested whether a model could reason about temporal, causal, spatial, or logical consequences before producing the edit.

The problem

There was no benchmark for measuring whether visual editing models can reason about the consequences of an instruction, rather than simply recognize what to change.

The idea

Build a benchmark around four reasoning categories, temporal, causal, spatial, and logical, and score each output on three axes together, whether the instruction's implicit reasoning was followed, whether the untouched parts of the image stayed consistent, and whether the result looked physically plausible.

How it works

Not an editing method; RISEBench is a curated evaluation set. Temporal Reasoning tests natural evolution such as decay or aging; Causal Reasoning tests the effect of an external force such as a bitten apple or a cracked egg; Spatial Reasoning tests geometric and 3D understanding such as object arrangement, assembly, or viewpoint; Logical Reasoning tests rule-based visual puzzles such as Sudoku or mazes. Each output is judged against three criteria, Instruction Reasoning, Appearance Consistency, and Visual Plausibility, by both human raters and an LMM-as-judge using GPT-4o.

Evidence

Nine editing models were evaluated in total. Note a version discrepancy found while fetching sources: the abstract stored in this entry (matching the paper's original release) states the best model, named there as GPT-4o-Image, reaches 28.8% accuracy. The arXiv v2 HTML full text fetched in this session instead names the top model GPT-4o-Native and states, quoting directly, "GPT-4o-Native achieves the highest performance with an accuracy of only 35.9%, followed by Gemini-2-Flash with the second-highest accuracy (10.9%)," where accuracy means achieving the maximum score on all three evaluation criteria simultaneously for a given test case. The weakest models trail far behind in both readings, with FLUX.1-Canny scoring 0.0% and EMU2 scoring 1.5% in the v2 text. Both versions agree on the qualitative headline: even the strongest model tested passes well under half of the reasoning-editing cases.

Limitations

The authors describe the benchmark as "an exploratory study" still in its early stages. They note that Visual Plausibility carries only a small weight in the overall score even though it has the highest measurement error against human judgment, an acknowledged weakness in how well the automated score reflects physical-law violations.

Why it matters

It reframed the bar for editing models from instruction-following to instruction-reasoning, and its headline result, the best model tested passing well under half the cases, is the number the field cites for how far reasoning-aware editing still has to go.

Abstract, in the authors' own words

Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but they still face challenges in General Visual Editing, particularly in following complex instructions, preserving appearance consistency, and supporting flexible input formats. To study this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE). RISEBench focuses on four key reasoning categories: Temporal, Causal, Spatial, and Logical Reasoning. We curate high-quality test cases for each category and propose an robust evaluation framework that assesses Instruction Reasoning, Appearance Consistency, and Visual Plausibility with both human judges and the LMM-as-a-judge approach. We conducted experiments evaluating nine prominent visual editing models, comprising both open-source and proprietary models. The evaluation results demonstrate that current models face significant challenges in reasoning-based editing tasks. Even the most powerful model evaluated, GPT-4o-Image, achieves an accuracy of merely 28.8%. RISEBench effectively highlights the limitations of contemporary editing models, provides valuable insights, and indicates potential future directions for the field of reasoning-aware visual editing. Our code and data have been released at https://github.com/PhoenixZ810/RISEBench.

Research line

Evaluation and benchmarksascendant

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.