core
GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
Dhruba Ghosh, Ludwig Schmidt · University of Washington, Meta AI (FAIR), UCLA · 2025-12 · arXiv:2512.16853
Why it matters
Shows the field's most-cited compositional benchmark has drifted up to 17.7% away from human judgment as models improved, and proposes a corrected version. Directly relevant to any claim in this atlas that rests on a GenEval number.
What this paper does
read: full textBefore this
GenEval became the field's most widely used compositional text-to-image benchmark, scoring images with specialized judges, a MaskFormer object detector, CLIP for color, and bounding-box math for position, against 553 templated prompts across six categories. At release it matched human judgment on 83% of images generated by Stable Diffusion v2.1, rising to 91% on images where annotators unanimously agreed, which made it a trusted proxy for compositional generation quality.
The problem
As generator models improved over roughly three years, GenEval's static judges and prompt set did not keep pace with newer model outputs. The paper formalizes this failure mode as benchmark drift, where a benchmark well aligned with human judgment at release becomes misaligned as the distribution of generated images shifts with model capability while the benchmark itself stays frozen.
The idea
Measure GenEval's drift directly with a large human study, show the drift is large enough that the benchmark is effectively saturated for current state-of-the-art models, then build a replacement, GenEval 2, with wider coverage and explicit compositionality control, plus a new scoring method, Soft-TIFA, designed to resist the same failure mode by scoring each semantic atom of a prompt separately rather than judging the image holistically.
How it works
The drift measurement collects over 23,000 human annotations, three per image, on GenEval outputs from eight popular text-to-image models spanning about three years, with 85.1% of data points unanimous across annotators. GenEval 2 replaces the 553 templated prompts with 800 prompts built from 40 objects, 18 attributes, 9 relations and counts from 2 to 7, composed into templates with explicitly controlled compositionality, from 3 to 10 semantic atoms with 100 prompts at each atomicity level. Soft-TIFA scores a generated image by generating one visual question per atom of the prompt, using a VQA model's probability of the correct answer for each question, then aggregating either by arithmetic mean across atoms for atom-level correctness, or geometric mean, which penalizes any single wrong atom and better captures whole-prompt correctness.
Evidence
GenEval's reported scores diverge from human judgment by up to 17.7% absolute for current models, for example Gemini 2.5 Flash Image scoring 75.4% by GenEval's automated judges against 93.1% by human judgment. This gap grew over roughly three years, and four state-of-the-art models released 2024 to 2025 all score above 94.8% on GenEval, indicating the benchmark is saturated. On GenEval 2, the same leading model, Gemini 2.5 Flash Image, reaches only 35.8% prompt-level accuracy despite 85.3% atom-level accuracy, exposing a compositional gap that per-atom scoring reveals but prompt-level GenEval did not. Soft-TIFA using the geometric mean reaches 94.5% AUROC against human judgment on combined prompts, against 92.4% for VQAScore and 91.6% for TIFA.
Limitations
The authors concede that avoiding future drift is far from guaranteed even for GenEval 2 and Soft-TIFA, and state it is entirely possible that greater distribution shift over a longer period could cause the geometric-mean Soft-TIFA variant to show the same drift VQAScore already shows. They frame continual audits of model-based text-to-image evaluation as necessary rather than optional, not something this paper's replacement benchmark alone solves permanently.
Why it matters
Because GenEval has been the primary evaluation in many prominent text-to-image papers, reported gains of 2 to 3 percentage points on GenEval between recent models are smaller than its measured 17.7% drift error, so such comparisons cannot be trusted at face value. This directly undercuts any claim in this atlas that rests on a GenEval delta between models evaluated at different points in time.
Abstract, in the authors' own words
Automating Text-to-Image (T2I) model evaluation is challenging; a judge model must be used to score correctness, and test prompts must be selected to be challenging for current T2I models but not the judge. We argue that satisfying these constraints can lead to benchmark drift over time, where the static benchmark judges fail to keep up with newer model capabilities. We show that benchmark drift is a significant problem for GenEval, one of the most popular T2I benchmarks. Although GenEval was well-aligned with human judgment at the time of its release, it has drifted far from human judgment over time -- resulting in an absolute error of as much as 17.7% for current models. This level of drift strongly suggests that GenEval has been saturated for some time, as we verify via a large-scale human study. To help fill this benchmarking gap, we introduce a new benchmark, GenEval 2, with improved coverage of primitive visual concepts and higher degrees of compositionality, which we show is more challenging for current models. We also introduce Soft-TIFA, an evaluation method for GenEval 2 that combines judgments for visual primitives, which we show is more well-aligned with human judgment and argue is less likely to drift from human-alignment over time (as compared to more holistic judges such as VQAScore). Although we hope GenEval 2 will provide a strong benchmark for many years, avoiding benchmark drift is far from guaranteed and our work, more generally, highlights the importance of continual audits and improvement for T2I and related automated model evaluation benchmarks.
Research line
Method note — the shared flow-matching interpolation
Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.