What these models actually produce
Released weights, run here on an A100 80GB, same prompts and same seed across models. Three prompts chosen to test different things rather than to flatter anyone.
How to read this, and how not to
Each model runs at its own recommended settings rather than a forced common configuration. That shows what you would actually get, and it means quality differences here cannot be attributed to architecture. Step counts and guidance differ between rows.
HiDream-O1 additionally ignored the requested 1024px and snapped to 2048, so it is rendering four times the pixels of the other rows. That alone can account for apparent detail differences.
The one comparison that is clean is LLaDA against its own distilled variant, because those share weights and differ only in distillation. Everything else is an observation about systems as shipped, not evidence about the ideas underneath them. See why benchmark numbers cannot rank these.
Text rendering
Whether the model can render specified text correctly. This is the capability Qwen-Image was built around, and where most earlier open models visibly failed.
a storefront window with a hand-painted sign reading "GENERATIVE VISION ATLAS", morning light, photograph
All three words rendered correctly in the hand-painted style asked for.
All three words correct, with more atmospheric lighting and incidental signage in the background that is also legible.
Renders the requested three words correctly on the main window, then hallucinates garbled text on every other sign in the scene: the awning reads TRKEC IUVN ATLAS and the hanging sign reads CATIGE & PRARTSRNIC ATLAS. A distinct failure from the row below, which garbles the requested text itself.
Garbles the middle word to VION and duplicates the phrase. Visible distillation artifacts across the image. Same weights as the row above, distilled to 2-4 steps.
Counting and spatial relations
Whether the model can count and place objects. This is what GenEval measures, and it is where all three models here fail.
three red cubes stacked on top of a blue sphere, to the left of a green cone, studio lighting
Colours correct, both spatial relations correct, but produced two cubes instead of three. The cone is a compound shape rather than a cone, and the sphere is squashed.
Clean geometry: a true cone, a true sphere, crisp cubes. But it produced four cubes instead of three, missing the count in the other direction from Qwen.
Also two cubes instead of three. Geometry is clean and the spatial relation is right. Three models, three attempts, three wrong counts.
Plain photographic quality
A prompt with no trick in it, to see baseline quality when nothing unusual is being asked.
a close-up portrait of an elderly fisherman with a weathered face, soft window light, 85mm lens
Convincing skin detail and lighting.
Convincing, with stronger environmental context and depth of field.
Convincing, rendered at 2048px rather than the requested 1024.
What running them actually taught us
- Counting fails in all three, and that is now a robust finding rather than an anecdote. Every model placed the objects correctly and coloured them correctly, and every model got the count wrong: two cubes, four cubes and two cubes against a requested three. These are three different architectures on three different latents, including one with no latent at all. LLaDA's paper concedes a weak counting score; the other two do not mention it, and show it anyway.
- Distillation costs more than throughput. LLaDA's Turbo variant is the same model compressed to two to four sampling steps, and it is the only run here that fails the text prompt outright. Anyone quoting a distilled model's speed should quote this alongside it.
- Text rendering is genuinely solved in open models. Two years ago this prompt would have produced letter-shaped noise from anything open. Both full models here get it right first time.
- Every one of these needs a different environment. Qwen runs on stock diffusers. LLaDA pins transformers 4.57.6 with diffusers 0.39. HiDream-O1 pins transformers 4.57.1 and requires a compiled flash-attention with no fallback path in its code. No two of the three coexist in one environment, which is a real cost of the current open ecosystem and part of why assembling them in one place is worth doing.