Open problems
The questions the field has not answered, each with the papers attacking it and the research lines that inherit it as a weakness. A problem marked partly addressed has real progress against it but no settled answer.
Medical imaging keeps its own open problems, since they turn on clinical evidence rather than benchmark scores. See the medical domain →
No paper has directly studied whether a text-conditioning mechanism behaves differently when the generator runs in a high-dimensional, semantically structured frozen-encoder latent (RAE, SVG) versus a low-dimensional, reconstruction-trained VAE latent. The two published text-to-image RAE-family systems (Scale-RAE, SVG-T2I) picked different conditioning mechanisms without a stated comparison against each other or against the VAE-latent baseline.
If conditioning mechanism and representation choice are truly independent design axes, results should transfer freely between them; if they interact (e.g. a much higher-dimensional or more anisotropic latent needs a different amount or kind of conditioning bandwidth than a compact VAE latent does), then RAE-family results that reuse an off-the-shelf conditioning recipe from the VAE-latent literature could be systematically under- or over-conditioning the model without anyone having measured it.
Approaches tried
- SVG-T2I's answer: single-stream joint attention (Unified Next-DiT) over concatenated Gemma2-2B text tokens and image tokens, i.e. the same MM-DiT family of mechanisms used with VAE latents (SD3, Qwen-Image), just applied unchanged to a frozen-DINOv3 latent.
- Scale-RAE's answer: the MetaQuery learnable-query-bridge mechanism (a Qwen-2.5 LLM jointly processes 256 learnable queries and the prompt, projected by a 2-layer MLP connector into the DiT) — also an unmodified transplant of a mechanism first developed for CLIP-feature/VAE-latent unified models, not a mechanism designed for the frozen-encoder latent's own geometry.
- No published ablation isolates conditioning mechanism from representation choice on a matched RAE-family backbone; this atlas is watching for one.
Should the noise source distribution for diffusion/flow training be a simple isotropic Gaussian, or something structured (a mixture, a learned distribution) that better matches the target latent's actual geometry?
A mismatched source distribution is one hypothesized cause of curved, hard-to-distill transport paths in high-dimensional semantic latents.
Approaches tried
- Gaussian-mixture source distributions
- data-adaptive learned source distributions
- structured couplings for the stochastic interpolant
Lines that carry it as a weakness
Is a learned latent space necessary for diffusion transformers at all, or can a correctly reparameterized pixel-space model (JiT) match latent-space performance with no tokenizer?
If pixels alone are enough once the objective is fixed, it reframes the entire representation-space research line as optimizing something that may not be structurally necessary — the sharpest open challenge to the RAE line's premise.
Approaches tried
- x-prediction instead of epsilon-prediction on large pixel patches (JiT)
Papers attacking it
Lines that carry it as a weakness
Representation-space generation, Single-stage pixel transformers, Cascaded and multiscale pixel diffusion
Semantic / foundation-model latents discard much of the high-frequency pixel detail (exact color, texture, fine structure) that faithful reconstruction — and, later, edit-region preservation — depends on.
This is the crux of whether semantic latents can ever support image editing: an editor must leave untouched pixels untouched, which requires detail the latent may not carry.
Approaches tried
- residual/detail branch alongside the semantic latent (SVG)
- detail-recovering decoder queries (later follow-up work, not yet in this atlas)
Papers attacking it
Lines that carry it as a weakness
In-context editing, Representation-space generation, Semantic-plus-detail hybrids, Editing in a representation latent, Medical: foundation-model latent
Rectified-flow-style objectives aim for straight, easy-to-distill transport paths, but anisotropic high-dimensional latents (like RAE's) can force curved, unstable trajectories even under a straight-path objective.
Curved paths make few-step distillation and fast sampling much harder — directly relevant to whether representation-space models can ever be as fast at inference as VAE-based ones.
Approaches tried
- drifting-field distillation for anisotropic latents
- cosine-similarity-based coupling for cleaner velocity fields
Papers attacking it
Lines that carry it as a weakness
Flow matching and rectified flow, Representation-space generation, Natively few-step objectives
Two independent groups agree DINOv2 is the best encoder to build a generative model on, and disagree about why. Nobody knows which property actually causes the gain.
Every representation-space generator has to pick a frozen encoder, and the choice is currently made by copying whoever published last. Without knowing which property matters, there is no principled way to pick the next encoder, and no way to know whether a better one would even be recognized.
Approaches tried
- Rank by alignment-target ablation: REPA's own table puts DINOv2-B first and MAE-L last, with SigLIP and CLIP in between
- Rank by generation quality in the latent itself: RAE finds DINOv2 gives the strongest generation even though MAE gives the best reconstruction, so reconstruction is the wrong proxy
- Explain it as semantic quality, which linear-probe accuracy would measure
- Explain it as patch-level spatial structure instead: a 27-encoder study finds spatial structure predicts the generation gain where global linear-probe accuracy does not
- Test whether the language-free scaling advantage Web-SSL reports for understanding transfers to generation at all — as of this writing nobody has run that experiment
Papers attacking it
Lines that carry it as a weakness
Contrastive language-image pretraining, Self-distillation representations, Representation-aligned latents, Representation-space generation
No editing method reliably guarantees that the regions a user did not ask to change come back unchanged.
This is the constraint that separates editing from generation. A model that produces a beautiful image which subtly alters a face, a logo or an instrument reading has failed at the task regardless of its sample quality, and in clinical or forensic settings it has failed dangerously.
Approaches tried
- Learn preservation statistically from annotated triplets where annotators were told to leave unrelated regions alone (MagicBrush) — weakest guarantee
- Condition architecturally, via inpainting-style masks or in-context concatenation
- Inject source attention and features to bias toward the original layout
- Derive an explicit mask from cross-attention and restrict the edit to it (MasaCtrl)
- Never regenerate the unedited content at all: freeze its key-value pairs and only attend to them (KV-Edit) — the only architecturally earned guarantee
- Add a reward term that scores region preservation directly and optimize it with RL (EditScore, and the identity-preservation reward head in Qwen-Image-2.0-RL)
Papers attacking it
Lines that carry it as a weakness
Editing in the VAE latent, Training-free attention manipulation, Editing in a representation latent
Diffusion transformers were built around low-dimensional latents (4-16 channels); RAE-style frozen encoders produce latents with hundreds to thousands of channels. Does standard diffusion training still work well at that dimensionality, and what has to change?
If high-dimensional semantic latents need fundamentally different noise schedules, samplers, or architectures, every technique tuned on low-dimensional VAE latents needs re-validating.
Approaches tried
- dimension-dependent noise-schedule shift (RAE)
- anisotropy-aware distillation (Distilling Drifting Transformers)
Papers attacking it
Lines that carry it as a weakness
A single latent space optimized for pixel-reconstruction fidelity is often a poor space for a generative prior to learn in, and pushing on one side tends to hurt the other.
This dilemma is the reason a decade of 'just make the autoencoder reconstruct better' work did not translate into better generation, and it motivates the entire representation-space generation line.
Approaches tried
- align VAE training to a frozen foundation model (VA-VAE)
- train VAE and diffusion model jointly through that alignment loss (REPA-E)
- stop training the encoder for reconstruction at all (RAE)
Papers attacking it
Lines that carry it as a weakness
VAE-latent diffusion, Representation-aligned latents, Semantic-plus-detail hybrids
Standard VAE latents (e.g. SD-VAE) are optimized purely for pixel reconstruction, so they carry little semantic structure — giving a diffusion model a low-information target to learn.
If the target latent is semantically impoverished, the generator has to rediscover semantics from scratch during training, which is exactly the compute REPA/RAE-style methods claw back.
Approaches tried
- auxiliary alignment loss (REPA)
- align the VAE itself (VA-VAE, REPA-E)
- drop the VAE, generate directly in a frozen foundation encoder's space (RAE, SVG)