Research line · emerging
Semantic-plus-detail hybrids
Take a foundation-model representation and give it back the low-level detail it discards, with a residual branch, detail queries, or a VAE compressing it.
What defines membership
Neither pure reconstruction codes nor pure semantic features are the right latent; the answer is an explicit combination of both.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
Before
TokenFlow · 2024-12strong-followup
Dual codebooks — one semantic, one pixel-level — kept aligned by a shared mapping, motivated by understanding and generation needing different granularity.
The idea
LV-RAE · 2026-02strong-followup
Augments RAE's semantic features with the missing colour and texture information, and hardens the decoder against latent perturbation.
FlatDINO · 2026-02strong-followup
Compresses DINOv2 features with a VAE into 32 1-D tokens: literally composing the two rival representations, at 8x fewer FLOPs.
Evidence
Latent Diffusability · 2026-06core
Trains tokenizers across the whole reconstruction-to-representation spectrum and measures which latent properties actually predict generation quality.
Improvement
PAE · 2026-05core
Shapes a tokenizer's manifold using priors distilled from foundation models rather than inheriting a frozen encoder; gFID 1.03 at 13x faster convergence.
DecQ · 2026-05strong-followup
Pulls fine detail from shallower foundation-model layers into the decoder, lifting reconstruction PSNR 19.13 to 22.76 dB without unfreezing the encoder.
Combination
Latent Forcing · 2026-02core
Runs latents and pixels together under separate noise schedules, treating the latent as a computational scratchpad rather than as the space generation happens in. The clearest counter-position to the tokenizer-free line.
CrossFlow · 2026-06emerging
A one-step flow from noisy latents to pixels, FID 1.62 at one function evaluation. A bridge between the spaces rather than a choice between them.
What it gets right
- Directly targets the one weakness the representation-latent line cannot argue away
- Empirically strong: several members report both better reconstruction and better generation than pure RAE
- Cheaper sequences than diffusing on dense foundation-feature grids
Where it is weak
- Youngest line here, with the least independent replication
- More moving parts, so ablations are harder to attribute
- No agreement yet on which hybridization mechanism is the right one
Reported numbers
As published, with the guidance method, budget, and model size that produced them. Seethe comparison page for why these cannot be ranked naively.
Competing answers
Open problems it has not solved
- Semantic / foundation-model latents discard much of the high-frequency pixel detail (exact color, texture, fine structure) that faithful reconstruction — and, later, edit-region preservation — depends on.
- A single latent space optimized for pixel-reconstruction fidelity is often a poor space for a generative prior to learn in, and pushing on one side tends to hurt the other.
- Should the noise source distribution for diffusion/flow training be a simple isotropic Gaussian, or something structured (a mixture, a learned distribution) that better matches the target latent's actual geometry?
Also in this line
GenFirst