Representation-space generation

RAE · 2025-10Representation-space generation

Replaces the VAE with a Representation Autoencoder: a frozen pretrained encoder (DINO, SigLIP, or MAE) plus a lightweight trained ViT decoder. The resulting latent is high-dimensional (width must be >= token dimension) and semanti…

PixNerd · 2025-07Single-stage pixel transformers

Replaces the pretrained VAE with a per-patch neural-field parameterisation, giving single-scale single-stage pixel diffusion with no tokenizer and no cascade, at ImageNet 256 FID 2.15 and 512 FID 2.84. Published four months before…

REPA · 2024-10Representation-aligned latents

Adds a simple auxiliary loss that aligns a diffusion transformer's intermediate hidden states to a frozen DINOv2's features during training. Speeds up SiT-XL training by 17.5x and sets a new ImageNet FID, with no change to the mod…

DINOv2 · 2023-04Self-distillation representations

Scales self-distillation-based self-supervised ViT training to produce general-purpose visual features that transfer without finetuning; becomes the default frozen encoder that REPA, VA-VAE, and RAE all align to or generate direct…

DiT · 2022-12VAE-latent diffusion

Replaces the UNet backbone of latent diffusion with a plain Vision Transformer operating on latent patches, showing that transformer scaling laws (more FLOPs, lower FID) hold for diffusion just as they do for language modeling. Th…

unCLIP / DALL-E 2 · 2022-04Representation-space generation

Generates a CLIP image embedding from a text prompt (the 'prior'), then decodes that embedding to pixels with a diffusion decoder. The first time a generative system's primary latent was a semantic, language-aligned feature space …

LDM / Stable Diffusion · 2021-12VAE-latent diffusion

Moves diffusion training from pixel space into the latent space of a pretrained autoencoder, cutting compute by orders of magnitude while keeping quality; introduces the VAE + UNet + cross-attention recipe every 'latent diffusion'…

MAE · 2021-11Self-distillation representations

Masks most of an image's patches and trains a ViT to reconstruct the missing pixels; one of the three encoder families (with DINO, CLIP/SigLIP) RAE ablates as candidate frozen representations, and the weakest of the three by RAE's…

GenFirst · 2026-08Semantic-plus-detail hybrids

Attacks the same latent-collapse problem as the representation-latent line but from the opposite direction: instead of freezing a semantic encoder, it trains generation-first under weak reconstruction pressure. Reports gFID 0.97 o…

Retinal FM tokenizers · 2026-08Medical: foundation-model latent

Tests four retinal foundation models as generative latent spaces and reports the cautionary result the representation-latent line most needs: generation inside a foundation model's space beats conventional latent diffusion when sc…

Latent Diffusability · 2026-06Semantic-plus-detail hybrids

The empirical grounding the whole latent-design argument lacked: trains a large family of tokenizers spanning reconstruction-only, representation-only, and hybrid regularization, then measures which latent properties actually pred…

STREAM · 2026-06Medical: foundation-model latent

The first genuine representation-space generator in medical imaging: it runs Riemannian flow matching directly inside a pretrained histopathology foundation model's patch-token feature space, which is L2-normalized and therefore l…

PAE · 2026-05Semantic-plus-detail hybrids

Asks the design question directly rather than picking a side: instead of inheriting a frozen foundation encoder (RAE) or training for reconstruction alone (VAE), it explicitly shapes a tokenizer's latent manifold using priors dist…

RPiAE · 2026-03Editing in a representation latent

Initializes from a representation encoder but fine-tunes it, built explicitly to repair editing quality. Its abstract names the same bottleneck PS-VAE found: frozen-encoder tokenizers suffer limited reconstruction fidelity, which …

Latent Forcing · 2026-02Semantic-plus-detail hybrids

Processes latents and pixels jointly under separate noise schedules, using the latent as a computational scratchpad rather than as the generative space. The explicit counter-position to the tokenizer-free line: not that latents ar…

DiDAE · 2026-01Editing in a representation latent

The cleanest existing case of editing inside a genuinely frozen representation latent. It finds editing directions by Procrustes alignment and SVD directly in a frozen CLIP embedding space, transplanting the InterFaceGAN and GANSp…

Scale-RAE · 2026-01Representation-space generation

Tests whether the RAE recipe survives the jump from ImageNet to open-vocabulary text-to-image: scales a SigLIP-2-frozen RAE decoder on web, synthetic, and text-rendering data. Scale improves general fidelity but targeted data comp…

PS-VAE · 2025-12Editing in a representation latent

Starts from DINOv2 and SigLIP2 features and does general instruction editing with a Transfusion-style transformer diffusing in a 96-channel latent. Its importance to this atlas is the reason it does not keep the encoder frozen: th…

REPA spatial-structure study · 2025-12Representation-aligned latents

Runs representation alignment against 27 different encoders and finds that what predicts a generation gain is patch-level spatial structure, not the global linear-probe accuracy the field had been using as a proxy. The most direct…

There is No VAE · 2025-10Single-stage pixel transformers

Pretrains an encoder self-supervised and then finetunes the whole thing end to end in pixels, reaching ImageNet 256 FID 1.58 at roughly 30 percent of DiT's training compute, and producing the first working pixel-space consistency …

SVG · 2025-10Representation-space generation

Independently arrives at the same core move as RAE within days: drop the VAE, generate directly in a frozen DINOv3 feature space, and add a lightweight residual branch specifically to recover the fine-grained pixel detail semantic…

DINOv3 · 2025-08Self-distillation representations

Scales DINO-style self-supervised training to 7B parameters and adds Gram anchoring to keep dense features high-quality over very long training, fixing a known DINOv2 dense-feature degradation issue; the encoder SVG generates dire…

REG · 2025-07Representation-aligned latents

Sharpens REPA's idea: rather than aligning hidden states to a frozen encoder as an auxiliary loss, it entangles a high-level semantic token directly into the denoising target alongside the low-level latents. A NeurIPS 2025 oral, a…

BLIP3-o · 2025-05Unified understanding and generation

A fully open study of unified-model design choices that lands on a diffusion transformer generating CLIP image features (not VAE latents), trained understanding-first then generation-second so understanding ability is preserved; s…

MetaQuery · 2025-04Encoder plus projectorUnified understanding and generation

Bridges a frozen, understanding-only multimodal LLM to a diffusion image decoder with a small set of learnable query tokens, so the MLLM's weights (and its reasoning/understanding ability) never have to change to gain generation. …

REPA-E · 2025-04Representation-aligned latents

Shows naive end-to-end VAE+diffusion training collapses the VAE's latent space, but training through the REPA alignment loss instead (with a batch-norm bridge and a stop-gradient on the diffusion loss's path back to the VAE) makes…

MAETok · 2025-02Representation-aligned latents

Argues the key variable for generation quality is latent *geometry*, not whether the autoencoder is variational: a masked-modeling-trained, purely discriminative latent with fewer Gaussian-mixture modes is both easier to generate …

SigLIP 2 · 2025-02Contrastive language-image pretraining

Extends SigLIP's sigmoid contrastive language-image objective with captioning, self-distillation and masked-prediction losses in one recipe, improving dense/localization features specifically; the frozen encoder Scale-RAE uses for…

VA-VAE / LightningDiT · 2025-01Representation-aligned latents

Names and measures the reconstruction-generation dilemma directly: a VAE tuned purely for reconstruction fidelity becomes a *harder* space for a diffusion model to learn in. Fixes it by adding a vision-foundation-model alignment l…

SiT · 2024-01Flow matching and rectified flowVAE-latent diffusion

Reframes diffusion and flow matching as two points in one family of stochastic interpolants, isolating objective, coupling, and sampler as independent design choices on the exact DiT backbone; SiT-XL beats DiT-XL at matched size a…

Emu2 · 2023-12Representation-space generationUnified understanding and generation

A 37B multimodal model that autoregressively predicts the next CLIP visual embedding alongside text tokens, then decodes embeddings back to pixels with a diffusion decoder — an autoregressive, MLLM-scale continuation of unCLIP's s…

dRAE · 2026-07Representation-space generationDiscrete-token autoregression

Discretizes the continuous semantic latent that representation autoencoders produce, using hyper-spherical quantization to avoid the codebook collapse that Euclidean quantization suffers on anisotropic features, and scales to a 13…

Distilling Drifting Transformers · 2026-06Natively few-step objectivesRepresentation-space generation

Identifies that RAE latents are severely anisotropic on a per-token basis, so the standard isotropic-Gaussian-to-data coupling forces curved, hard-to-distill denoising trajectories. Proposes a 'drifting field' distillation method …

DecQ · 2026-05Semantic-plus-detail hybrids

Attacks RAE's core weakness without giving up its frozen encoder: lightweight detail-condensing queries pull fine-grained information out of *shallower* foundation-model layers and feed it to the decoder alongside the semantic tok…

RAEv2 · 2026-05Representation-space generation

Fixes RAE's rough edges: multilayer representation aggregation for a richer frozen feature, explicit combination of RAE with a REPA-style alignment loss, and a re-parameterized DiT output that fixes classifier-free guidance (which…

PixelREPA · 2026-03Single-stage pixel transformersRepresentation-space generation

Finds that REPA, which reliably accelerates latent diffusion transformers, actively hurts JiT, and fixes it with a masked adapter. Valuable to the atlas as a negative result connecting two lines: representation alignment is not a …

FlatDINO · 2026-02Semantic-plus-detail hybrids

Literally composes the two rival representations: it takes DINOv2 patch features and compresses them with a VAE into 32 continuous 1-D tokens, an 8x shorter sequence. Reports gFID 1.80 on ImageNet 256 at 8x fewer FLOPs per forward…

LV-RAE · 2026-02Semantic-plus-detail hybrids

Augments RAE's semantic features with the low-level colour and texture information a frozen semantic encoder discards, and separately hardens the decoder against latent perturbation through fine-tuning with noise injection — a fix…

SVG-T2I · 2025-12Representation-space generation

Scales SVG's frozen-DINOv3 recipe from ImageNet to text-to-image, reporting a 0.75 GenEval score and 85.78 DPG-Bench with full open-source release — the direct competitor to Scale-RAE, published about a month apart on essentially …

JiT · 2025-11Single-stage pixel transformers

Argues pixel-space diffusion was never unstable because of pixels — it was unstable because of epsilon-prediction. Switching to x-prediction on large patches lets a plain ViT ('Just image Transformers') match tokenizer-based DiT/P…

VFM-VAE · 2025-10Representation-space generation

A useful negative result for the hybrid camp: distilling foundation-model features into a trained tokenizer weakens the representation's robustness, so this work keeps the foundation encoder frozen and redesigns only the decoder. …

GigaTok · 2025-04Discrete-token autoregression

Scales a visual tokenizer to 3B parameters and finds the reconstruction-versus-generation tension reappears at scale: better reconstruction does not monotonically improve downstream autoregressive generation unless the latent stay…

Web-SSL · 2025-04Self-distillation representations

Trains a family of pure self-supervised (no language) vision models from 0.3B to 7B parameters on the same web data CLIP uses, and finds visual SSL keeps improving with scale where CLIP-style training saturates — evidence against …

TokenFlow · 2024-12Semantic-plus-detail hybrids

Argues understanding and generation genuinely need different granularities of visual information, and resolves the conflict with a dual-codebook design — a CLIP-style semantic encoder and a pixel encoder, kept aligned through a sh…

DC-AE · 2024-10VAE-latent diffusion

Pushes autoencoder spatial compression up to 128x (vs. the standard 8x) without losing reconstruction quality, via residual autoencoding and a staged high-resolution adaptation schedule — up to 19x faster inference than SD-VAE-f8 …

Any-OPD · 2026-08Natively few-step objectives

Distills a 12B model into a 2.5B one while treating the teacher as a black-box sampler, requiring no shared latent space between them — relevant to the atlas because it makes distillation independent of the representation choice.

UniSpace · 2026-08Editing in a representation latent

An 8B mixture-of-experts unified understanding, generation and editing model built on a reparameterized frozen vision-transformer feature space rather than a VAE pathway. Flagged as needing an independent verification pass before …

Tokenizer Post-Training · 2025-09Discrete-token autoregression

Argues tokenizers are trained on the wrong distribution: they see real images but must later decode *generated* latents. Post-training the tokenizer on the generative distribution closes that train/test gap and improves final samp…