Research line · dominant
Self-distillation representations
Learn visual features with no labels, no language and no pixel reconstruction, by having a network predict its own slowly-updated teacher.
What defines membership
The best visual representation comes from a teacher-student objective in latent space, which produces spatially structured features that transfer without fine-tuning.
How the line developed
Read top to bottom: what came before, the idea itself, the evidence for it, what improved, and where it breaks.
Before
MAE · 2021-11landmark
Masked pixel reconstruction: scalable and a strong fine-tuning initializer, but its frozen features turn out weak for both alignment and generation.
The idea
DINOv2 · 2023-04landmark
The model that made this bet decisively: features good enough to use frozen, and the default alignment target ever since.
Evidence
Web-SSL · 2025-04strong-followup
Shows language-free self-supervision keeps improving with scale where contrastive language training saturates, at least for understanding.
Improvement
I-JEPA · 2023-01core
Predicts masked regions in latent space instead of pixel space, on the argument that pixel detail wastes capacity.
ViT Registers · 2023-09core
Removes the high-norm artifact tokens that pollute dense feature maps, now standard across the family.
At scale
DINOv3 · 2025-08core
Scales to 7B with Gram anchoring to stop dense features degrading over long training.
What it gets right
- Wins the encoder comparison in both REPA's and RAE's own ablations
- Needs no paired captions, so it scales with raw image data
- Produces the spatially structured features that predict generation gains
Where it is weak
- No language grounding, so it cannot be prompted directly
- Whether its advantage for generation is semantic quality or spatial structure is disputed, with two groups offering different explanations
- The claim that language-free scaling wins has not been tested for generation
Competing answers
Also in this line
V-JEPA 2