Separate domain
Medical imaging
Generative models applied to clinical images. This is kept apart from the rest of the atlas on purpose. The constraints are different, the evidence bar is different, and the field has arrived at different answers about where generation should happen.
A better distribution distance means little here. What counts is whether a downstream diagnostic model improves, whether a radiologist can tell the image is synthetic, and whether the model memorized a patient.
The finding that organizes this domain
The choice of generation space here splits by task constraint, not by era. Reconstruction and inverse problems stay in pixel space because the solver needs an invertible mapping back to the measurement, so a latent would break it. Volumetric work trains its own autoencoder because full-resolution 3D will not fit in memory. Most two-dimensional work borrows Stable Diffusion's natural-image autoencoder, and that choice is almost never argued.
That last point is the open question. Nobody has tested whether an autoencoder trained on photographs preserves the fine texture that separates a lesion from an artifact, and the standard evaluation metrics come from the same photograph-trained feature family, so they would not reveal it.
Where medical generation happens
Four answers, each chosen for a stated reason.
Medical: pixel space
dominantsince 2021-08Generate or reconstruct medical images directly in pixels, with no learned latent.
Why this space: For inverse problems the solver needs an invertible image-to-measurement mapping, so a latent is not merely unnecessary but structurally disallowed.
Medical: borrowed natural-image latent
dominantsince 2022-11Fine-tune Stable Diffusion on medical data while keeping its natural-image VAE frozen.
Why this space: The pretrained natural-image autoencoder is good enough for medical images, and the transfer-learning saving outweighs any domain mismatch.
Medical: domain-trained latent
ascendantsince 2022-09Train an autoencoder on medical data itself, then run diffusion in that latent.
Why this space: A latent is necessary for 3D volumes, but it must be trained on the target anatomy rather than borrowed from natural images.
Medical: foundation-model latent
emergingsince 2026-06Generate inside a medical foundation model's feature space, the RAE idea applied to clinical images.
Why this space: A medical vision foundation model's representation is a better generative substrate than either a borrowed natural-image latent or a reconstruction-trained domain latent.
Other approaches
Open problems
Most medical image generation runs inside an autoencoder trained on natural photographs, and nobody has tested whether that latent preserves clinically relevant detail.
If a borrowed natural-image latent silently discards the fine texture that distinguishes a lesion from an artifact, then every downstream result built on it is measuring the wrong thing — and the standard evaluation metrics, which come from the same ImageNet feature family, would not reveal it.
Approaches tried
- Train the autoencoder on the target modality instead of borrowing one (MAISI)
- Generate inside a medical foundation model's feature space (STREAM)
- Evaluate with radiomics-native rather than ImageNet-derived features (Fréchet Radiomic Distance)
- Run the controlled ablation nobody has published: pixel versus borrowed-VAE versus domain-VAE versus foundation latent, on one modality with a downstream clinical task
Papers (27)
Prioritized by strength of evidence rather than by benchmark score.
landmark
Ktena et al. · 2024-04Medical: pixel space
The strongest downstream evidence in medical generative imaging: pixel-space cascaded diffusion models, with no VAE anywhere, generate synthetic training data that improves both in-distribution and out-of-distribution diagnostic accuracy an…
Ribeiro et al. · 2023-06Medical: causal counterfactuals
The direct successor to the deep structural causal model framework, from the same group, using hierarchical VAE-based causal mechanisms rather than diffusion. Its importance to this atlas is the evaluation: it judges counterfactuals against…
RoentGen · 2022-11Medical: borrowed natural-image latent
Adapts Stable Diffusion to chest X-rays by fine-tuning the U-Net and text encoder while freezing Stable Diffusion's natural-image VAE, and shows the synthetic images improve downstream classifiers by up to five points. The frozen natural-im…
Score-based MRI recon · 2021-10Medical: pixel space
Reconstructs undersampled MRI by alternating score-model sampling with a data-consistency projection against the measured k-space. The pixel space here is not a preference but a structural requirement: the solver needs an invertible mapping…
Deep SCM · 2020-06Medical: causal counterfactuals
Builds counterfactual image generation on normalizing flows and variational inference rather than diffusion, because the abduction-action-prediction procedure counterfactuals require needs a tractable invertible model. A different lineage f…
core
Retinal FM tokenizers · 2026-08Medical: foundation-model latent
Tests four retinal foundation models as generative latent spaces and reports the cautionary result the representation-latent line most needs: generation inside a foundation model's space beats conventional latent diffusion when scored by th…
STREAM · 2026-06Medical: foundation-model latent
The first genuine representation-space generator in medical imaging: it runs Riemannian flow matching directly inside a pretrained histopathology foundation model's patch-token feature space, which is L2-normalized and therefore lives on a …
PCGM · 2025-09Medical: causal counterfactuals
Combines an explicit causal graph with voxel-level anatomical constraint masks, applied through a 3D ControlNet, for counterfactuals on 3D brain MRI. Offers evidence beyond distribution metrics: measurements taken from the generated counter…
PixCell · 2025-06Medical: borrowed natural-image latent
A diffusion foundation model for pathology built on Stable Diffusion 3's natural-image VAE, conditioned on UNI-2h pathology foundation-model embeddings because pathology images have no captions to prompt with. The foundation model supplies …
Fréchet Radiomic Distance · 2024-12
Argues that evaluating medical generative models with ImageNet-derived perceptual features, the same family Stable Diffusion's VAE and standard FID come from, misses clinically relevant structure. Radiomics-native features detect distributi…
DiffChest · 2024-09Medical: causal counterfactuals
A self-conditioned diffusion model, not a structural causal model, trained on 515,704 chest radiographs from 194,956 patients across the US and Europe to generate patient-specific counterfactual visualisations of confounders. Validated with…
MAISI · 2024-09Medical: domain-trained latent
Synthesizes full 3D CT volumes with anatomical control, using a domain-trained 3D VAE-GAN rather than a natural-image autoencoder, compressing 512-cubed volumes to 128-cubed and splitting tensors across GPUs. The clearest case in medical im…
SynthRAD2023 · 2024-03
A multi-centre benchmark for generating CT from MRI or cone-beam CT for radiotherapy planning. Its most useful finding for this atlas is negative: image-similarity metrics did not correlate with clinical dose accuracy, so a method can look …
RadEdit · 2023-12Medical: borrowed natural-image latent
Uses masked diffusion editing to synthesize the dataset shifts a deployed model will face, so a vision model can be stress-tested without collecting new clinical data. Editing as an evaluation instrument rather than as a content-creation to…
BiomedJourney · 2023-10Medical: borrowed natural-image latent
Learns disease-progression editing from pairs of longitudinal radiology reports, using a language model to turn two visits into an edit instruction. Latent diffusion, and one of the few genuine image-editing systems in the medical literatur…
Dar et al. memorization · 2023-07
Shows 3D latent diffusion models memorize and reproduce training samples in coronary CT angiography and knee MRI, detected with a contrastive detector. The patient-privacy constraint that makes medical synthetic data harder than the natural…
MONAI Generative Models · 2023-07Medical: domain-trained latent
The open-source infrastructure most medical generative work is built on, providing reference implementations of diffusion, autoregressive and adversarial models for 2D and 3D medical data. Infrastructure rather than a claim about which spac…
Singla et al. · 2022-12Medical: causal counterfactuals
A GAN-based counterfactual explainer for chest X-ray classifiers with no structural causal model anywhere in it, carrying the rarest evidence in this section: a reader study with diagnostic radiology residents, in which counterfactual expla…
Pinaya brain-MRI LDM · 2022-09Medical: domain-trained latent
Applies latent diffusion to 3D brain MRI conditioned on age, sex and brain structure volumes, and releases a synthetic dataset of 100,000 brain images. An early demonstration that the latent-diffusion recipe transfers to volumetric medical …
SynDiff · 2022-07
Translates between imaging modalities (multi-contrast MRI, MRI to CT) with an adversarial diffusion model trained without paired data, using cycle consistency. The argument made is diffusion versus GAN fidelity, not a claim about which spac…
Diff-SCM · 2022-02Medical: causal counterfactuals
The first diffusion-based deep structural causal model: forward diffusion for abduction, then anti-causal gradient guidance through the reverse process for action and prediction. Its experiments are MNIST and ImageNet only, with no medical …
CSGM-MRI · 2021-08Medical: pixel space
Uses a score-based generative prior for compressed-sensing MRI and shows it generalizes out of distribution better than end-to-end supervised reconstruction, which fails when the anatomy or sampling pattern shifts.
strong followup
Causal-Adapter · 2025-09Medical: causal counterfactuals
Inserts structural-causal-model interventions into a frozen text-to-image diffusion model, tested on ADNI brain MRI. Evaluated on effectiveness, composition, realism and minimality plus a counterfactual latent divergence measure, with no cl…
MedEdit · 2024-07Medical: pixel space
Induces pathology into healthy brain MRI while preserving the surrounding scan. Notable for its evaluation: a board-certified neuroradiologist judged the generated strokes indistinguishable from real ones, which is a stronger bar than any d…
StylEx (medical) · 2024-04Medical: causal counterfactuals
StyleGAN-based counterfactual attribute discovery across eight tasks in retinal fundus, external eye and chest radiograph imaging, validated by an interdisciplinary expert panel. States explicitly that it is not designed to infer causality,…
StainDiffuser · 2024-03
Virtual staining from H&E to immunohistochemistry with a dual diffusion model that trains staining and cell-segmentation jointly. Explicitly motivated by the small-data regime of pathology, where thousands rather than millions of paired sam…
Shortcut counterfactuals · 2023-12Medical: causal counterfactuals
Classifier-guided diffusion with no structural causal model, used to detect and quantify shortcut features such as pacemakers in medical classifiers across two chest X-ray datasets and a skin lesion dataset. Evidence is classifier-behaviour…