core

Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis

Salman Ul Hassan Dar, Arman Ghanaat, Jannik Kahmann · Heidelberg University · 2023-07 · arXiv:2307.01148

Why it matters

Shows 3D latent diffusion models memorize and reproduce training samples in coronary CT angiography and knee MRI, detected with a contrastive detector. The patient-privacy constraint that makes medical synthetic data harder than the natural-image case.

What this paper does

read: full text

Before this

3D latent diffusion models had become state of the art for medical image synthesis and were being promoted as a route to open data sharing without compromising patient privacy, but whether they actually memorize training data had gone largely unexamined in the medical imaging community. Prior memorization work looked only at 2D pixel-space models using pixel-level correlation, which misses rotated or flipped near-copies that data augmentation can still produce.

The problem

The paper asks whether 3D latent diffusion models trained on medical imaging datasets reproduce identifiable training samples, which would undermine the privacy rationale for releasing synthetic data.

The idea

Train self-supervised contrastive encoders so that augmented versions of the same image cluster together in embedding space, then use nearest-neighbor distance in that space to flag generated samples that are near-duplicates, including rotated or flipped copies, of training images.

How it works

The generative model is a 3D latent diffusion model architecture adopted directly from prior work, trained separately on 242 photon-counting coronary CT angiography sub-volumes and 904 knee MRI volumes. The paper gives no explicit reasoning for using a latent rather than a pixel-space model; the autoencoder is carried over as inherited 3D latent diffusion practice rather than argued for in this paper, and the absence of any stated rationale is itself notable given how load-bearing the memory argument is elsewhere in the 3D medical literature. Memorization is detected with a contrastive encoder reduced to 32-dimensional embeddings, using mean squared embedding distance between generated and training samples to surface candidate copies, which two annotators then manually classified.

Evidence

This is not downstream clinical or reader-study evidence in the ordinary sense, but it is stronger than a bare distribution metric because it directly compares generated samples against real training identities. On coronary CT angiography, 59 percent of the 242 training volumes, 142 of 242, were manually confirmed as memorized among 1,000 generated samples. On knee MRI, manual review of 150 randomly sampled copy candidates out of 3,600 generated samples found 33 percent were confirmed copies. Adding data augmentation during training reduced coronary CT memorization from 59 percent to 41 percent, but did not eliminate it.

Limitations

The paper concedes it does not test or propose any mitigation strategy for memorization, and states that understanding the underlying causes is left to future work. Its definition of a copy is restricted to rotated and flipped variants for simplicity, so it likely undercounts memorization under other deformations. Only two modalities and two relatively small training sets are tested, and only 150 of 3,600 MRI candidates were manually reviewed.

Why it matters

It shows that the small training sets typical of medical imaging make 3D latent diffusion models prone to memorizing patient data at what the authors call an alarming and large scale, directly undercutting the privacy justification for treating synthetic data as a substitute for real patient data in sharing and research.

Abstract, in the authors' own words

Generative latent diffusion models have been established as state-of-the-art in data generation. One promising application is generation of realistic synthetic medical imaging data for open data sharing without compromising patient privacy. Despite the promise, the capacity of such models to memorize sensitive patient training data and synthesize samples showing high resemblance to training data samples is relatively unexplored. Here, we assess the memorization capacity of 3D latent diffusion models on photon-counting coronary computed tomography angiography and knee magnetic resonance imaging datasets. To detect potential memorization of training samples, we utilize self-supervised models based on contrastive learning. Our results suggest that such latent diffusion models indeed memorize training data, and there is a dire need for devising strategies to mitigate memorization.

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.