core

MAISI: Medical AI for Synthetic Imaging

Pengfei Guo, Can Zhao, Dong Yang · NVIDIA, National Cancer Institute · 2024-09 · arXiv:2409.11169

Why it matters

Synthesizes full 3D CT volumes with anatomical control, using a domain-trained 3D VAE-GAN rather than a natural-image autoencoder, compressing 512-cubed volumes to 128-cubed and splitting tensors across GPUs. The clearest case in medical imaging of a latent chosen for an explicitly stated reason: full-resolution 3D volumes do not fit in memory otherwise.

What this paper does

read: full text

Before this

Prior CT synthesis work was limited to volumes at or below 512-cubed voxels, often generated as independent 2D slices that caused 3D structural inconsistency across slices, or trained on dedicated single-organ datasets with weak generalization and fixed output dimensions and voxel spacing.

The problem

The paper addresses generating full-resolution 3D CT volumes larger than 512-cubed voxels, with flexible dimensions and voxel spacing, generalizing across body regions and 127 anatomical structures, within GPU memory limits that 3D convolutional networks would otherwise exceed.

The idea

Compress CT volumes with a domain-trained 3D VAE-GAN into a much smaller latent volume, run a latent diffusion model conditioned on body region and voxel spacing in that space, and add ControlNet conditioning on anatomical segmentation masks for controllable synthesis, splitting large feature maps across GPUs to fit in memory.

How it works

The VAE-GAN is trained on 39,206 3D CT and 18,827 3D MRI volumes, so the latent is domain-trained on medical data rather than borrowed from a natural-image autoencoder, using combined perceptual, adversarial, and L1 reconstruction loss in voxel space. The paper gives an explicit, quoted reason for compressing at all, that generating high-resolution 3D volumes exceeding 512-cubed voxels poses a significant challenge due to the substantial memory demands imposed by 3D convolution networks, and it identifies the cause as large feature maps rather than model weight count, unlike language models. To handle this it introduces tensor splitting parallelism, dividing feature maps into overlapping segments distributed across GPUs or processed sequentially within one GPU to reduce peak memory. The paper does not state an explicit numeric compression ratio for the latent. A time-conditional U-Net diffusion model with an L1 denoising objective operates in this latent space, conditioned on a one-hot body-region code and voxel spacing, with ControlNet added for 127-structure segmentation conditioning.

Evidence

Evidence is a mix of distribution metrics and downstream segmentation task performance, with no radiologist reader study conducted. On an unseen external dataset, autoPET 2023, MAISI's diffusion model reaches average FID 6.083 against a retrained LDM baseline's 12.379 and HA-GAN's 13.757. For downstream tumor segmentation trained on synthetic data, Dice similarity coefficient improves by about 4 to 6.5 percent across liver, lung, pancreas, colon, and bone lesion tasks, for example lung tumor Dice rising from 0.494 to 0.585 in one cross-validation fold. Out-of-distribution VAE reconstruction reaches LPIPS 0.038, SSIM 0.978, and PSNR 37.266 without any additional training on the new dataset.

Limitations

The paper concedes that demographic variation in generated anatomy, such as age, ethnicity, and gender, has not been extensively validated, that the high-resolution generation still demands substantial compute that could limit accessibility, and that small organs show a pronounced synthetic-versus-real performance gap, for example gallbladder, duodenum, and pancreas Dice scores of 0.47 to 0.70 on synthetic data against 0.75 to 0.80 on real data for major organs. No clinical validation or reader study is conducted.

Why it matters

It is the clearest case in medical imaging of a generation-space choice justified by a stated, concrete constraint rather than inherited convention. Full-resolution 3D volumes do not fit in GPU memory without compression, and the paper both states this and engineers around it with tensor splitting parallelism.

Abstract, in the authors' own words

Medical imaging analysis faces challenges such as data scarcity, high annotation costs, and privacy concerns. This paper introduces the Medical AI for Synthetic Imaging (MAISI), an innovative approach using the diffusion model to generate synthetic 3D computed tomography (CT) images to address those challenges. MAISI leverages the foundation volume compression network and the latent diffusion model to produce high-resolution CT images (up to a landmark volume dimension of 512 x 512 x 768 ) with flexible volume dimensions and voxel spacing. By incorporating ControlNet, MAISI can process organ segmentation, including 127 anatomical structures, as additional conditions and enables the generation of accurately annotated synthetic images that can be used for various downstream tasks. Our experiment results show that MAISI's capabilities in generating realistic, anatomically accurate images for diverse regions and conditions reveal its promising potential to mitigate challenges using synthetic data.

Research line

Medical: domain-trained latentascendant

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.