core

Transfer between Modalities with MetaQueries

Xichen Pan, Satya Narayan Shukla, Aashu Singh · NYU, Meta AI · 2025-04 · arXiv:2504.06256

Why it matters

Bridges a frozen, understanding-only multimodal LLM to a diffusion image decoder with a small set of learnable query tokens, so the MLLM's weights (and its reasoning/understanding ability) never have to change to gain generation. Trains only on paired image-caption data with a standard diffusion loss.

What this paper does

read: full text

Before this

Prior unified models tried to make a single autoregressive backbone jointly handle text and pixel outputs by fine-tuning a multimodal LLM to also produce image tokens or embeddings, which required careful architecture design, data and loss balancing across modalities, and multi-stage training recipes.

The problem

Fine-tuning an MLLM to add generation risks degrading the understanding ability it already has, because optimizing one capability can compromise the other when both are trained into the same weights.

The idea

Insert a fixed set of learnable query tokens between a frozen MLLM and a diffusion image decoder. The queries are fed through the frozen MLLM so its output at those positions carries the model's understanding and reasoning about the prompt, and a small trained connector maps that output into the diffusion decoder's conditioning space, so the MLLM's weights never have to change to gain generation.

How it works

MetaQueries are randomly initialized learnable embeddings matched to the MLLM's hidden dimension, fed into the frozen MLLM under causal masking alongside the prompt. The MLLM's output at the query positions becomes a conditioning sequence, passed through a trainable 24-layer bidirectional-attention transformer connector that projects it into the diffusion decoder's conditioning space in place of the decoder's usual text conditioning. Base experiments use 64 query tokens; the main models scale this to 256, and 512 gives a further small gain. MLLM backbones tested range from LLaVA-OneVision-0.5B up to Qwen2.5-VL 7B; diffusion decoders are Stable Diffusion v1.5 and Sana-1.6B. Only the queries, the connector and the diffusion decoder are trained, with a standard diffusion denoising loss on paired image-caption data; the MLLM itself is frozen throughout. Pretraining uses 25 million image-caption pairs for 8 epochs, and instruction tuning adds 2.4 million naturally occurring image pairs for 3 epochs, after which the model can be adapted to editing or subject-driven generation with as few as 1,000 fine-tuning steps.

Evidence

With a Qwen2.5-VL 7B backbone and a Sana decoder, MetaQuery-XL reaches COCO FID 8.69 (with an SD v1.5 decoder), state of the art among SD v1.5-based unified models, ahead of MetaMorph's 11.8 and Emu's 11.66, and MJHQ-30K FID 6.02 with Sana, GenEval 0.80 and DPG-Bench 82.05. On the WISE knowledge-and-reasoning generation benchmark MetaQuery-XL scores 0.55 overall against Janus-Pro-7B's 0.35. Understanding is preserved at MME-Perception 1685.2, MMBench 83.5, MMMU 58.6 and MM-Vet 66.6 with the 7B backbone. On DreamBench subject-driven generation, without any per-subject fine-tuning, MetaQuery reaches DINO 0.737 and CLIP-I 0.852, ahead of Kosmos-G's 0.694 DINO score.

Limitations

The authors concede a persistent prompt-alignment gap against Janus-Pro-7B on GenEval and DPG-Bench, which they attribute to differing failure modes between diffusion-based and autoregressive generation rather than to the query-bridge design itself. They also concede that freezing the MLLM costs a small amount of prompt alignment, 0.56 versus 0.58 GenEval compared with full fine-tuning, though visual quality is comparable. They state that closing the remaining gap to leading proprietary systems is likely primarily a data-scaling problem given their 25 million pair training set, rather than an architectural limitation.

Why it matters

MetaQuery shows that a frozen MLLM's reasoning and world knowledge can be transferred to image generation through a lightweight learned interface alone, without ever updating the MLLM's weights, establishing the query-bridge pattern as a way to add generation to any frozen understanding model cheaply while provably preserving its original capabilities, in contrast to Emu2's approach of training the whole multimodal backbone end to end.

Abstract, in the authors' own words

Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation.

Research lines

Encoder plus projectordominant Unified understanding and generationascendant

Design-axis choices

Representation

Semantic / foundation latent

Conditioning & control

Learnable query-token bridge

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

builds_on Hierarchical Text-Conditional Image Generation with CLIP Latents — MetaQuery continues unCLIP's idea of generating in a CLIP-like semantic feature space, now mediated by a frozen multimodal LLM instead of a frozen CLIP text encoder.

Built on by

Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders uses_architecture_from this — Scale-RAE's paper states it explicitly: "We adopt the MetaQuery architecture for text-to-image (T2I) generation." It prepends 256 learnable query tokens to the prompt, processes them jointly with a Qwen-2.5 1.5B LLM, and projects the result through a 2-layer MLP connector into the DiT backbone — the same query-bridge mechanism MetaQuery introduced, applied on top of RAE's frozen-encoder latent.

Challenges / competes with

competes_with BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset — BLIP3-o and MetaQuery both connect a frozen/near-frozen MLLM to a diffusion decoder over CLIP-like features, published within weeks of each other.