core

BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

Jiuhai Chen, Zhiyang Xu, Xichen Pan · Salesforce Research, University of Maryland · 2025-05 · arXiv:2505.09568 · code

Why it matters

A fully open study of unified-model design choices that lands on a diffusion transformer generating CLIP image features (not VAE latents), trained understanding-first then generation-second so understanding ability is preserved; ships model, code, and both pretraining and instruction-tuning data.

What this paper does

read: full text

Before this

Unified models that could both understand and generate images had converged on two competing designs for the generation side, autoregressive prediction of discrete VQ tokens, or diffusion over VAE latents borrowed from ordinary text-to-image models. Design choices for the understanding side had been studied extensively, but the space of representation, objective and training-order choices for unifying generation with understanding had not.

The problem

It was unclear which image representation a unified model should generate in, which generative objective to use over it, and in what order to train understanding and generation so that adding generation does not degrade the understanding ability the base MLLM already has.

The idea

Train a diffusion transformer to generate semantically rich CLIP image features rather than VAE latents, bridge it to a frozen multimodal LLM through a small learnable query vector, and train understanding first, generation second, so generation is added without touching the weights that give the model its understanding ability.

How it works

A CLIP encoder turns each target image into 64 fixed-length semantic embeddings; these embeddings, not VAE latents, are what the diffusion transformer learns to produce. The MLLM backbone is Qwen2.5-VL-7B-Instruct for the 8B model and Qwen2.5-VL-3B-Instruct for the 4B model. A learnable query vector is appended to the prompt embedding sequence and learns to attend over the prompt and extract the semantic content needed for generation; this conditions a Lumina-Next-style diffusion transformer (Next-DiT with 3D rotary position embeddings, sandwich RMSNorm and grouped-query attention) trained with a flow-matching loss to predict the velocity field toward the target CLIP features. During this training the autoregressive MLLM backbone is kept entirely frozen; only the diffusion transformer module, 1.4B trainable parameters for the 8B model, is updated. Pretraining uses roughly 25 million open-source images (CC12M, SA-1B, JourneyDB) plus 30 million proprietary images, captioned by Qwen2.5-VL-7B-Instruct at an average of 120 tokens per caption. A second instruction-tuning stage fine-tunes on BLIP3o-60k, about 60,000 prompt-image pairs generated by prompting GPT-4o with diverse scene, object and gesture descriptions.

Evidence

BLIP3-o 8B reaches a GenEval score of 0.84, ahead of Janus-Pro-7B's 0.80, Janus 1.3B's 0.61 and Show-o 1.3B's 0.68, and a WISE knowledge-generation score of 0.62. On DPG-Bench it scores 81.60, behind Janus-Pro's 84.19. Understanding is preserved at MME-Perception 1682.6, MMMU 50.6 (versus Janus-Pro's 41.0), MMBench 83.5 (versus Janus-Pro's 79.2), VQAv2 83.1 and MM-Vet 66.6. Human evaluators preferred BLIP3-o 8B over Janus-Pro on both visual quality and prompt alignment, with p-values of 5.05e-06 and 1.16e-05.

Limitations

The authors concede the BLIP3o-60k instruction-tuning set does not fully resolve hard cases such as complex human gesture generation, landmark rendering and legible text in images, even though it substantially improves overall quality. They also concede DPG-Bench trails Janus-Pro, and separately note that FID can be misleading as a generation metric, pointing out that GPT-4o itself scores only around 30 FID under their evaluation despite strong perceived quality.

Why it matters

BLIP3-o is a fully open, systematically ablated demonstration that generating in a semantically rich CLIP feature space, rather than a VAE latent space, paired with a frozen MLLM and a lightweight query bridge, gives strong generation quality while provably preserving understanding performance, and it ships the model, code and both training datasets so the design study can be directly reproduced and extended.

Abstract, in the authors' own words

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.

Research line

Unified understanding and generationascendant

Design-axis choices

Representation

Semantic / foundation latent

Conditioning & control

Learnable query-token bridge

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

builds_on Generative Multimodal Models are In-Context Learners — BLIP3-o's diffusion-over-CLIP-features approach continues the semantic-latent generation lineage Emu2 established with autoregressive CLIP-feature regression.

Built on by

Nothing recorded yet.

Challenges / competes with

competes_with Transfer between Modalities with MetaQueries — BLIP3-o and MetaQuery both connect a frozen/near-frozen MLLM to a diffusion decoder over CLIP-like features, published within weeks of each other.