Conditioning & control

adaLN / adaLN-Zero modulation

Condition every transformer block by regressing per-channel scale-and-shift (and, in adaLN-Zero, a gating factor initialized to zero) from a single fused embedding of timestep and class label, instead of attending over a token sequence at all.

Used by: Scalable Diffusion Models with Transformers

Adapter-based spatial conditioning

Freeze a pretrained text-to-image diffusion model entirely and add a small trainable side network (a duplicated-and-shrunk encoder branch, or a lightweight external feature extractor) whose output is injected into the frozen backbone, so a new control signal (edges, depth, pose) can be learned without touching or risking the base model's weights.

CLIP text encoder as conditioning source

Use the text tower of an image-text contrastive model (CLIP) as the source of prompt embeddings, so the conditioning vector already lives in a space jointly trained to align with image content.

Cross-attention conditioning

Condition a generator on text/image context via dedicated cross-attention layers between the generation stream and a separately encoded context sequence.

Decoupled cross-attention (image-prompt adapter)

Add a second, separate cross-attention branch with its own key/value projections for an image prompt's embedding, computed in parallel with the frozen model's existing text cross-attention and summed into the same query, so image and text conditioning no longer compete for one shared cross-attention layer.

Identity-preserving face conditioning

Combine a semantic face-embedding branch (from a face-recognition encoder) with a spatial landmark/structure branch, injected through some mix of decoupled cross-attention and adapter-style side networks, to preserve a specific person's identity across generations without per-identity fine-tuning.

Joint (shared) attention conditioning

Concatenate condition and generation tokens into one sequence and let ordinary self-attention mix them, instead of routing conditioning through separate cross-attention layers.

Native autoregressive token conditioning

Discretize images into tokens from the same vocabulary space text uses (or a space a single decoder-only transformer treats uniformly), and generate images by ordinary next-token prediction conditioned on whatever text or image tokens already sit earlier in the same sequence — there is no separate conditioning pathway because there is no separate architecture for condition versus target.

Learnable query-token bridge

A fixed-size set of learnable query tokens is prepended to the prompt and processed jointly by a (frozen or lightly-tuned) multimodal LLM; the resulting query embeddings, not the LLM's own hidden states over the prompt, are projected by a small connector and handed to a diffusion decoder as its conditioning signal.

In-context sequence concatenation

Concatenate a reference image's tokens directly onto the generation sequence and let ordinary attention mix them, with no adapter, inversion, or task-specific head.

Used by: OminiControl: Minimal and Universal Control for Diffusion Transformer

Frozen large-language-model text encoder (T5-style)

Replace or supplement the CLIP text tower with a much larger text-only pretrained language model (T5) as the source of prompt embeddings, on the finding that scaling the text encoder improves fidelity and prompt alignment more than scaling the image diffusion model does.

Vision-language model as text encoder

Condition on prompt embeddings from a multimodal VLM rather than CLIP or T5, so the conditioning signal already carries visual grounding and instruction understanding.