Mechanism
The generation stream (a UNet’s spatial feature map, in the originating system) supplies queries; a context sequence encoded entirely separately — LDM names “text or bounding boxes” as the general case, and the Stable Diffusion models built on it use CLIP text tokens specifically — supplies keys and values at one or more dedicated cross-attention layers spliced into the network. LDM’s own description: “by introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs.” No other coupling between the two pathways exists; the image stream never sees the raw text tokens, only what the cross-attention layers pull from them at each layer they’re inserted into.
Trade-offs
Parameter cost: small and fixed — one extra K/V projection pair per cross-attention layer, independent of how large the text encoder is. Flexibility: high — because the coupling is confined to a few well-defined layers, this is the mechanism that later let ControlNet, T2I-Adapter, and IP-Adapter bolt new conditioning signals onto a frozen pretrained model — see adapter-based conditioning. Quality ceiling: SD3’s own ablation reports that giving text and image tokens joint self-attention instead “significantly outperforms the cross-attention… variant” at production text-to-image scale — the field’s clearest internal evidence that the modularity cross-attention buys costs measurable quality once compute for a fully joint mechanism is affordable.