core

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

· Alibaba (Qwen) · 2024-09 · arXiv:2409.12191

Why it matters

Processes images at their native resolution and aspect ratio rather than resizing to a fixed square, and introduces a rotary position encoding decomposed across time, height and width. Directly load-bearing for this atlas: its successor is the text encoder inside Qwen-Image.

What this paper does

read: full text

Before this

Prior large vision-language models processed images at a fixed predetermined resolution, resizing or padding every input to something like a fixed square before encoding, typically through a static frozen CLIP-style vision encoder. Position encoding for multimodal sequences relied on plain 1D RoPE, designed for sequential text and not for the two spatial dimensions of an image or the added temporal dimension of video.

The problem

Fixed-resolution preprocessing loses detailed information in high-resolution images and wastes tokens on low-resolution ones, so the model perceives visual detail with less sensitivity to scale than human vision, and a 1D position scheme cannot cleanly represent the joint temporal, height, and width structure of interleaved image, video, and text sequences.

The idea

Let the vision encoder process each image at its own native resolution and aspect ratio, producing a number of visual tokens proportional to the image's actual size rather than a fixed count, and decompose rotary position embeddings into separate temporal, height, and width components so one scheme covers text, images, and video uniformly.

How it works

The vision encoder is a roughly 675M-parameter ViT initialized from a DFN ViT, with absolute position embeddings removed and replaced by 2D-RoPE inside the encoder to represent patch position. After the ViT, an MLP layer compresses each group of 2 by 2 adjacent patch tokens into a single token before it enters the language model, so fusion is an MLP projector feeding a compressed but still fine-grained stream of visual tokens directly into the LLM's input sequence, in the encoder-projector family rather than cross-attention. The LLM backbone is Qwen2, at 1.5B, 7.6B, or 72B parameters for the 2B, 7B, and 72B model variants respectively. Training proceeds in three stages. Stage one trains only the ViT on roughly 600B tokens of image-text data while the LLM stays frozen, stage two unfreezes all parameters, vision encoder and LLM together, across a wider roughly 800B-token data mix, and stage three freezes the ViT again and instruction-tunes the LLM on dialogue, VQA, and agent-style data. M-RoPE assigns text tokens identical position ids across all three axes, equivalent to ordinary 1D RoPE, gives image tokens a constant temporal id with height and width ids that vary per patch position, and gives video frames an incrementing temporal id per frame with height and width following the image pattern, with each new modality's position numbering continuing from one past the maximum id used by the preceding modality.

Evidence

The 72B model reaches 96.5 on DocVQA against GPT-4o's 92.8 and Claude 3.5 Sonnet's 95.2, 70.5 on MathVista against GPT-4o's 63.8, and 877 on OCRBench against a prior state of the art of 852. On MMMU it scores 64.5, below GPT-4o's 69.1. On MathVision it scores 25.9, below GPT-4o's 30.4.

Limitations

The paper concedes MMMU performance still lags GPT-4o and leaves room for improvement on complex multi-discipline problems. It concedes vision-language navigation performance falls significantly behind specialized VLN models, scoring 51.7% versus a prior state of the art of 79.0% on R2R, attributing this to incomplete and unstructured map information, and states that accurately modeling maps and locations in a 3D environment remains a major challenge. Long-video evaluation was capped at 768 frames per video.

Why it matters

Native dynamic resolution plus M-RoPE became the standard recipe for the Qwen-VL line, and this atlas records that the next generation in this line, Qwen2.5-VL, is the model actually used as the text encoder inside Qwen-Image and as the instruction parser inside Step1X-Edit. That downstream role is not claimed anywhere in this Qwen2-VL paper itself; it applies to the successor model, with Qwen2-VL establishing the architecture that made it possible.

Abstract, in the authors' own words

We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism, which enables the model to dynamically process images of varying resolutions into different numbers of visual tokens. This approach allows the model to generate more efficient and accurate visual representations, closely aligning with human perceptual processes. The model also integrates Multimodal Rotary Position Embedding (M-RoPE), facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos, enhancing the model's visual perception capabilities. To explore the potential of large multimodal models, Qwen2-VL investigates the scaling laws for large vision-language models (LVLMs). By scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the amount of training data, the Qwen2-VL Series achieves highly competitive performance. Notably, the Qwen2-VL-72B model achieves results comparable to leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal benchmarks, outperforming other generalist models. Code is available at https://github.com/QwenLM/Qwen2-VL .

Research line

Native-resolution vision-language modelsascendant

Method note — the shared flow-matching interpolation

Every flow-matching / rectified-flow paper in this atlas trains toward a straight-line path between a noise sample x₀ and a data sample x₁:

x_t = (1-t)\,x_0 + t\,x_1, \qquad v_\theta(x_t, t) \approx x_1 - x_0

Builds on

Nothing recorded yet.

Built on by

Nothing recorded yet.