strong-followup
Otter: A Multi-Modal Model with In-Context Instruction Tuning
· NTU S-Lab · 2023-05 · arXiv:2305.03726
Why it matters
Builds directly on Flamingo's perceiver resampler and gated cross-attention design and adds in-context instruction tuning. Evidence the mechanism had genuine uptake beyond its originating lab.
What this paper does
read: full textBefore this
By mid-2023, Flamingo-style architectures (frozen vision encoder, frozen LLM, trainable Perceiver resampler plus gated cross-attention layers inserted into the LLM) existed only as Flamingo itself and the open reproduction OpenFlamingo. It was untested whether other groups could take that exact architecture and get new capabilities out of it by changing only the training data and objective, rather than the architecture.
The problem
Existing instruction-tuned multimodal models responded to single instructions or used prior dialogue turns, but did not exploit interleaved image-and-text in-context examples to teach the model a task at inference time the way text-only LLMs use in-context examples.
The idea
Keep OpenFlamingo's Perceiver resampler and gated cross-attention architecture unchanged, and instruction-tune it on a purpose-built dataset, MIMIC-IT, whose examples are query-instruction-answer triplets paired with in-context image-text example triplets, so the model learns to use multimodal in-context demonstrations.
How it works
Otter is built on OpenFlamingo with a frozen CLIP ViT-L/14 vision encoder and a frozen LLaMA-7B language model. Only the Perceiver resampler, the cross-attention layers inserted into the language model, and the input/output embeddings of the language model are fine-tuned, roughly 1.3B trainable parameters. Training uses the MIMIC-IT dataset (over 3 million multimodal instruction-response pairs, about 2.2 million unique instructions), built from sources including VQAv2, GQA, LLaVA-style instruction data, and a panoptic video scene graph dataset (4-8 frames per video), with in-context examples selected via heuristics for each entry.
Evidence
The fetched full text contains no quantitative benchmark numbers or standard metric comparisons; results are presented qualitatively as side-by-side generations comparing Otter to OpenFlamingo on instruction-following and in-context visual QA examples. This is a genuine limitation of what the paper reports, not a gap in fetching.
Limitations
The paper explicitly concedes language hallucination: because Otter is built on OpenFlamingo, which depends on LLaMA, it inherits LLaMA's hallucination tendency, and the paper notes the model may hallucinate language unrelated to the image. No other limitations or failure cases are discussed in the fetched text.
Why it matters
Otter is evidence that Flamingo's gated cross-attention design was portable: an outside lab could pick up the same frozen-encoder/frozen-LLM/cross-attention-adapter architecture, unchanged, and extend it with a new instruction-tuning recipe and dataset. That supports treating cross-attention VLMs as a genuine architectural line with outside uptake rather than one lab's isolated result, though the paper's own evidence for the resulting capability gain is qualitative only.
Abstract, in the authors' own words
Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the \textbf{Otter} model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the \textbf{MIMIC-IT} (\textbf{M}ult\textbf{I}-\textbf{M}odal \textbf{I}n-\textbf{C}ontext \textbf{I}nstruction \textbf{T}uning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.
Research line
Builds on
Nothing recorded yet.
Built on by
Nothing recorded yet.
This is a strong-followup entry — a short-form summary. Full paper-page explanations (before/problem/ core idea/representation/architecture/objective/conditioning/training/inference/results/ ablations/limitations) are written for landmark and core papers first; see PROJECT_STATE.md for the schedule.