Source-linked AI summary
F-LMM: Grounding Frozen Large Multimodal Models
Size Wu, Sheng Jin, Wenwei Zhang, Lumin Xu, Wentao Liu, Wei Li, Chen Change Loy
TL;DR
Grounding LMMs can compromise general knowledge and instruction-following ability when fine-tuned on segmentation and grounding data, while mask-annotated conversation data is costly to scale. F-LMM instead decodes grounding masks from frozen LMM attention using a small trainable mask head and keyword selector. It reports competitive grounding with preserved conversational ability, and the attention formulation omits residual and feedforward layers for brevity.
Problem
Fine-tuning grounding LMMs on segmentation data and special tokens can diminish general knowledge and instruction-following, while mask-annotated conversation data is costly and hard to scale.
Source: §1 · p. 2Method
F-LMM freezes the LMM and uses attention-derived word-pixel correspondences with a trainable mask decoder, SAM-based refiner, and keyword selector to produce grounding masks.
Source: §1 · pp. 2–3, §3.2 · p. 5Results
F-LMM preserves off-the-shelf LMM question-answering ability while achieving competitive referring segmentation and phrase-grounding results, with the best balance of grounding and chat among compared methods.
Source: §1 · p. 3, §4.2 · p. 7Takeaways & Limitations
The combination of conversational and grounding capabilities shows promise for complex perception and reasoning tasks.
Source: §5 · p. 8The attention formulation omits residual and feedforward layers for brevity.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Endowing Large Multimodal Models (LMMs) with visual grounding capability can significantly enhance AIs' understanding of the visual world and their interaction with humans. However, existing methods typically fine-tune the parameters of LMMs to learn additional segmentation tokens and overfit grounding and segmentation datasets. Such a design would inevitably cause a catastrophic diminution in the indispensable conversational capability of general AI assistants. In this paper, we comprehensively evaluate state-of-the-art grounding LMMs across a suite of multimodal question-answering benchmarks, observing drastic performance drops that indicate vanishing general knowledge comprehension and weakened instruction following ability. To address this issue, we present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations -- a straightforward yet effective design based on the fact that word-pixel correspondences conducive to visual grounding inherently exist in the attention mechanism of well-trained LMMs. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits, which a SAM-based mask refiner can further optimise. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data, but achieves competitive performance on referring expression segmentation and panoptic narrative grounding benchmarks while completely preserving LMMs' original conversational ability. Additionally, with instruction-following ability preserved and grounding ability obtained, F-LMM can be directly applied to complex tasks like reasoning segmentation, grounded conversation generation and visual chain-of-thought reasoning. Our code can be found at https://github.com/wusize/F-LMM.
1. Introduction
F-LMM grounds frozen LMMs by decoding spatial cues already present in their word-image attention, avoiding the fine-tuning and special segmentation tokens used by prior designs. It preserves conversational ability while achieving competitive grounding results without costly mask-annotated conversation data.
1. Introduction
- Prior grounding designs fine-tune LMMs to model special segmentation tokens on elementary grounding data, which the authors say sacrifices general knowledge comprehension and instruction-following ability.High-quality mask-annotated conversation data is costly and hard to scale, and LLaVA-G still lags general-purpose LMMs on multimodal understanding.Source: §1 · p. 2
- F-LMM uses geometric and spatial cues in frozen LMM attention maps as segmentation priors, without tuning LMM weights or learning a special segmentation token.The paper visualizes stacked attention maps using K-Means clustering to reveal coarse masks for phrases such as “two girls” and “the plate.”Source: §1 · pp. 2–3
- Its only trainable components are a CNN mask decoder, a SAM-based mask refiner, and a linear keyword selector, trained with RefCOCO(+/g) and PNG rather than grounded conversation data.The decoder translates stacked attention maps to mask logits; the refiner uses additional image and language cues to refine semantic-agnostic masks.Source: §1 · p. 3
- F-LMM maintains general question-answering ability while achieving competitive referring segmentation and phrase-grounding results, offering the best balance of grounding and chat among compared methods.The authors also report better or comparable results on reasoning segmentation, grounded conversation generation, and visual chain-of-thought reasoning.Source: §1 · p. 3
2. Related Work
The related work reviews LMM architectures and segmentation tasks, then contrasts grounding approaches that fine-tune LMMs or add mask tokens with F-LMM's frozen-backbone design.
Grounding Large Multimodal Models
- Grounding LMMs localize language in conversations through predicted box coordinates or special segmentation tokens decoded by a segmentation head, while F-LMM avoids fine-tuning LMM parameters and grounded instruction-tuning data.The paper focuses on grounding LMMs with segmentation ability for visual perception.Source: §2 · p. 3
Visual Segmentation
- Referring expression segmentation uses free-form descriptions to segment objects, while panoptic narrative grounding segments masks for key phrases or words in a sentence.The paper evaluates these tasks and also tests segmentation in reasoning-intensive scenarios; standard segmentation instead uses predefined object classes.Source: §2 · p. 3
3. Method
F-LMM uses frozen LMM word-image attention as segmentation priors, turns them into masks with a trainable mask head, and selects grounding words for conversation.
3. Method
- F-LMM’s pipeline uses frozen-LMM word-image attention maps as mask-head priors while a keyword selector discovers object nouns in text.The mask head contains a decoder that translates attention weights to mask logits and a refiner that optimizes the decoder’s predictions.Source: §3 · pp. 3–4, Figure 3 · p. 4
3.1. Segmentation Priors from Frozen LMM
- Word-image attention maps from frozen LMMs provide spatial and geometric segmentation priors for grounding objects.The maps can be combined across layers and heads; objects’ shapes and locations are visible in some individual maps.Source: §3.1 · p. 4
3.2. Visual Grounding with Mask Head
- The mask decoder maps stacked attention maps to mask logits, and a SAM-based refiner uses image and language cues to refine the predicted masks.For objects described by multiple words, their attention maps are averaged or max-combined before normalization and decoding.Source: §3.2 · p. 5
3.3. Keyword Selector for Grounded Conversation
- A linear keyword selector predicts which words should be grounded, grouping adjacent positive tokens as one visual object.At inference, tokens scoring above λ = 0.3 are selected, and their attention maps and text embeddings are passed to the mask head.Source: §3.3 · p. 6
4. Experiments
Across standard and complex tasks, F-LMM balances conversational and grounding abilities, with results on complex tasks extending to reasoning segmentation, grounded conversation generation, and visual chain-of-thought. Ablations and analyses examine its design choices and the relationship between backbone chat and grounding performance.
4.1. Implementation Details
- Experiments use open-source LMM backbones spanning 1.3B–8B, a lightweight U-Net mask decoder, and SAM ViT-L; training uses RefCOCO(+/g) and PNG.The reported setup trains on about 190k samples using eight A800-40G GPUs for about 20 hours per training round.Source: §4.1 · p. 6
4.2. Standard QA and Grounding Tasks
- F-LMM offers the best balance of conversational and grounding abilities among compared methods, with comparable RES and PNG results and preserved question-answering performance.Existing grounding LMMs score near zero on MMBench and MME and lag general-purpose LMMs on MMVet and LLaVA-In-the-Wild.Source: §4.2 · p. 7
4.3. Complex Scenarios
- F-LMM performs reasoning segmentation without training on that task, excels on long-sentence cases, and achieves best zero-shot grounded-conversation results comparable to GCG-finetuned models.With VisCoT prompting, it outperforms VisCoT-7B and improves resistance to object hallucinations on POPE.Source: §4.3 · p. 7
4.4. Ablation Study
- The mask refiner adds a 12.6 performance gain on PNG; averaging multiword attention beats max by 0.7, and input normalization adds 0.5.Box and text prompts add 0.3 and 1.2, respectively; performance increases with SAM model size, with ViT-L chosen for cost-performance balance. The keyword selector beats SpaCy in F1 and precision while achieving comparable recall.Source: §4.4 · pp. 7–8, Table 5 · p. 7, Figure 5 · p. 7
4.5. Analysis & Visualisation
- Across ten models, stronger conversational ability correlates positively with grounding ability; attention-map pixels cluster roughly by object before the decoder and refiner produce masks.The analysis uses average ranks for chat and grounding benchmarks and visualizes the attention maps with KMeans clustering.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8
5. Conclusion
F-LMM grounds fully frozen LMMs by using spatial cues in self-attention with a CNN mask decoder and SAM-based refiner. It achieves competitive visual grounding without sacrificing pretrained conversational abilities, and shows promise on complex perception and reasoning tasks.
5. Conclusion
- F-LMM combines frozen LMMs with a CNN mask decoder and SAM-based refiner to deliver competitive grounding while preserving pretrained conversational abilities.The conclusion identifies reasoning segmentation, grounded conversation generation, and visual chain-of-thought reasoning as promising applications.Source: §5 · p. 8
A1. Appendix
The appendix points readers to expanded benchmark results, mask-decoder details, dataset summaries, visualizations, and the paper’s broader-impact and limitations discussions.
A1. Appendix
- The appendix covers expanded benchmark results, mask-decoder architecture, dataset summaries, visualizations, and broader-impact and limitations discussions.The listed visualizations include failure cases on general question-answering tasks and examples of reasoning segmentation, grounded conversations, and visual CoT.Source: A1. Appendix · p. 9
A1.1. Benchmark results
The appendix expands question-answering and grounding results across additional benchmarks and subsets. It also reports that segmenting plural object nouns is more challenging than singular nouns for all tested models.
A1.1. Benchmark results
- Expanded results cover more QA benchmarks, all RefCOCO(+/g) subsets, and PNG mask recalls by singular and plural nouns; plural nouns are harder for every tested model.The QA scores are zero or near zero for existing grounding LMMs, attributed in the passage to corrupted instruction-following abilities.Source: A1.1. Benchmark results · p. 9
A1.2. Mask Decoder
The mask decoder uses a 3-stage U-Net architecture with repeated downsampling and upsampling. It contains 8M parameters.
A1.2. Mask Decoder
- The mask decoder is based on a 3-stage U-Net [62] with feature maps downsampled and upsampled three times.It contains 8M parameters.Source: A1.2. Mask Decoder · p. 9
- Downsampling uses two convolutional layers with kernel sizes 2 and 1, respectively.Source: A1.2. Mask Decoder · p. 9
- Upsampling uses bilinear interpolation followed by two convolutional layers with kernel size 1.Source: A1.2. Mask Decoder · p. 9
A1.3. Dataset Usage
F-LMM uses RefCOCO and PNG for segmentation without additional grounded instruction-tuning datasets. The training text formats differ by dataset: PNG uses image narratives, while RES expressions are concatenated into a sentence.
A1.3. Dataset Usage
- F-LMM uses RefCOCO and PNG for segmentation without additional grounded instruction-tuning datasets.Source: A1.3. Dataset Usage · p. 9
- PNG examples format an image narrative as a user prompt followed by the model’s description, with grounding keywords annotated in the training set.Source: A1.3. Dataset Usage · p. 9
- For RES training, multiple referring expressions for an image are concatenated into a single sentence in PNG format.Source: A1.3. Dataset Usage · p. 9
A1.4. Visualisation
The visualizations illustrate F-LMM grounding conversational answers and question-relevant content, including in visual chain-of-thought examples. They also show attention-map differences across LMMs and provide detailed benchmark comparisons.
A1.4. Visualisation
- F-LMM examples show question answers grounded by localized masks, including reasoning segmentation and conversational phrases.In reasoning segmentation, the answer to an image question becomes the grounding target; in conversation examples, key phrases are localized.Source: A1.4. Visualisation · pp. 9–10
- The appendix includes detailed comparisons for question-answering, referring expression segmentation, and panoptic narrative grounding; the mask decoder visualization depicts a 3-stage U-Net [62].Source: Table A1 · p. 10, Table A2 · p. 10, Table A3 · p. 11
- Visual chain-of-thought examples use a grounded answer’s mask bounding box to crop an image region for the LMM’s final answer.The examples include locating document text such as a fax number, even though the mask head was not trained on VisCoT data.Source: A1.4. Visualisation · pp. 10–11
- DeepSeekVL-7B and LLaVA-1.5-7B attention maps show similar object shapes and locations but different patterns and textures; LLaVA’s attention is more concentrated with higher amplitudes.Source: A1.4. Visualisation · p. 11
A1.5. Broader Impact
The broader-impact discussion frames F-LMM as a way to combine visual grounding with general language capability, while noting possible social and inherited-model risks. The limitations section identifies inherited biases, untested modalities, and model-size constraints.
A1.5. Broader Impact
- F-LMM is presented as combining visual grounding with broad language capability, supporting applications that need both specialized multimodal skills and general language understanding.The discussion names assistive tools and interactive robotics as examples.Source: A1.5. Broader Impact · p. 11
- The authors identify potential bias propagation, human-labor displacement, and privacy or ethical concerns as possible negative impacts of visually grounding LMMs.Source: A1.5. Broader Impact · pp. 11–12
- The discussion proposes access controls, usage policies, and transparency about model capabilities, limitations, training, and datasets as safeguards against misuse.Source: A1.5. Broader Impact · p. 12
A1.6. Limitations
- Because F-LMM uses frozen pretrained LMMs, it inherits their biases and limitations.Source: A1.6. Limitations · p. 13
- The work focuses on vision-language interactions and does not explore video, audio, or 3D point clouds.Source: A1.6. Limitations · p. 13
- Experiments cover LMMs up to 8 billion parameters because of limited computing resources, leaving larger models unevaluated.Source: A1.6. Limitations · p. 13