PaperlayerView exact revision on arXiv

Source-linked AI summary

MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning

Zirui Cheng, Xun Xu, Tiankai Chen, Fady Rezk, Bowen Zheng, Xiaodong Shi, Shijie Li, Kangkang Lu, Bharadwaj Veeravalli, Nancy F. Chen

arXiv:2608.12724v1cs.LG

TL;DR

Few-shot multimodal ICL depends on selecting relevant, well-covered demonstrations, yet labeled examples are scarce and abundant unlabeled data is difficult to exploit. MAG uses graph-based relevance propagation to select unlabeled samples for pseudo-labeling before multimodal demonstration selection, and consistently outperforms labeled-data retrieval across eight benchmarks with limited pseudo-labeling.

  • Problem

    Few-shot multimodal ICL requires high-quality, relevant, and well-covered demonstrations, but labeled data is scarce while unlabeled multimodal data is difficult to exploit.

  • Method

    MAG uses two-stage graph-based relevance propagation to select a compact unlabeled subset for pseudo-labeling, then combines visual and textual relevance to choose demonstrations.

  • Results

    Across eight multimodal benchmarks, MAG consistently outperforms methods retrieving from labeled data alone, with substantial gains under limited pseudo-labeling budgets.

  • Takeaways & Limitations

    Unlabeled multimodal data can support more robust and informative demonstration construction for multi-modal ICL when selected through relevance-guided pseudo-labeling.

Abstract

from arXiv · show

Few-shot in-context learning (ICL) with multi-modal large language models (MLLMs) enables task adaptation without parameter updates, but its performance is highly sensitive to the quality and coverage of the selected demonstrations. While unlabeled multi-modal data is abundant, it remains elusive how to exploit them for ICL. We propose MAG (MAnifold-Guided semi-supervised in-context demonstra- tion selection), an efficient framework that leverages unlabeled data to improve multi-modal ICL. MAG formulates demonstration selection as a semi-supervised propagation problem on a multi-modal graph and adopts a two-stage strategy: (i) relevance score propagation identifies a compact set of high-impact unlabeled samples for pseudo-labeling, reducing MLLM inference cost; (ii) multi-modal relevance is used to select the final demonstrations. We show that textual represen- tations are more effective for relevance propagation, while both visual and textual modalities are crucial for high-quality demonstration selection. Experiments on eight multi-modal benchmarks demonstrate that MAG consistently outperforms strong baselines in label-scarce regimes, achieving significant gains with a limited pseudo-labeling budget.

1 Introduction

MAG addresses label scarcity in few-shot multi-modal ICL by using abundant unlabeled multi-modal data to construct more informative demonstrations. Its training-free, two-stage graph-based framework efficiently identifies pseudo-labeling candidates and selects demonstrations using complementary modality roles.

  • Motivation: Label scarcity limits few-shot multi-modal ICL, whereas abundant unlabeled multi-modal data offers a resource for constructing more informative and robust demonstrations.The paper motivates a semi-supervised formulation to exploit unlabeled visual samples paired with textual data when labeled interactions or task data are limited.
  • Method: MAG uses two stages: graph-based relevance propagation first selects a compact query-relevant unlabeled subset for pseudo-labeling, then multi-modal relevance selects final demonstrations.This design balances scalability with demonstration quality while reducing the need for costly MLLM inference over all unlabeled samples.
  • Method: Textual descriptions provide stable relevance propagation, while combining visual and textual modalities is essential for high-quality final demonstration selection.Late fusion is more effective in the second stage because it better supports fine-grained alignment between demonstrations and the query.
  • Method: MAG is learning-free and avoids fine-tuning or auxiliary-model training, enabling rapid deployment across diverse tasks.The method is designed as a training-free alternative to learning-based approaches.
  • Results: Across 8 diverse benchmarks, MAG delivers consistent and substantial gains, with particularly large improvements on reasoning-intensive tasks.These results support leveraging unlabeled multi-modal data for semi-supervised demonstration construction.

2 Related Work

Related work covers in-context learning, multi-modal in-context learning, and semi-supervised label propagation. These lines of research motivate improving demonstration selection and exploiting similarity graphs with unlabeled data.

  • In-Context Learning (ICL): ICL adapts large language models to new tasks by conditioning on demonstrations without parameter updates.Prior work studies ICL as implicit gradient descent or Bayesian inference, and examines demonstration selection, ordering, and formatting.
  • Multi-Modal In-Context Learning: Vision-language models established shared representation spaces that enable multi-modal ICL with systems including Flamingo, GPT-4V, and LLaVA.MM-ICL performance remains highly sensitive to demonstration selection, motivating learning-free methods based on improved retrieval.
  • Semi-Supervised Learning and Label Propagation: Label propagation diffuses information from labeled to unlabeled samples over similarity graphs under the assumption that nearby samples share labels.Modern variants use deep representations, efficient diffusion on region manifolds, and pseudo-labeling in deep learning.

3 Methodology

MAG treats demonstration selection under label-scarce conditions as a two-stage semi-supervised graph-propagation problem. It first expands the candidate pool with relevant pseudo-labeled samples, then uses complementary visual and textual graphs for query-conditioned selection.

  • Stage 1: Manifold-Guided Pseudo-Labeling: Stage 1 generates image descriptions, combines them with task text, and builds a textual k-NN manifold graph over labeled and unlabeled samples.Textual representations are produced with the MLLM and encoded using a pre-trained text encoder.
  • Stage 2: Multi-Modal Demonstration Selection: Stage 2 constructs separate visual and textual k-NN graphs over DE, propagates query-conditioned relevance from each modality, and fuses scores with β to rank demonstrations.The query initializes each modality’s relevance vector from its k nearest neighbors, after which the top-k candidates are provided to the MLLM.
  • Overview: MAG selects effective in-context demonstrations when labeled data are scarce by combining manifold-guided pseudo-labeling with query-conditioned multi-modal graph selection.The first component expands the candidate pool while controlling cost; the second selects relevant demonstrations from it.
  • Stage 1: Manifold-Guided Pseudo-Labeling: Relevance propagation initializes labeled nodes at 1 and unlabeled nodes at 0, then iteratively spreads scores using normalized adjacency with propagation strength α.The normalized adjacency is ˆW (t) = (D(t))−1/2W (t)(D(t))−1/2.
  • Stage 1: Manifold-Guided Pseudo-Labeling: The top-K unlabeled samples by relevance are pseudo-labeled with the MLLM and combined with labeled data to form the expanded candidate pool DE.Sparse graph structure supports scalable score propagation over very large unlabeled pools.
  • Efficient Incremental Inference: For incremental inference, MAG removes the previous query and connects the new query to the top K nodes, requiring inner products with only |DE| samples.This avoids recomputing k-nearest neighbors for all graph nodes.

4 Experiments

Across eight multimodal benchmarks, MAG improves few-shot in-context learning by exploiting unlabeled data through manifold-guided pseudo-labeling and graph-based demonstration selection. Ablations show that graph propagation and complementary visual-textual information are central to its performance.

  • Experimental Setup: MAG evaluates eight benchmarks spanning visual emotion recognition, scene text understanding, visual reasoning, and visual question answering.Each dataset uses 15 labeled examples, 585 unlabeled candidates, and 300 test samples.
  • Main Results: With only 15 labeled samples, MAG consistently outperforms all baselines across the eight benchmarks.Against MMICES, MAG gains +30.5% on CLEVR, +7.1% on MMStar, and +25.1% on TextOCR.
  • Main Results: MAG also improves over standard few-shot prompting by +23.8% on TextOCR, +30.8% on CLEVR, and +15.3% on EmoSet.These gains are attributed to relevance-guided pseudo-labeling and graph propagation.
  • Ablation Studies: Graph propagation outperforms similarity retrieval, surpassing Top-K+MDL by +34.3% on CLEVR and +27.1% on TextOCR.The results support modeling global relational structure rather than relying only on local similarity.
  • Ablation Studies: Manifold-guided pseudo-labeling is critical: EmoSet reaches 76.3%, versus 72.0% without pseudo-labeling and 73.1% with random pseudo-labels.Random pseudo-labeling can also degrade performance, producing 54.7% on GQA versus 56.4% without it.
  • Modality Analysis: Both modalities are necessary: text-only graphs reduce GQA to 44.2% versus 58.0%, while visual-only graphs reduce EmoSet to 60.2% versus 76.3%.Textual representations nevertheless achieve the best performance on 6 of 8 datasets, while visual representations perform slightly better on CLEVR and OKVQA.

5 Conclusion … B Additional Experimental Analysis

MAG leverages unlabeled multi-modal data through graph-based relevance propagation, enabling efficient pseudo-labeling and query-conditioned demonstration selection in label-scarce settings. Its procedure constructs textual and visual representations to expand the demonstration pool and produce predictions.

  • 5 Conclusion: MAG models demonstration selection as relevance propagation on multi-modal graphs to exploit unlabeled data under label-scarce conditions.The framework identifies high-impact unlabeled samples for pseudo-labeling and selects globally relevant demonstrations across visual and textual modalities.
  • 5 Conclusion: MAG identifies a compact set of high-impact unlabeled samples for pseudo-labeling before selecting query-conditioned demonstrations.This two-stage design enables efficient use of unlabeled data while combining visual and textual relevance.
  • A Algorithm Psuedo Code: Algorithm 1 summarizes expanded-pool construction, final query-conditioned selection, and prediction.The algorithm takes labeled data, unlabeled data, and a query as input and outputs a predicted answer.
  • A Algorithm Psuedo Code: MAG generates textual descriptions for every labeled and unlabeled image before building a textual graph over both datasets.Text embeddings support Stage 1 relevance scoring for unlabeled samples.
  • A Algorithm Psuedo Code: Stage 1 pseudo-labels selected unlabeled samples with the MLLM and adds them to the expanded pool alongside labeled data.The algorithm computes relevance scores for unlabeled samples, pseudo-labels selected examples, and forms DE = DL ∪ DP.
  • A Algorithm Psuedo Code: Stage 2 builds visual and textual graphs over the expanded pool for multi-modal query-conditioned selection.The visual graph uses image embeddings, while the textual graph uses text embeddings.

B.1 Multi-Modal Backbone Analysis

MAG remains robust across seven multi-modal backbones and eight benchmarks, but performance varies substantially with backbone capabilities. Models balancing cross-modal perception and reasoning, especially Gemini-2.0-Flash and GPT-5, produce the most consistent results.

  • Overall trends: MAG shows substantial performance variance across seven backbones evaluated on eight benchmarks, although every model benefits from MAG.The variation reflects differences in multi-modal reasoning and perception capabilities.
  • Balanced vs. specialized performance: Gemini-2.0-Flash delivers the most balanced results across perception-heavy tasks such as TextOCR and CLEVR and reasoning-intensive benchmarks including GQA and OKVQA.Gemini-3.0-Flash instead excels on MMStar and CLEVR but drops notably on TextOCR, indicating a perception–recognition trade-off.
  • Effect of backbone capability: GPT-5 has clear advantages on high-level reasoning tasks such as GQA and OKVQA while remaining competitive on CLEVR and MatchingMI.GPT-4o and GPT-4.1 are limited on multi-modal understanding tasks including MMStar and TextOCR.
  • Open-source vs. proprietary models: Qwen-VL variants are competitive on MatchingMI and GQA but unstable on fine-grained semantic tasks, with Qwen-VL-Plus notably dropping on Emotion6.The results indicate that emotion recognition remains challenging for some open-source models.
  • Key takeaway: MAG’s effectiveness is amplified by backbones with strong cross-modal alignment and reasoning, particularly models balancing perception and reasoning such as Gemini-2.0-Flash and GPT-5.These backbones tend to yield the most consistent gains across diverse benchmarks.

B.2 Scalability Analysis

MAG scales through sparse graph propagation rather than dense matrix inversion, with computational complexity typically ranging from O(n log n) to O(n^1.5) and memory consumption of O(nk). Empirical measurements show near-linear latency and memory growth as sample counts increase, supporting scalability to larger unlabeled datasets.

  • Computational complexity: MAG solves a sparse linear system because its k-nearest-neighbor affinity matrix contains mostly non-zero edges, avoiding dense matrix inversion.The main computational cost is solving the system involving (I −α ˆW (m)).
  • Computational complexity: Iterative methods and sparse Cholesky decomposition yield complexity typically ranging from O(n log n) to O(n^1.5), below dense inversion’s O(n^3).The cited methods include Conjugate Gradient and Lanczos.
  • Memory complexity: Memory consumption scales as O(nk) because only non-zero graph edges are stored.This linear scaling follows from the sparse affinity structure.
  • Empirical validation: Theoretical and empirical results demonstrate that MAG remains efficient and scalable for larger unlabeled datasets.Empirical measurements are reported in Table 6.
  • Empirical validation: As sample counts increase, graph edges grow linearly, latency increases approximately linearly, and memory usage shows nearlinear growth with minor implementation overhead.Doubling the samples roughly doubles runtime, confirming that the method avoids cubic complexity.

B.3 Pseudo-Label Accuracy Analysis

Pseudo-label accuracy is strongly positively correlated with final performance, and the proposed selection strategy outperforms random selection on most datasets. Modality-specific disentanglement and late fusion improve robustness to cross-modal inconsistencies, semantic gaps, and noisy inputs.

  • Pseudo-label selection: Pseudo-label accuracy shows a strong positive correlation with final performance.The paper presents a quantitative comparison of pseudo-label selection strategies in Table 7.
  • Pseudo-label selection: In 4 out of 5 datasets, the proposed method outperforms random selection in both pseudo-label quality and downstream performance.This supports the effectiveness of the proposed selection strategy.
  • Robustness to Real-World Noise: Modality-specific disentanglement processes each modality independently, preventing noise in one modality from contaminating the other during propagation.The method addresses cross-modal inconsistencies such as low-quality images.
  • Robustness to Real-World Noise: Across seven datasets, late fusion consistently outperforms alternative designs by selectively integrating reliable steady-state representations from each modality.This design improves robustness to semantic gaps and noisy inputs.

C Qualitative Analysis · C.1 Comparison with Baselines

Figure 4 qualitatively compares MAG with VICL, MMICES, and CVR-LLM across four multimodal tasks, showing consistently more accurate predictions through semantically relevant, graph-propagated demonstrations.

  • C.1 Comparison with Baselines: MAG outperforms strong baselines qualitatively across emotion recognition, visual QA, reasoning, and OCR.The comparison covers EmoSet, GQA, MMStar, and TextOCR.
  • C.1 Comparison with Baselines: MAG identifies subtle affective cues such as disgust, whereas baselines produce generic interpretations.
  • C.1 Comparison with Baselines: Graph-based selection helps MAG reject reasoning options that are linguistically plausible but visually inconsistent.
  • C.1 Comparison with Baselines: MAG accurately recognizes scene text in the OCR comparison, including the visible text “REINICIO.”
  • C.1 Comparison with Baselines: The qualitative comparison evaluates MAG against VICL, MMICES, and CVR-LLM.
  • C.1 Comparison with Baselines: MAG’s improved predictions result from selecting semantically relevant demonstrations through graph-based propagation.

C.2 Ablation Visualizations

Qualitative ablations show that randomizing either pseudo-label selection or demonstration selection weakens semantic alignment and task performance. The full MAG pipeline instead selects coherent demonstrations and supports accurate, robust multi-modal reasoning.

  • Ablation setup: Figures 5–7 compare full MAG against randomized Stage 1 pseudo-label selection and randomized Stage 2 demonstration selection.The visualizations cover EmoSet, GQA, MMStar, and OCR tasks.
  • Stage 1 ablation: Randomizing Stage 1 produces less semantically aligned demonstrations, less stable emotion recognition, weaker objectcentric reasoning, and more corrupted OCR outputs.Random selection also reduces incorporation of informative unlabeled samples.
  • Stage 2 ablation: Randomizing Stage 2 introduces noisy visual-text associations that harm multi-choice reasoning and scene-text understanding.Replacing graph-based demonstration selection with random selection further degrades performance.
  • Full model: The full model consistently selects semantically coherent demonstrations and produces accurate predictions across all task types.Figure 5 visualizes the full MAG method across EmoSet, GQA, MMStar, and OCR tasks.
  • Ablation conclusion: Both relevance-guided pseudo-labeling in Stage 1 and graph-based demonstration selection in Stage 2 are essential for robust multi-modal reasoning.The qualitative ablations verify the contribution of both selection stages.

D Evaluation Protocol · E Prompts for ICL

The evaluation uses exact ground-truth containment in generated responses, with average accuracy reported across samples. ICL prompts are task-specific, including a constrained visual emotion recognition prompt for EmoSet and Emotion6.

  • D Evaluation Protocol: D Evaluation Protocol: Predictions are correct when the ground-truth answer y appears in the model response r.The criterion is defined as exact containment of the ground-truth text in the generated response.
  • D Evaluation Protocol: D Evaluation Protocol: The evaluation assigns 1 if y ⊆ r and 0 otherwise.This binary rule formalizes correctness using containment.
  • D Evaluation Protocol: D Evaluation Protocol: A response is incorrect when it does not contain the ground-truth answer as a substring.Correctness depends on substring containment rather than exact response matching.
  • D Evaluation Protocol: D Evaluation Protocol: Results are reported as average accuracy over all evaluation samples.The reported metric aggregates correctness across the complete evaluation set.
  • D Evaluation Protocol: D Evaluation Protocol: The containment protocol supports open-ended multi-modal generation with additional explanatory text.A response can include extra explanation while remaining correct if it contains the answer.
  • E Prompts for ICL: E Prompts for ICL: The paper specifies the prompts used for in-context learning across different tasks.The prompt descriptions are organized by task.
  • E Prompts for ICL: E Prompts for ICL: For visual emotion recognition on EmoSet and Emotion6, the prompt requests only an emotion category from a provided label list.It asks the model to predict the image’s emotion category using the image and its description, without extra commentary or chattiness.
Loading 2608.12724v1…