PaperlayerView exact revision on arXiv

Source-linked AI summary

ActICL: Post-Training for Agentic Multimodal In-Context Learning

Yanshu Li, Ruixiang Tang, Junyi Jessy Li, Elias Stengel-Eskin

arXiv:2610.19475v1cs.CVcs.CL

TL;DR

Single-step retrieval can yield noisy or redundant multimodal ICL contexts, and prior context-aware methods do not train MLLMs to guide progressive retrieval according to their information needs. ActICL post-trains an MLLM agent to reason through iterative search, reading, curation, and context commitment, jointly optimizing acquisition and answering. ActICL gains 11.6 percentage points over standard similarity-based retrieval in adaptive-shot settings, averaged across seven benchmarks and three MLLMs, and beats ContextNav by 2.1 points on average for frontier models.

  • Problem

    Single-step retrieval can yield noisy or redundant multimodal ICL contexts, and prior context-aware methods do not train MLLMs to guide progressive retrieval according to their information needs.

    Source: §1 · pp. 1–2

  • Method

    ActICL post-trains an MLLM agent to reason through iterative search, reading, curation, and context commitment, jointly optimizing acquisition and answering.

    Source: §1 · p. 2

  • Results

    ActICL gains 11.6 percentage points over standard similarity-based retrieval in adaptive-shot settings, averaged across seven benchmarks and three MLLMs, and beats ContextNav by 2.1 points on average for frontier models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ActICL-trained agents can construct contexts for frontier models, extending context-acquisition benefits beyond the agent's own answering.

    Source: §4.2 · p. 8, §5 · p. 9

Abstract

from arXiv · show

Multimodal in-context learning (ICL) is an important capability of multimodal large language models (MLLMs). Its performance is closely tied to the demonstrations it uses, making demonstration retrieval a core problem for ICL. Common methods use fixed scoring methods to retrieve all the demonstrations at once, which may introduce redundancy and noise. Context-aware retrieval methods have emerged to address this problem by allowing incremental retrieval from a growing context. However, none of them lets the MLLM learn how to judge what new demonstrations it needs and then carry out progressive retrieval on its own. To this end, we introduce ActICL, which trains an MLLM agent to acquire and use an ICL context via tool-augmented reasoning. An ActICL agent analyzes its own information gaps during iterative reasoning and tool-use actions. Based on these gaps, it performs targeted retrieval to acquire a set of initial candidates. It can then use the existing context to further filter the candidates and refine the context, or it can iteratively perform a new retrieval. When the agent judges that the context is sufficient for the task, or when the resource limit is reached with the user request met, it sends the acquired context to an answering model, either the agent itself or another MLLM. We use a two-stage post-training procedure that jointly optimizes the agent's acquisition and answering. With an adaptive number of demonstrations, ActICL improves average performance by 11.6 percentage points over standard similarity-based retrieval across seven benchmarks and three MLLMs, and by 4.4 points compared to the strongest prior baseline. The contexts acquired by ActICL agents can also be used effectively by frontier models. When GPT-5.6-Sol performs 4-shot ICL, ActICL's contexts improve its performance by 3.5 points over standard retrieval, averaged over three benchmarks.

1 INTRODUCTION

Multimodal ICL depends on demonstrations that form a useful context, but single-step retrieval can produce noisy, redundant, or suboptimal examples. ActICL trains an MLLM agent to acquire and use context through iterative tool use, improving performance across benchmarks and models.

1 INTRODUCTION

  • Single-step retrieval scores demonstrations independently, which can yield noisy or suboptimal contexts and reduce diversity through dominant-feature shortcuts.Prior work also explores context-aware selectors and agentic workflows, but selectors can generalize poorly to unseen tasks and inference-time workflows do not learn from feedback what makes contexts useful.Source: §1 · p. 1
  • The central question is how to train an MLLM to adaptively acquire useful context according to its reasoning and evolving information needs.Source: §1 · p. 2
  • ActICL trains an MLLM agent to reason through search, read, curate, and solve actions, progressively retrieving, inspecting, revising, and committing demonstrations.Its two-stage post-training jointly optimizes acquisition and answering using rejection-sampling SFT and RL rewards combining answer quality with resource penalties.Source: §1 · p. 2
  • ActICL improves accuracy over standard similarity-based retrieval by 10.2 points in 4-shot and 11.6 points in adaptive-shot settings, averaged across three models and seven benchmarks.It exceeds ProMSA by 3.0 and 4.4 points, respectively.Source: §1 · p. 2

2 RELATED WORK

Prior work on multimodal agents uses tool-based reasoning to gather evidence, while multimodal ICL retrieval methods construct contexts largely through static criteria or test-time refinement. ActICL instead trains agents to reason about what to retrieve, inspect, and retain.

2 RELATED WORK

  • Multimodal search agents use trained policies and tool calls to gather evidence, and ActICL extends these frameworks to retrieving complete demonstrations for ICL.Examples include multi-turn search, grounding and cropping, and entity-based visual question answering.Source: §2 · p. 3
  • Prior ICL retrieval work uses static retrieval or context-aware refinement, but ContextNav's test-time reasoning does not learn context utility and filters similarity-retrieved candidates rather than guiding retrieval as a whole.ActICL trains MLLM agents to determine what to retrieve, inspect, and retain.Source: §2 · p. 3

3 METHOD: ACTICL

ActICL trains an MLLM agent to iteratively acquire and revise multimodal ICL context under resource constraints, then jointly optimizes acquisition and answering through SFT and RL.

3.1 OVERVIEW

  • An acquisition is an agent run that iteratively selects demonstrations from a bank; its retained set is supplied with the query and ICL instructions to an answering model.The answering model may be the agent's own MLLM or a different model.Source: §3.1 · p. 3

3.2 CONTEXT ACQUISITION PIPELINE

  • The search, read, curate, and solve tools respectively retrieve previews, reveal full demonstrations, add or remove retained examples, and commit an admissible context.The pipeline also tracks action and token budgets; a context must fit its budget and, for fixed-count requests, contain exactly the requested number of demonstrations.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 CONTEXT-AWARE RETRIEVAL

  • The agent formulates retrieval goals that target visual, textual, or joint information needs, using reference images and descriptions of desired visual content or demonstration tasks.Available channels match image-to-image, text-to-image, and text-to-text; rankings from enabled channels are combined by reciprocal-rank fusion.Source: §3.3 · p. 5

3.4.1 TRAINING DATA

  • Training uses 20k ICL instances spanning diverse multimodal tasks and demonstration banks of 512, 1024, or 4096 examples, sampled through task-matched, similarity-based, and random routes.Source: §3.4.1 · p. 5

3.4.2 REJECTION-SAMPLING SUPERVISED FINE-TUNING (SFT)

  • Rejection-sampling SFT retains valid trajectories with admissible contexts and fully correct answers, then jointly supervises acquisition and answering with token-level cross-entropy.Each backbone retains 4k trajectory–answer pairs from its own rollouts.Source: §3.4.2 · p. 6

3.4.3 JOINT REINFORCEMENT LEARNING (RL)

  • Joint RL rewards valid acquisitions according to mean answer quality minus context and acquisition token costs, while failed commitments receive a fixed negative return.Separate acquisition and answer advantages are computed; answer rewards use a mixed baseline, and the two objectives share parameters.Source: §3.4.3 · p. 6

4 EXPERIMENTS

Across seven benchmarks and three backbones, ActICL improves multimodal ICL with fixed-shot and adaptive context acquisition, and its trained models also strengthen reasoning over fixed interleaved image–text contexts. Ablations show benefits from both training stages and adaptive, context-aware retrieval.

4.1 SETUP

  • The evaluation uses three MLLM backbones and seven benchmarks spanning visual understanding, structured and quantitative reasoning, and context-dependent multimodal tasks.The benchmarks are disjoint from the training data, and test-time system prompts and benchmark ICL instructions remain fixed.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 4-shot ActICL beats all baselines across seven benchmarks and three backbones; adaptive acquisition adds 1.1–1.8 percentage points using an average of 3.54 shots.The fixed-shot gains average 8.2–11.4 points over MMICES and 2.8–3.2 points over ProMSA.Source: §4.2 · p. 7
  • ActICL achieves the best results across all backbones and settings on fixed 4- and 8-shot multimodal contexts, with a largest gain of 5.1 points over the strongest baseline.The comparison uses MMICES-built prompts on three multimodal ICL tasks and two multi-image understanding tasks.Source: §4.2 · p. 8, Table 2 · p. 8
  • ActICL-built 4-shot contexts lead across nine frontier-model and benchmark combinations, exceeding ContextNav by 0.8–3.1 percentage points.The contexts are constructed by ActICL-trained Qwen3.5-9B for larger open-source and frontier closed-source models.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 ABLATION STUDIES

  • SFT raises the average score from 0.582 to 0.657, while joint RL alone reaches 0.636; the full ActICL is best on every benchmark.Removing answering or acquisition from joint RL after SFT lowers the average by 2.5 or 2.0 points, respectively.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Baselines gain at most 1.9 points beyond four demonstrations and may decline, while ActICL improves at every tested count, including unseen counts.Adaptive acquisition uses 3.93 and 3.26 demonstrations on average, surpassing fixed 4-shot contexts on both models and matching eight shots on Qwen3.5-9B.Source: §4.3 · p. 9, Figure 3 · p. 9
  • Replacing the agent’s evolving goals with a fixed query-based goal lowers average score by 2.8 points; restricting retrieval to text or visual input lowers it by 1.6 or 1.8 points.Text-only retrieval loses 3.9 points on CharXiv, while visual-only retrieval loses 2.2 points on MathVision.Source: §4.3 · p. 9

5 CONCLUSION

ActICL post-trains MLLM agents to acquire and use multimodal ICL contexts through reasoning and tool use. The appendix details the retrieval, observation, and resource rules governing that process.

5 CONCLUSION

  • ActICL post-trains MLLM agents to acquire and use multimodal ICL contexts through reasoning and tool use.The trained agents can also construct contexts for frontier models.Source: §5 · p. 9

A ENVIRONMENT AND TOOL PROTOCOL

A.1 RETRIEVAL AND CANDIDATE PREVIEWS

  • Retrieval previews expose at most two images within 64 visual tokens and up to 128 input-text tokens for candidate screening.For multi-image inputs, the environment retains the first and last images; long text is represented by a prefix and suffix.Source: §A.1 · p. 16

A.2 TOOL EXECUTION AND OBSERVATIONS

  • Stable candidate numbers let the agent refer to records across read and curate actions, while observations preserve acquired examples, previews, history, and remaining resources.Accepted reads move a candidate from the preview pool to the acquired set; the context remains a subset of acquired demonstrations.Source: §A.2 · p. 16

A.3 RESOURCE ACCOUNTING AND COMMITMENT

  • The environment admits calls only when the complete input and reserved generation allowance fit the per-call window and cumulative acquisition budget.A committed context must fit Bctx, and the environment reserves enough capacity for a fresh answering call.Source: §A.3 · p. 17

B TRAINING DATA CONSTRUCTION

The training data construction prepares multimodal records, partitions them into disjoint pools, and samples training instances from those sources. It preserves test-time inputs and targets while withholding auxiliary supervision unavailable at test time.

B.1 SOURCE PREPARATION

  • The construction draws records from released training splits and Meta-Dataset meta-training classes, retaining original inputs, options, and targets while keeping auxiliary supervision offline.Sources cover knowledge and commonsense VQA, text-rich images, mathematical and diagram reasoning, and visual reasoning including multi-image queries.Source: §B.1 · p. 18

B.2 DEDUPLICATION AND POOL PARTITIONING

  • The pipeline removes exact duplicates and assigns each remaining record to one of the demonstration, training-query, or development pools.The 1,000 development queries are disjoint from training queries and bank records.Source: §B.2 · p. 18
  • BabyVision’s 388 questions are partitioned into 208 bank records, 140 training queries, and 40 development queries.One hundred training queries form 200 RL instances, with 40 accepted SFT pairs targeted.Source: §B.2 · p. 19

B.3 ICL INSTANCE CONSTRUCTION

ICL instances pair a query with a sampled bank and resource request, using multiple routes to supply demonstrations. Collection rollouts validate feasible configurations and provide candidates for rejection-sampling SFT.

B.3 ICL INSTANCE CONSTRUCTION

  • Each instance fixes its query, bank, count request, and resource tier before rollout, while its bank combines a 256-record core with a sampled background.Core routes include source-matched examples, input-similarity retrieval, and random cross-source sampling; duplicates are removed and route provenance stays hidden from the agent.Source: §B.3 · pp. 19–20, Table 7 · p. 20
  • The sampler jointly selects count requests and resource tiers, rejecting fixed-count configurations that fail minimum context or call-budget feasibility checks.For 15% of instances, visual concept binding uses two to six classes with random neutral labels inferred from demonstrations.Source: §B.3 · p. 20, Table 8 · p. 20

B.4 COLLECTION ROLLOUTS AND INSTANCE ACCEPTANCE

  • Collection rollouts certify feasible paths and supply SFT candidates; acceptance depends on a valid admissible path, not on the initial answer being correct.The resulting manifest contains 20,000 instances over approximately 10,000 distinct queries.Source: §B.4 · pp. 20–21

C REJECTION-SAMPLING SFT

  • Each backbone retains 4,000 SFT trajectory–answer pairs that meet tool-validity, resource-compliance, context-admissibility, and full-answer-score criteria.The pairs come from that backbone’s own collection rollouts and are limited to at most two per query.Source: C REJECTION-SAMPLING SFT · p. 21

D JOINT REINFORCEMENT LEARNING

Joint RL samples acquisition trajectories and answers under each instance’s count request and budgets. Its objective rewards answer quality while accounting for context and acquisition costs, with leave-one-out comparisons for both acquisition and answering.

D JOINT REINFORCEMENT LEARNING

  • For each instance, the frozen policy samples K acquisition trajectories and M fresh answers per valid context; failed trajectories receive no answers.The same policy generates acquisition and answering tokens, and K and M are at least two for leave-one-out baselines.Source: D JOINT REINFORCEMENT LEARNING · p. 21
  • The return bounds context size and cumulative acquisition cost alongside mean task score, while assigning failures a separate negative return.Choosing κ greater than λctx + λacq makes every valid return exceed the failure return.Source: D JOINT REINFORCEMENT LEARNING · pp. 21–22
  • A common ramp scales both cost weights so the policy first learns admissible contexts and is then pushed toward efficiency.The ramp depends on the fraction of the 20,000 instances sampled and stays fixed within each rollout batch.Source: D JOINT REINFORCEMENT LEARNING · p. 22
  • Acquisition advantages compare each path with the other trajectories for the same instance, including failed trajectories in the baseline.Answer advantages use leave-one-out means; the trainer does not apply further group centering or standard-deviation normalization.Source: D JOINT REINFORCEMENT LEARNING · p. 22
  • The clipped-surrogate training loop keeps the rollout snapshot, log probabilities, returns, and advantages fixed across four minibatch updates.A fixed Lref avoids inversely weighting longer trajectories, and the objective has no KL, entropy, or auxiliary retrieval-hit term.Source: D JOINT REINFORCEMENT LEARNING · pp. 22–23

E IMPLEMENTATION DETAILS

ActICL uses three Qwen backbones with shared acquisition and answering parameters, updating language parameters while freezing visual components. The appendix lists training settings and evaluation-pool partitioning.

E IMPLEMENTATION DETAILS

  • Three Qwen backbones share parameters between acquisition and answering; all language-backbone parameters are updated while visual encoders, mergers, and projections are frozen.Training uses AdamW on 8 NVIDIA H200 GPUs.Source: E IMPLEMENTATION DETAILS · p. 23
  • Training settings are listed in Tables 9 and 10, and each run stores tool schemas, index mappings, processor settings, and prompt templates.Source: E IMPLEMENTATION DETAILS · p. 23
  • Table 11 summarizes evaluation pools: benchmarks with released splits retain native bank/query boundaries, while others target a 6:4 bank/query ratio within each stratum.Source: Table 11 · p. 24

F.1 BENCHMARK POOLS AND PARTITIONS

The evaluation pools are constructed to prevent overlap and preserve benchmark-specific support conventions. For benchmarks without released splits, the authors stratify support/query partitions and combine eligible supports into a bank without exposing query-subcategory filters.

F.1 BENCHMARK POOLS AND PARTITIONS

  • For VL-ICL Bench image-to-text tasks, candidate records are checked against frozen training-bank, training-query, and development manifests, and exact duplicates are removed from evaluation.This includes evaluation-bank records released by a benchmark as a training split.Source: §F.1 · p. 24
  • Benchmarks without released splits target 60% support and 40% query records, stratified by subcategory and balanced for difficulty within subtype where feasible.Seed 42 is used; sparse secondary strata are balanced at their parent category, and small-stratum rounding may prevent an exact ratio.Source: §F.1 · p. 24
  • Exact duplicates are removed across bank/query boundaries for every benchmark, including those with released splits.Source: §F.1 · p. 24
  • Eligible supports are merged into one bank per benchmark subject to native task compatibility, with no query-subcategory filter, training-route quotas, or cross-source additions.CharXiv uses its labeled validation set because public test labels are unavailable; MathVision testmini is not added separately because it is contained in the full release.Source: §F.1 · p. 24
  • VL-ICL preserves each subtask’s admissible support construction and label/operator convention, and comparisons match the number of examples actually supplied.A native VL-ICL shot with two class-specific supports counts as two ActICL demonstrations.Source: §F.1 · p. 24

F.2 BASELINES

Acquisition baselines use the same evaluation banks and query partitions as ActICL, while additional baselines compare context construction and supplied-context fine-tuning. The comparisons align key evaluation conditions and report the best-performing fine-tuning data size.

F.2 BASELINES

  • All acquisition baselines share ActICL’s evaluation banks, query partitions, image processing, backbones, and answering prompt.Source: §F.2 · p. 24
  • Random sampling draws demonstrations uniformly, whereas MMICES selects them by multimodal similarity using the same SigLIP encoder as Appendix A.1.Source: §F.2 · p. 25
  • ContextNav uses Gemini3.5-Flash as its policy and passes its constructed context to the answering model.Source: §F.2 · p. 25
  • LSD trains a retriever with answering-model feedback, using ActICL’s data sources and LSD’s original data-construction method.Source: §F.2 · p. 25
  • ProMSA is adapted to multimodal ICL and trained with SFT and RL on ActICL’s data at comparable training cost; it answers with its own trained model.Source: §F.2 · p. 25
  • LoRA, DARA, and COMM strengthen supplied-context baselines; their 4- and 8-shot training sequences use retrieved interleaved examples and ground-truth query targets.The selected baseline configuration uses about 35% of ActICL’s training data, chosen from six data sizes capped at ActICL’s volume.Source: §F.2 · p. 25

F.3 SCORING AND INFERENCE

The evaluation uses benchmark-default metrics and greedy acquisition and answering, with failures retained in scoring. Separate experiments test fixed supplied contexts and ActICL-constructed contexts for external answerers.

F.3 SCORING AND INFERENCE

  • Each benchmark is scored with its released answer extraction and default metrics, including benchmark-specific VQA, reasoning, graph, math, discipline, and image-to-text measures.External judges are used only for evaluation in their released configurations.Source: §F.3 · p. 25
  • Main-text results average three random seeds; inference greedily acquires one context and generates one greedy answer in a fresh call.Acquisition failures remain in the denominator with zero task score rather than being excluded.Source: §F.3 · p. 25
  • Supplied-context tests disable acquisition and give the base backbone, LoRA, DARA, and ActICL identical MMICES-built 4- and 8-shot prompts on three benchmarks.MuirBench and BLINK are evaluated using their released interleaved image–text presentation.Source: §F.3 · p. 25
  • For external answerers on PerceptionBench, MindCube, and MMMU-Pro, Qwen3.5-9B constructs 4-shot contexts; answerer configurations are held fixed across context methods.The comparisons include ActICL, MMICES, and ContextNav.Source: §F.3 · p. 25

G ADDITIONAL ABLATION STUDIES AND ANALYSES

The ablations show how ActICL’s training choices, action set, and training-bank construction affect accuracy, efficiency, and robustness. The prompts and case trajectory illustrate how the agent identifies missing guidance, revises its context, and commits it under resource constraints.

G ADDITIONAL ABLATION STUDIES AND ANALYSES

  • Under four-shot requests, full SFT plus joint RL averages 0.680, versus 0.676 for SFT and 0.674 for joint RL alone; RL adds 0.4 points after SFT.The gain from joint RL after SFT is 5.2 points under adaptive requests, where the agent controls how many demonstrations to commit.Source: G ADDITIONAL ABLATION STUDIES AND… · p. 25, Table 12 · p. 26
  • After SFT, one-part RL lowers the average to 0.660 without acquisition and 0.669 without answering, consistent with interference from shared parameters.Removing acquisition lowers the average by 2.0 points, while removing answering lowers it by 1.1 points under four-shot requests.Source: Table 12 · p. 26, G ADDITIONAL ABLATION STUDIES AND… · p. 26
  • Removing read lowers average accuracy from 0.709 to 0.685; removing curate lowers it to 0.693, while add-only curate reaches 0.700.Read removal causes the largest loss on CharXiv (3.0 points); curate removal loses most on MME-R (2.0 points). Across main experiments, the agent retains 74.2% of demonstrations it reads and removes 0.85 per query on average.Source: Table 13 · p. 26, G ADDITIONAL ABLATION STUDIES AND… · p. 26
  • Removing resource costs changes average accuracy from 0.709 to 0.703 but raises committed demonstrations from 2.62 to 2.94 and acquisition tokens by 24%.Applying costs from the first rollout batch or removing the fixed reference each lowers average accuracy to 0.693; the cost schedule and fixed reference each contribute about 1.5 accuracy points.Source: Table 14 · p. 26, G ADDITIONAL ABLATION STUDIES AND… · pp. 26–27
  • On the mixed bank, ActICL scores 0.693 after a 1.6-point drop and remains above baselines evaluated on clean single-benchmark banks.The three-route bank construction combines the accuracy and robustness strengths of the Source-only and Random variants; ActICL leads on every benchmark in both bank settings.Source: Table 15 · p. 27, G ADDITIONAL ABLATION STUDIES AND… · p. 27
  • When the bank lacks examples for a needed convention or operation combination, the agent must infer the missing rule; query-detail misreads likewise cannot be fixed by adding demonstrations.The agent cannot retrieve evidence absent from its supplied bank, and more examples of the same procedure do not supply query-specific visual evidence.Source: G ADDITIONAL ABLATION STUDIES AND… · pp. 27–28

H PROMPTS AND INPUT CONSTRUCTION

  • The prompts direct the agent to search for missing guidance, read promising previews, curate useful nonredundant examples, and solve when the context is admissible and further acquisition is not worth its cost.Search channels are matched to the missing guidance; admissibility requires fitting the context-token budget and, for fixed-count requests, retaining exactly the requested number of demonstrations.Source: H PROMPTS AND INPUT CONSTRUCTION · p. 28

I CASE TRAJECTORY

  • In the chart-query trajectory, the agent searches for percentage-growth guidance, reads a candidate, replaces a subsumed demonstration, then commits the revised context for answering.The committed context contains demonstrations 1 and 7 in first-read order.Source: I CASE TRAJECTORY · p. 31

Related papers

Loading 2610.19475v1…