PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Policy Internalization for Conversational Agents

Zhenhailong Wang, Jiateng Liu, Amin Fazel, Ritesh Sarkhel, Xing Fan, Xiang Li, Chenlei Guo, Heng Ji, Ruhi Sarikaya

arXiv:2510.09474v2cs.CLcs.AI

TL;DR

Lengthy, reasoning-intensive policies can raise prompt costs and challenge consistent adherence, while multimodal policy internalization remains unexplored. The paper introduces MPI, datasets, and TriMPI; TriMPI improves benchmark accuracy, generalization to policy updates, and robustness to forgetting.

  • Problem

    Long, reasoning-intensive in-context policies impose substantial fixed prompt costs and can be difficult to follow consistently, while multimodal policy internalization has not been explored.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MPI, two datasets spanning synthetic decision-making and real-world tool use, and TriMPI’s three-stage training framework with policy-grounded PolicyRollout.

    Source: §1 · p. 2

  • Results

    TriMPI achieves 70.7% and 79.4% absolute gains over the CoT SFT baseline and in-context setting, respectively, and improves policy-update generalization and robustness to forgetting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The two benchmarks support MPI training and evaluation across analytical and real-world policy settings.

    Source: §1 · p. 2

    The best-performing model still exhibits perception and reasoning errors on ClevrPolicy.

    Source: §5.6 · p. 9

Abstract

from arXiv · show

Modern conversational agents like ChatGPT and Alexa+ rely on predefined policies specifying metadata, response styles, and tool-usage rules. As these LLM-based systems expand to support diverse business and user queries, such policies, often implemented as in-context prompts, are becoming increasingly complex and lengthy, making faithful adherence difficult and imposing large fixed computational costs. With the rise of multimodal agents, policies that govern visual and multimodal behaviors are critical but remain understudied. Prior prompt-compression work mainly shortens task templates and demonstrations, while existing policy-alignment studies focus only on text-based safety rules. We introduce Multimodal Policy Internalization (MPI), a new task that internalizes reasoning-intensive multimodal policies into model parameters, enabling stronger policy-following without including the policy during inference. MPI poses unique data and algorithmic challenges. We build two datasets spanning synthetic and real-world decision-making and tool-using tasks and propose TriMPI, a three-stage training framework. TriMPI first injects policy knowledge via continual pretraining, then performs supervised finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement learning extension that augments rollouts with policy-aware responses for grounded exploration. TriMPI achieves notable gains in end-to-end accuracy, generalization, and robustness to forgetting. As the first work on multimodal policy internalization, we provide datasets, training recipes, and comprehensive evaluations to foster future research. Project page: https://mikewangwzhl.github.io/TriMPI.

1 INTRODUCTION

Long, reasoning-intensive policies can impose substantial prompt costs and be difficult for models to follow consistently. The paper introduces multimodal policy internalization, two datasets, and TriMPI to train policy-compliant behavior without providing the policy at inference.

1 INTRODUCTION

  • 20× to 250× higher input-token costs arise from policy prompts relative to user queries, and models can struggle to follow expanding, reasoning-intensive policies consistently.Source: §1 · pp. 1–2
  • MPI targets policy-compliant multimodal responses without an in-context policy, addressing a gap left by prompt compression and text-only policy-alignment research.Source: §1 · p. 2
  • ClevrPolicy supports controlled study of complex, multi-hop decision policies, while GTAPolicy covers real-image tool-use instructions in a low-data regime.Source: §1 · p. 2
  • TriMPI combines policy-context continual pretraining, chain-of-thought supervised finetuning, and reinforcement learning with PolicyRollout’s policy-grounded rollout augmentation.Source: §1 · pp. 2–3
  • TriMPI achieves 70.7% and 79.4% absolute gains over the CoT SFT baseline and in-context setting, respectively, while improving generalization to policy updates and robustness to forgetting.Source: §1 · p. 3

2 PROBLEM FORMULATION

MPI frames multimodal agent behavior as producing a response to text and visual inputs under a predefined policy. Internalization embeds policy knowledge in model parameters so the policy need not be present in the inference prompt.

2 PROBLEM FORMULATION

  • MPI maps a text query and visual inputs to a textual response, which may be natural language or structured tool-calling code.Source: §2 · p. 3
  • The policy context can include both textual and visual instructions governing the response.Source: §2 · p. 3
  • Internalization places policy knowledge in model parameters to support policy-compliant generation without the policy in-context at inference.Source: §2 · p. 3

3 DATASET CREATION

The section introduces ClevrPolicy and GTAPolicy as benchmarks for multimodal policy internalization, then evaluates off-the-shelf models with policies included in-context.

3.1 CLEVRPOLICY DATASET

  • ClevrPolicy uses visual decision-making tasks with binary decision trees converted into natural-language policies, with tree depth controlling policy complexity.ClevrPolicy-T uses text-only policies; ClevrPolicy-M includes image demonstrations in decision-node conditions.Source: §3.1 · p. 4

3.2 GTAPOLICY DATASET

  • GTAPolicy targets tool use with real-world images and queries in a low-data regime, using policies built from descriptions of 13 tools and 24 tool-calling rules.Its single-turn tasks include visual inputs, a user profile, query, interaction history, and an expected JSON-formatted tool call.Source: §3.2 · p. 4

3.3 ZERO-SHOT IN-CONTEXT RESULTS ON CLEVRPOLICY AND GTAPOLICY

  • 60% was the best tool accuracy on GTAPolicy, while models struggled as ClevrPolicy policy depth increased, particularly Claude-4 at N=6 on ClevrPolicy-M.The in-context evaluation uses the policies directly in the inference prompts.Source: §3.3 · p. 5

4 MULTIMODAL POLICY INTERNALIZATION (MPI) ALGORITHMS

The section compares SFT baselines with TriMPI, a three-stage framework for policy internalization, and details PolicyRollout’s use of policy-aware samples during reinforcement-learning rollouts.

4.1 BASELINES

  • Direct SFT learns input-to-answer mappings without intermediate reasoning, while CoT SFT trains on data that reasons over policy rules before answering.The section positions these as baselines for multimodal policy internalization.Source: §4.1 · p. 5

4.2 TRIMPI: A THREE-STAGE TRAINING STRATEGY FOR MPI

  • TriMPI combines visually masked continual pretraining, CoT supervised finetuning, and reinforcement learning to inject policy knowledge before policy-free inference.VM-CPT trains on concatenated policy, visual-input, query, reasoning, and answer sequences while excluding visual tokens from the next-token loss.Source: §4.2 · pp. 5–6
  • The RL stage uses GRPO or DAPO with format and accuracy rewards; the authors identify insufficient policy-grounded exploration as a challenge for complex policies.Adding the policy during training alone is described as ineffective because it creates a mismatch with inference, where the policy is absent.Source: §4.2 · p. 6

4.3 POLICYROLLOUT (PORO): POLICY-AWARE REINFORCEMENT LEARNING

  • PolicyRollout augments no-policy responses with policy-conditioned responses for group-based advantage estimation, while applying the policy gradient only to the no-policy path.The figure caption says this enables more policy-aware exploration without a training–inference gap, with gains especially on complex policies.Source: §4.3 · pp. 6–7, Figure 5 · p. 6
  • Table 2 reports significant TriMPI improvements over in-context and SFT baselines, with competitive or stronger DAPO performance using fewer update steps.The table covers main results and ablations; its metrics are percentages.Source: Table 2 · p. 7

5 EXPERIMENTS

TriMPI is evaluated on task performance, efficiency, policy updates, policy knowledge, and retention of general reasoning. The experiments report gains over baselines alongside remaining failure modes.

5.1 MAIN RESULTS

  • 70.7% and 79.4% absolute gains in accuracy over the CoT SFT baseline and in-context setting, respectively, were achieved by the best-performing model.Qualitative examples show better alignment in intermediate thoughts and final answers; gains were more pronounced on complex policies and persisted on 3B models.Source: §5.1 · pp. 7–8
  • 93.9% fewer prompt tokens and 85.7% lower prefill inference time were observed after policy removal on Qwen2.5-VL-7B.Figure 6 reports prompt-token and prefill-time metrics before and after internalization.Source: Figure 6 · p. 8, §5.1 · p. 8

5.2 POLICY OVERRIDE: EVALUATING GENERALIZATION TO POLICY UPDATES

  • TriMPI consistently outperforms baselines and ablated settings when unseen policy updates are supplied in-context at inference.The override setting changes ClevrPolicy conditions or GTAPolicy tool-calling rules.Source: Table 3 · p. 8, §5.2 · pp. 8–9

5.3 POLICY REFERRAL: EVALUATING POLICY KNOWLEDGE INJECTION

  • TriMPI achieves more accurate referral to the original policy in intermediate reasoning, indicating that it internalizes policy knowledge beyond end-task behavior.The evaluation uses a judge to score policy referral on a subset of 100 test responses from each dataset.Source: Table 3 · p. 8, §5.3 · p. 9

5.1 MAIN RESULTS

  • The RL stage contributes most improvement over SFT baselines; VM-CPT helps both SFT and RL, and PolicyRollout adds gains over GRPO and DAPO.The RL stage also better leverages non-CoT data, while VM-CPT's effect is more pronounced in RL through more grounded exploration.Source: §5.1 · p. 8

5.4 POLICY IN-CONTEXT: EVALUATING DISTINCTIVE IMPACT ON POLICY FOLLOWING

  • TriMPI still outperforms baselines when the policies are placed back into the inference context.This setting tests whether the methods retain distinctive policy-following performance when policy inputs are present.Source: Table 4 · p. 9, §5.4 · p. 9

5.5 EVALUATING ROBUSTNESS TO CATASTROPHIC FORGETTING

  • TriMPI consistently preserves strong general reasoning abilities across settings and achieves the best overall performance after internalization.Baselines degrade significantly on GTAPolicy but retain strong performance on ClevrPolicy; the paper attributes the discrepancy to GTAPolicy's small dataset and overfitting risk.Source: §5.5 · p. 9

5.6 ERROR ANALYSIS

  • 7.07 Average Absolute Difference for TriMPI, versus 38.72 and 39.74 for GRPO and the SFT baseline, marks a smaller Condition-ID discrepancy.Remaining errors include missed occluded objects, hallucinated Conditions, and misuse of interaction history when selecting tools.Source: §5.6 · pp. 9–10

6 RELATED WORK

The related work covers prompt compression and deliberative alignment, then connects PolicyRollout to rollout-diversification methods for multimodal reinforcement learning.

6.1 PROMPT COMPRESSION AND DELIBERATIVE ALIGNMENT

  • Prompt compression targets templates and demonstrations requiring little reasoning, while deliberative alignment internalizes safety-specification knowledge and emphasizes reasoning over that knowledge.MPI extends this high-level motivation to multimodal models and conversational-agent tasks beyond safety.Source: §6.1 · p. 10

6.2 IMPROVING GRPO FROM THE ROLLOUT PERSPECTIVE

  • PolicyRollout augments RL rollouts with in-context policy guidance for grounded exploration, rather than altering visual inputs as in NoisyRollout and R1-ShareVL.The paper presents this as a general approach to adding guidance during RL while retaining the original optimization task.Source: §6.2 · p. 10

7 CONCLUSION AND LIMITATIONS

The conclusion presents MPI, two benchmarks, and TriMPI, while identifying dataset diversity, continual pretraining, and mixed response formats as limitations.

7 CONCLUSION AND LIMITATIONS

  • The paper identifies scaling to more diverse real-world images and tasks, improving continual pretraining beyond visual-token masking, and handling mixed response formats as remaining limitations.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The supplementary material includes a source-code archive and instructions for reproducing the results.

8 REPRODUCIBILITY STATEMENT

  • The supplementary material provides an anonymous source-code archive with training and evaluation instructions for reproducing the results.Dataset creation, evaluation metrics, policy examples, implementation details, and Policy Referral judge prompts are also specified by section or figure.Source: §8 · p. 11

9 ETHICS STATEMENT

The ethics statement describes the research scope and data, while the appendix reports limited uses of language models in writing and coding.

9 ETHICS STATEMENT

  • The work uses publicly available datasets and no human subjects or private user data; GTAPolicy uses fully synthesized user profiles.The authors say code and data will be made publicly available for research purposes in the near future.Source: §9 · p. 11

APPENDIX

  • Table 5 summarizes training hyperparameters by stage, with Tables 6 and 7 providing RL-step counts and exact dataset sizes.BS denotes per-device batch size multiplied by number of devices; for RL it is the update batch size.Source: Table 5 · p. 15

A USE OF LARGE LANGUAGE MODELS

B IMPLEMENTATION DETAILS

The implementation details describe the model checkpoints, policy construction, and training and testing data for ClevrPolicy and GTAPolicy.

B IMPLEMENTATION DETAILS

  • The listed Qwen2.5-VL checkpoints are 3B and 7B; the Claude versions are claude-3-7-sonnet-20250219 and claude-sonnet-4-20250514.RL uses rollout batch sizes of 384 for ClevrPolicy and 256 for GTAPolicy; SFT uses four H100 GPUs, while CPT and RL use eight.Source: B IMPLEMENTATION DETAILS · p. 15

C.1.1 POLICY GENERATION IN CLEVRPOLICY

  • ClevrPolicy policies are binary decision trees whose visual conditions lead to response outcomes and require reasoning across textual policy sections.True-edge child nodes avoid reusing the parent’s attribute type; the resulting policies have 4, 16, and 55 unique target responses at N = 2, 4, and 6, respectively.Source: §C.1.1 · pp. 15–16
  • ClevrPolicy-M can express a decision-node attribute value with an image crop rather than natural language, adding difficulty to policy following.Source: §C.1.1 · p. 16

C.1.2 TRAINING DATA CREATION IN CLEVRPOLICY

  • ClevrPolicy-T and ClevrPolicy-M each contain 20K non-CoT training QA pairs per task and 2K unseen test instances across policies with N = 2, 4, or 6.Ground truth is verified against scene graphs and decision trees; the trees are unavailable to the model during training, and evaluation uses exact-match accuracy.Source: §C.1.2 · p. 16

C.2.1 POLICY CREATION IN GTAPOLICY

  • GTAPolicy defines 13 tool types and 24 user-profile-conditional rules, with versioned tools such as OCR v1 and OCR v2.The versioning and conditional rules are intended to test policy-following rather than reliance on prior knowledge.Source: §C.2.1 · pp. 16–17

C.2.2 TRAINING DATA CREATION IN GTAPOLICY

  • GTAPolicy converts GTA’s multi-turn examples into 451 single-turn training instances and 106 test instances, each expecting a JSON tool call.Tool names are evaluated by exact match; argument metrics vary by type, including IoU for boxes and text similarity for free-form text.Source: §C.2.2 · p. 17

C.3 EVALUATION METRICS AND RLVR REWARDS

The appendix specifies evaluation and training procedures and reports results on policy complexity, qualitative behavior, general capabilities, safety, and use of non-CoT data.

C.3 EVALUATION METRICS AND RLVR REWARDS

  • ClevrPolicy accuracy uses exact outcome matching, while GTAPolicy combines exact-match tool accuracy with argument metrics suited to argument type.Argument scoring averages normalized field scores; examples of full policies are provided for ClevrPolicy-T, ClevrPolicy-M, and GTAPolicy.Source: §C.3 · p. 17, §C.4 · p. 17

E ADDITIONAL RESULTS ON VARYING POLICY COMPLEXITY AND MODEL SIZE

D.1 COT DATA GENERATION

  • ClevrPolicy CoT data uses forward reasoning with answer filtering, whereas GTAPolicy uses reverse CoT because its dataset is small.The CoT SFT objective trains on concatenated reasoning annotation C and answer A; approximately 2.5K CoT instances are obtained per ClevrPolicy setting and 451 for GTAPolicy.Source: §D.1 · p. 18, §D.2 · p. 18

F QUALITATIVE ANALYSIS

H ROBUSTNESS TO CATASTROPHIC FORGETTING ON SAFETY

I ILLUSTRATION OF THE POLICY REFERRAL EVALUATION

K RL EFFECTIVELY LEVERAGES NON-COT DATA

L ADDITIONAL RELATED WORK

  • Textual-inversion personalization learns lightweight embeddings for visual concepts, but supported concepts are typically limited to one or a few personal photos.The passage contrasts this scope with the greater complexity of policies for multimodal conversational agents.Source: L ADDITIONAL RELATED WORK · pp. 21–22

M ERROR EXAMPLES

The appendix presents examples of reasoning and perception errors on ClevrPolicy and tool-selection errors on GTAPolicy.

M ERROR EXAMPLES

  • A ClevrPolicy reasoning error branches to a Condition that does not exist in the current policy.Source: Figure 11 · p. 23
  • A ClevrPolicy perception error incorrectly reports that the image contains no small cyan objects.Source: Figure 12 · p. 23
  • On GTAPolicy, the model calls ImageDescription instead of RegionAttributeDescription after TextToBbox localizes the requested region.The incorrect call captions the whole image and disregards the prior tool output.Source: Figure 13 · p. 24

Related papers

Loading 2510.09474v2…