PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing and Internalizing Complex Policy Documents for LLM Agents

Jiateng Liu, Zhenhailong Wang, Xiaojiang Huang, Yingjie Li, Xing Fan, Xiang Li, Chenlei Guo, Ruhi Sarikaya, Heng Ji

arXiv:2510.11588v1cs.AI

TL;DR

Policy documents consume substantial prompt space, while their complex rules make reliable internalization difficult to evaluate and achieve. The paper introduces CC-Gen and CAP-CPT, which categorizes policy specifications to guide targeted continued-pretraining data. CAP-CPT improves baseline performance across scenarios and reduces input length substantially on CC-Gen.

  • Problem

    Policy documents consume substantial input context, and their complex specifications pose distinct challenges that prior generic prompt-compression work has not systematically examined.

    Source: §1 · pp. 1–2

  • Method

    CC-Gen varies four policy-task complexity dimensions, while CAP-CPT categorizes policy specifications and uses category-targeted data for continued pretraining.

    Source: §1 · p. 2

  • Results

    97.3% input token compression on the synthetic benchmark accompanies baseline performance gains across scenarios on Qwen-3-32B.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings underscore the importance of explicitly modeling policy complexity and present a scalable approach for policy internalization.

    Source: §6 · p. 10

    The evaluation is limited to text-only, single-turn agent tasks, leaving multimodal and multi-turn settings for future work.

    Source: M LIMITATION AND FUTURE WORK · p. 42

Abstract

from arXiv · show

Large Language Model (LLM)-based agentic systems rely on in-context policy documents encoding diverse business rules. As requirements grow, these documents expand rapidly, causing high computational overhead. This motivates developing internalization methods that embed policy documents into model priors while preserving performance. Prior prompt compression work targets generic prompts, but agentic policy documents span multiple complexity levels and require deeper reasoning, making internalization harder. We introduce CC-Gen, an agentic benchmark generator with Controllable Complexity across four levels, enabling systematic evaluation of agents' ability to handle complexity and offering a unified framework for assessing policy internalization. Our analysis shows that complex policy specifications governing workflows pose major reasoning challenges. Supporting internalization with gold user agent interaction trajectories containing chain-of-thought (CoT) annotations via supervised fine-tuning (SFT) is data-intensive and degrades sharply as policy complexity increases. To mitigate data and reasoning burdens, we propose Category-Aware Policy Continued Pretraining (CAP-CPT). Our automated pipeline parses policy documents to extract key specifications, grouping them into factual, behavioral, and conditional categories, and isolating complex conditions that drive workflow complexity. This guides targeted data synthesis and enables agents to internalize policy information through an autoregressive pretraining loss. Experiments show CAP-CPT improves SFT baselines in all settings, with up to 41% and 22% gains on Qwen-3-32B, achieving 97.3% prompt length reduction on CC-Gen and further enhancing tau-Bench with minimal SFT data.

1 INTRODUCTION

Policy documents consume substantial prompt space, yet agents can struggle to follow their complex specifications, motivating systematic study of policy complexity and internalization. The paper introduces CC-Gen and CAP-CPT, reporting performance gains and substantial prompt compression.

Motivation and research gap

  • Policy prompts can dominate agent inputs and impose computational overhead, while even short policy documents can cause severe performance degradation.In τ-Bench, policy documents account for roughly 35% of input tokens; agents can degrade severely with documents as short as 1K tokens.Source: §1 · pp. 1–2

Benchmarking complexity and SFT

  • CC-Gen controls four complexity dimensions, and initial results identify workflow complexity as the strongest performance challenge.The paper also reports that SFT with 1K–30K gold chain-of-thought trajectories remains data-intensive and leaves substantial performance gaps at high complexity.Source: §1 · p. 2

CAP-CPT

  • CAP-CPT categorizes policy specifications as factual, behavioral, or conditional and tailors generated training data to each category.Its pipeline also simulates scenarios for complex conditions, adds role-model demonstrations for behavioral rules, and applies continued pretraining with an autoregressive loss.Source: §1 · pp. 2–3

Experimental results

  • 97.3% input token compression on the synthetic benchmark accompanies performance gains over baselines across scenarios on Qwen-3-32B.The reported comparison is over 10% across all scenarios when CAP-CPT is combined with SFT; the approach also improves Tau-Bench with limited SFT data.Source: §1 · p. 3

Contributions

  • The authors characterize policy complexity, identify complex workflows as a key internalization challenge, and report that CAP-CPT remains broadly applicable with minimal policy assumptions.Their contributions include controllable-complexity benchmarks and targeted continued-pretraining data.Source: §1 · p. 3

2 COMPLEXITY CHARACTERIZATION OF LLM-BASED AGENTIC TASKS

The paper defines a controlled, text-only single-turn agent setting and introduces CC-Gen to vary four dimensions of policy-governed task complexity. Experiments find that task and workflow complexity reduce performance, with workflow complexity producing the sharper decline.

2.1 LLM-BASED AGENTIC TASK SETTING

  • The benchmark setting is text-only and single-turn, with database access as the controlled external environment for tool-using agents.Agents maintain a trajectory of reasoning, actions, and tool observations until the query is resolved or the iteration limit is reached.Source: §2.1 · p. 3

2.2 CC-GEN: AGENTIC BENCHMARK GENERATOR WITH CONTROLLABLE COMPLEXITIES

  • CC-Gen controls task, workflow, environmental, and query complexity and supports systematic evaluation of internalization, including policy-referral and policy-override tasks.Its generated benchmarks include policy documents, databases and executable tools, and user queries that can be paired with gold trajectories.Source: §2.2 · pp. 3–4, Table 1 · p. 4

2.3 BENCHMARKING AGENT PERFORMANCE WITH CONTROLLED COMPLEXITY

  • Performance consistently declines as task and workflow complexity rise, with some models reaching zero in the most challenging workflow settings.Qwen-3 models are more resilient than Claude-3.5 under high-complexity conditions.Source: §2.3 · pp. 4–5, Table 2 · p. 4

3 INTERNALIZING COMPLEX AGENTIC POLICY DOCUMENTS

The section frames policy internalization as replacing policy text with a policy identifier that cues the model to recall and apply the relevant rules. It contrasts data-intensive Gold-CoT SFT with CAP-CPT, which categorizes policy specifications and creates targeted training data for continued pretraining.

3 INTERNALIZING COMPLEX AGENTIC POLICY DOCUMENTS

  • Internalization replaces policy text P with a unique policy identifier pid, which acts as a retrieval cue for recalling and applying the relevant policy.The objective is to align outputs given the full policy with outputs given its identifier.Source: §3 · p. 5

3.1 BASELINE: SFT WITH GOLD COT-ENHANCED INTERACTION TRAJECTORIES

  • Gold-CoT SFT trains on 1K–30K interaction trajectories with manually curated reasoning and ground-truth actions and observations.The autoregressive loss is applied to reasoning and action tokens, with separate training sets at 1K, 5K, 10K, 20K, and 30K samples.Source: §3.1 · p. 5

3.2 OUR APPROACH: CATEGORY-AWARE POLICY CONTINUED PRETRAINING

  • Gold-CoT SFT is data-intensive and can lose up to 46% as workflow complexity rises from level (1) to level (3) on Qwen-2.5-32B.The section identifies limited full-trajectory data and complex-policy reasoning demands as its two main limitations.Source: §3.2 · p. 5
  • CAP-CPT categorizes policy specifications as factual, behavioral, simple conditional, or complex conditional, based on their role in agent reasoning and internalization.An LLM extracts and classifies specifications and identifies each specification's valid scope; an optional manual check may be used for complex cases.Source: §3.2 · pp. 5–6
  • CAP-CPT tailors data by category: paraphrases and QA support factual recall, role-model responses support behavioral rules, and scenario simulations exercise conditional workflows.It also incorporates SFT trajectories as auxiliary data and applies continued pretraining with an autoregressive language-modeling loss.Source: §3.2 · p. 6, Figure 2 · p. 6

4 EVALUATION OF POLICY DOCUMENT INTERNALIZATION

The evaluation tests policy internalization across workflow complexity, supportive policy tasks, instruction following, and τ-Bench. CAP-CPT + SFT improves task completion and narrows complexity gaps, while broader policy tasks remain challenging relative to in-context prompting.

4.1 EXPERIMENT SETTINGS

  • Experiments vary workflow complexity, task counts, and SFT data size, and also evaluate policy substitution, override, referral, and general instruction following.Task completion is measured by success rate; policy QA uses a rescaled 0–100 score, and instruction following uses average accuracy.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • CAP-CPT + SFT consistently outperforms strong baselines and reduces high- versus low-complexity performance gaps by 37% on Qwen-2.5-32B and 21% on Qwen-3-32B.Table 3 covers workflow complexity and SFT trajectory sizes from 1K to 30K; on Qwen-2.5-32B, the method also surpasses full-policy prompting.Source: Table 3 · p. 7, Figure 3 · p. 7, §4.2 · p. 8
  • On policy referral, substitution, and override, CAP-CPT outperforms SFT baselines but remains below in-context prompting on most tasks; general instruction following is largely preserved.Full substitution is harder than partial override, and policy-referral scores decline as SFT data increases.Source: Table 4 · p. 8, §4.2 · p. 8

4.3 ABATION STUDY

  • The full CAP-CPT approach performs best across data settings, outperforming generated-data SFT and the variant that removes scenario-simulation data.The evaluation also reports consistent internalization across multiple distinct policies with different complexity levels.Source: §4.3 · pp. 8–9, Table 5 · p. 9

4.4 APPLICATION ON τ-BENCH

  • On τ-Bench, CAP-CPT reaches 28.70% success, above the 26.96% in-context baseline, while reducing overall input length by 34.8%.The approach was evaluated with 282 successful self-generated SFT trajectories from 500 training samples; policy analysis and data generation ran without external LLM APIs.Source: §4.4 · p. 9

5 RELATED WORK

The related work distinguishes this study from research on internalizing general safety rules and behaviors, noting that agentic policies also involve complex workflow constraints. It also situates the work alongside prompt compression, knowledge injection, and continued pretraining.

5 RELATED WORK

6 CONCLUSION

The conclusion presents CC-Gen as a way to evaluate policy internalization across controllable complexity levels and identifies workflow depth as a primary source of performance degradation. It summarizes CAP-CPT as an automated, policy-analysis-based approach that reduces SFT data demands and narrows complexity-related performance gaps.

6 CONCLUSION

  • CC-Gen provides a controllable-complexity benchmark generator for systematic evaluation of agents and policy-internalization algorithms.The conclusion says the work characterizes distinct forms of policy complexity.Source: §6 · p. 9
  • The analysis identifies workflow depth as a primary driver of performance degradation and highlights limits of in-context methods and data-intensive SFT.Source: §6 · p. 9
  • CAP-CPT uses explicit policy identifiers and automated policy analysis to generate training data, reducing SFT data demands and narrowing complexity-related performance disparities.The conclusion characterizes the approach as a scalable solution for policy internalization.Source: §6 · p. 10

7 REPRODUCIBILITY STATEMENT

The paper describes its reproducibility resources, training setup, and experimental-run scope.

7 REPRODUCIBILITY STATEMENT

  • The supplementary archive includes the data generator and detailed training and evaluation instructions for reproducing the results.The authors also say they will publicly release the full codebase and data, including the benchmark generator.Source: §7 · p. 10
  • Qwen-2.5-32B and Qwen-3-32B were trained with LlamaFactory on eight H100 GPUs.The authors cite Zheng et al. (2024) for LlamaFactory.Source: §7 · p. 10
  • All reported experimental results are based on a single run.Further experimental details are provided in Section 4 and Appendix D.Source: §7 · p. 10

8 ETHICS STATEMENT

The study presents itself as fundamental research on internalizing complex policies and reports using only synthetic profiles, with no human subjects or private user data.

8 ETHICS STATEMENT

  • The work focuses on improving internalization of complex policy documents in language models.The authors characterize it as fundamental research.Source: §8 · p. 10
  • No human subjects or private user data were involved in the study.Source: §8 · p. 10
  • The dataset consists entirely of synthetically generated user profiles and does not rely on real user data.Source: §8 · p. 10
  • The authors state that, to their knowledge, the research raises no ethical concerns.Source: §8 · p. 10

A BENCHMARK DEVELOPMENT AND PROBING EXPERIMENTS

This section presents CC-Gen benchmark development and probing experiments, alongside policy analysis that classifies policy specifications and assigns valid task scopes. It notes that policy analysis may be less reliable for τ-bench policies than for templated generated policies.

A BENCHMARK DEVELOPMENT AND PROBING EXPERIMENTS

  • CC-Gen combines environments, templated policy documents, tools, queries, and reference trajectories, while allowing users to control benchmark complexity.Its dimensions count databases for environment complexity, tasks and arguments for task complexity, and nested logical depth for workflow complexity; the dimensions are separated to isolate their effects.Source: A BENCHMARK DEVELOPMENT AND PROBING… · pp. 13–14

B DATA EXAMPLES FOR GENERATED POLICY DOCUMENTS

  • A generated-policy example demonstrates a benchmark instance with environmental complexity 3, task-level complexity 5, and workflow complexity 1.The sample specifies profile access across three layers, tools, policies, and multiple tasks.Source: B DATA EXAMPLES FOR GENERATED POLICY… · pp. 15–16

A BENCHMARK DEVELOPMENT AND PROBING EXPERIMENTS

  • Probing results identify workflow complexity as the greatest reasoning challenge, task complexity next, and environmental complexity as relatively minor for LLM agents.Experiments use Qwen-3-8B and report both Success Rate (SR) and Partial Success Rate (PSR); the authors hypothesize that environment changes add tool-use commands without substantially increasing reasoning burden.Source: A BENCHMARK DEVELOPMENT AND PROBING… · pp. 14–15, Table 8 · p. 15, Table 7 · p. 15

C POLICY ANALYSIS DETAILS

  • Policy analysis uses the model itself to classify specifications as factual, behavioral, simple conditional, or complex conditional, and identify task scope and complex rules.The analysis prompt first identifies user-facing tasks, then classifies each specification and extracts complex workflow rules with their complexity and valid scope.Source: C POLICY ANALYSIS DETAILS · pp. 19–20
  • Complex workflows are hierarchical if-else structures whose complexity level depends on tree depth; multiple conditions without branching hierarchy are not necessarily complex workflows.The passage gives a multi-condition example without branching hierarchy and assigns it complexity 2.Source: C POLICY ANALYSIS DETAILS · pp. 21–22
  • Policy analysis is framed as splitting documents into clear specification chunks and classifying each into four types without omissions or overlapping content.The templates distinguish factual, behavioral, simple workflow, and complex workflow specifications, with valid scope recorded for each.Source: C POLICY ANALYSIS DETAILS · pp. 21–22
  • Templated generated policies were easy to analyze, but policy analysis on τ-bench may be inaccurate without human double-checking.The authors say they will report policy-analysis F1 in Appendix I and analyze its effects on overall performance.Source: C POLICY ANALYSIS DETAILS · p. 22

D MORE COMPREHENSIVE EXPERIMENTAL SETTINGS AND RESULTS

The appendix varies task and workflow complexity and training-data scale to examine policy internalization. CAP-CPT improves over baselines across settings, while Qwen-3-32B remains only comparable to prompting.

D MORE COMPREHENSIVE EXPERIMENTAL SETTINGS AND RESULTS

  • Experiments vary workflow complexity from levels 1 to 3 and task-level complexity across levels 3, 5, 8, and 12, using 1K–30K SFT samples.For τ-Bench, Qwen-3-32B self-generated 282 SFT samples from 500 available training samples; CPT scenario simulation can reach 125K samples for a task-level (5), workflow-level (2) policy.Source: D MORE COMPREHENSIVE EXPERIMENTAL… · p. 22
  • Qwen-2.5-32B shows consistent CAP-CPT gains, most evident in data-sparse and high-complexity scenarios where its baseline performance is weak.With sufficient SFT data, task-completion performance can reach a reasonable level.Source: D MORE COMPREHENSIVE EXPERIMENTAL… · p. 22
  • For Qwen-3-32B, SFT generally diminishes prior knowledge and brings limited gains across data scales; CAP-CPT improves results but does not surpass Qwen-2.5-32B.Its final performance remains comparable to the prompting baseline.Source: D MORE COMPREHENSIVE EXPERIMENTAL… · pp. 22–23
  • Table 10 covers workflow level (1) task variants for Qwen3-32B and Qwen2.5-32B, retaining original Task (5) results and adding Task (3/8/12) entries.The caption notes blank cells for later fill and prompting accuracy where available.Source: Table 10 · p. 23
  • CAP-CPT consistently outperforms the Qwen-2.5-32B baseline across settings and narrows gaps in high-complexity and data-sparse scenarios.For Qwen-3-32B, internalization yields performance comparable to the prompting baseline.Source: Figure 5 · p. 23

E EVALUATION FRAMEWORK OF POLICY DOCUMENT INTERNALIZATION

The framework evaluates policy internalization across task completion, policy knowledge and changes, and general instruction following. It also examines how complexity and training-data limitations affect performance.

E EVALUATION FRAMEWORK OF POLICY DOCUMENT INTERNALIZATION

  • The framework covers task completion, policy referral, policy substitution, policy override, and general instruction following, extending evaluation beyond ordinary user queries.Task completion is measured by success rate, referral by scored answers, and substitution and override by task success.Source: E EVALUATION FRAMEWORK OF POLICY… · pp. 23, 26
  • Across complexity settings, the approach consistently outperforms the baseline and narrows performance gaps in high-complexity and data-sparse cases.Input-token compression reaches 97.3% at high workflow complexity.Source: Figure 7 · p. 24, Figure 6 · p. 24, Figure 8 · p. 25

F INTUITIVE UNDERSTANDING OF OUR OBSERVATIONS

  • For Qwen-3-32B on Task (3), Workflow (5), self-generated CoT provides noticeable gains; combined with Multi-Granular CPT, it achieves the highest performance, with prompting performance of 0.53.Source: Table 12 · p. 27

F.1 WHY OUR CAP-CPT APPROACH WORKS WELL

  • Complex conditional specifications are difficult to learn from randomly sampled interaction trajectories, which may miss nuanced cases even in large datasets.Standard CPT based on paraphrases or QA pairs can support recall without fostering logical generalization, compositional reasoning, or relation specificity; CAP-CPT targets complex conditions with scenario-simulation data and continued pretraining.Source: §F.1 · p. 27

F.2 TRAINING WITH STRONGER MODELS DOES NOT YIELD BETTER PERFORMANCE

Internalization gains differ by model, and the approach is also tested on mixed policies, ablations, and τ-bench. On τ-bench, combined CAP-CPT and SFT surpass prompting while compressing input tokens.

F.2 TRAINING WITH STRONGER MODELS DOES NOT YIELD BETTER PERFORMANCE

  • The weaker Qwen-2.5-32B improves dramatically toward near-100% success, while stronger Qwen-3-32B remains close to its original performance after internalization.The authors attribute the divergence to stronger prior knowledge leaving less room for gains and greater fine-tuning fragility.Source: §F.2 · p. 28

G MULTIPLE POLICY INTERNALIZATION

  • Jointly fine-tuning on data from four policies preserves strong performance on each individual policy, though scaling to many policies remains computationally challenging.The experiment uses Qwen-3-32B and four policy documents with different task-level complexities.Source: G MULTIPLE POLICY INTERNALIZATION · p. 28

H MORE DETAILS ON ABLATION STUDY

  • Removing scenario-simulation data improves policy-override performance, while using CAP-CPT data for SFT slightly improves policy-referral scores.The authors associate reduced CPT with less rigid policy memorization for overrides and SFT with stronger memorization of patterns useful for referral, while noting potential generalization and forgetting costs.Source: H MORE DETAILS ON ABLATION STUDY · p. 28, Table 14 · p. 29

I APPLICATION TO τ-BENCH

  • On τ-bench, policy analysis achieves 100% F1 for high-complexity conditional specifications, while factual and behavioral specifications have lower recall.Factual specifications have 75% F1 (100% precision, 60% recall); behavioral specifications have 66.7% F1 (0.86 precision, 0.55 recall), without manual correction before data generation and training.Source: I APPLICATION TO τ-BENCH · p. 29
  • 34.8% overall input compression accompanies slightly improved performance over prompting for CAP-CPT on Qwen3-32B over τ-bench.SFT on 282 self-generated examples underperforms prompting alone; adding CAP-CPT data with CPT+SFT surpasses prompting.Source: I APPLICATION TO τ-BENCH · pp. 29–30, Table 15 · p. 30

J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH

This section illustrates a τ-Bench policy error in which a state-of-the-art LLM mishandles a complex airline-policy task, including using a payment identifier that does not match the user-confirmed gift card. It also situates policy internalization among related work on prompt compression, alignment, continued pretraining, and knowledge injection, and states how LLMs assisted with writing.

J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH

  • The agent identified the Houston–Denver reservation as OBUT9V and summarized the requested return-flight and baggage changes before asking for confirmation.The user confirmed the changes before the agent called the update tool.Source: J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH · pp. 33, 38
  • The agent called HAT229 plus HAT266 the fastest return option, then recommended HAT290 plus HAT175 after noting HAT229 had zero economy seats.The proposed alternative matched the user’s economy constraint, but it was not the fastest option the agent initially identified.Source: J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH · pp. 36–37
  • The final update used payment_id gift_card_627644, not the confirmed gift card gift_card_6276644.The agent’s confirmation summary names gift-card-6276644, but the update call contains the different identifier gift_card_627644.Source: J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH · p. 38
  • The agent charged for an extra checked bag even though the Silver member traveling in economy was entitled to two free checked bags.The example says the agent had successfully updated the flights but made an error when updating baggage.Source: J ERROR EXAMPLES OF SOTA LLMS ON τ-BENCH · p. 40

K.1 PROMPT COMPRESSION FOR LARGE LANGUAGE MODELS

  • PromptIntern progressively internalizes input tokens, but does not explicitly address the reasoning challenges of complex policy-document structure.The paper's special-token approach aims to combine interpretable hard prompts with the flexible learning associated with soft prompts.Source: §K.1 · p. 40

K.2 DELIBERATE ALIGNMENT

  • Deliberative alignment internalizes general safety rules and behaviors through additional training or test-time deliberation, but does not cover broader agentic policies or workflow-level constraints.The cited approaches are Guan et al. (2024) and Zhang et al. (2025a).Source: §K.2 · p. 40

K.3 CONTINUED PRETRAINING FOR LARGE LANGUAGE MODELS

  • CAP-CPT relies on continued pretraining for generalizable learning and to mitigate catastrophic forgetting associated with pure SFT, using targeted data and policy-grounded question–answer pairs.The passage also notes CPT's role in adapting to new data distributions and describes replay, regularization, and modular architectures as existing approaches, while identifying gaps in long-horizon simulations and standardized vertical-forgetting benchmarks.Source: §K.3 · pp. 40–41

K.4 KNOWLEDGE INJECTION FOR LARGE LANGUAGE MODELS

  • Policy internalization differs from knowledge injection by requiring deep interpretation and practical application of rules, so the paper proposes a CPT-based approach tailored to policy complexity.The passage describes dynamic retrieval, static embedding, modular adapters, and prompt optimization as knowledge-injection paradigms, with hybrids combining approaches.Source: §K.4 · p. 41

L ETHICAL STATEMENT ON LLM ASSISTANCE

  • ChatGPT-5 was used primarily for language refinement, and human authors reviewed and rewrote all model-generated content.The authors state this was to ensure accuracy, originality, and adherence to research-integrity standards.Source: L ETHICAL STATEMENT ON LLM ASSISTANCE · pp. 41–42

M LIMITATION AND FUTURE WORK

The paper identifies limits in benchmark scope, training recipe, challenging task variants, and the risks of policy-specific interference. It outlines extensions to broaden evaluation and improve robustness and alignment.

Scope of the benchmark

  • The benchmark is text-only and single-turn, so its complexity characterization primarily reflects policy documents and associated agent tasks.The authors identify multi-turn planning and repair, multimodal inputs, and user-intent distributions as future extensions.Source: M LIMITATION AND FUTURE WORK · p. 42

Training recipe

  • The training recipe uses continued pretraining followed by SFT but omits reinforcement-learning stages that could leverage the trajectories.The authors propose adding RL fine-tuning on top of CAP-CPT+SFT as a future extension for improved alignment.Source: M LIMITATION AND FUTURE WORK · p. 42

Challenging task variants

  • Despite strong average gains, models remain brittle on policy-substitute, policy-override, and policy-referral tasks.The authors report that scaling data may improve fixed-set scores but offers limited broader gains when override granularity and referral formats are under-specified.Source: M LIMITATION AND FUTURE WORK · p. 42

Fragility of strong priors

  • Stronger reasoning models can be more prone to policy-specific interference and forgetting, and the authors lack guarantees against negative transfer or general instruction-following regressions.Future directions include selective internalization via policy identifiers, prior-preservation regularizers, and continual-learning safeguards.Source: M LIMITATION AND FUTURE WORK · p. 42

Related papers

Loading 2510.11588v1…