PaperlayerView exact revision on arXiv

Source-linked AI summary

Smarter by the Moment: Environment-Driven Dynamic Policies for Continual LLM Improvement

Ting-Wei Chang, Po-Chun Chen, Hen-Hsen Huang, Hsin-Hsi Chen

arXiv:2609.16800v1cs.CL

TL;DR

As environments and requirements evolve, LLMs need continual adaptation to new challenges, but incremental weight updates impose considerable computational overhead, motivating memory-based alternatives. DRPG retrieves successful examples for the agent and uses a separate policy generator to synthesize guidelines from correct and incorrect prior cases plus binary environment feedback. DRPG outperforms strong baselines on the majority of model–dataset configurations, significantly overall against Self-StreamICL, with the largest gains on structured prediction tasks.

  • Problem

    As environments and requirements evolve, LLMs need continual adaptation to new challenges, but incremental weight updates impose considerable computational overhead, motivating memory-based alternatives.

  • Method

    DRPG retrieves successful examples for the agent and uses a separate policy generator to synthesize guidelines from correct and incorrect prior cases plus binary environment feedback.

  • Results

    DRPG outperforms strong baselines on the majority of model–dataset configurations, significantly overall against Self-StreamICL, with the largest gains on structured prediction tasks.

  • Takeaways & Limitations

    Policy-level guidance is most beneficial for systematic, recurring errors and remains modular across retrieval strategies, policy continuity, and generator model families.

  • Takeaways & Limitations

    The evaluation assumes clean binary correctness feedback at every step, whereas real deployments may provide sparse or imperfect feedback requiring additional verification.

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable progress across diverse domains, but continual adaptation to evolving tasks and environments remains a key challenge. Existing memory-augmented approaches retrieve individual past examples as direct references, but do not explicitly synthesize actionable strategies from them, causing the same types of errors to recur. We propose Dynamic Retrieval-based Policy Generation (DRPG), a framework that integrates memory-based retrieval with a dynamic policy generator, leveraging historical data and environment feedback to produce task-specific policies for continual LLM improvement. We evaluate DRPG across six benchmarks spanning text-to-SQL, question answering, medical diagnosis, and Python programming, using seven LLMs from both proprietary and open-weight families. DRPG outperforms strong baselines across most datasets and models. Further analysis demonstrates that DRPG's policy generation is robust to retrieval strategy, operates effectively without prior policy continuity, and can leverage smaller or cross-family models as cost-efficient policy generators. We also find that the benefit of policy-level guidance depends on task characteristics, offering practical insights into when and under what conditions this mechanism is most effective.

1 Introduction

LLMs need cost-efficient continual adaptation as environments evolve, but existing memory methods mainly retrieve individual examples rather than synthesize feedback-informed strategies. DRPG addresses this gap with dynamically generated policies and shows broad gains across benchmarks and models.

  • The framework uses retrieved successful examples for prediction while generating policies from both correct and incorrect prior cases, with binary environment feedback stored for future adaptation.
  • Existing memory-based methods guide predictions with individual retrieved examples, while Dynamic Cheatsheet does not use environment feedback to decide what to store.These limitations constrain strategy synthesis and can reduce effectiveness for smaller models.
  • Prior approaches do not explicitly synthesize feedback-informed strategies, allowing similar mistakes to recur even after related errors have been encountered.
  • DRPG augments memory-based retrieval with a dynamic policy generator that synthesizes actionable strategies from past experiences and environment feedback.The framework is evaluated across six benchmarks and seven LLMs.
  • DRPG outperforms strong baselines across most model–dataset configurations, with policy generation most effective on tasks containing systematic, recurring error patterns.The result is reported across six benchmarks and seven LLMs.

2 Related Work

Related work spans parameter updates, memory-augmented retrieval, reflection, and policy induction for improving LLM behavior. DRPG combines continual cross-query policy synthesis with environment feedback, distinguishing it from methods that reflect on single instances or produce fixed guidance.

  • LLMs Improving in Online Settings: Online parameter updates adapt models incrementally but impose substantial computational overhead for modern LLMs, motivating memory-based alternatives.
  • Memory-Augmented Methods: Memory-augmented methods retrieve recent or correct examples, while Dynamic Cheatsheet updates memory using internal model signals rather than external feedback.
  • Reflection and Feedback: Reflection methods such as Agent-Pro, SAGE, Reflexion, and Self-Refine improve behavior through self-feedback or iterative refinement, but differ in feedback source and accumulation scope.Reflexion repeatedly interacts with the same dataset, while Self-Refine does not use ground-truth external feedback.
  • Policy-Level Guidance: LEAP, SALAM, and Induct-Learn extract principles, mistake guidelines, or pseudo-instructions from correct and incorrect examples to guide later inference.
  • DRPG’s Distinction: DRPG’s novelty lies in continually synthesizing reusable, feedback-grounded policies across queries as the stream evolves, unlike single-instance reflection or fixed offline guidance.

3 Dynamic Retrieval-based Policy Generation (DRPG) Framework

DRPG combines memory retrieval with a separate policy generator that distills feedback-labeled past experiences into actionable guidance for each new input. The agent then answers using both retrieved correct examples and the generated policy, while new outcomes update memory.

  • DRPG retrieves correct past examples, contrasts correct and incorrect cases to synthesize actionable policies, conditions the agent on both resources, and stores binary feedback for future updates.The framework repeats this process at each time step in an online stream.
  • Agent: The agent retrieves top-k previous examples with positive environmental feedback and uses them as few-shot context for the current input.The retriever operates over past interactions before the current time step.
  • Policy Generator: The policy generator distills retrieved correct and incorrect examples into a concise set of task-specific guidelines, such as up to five actionable bullet points.Its retriever may be the same as or different from the agent’s retriever, with contrastive retrieval used by default.
  • Environment interaction: After producing an answer, the environment returns binary correctness feedback, which is stored with the input and output in the memory database.This feedback determines whether later retrievals can use each past experience as a correct or incorrect case.
  • Prompt Design: The prompt structure consistently includes role assignment, reference materials, and task-specific instructions for both the agent and policy generator.The policy-generator prompt includes retrieved correct and incorrect examples and asks for actionable bullet points.

4 Experiments

The experiments evaluate DRPG across four task categories, six benchmarks, and multiple LLM families using standardized dataset metrics and online-memory baselines. The setup compares DRPG with zero-shot, self-refinement, and streaming retrieval methods under shared retrieval and prompt configurations.

  • Datasets: The evaluation spans six benchmarks across text-to-SQL, multi-hop question answering, medical diagnosis, and Python programming.The text-to-SQL tasks are Spider, CoSQL, and BIRD; the remaining benchmarks are HotpotQA, DDXPlus, and DS-1000.
  • Evaluation Metrics: The experiments use execution accuracy for text-to-SQL, exact match for HotpotQA, accuracy for DDXPlus, and pass@1 for Python programming.These are the standard evaluation metrics adopted for the respective datasets.
  • Baselines: DRPG is compared with zero-shot, Self-Refine, Self-StreamICL, and other streaming or non-streaming baselines.Self-StreamICL serves as the main baseline and retrieves similar correctly answered cases from external memory.
  • Experimental Setup: DRPG uses the same agent configuration as Self-StreamICL while adding a dynamically generated policy based on retrieved past examples.The default policy-generator retrieval uses equal numbers of correct and incorrect cases, with k fixed by benchmark.
  • Models: The study covers Gemini, Llama, and Mistral models, including proprietary API-based and open-weight systems with varied sizes and capability levels.This model selection is intended to assess generalizability across families and model capabilities.

5 Results and Analysis

DRPG improves continual LLM performance by synthesizing actionable policies from retrieved experiences and feedback, with benefits varying by task structure and model pairing. Across six benchmarks and seven LLMs, it generally outperforms retrieval and refinement baselines, while remaining robust to several design choices.

  • 5.1 Performance of DRPG Framework: DRPG wins 29/42 model–dataset combinations against Self-StreamICL, compared with 35/42 against both Self-Refine and Zero-shot.The overall comparison spans six benchmarks and seven LLMs.
  • 5.1 Performance of DRPG Framework: DRPG’s strongest gains occur on text-to-SQL, winning 18/21 configurations against Self-StreamICL, and on HotpotQA, winning 5/7.Llama-4-maverick is a notable HotpotQA exception, where the generated policy can override correct reasoning and reduce performance.
  • 5.2 Task-Dependent Effectiveness of Policy Generation: DRPG performs on par with Self-StreamICL on DDXPlus and DS-1000, where policies capture narrow, instance-specific patterns rather than broadly applicable strategies.The cited comparisons report 3/7 wins on each dataset, with differences not significant in either direction for DDXPlus and DS-1000.
  • 5.2 Task-Dependent Effectiveness of Policy Generation: Policy-level guidance is most beneficial when errors have compositional, recurring structure that can be compressed into high-level rules.Spider policies include cross-query guidelines such as schema validation and avoiding ambiguous or redundant results, producing an 8.5-point improvement over Self-StreamICL.
  • 5.3 Ablation Studies: The agent’s capability remains the dominant performance factor, but smaller or cross-family policy generators usually provide auxiliary gains and can avoid some same-model failure modes.On HotpotQA, maverick scores 50.60 with its own policy generator versus 63.73 with scout and 63.07 with mistral-small.
  • 5.3 Ablation Studies: Policy generation remains effective with correct-only or wrong-only retrieval and without prior-policy input, while removing correctness feedback consistently hurts performance across all six benchmarks.These findings indicate robustness to retrieval composition and stateless operation, while isolating environment feedback as useful.
  • 5.4 Cost Analysis: DRPG adds one retrieval and one policy-generator call per query, making its relative latency higher than Self-StreamICL and Zero-shot in the representative cost comparison.The representative configuration is mistral-medium × Spider with N=2,147; latency is intended only as a relative comparison because runs used free-tier APIs.

6 Conclusion

DRPG augments memory retrieval with a dynamic policy generator for continual LLM improvement in online settings. It outperforms strong baselines on most model–dataset configurations, with the largest gains on structured tasks and weaker benefits for instance-specific knowledge.

  • DRPG augments memory-based retrieval with a dynamic policy generator for continual LLM improvement in online settings.
  • Across six benchmarks and seven LLMs, DRPG outperforms strong baselines on most configurations and significantly outperforms Self-StreamICL overall.
  • Policy-level guidance is most effective for systematic, recurring errors capturable by high-level rules and performs on par with retrieval on instance-specific tasks.
  • Ablations show robustness to retrieval strategy, stateless operation, and use of a separate potentially smaller policy-generation model.

Limitations

The evaluation uses clean binary correctness feedback at every step in a controlled online setting, so its conclusions should not be extrapolated to noisier real deployments. The study also leaves several robustness dimensions and reasoning-model evaluations for future work.

  • DRPG is evaluated with clean binary correctness feedback at every step, whereas real deployments may provide noisy, delayed, partial, non-binary, or sparse feedback.The authors identify adapting memory and policy generation to feedback-receiving interactions as a natural extension.
  • The study does not evaluate reasoning LLMs such as OpenAI GPT-o3 or DeepSeek R1, and DRPG’s effectiveness may vary on tasks beyond the four benchmark categories.Experiments also use one fixed data ordering and temperature, without reporting multi-run variance or data-order sensitivity.

B Experimental Cost

DRPG’s cost–performance trade-off is evaluated across all 42 model–dataset configurations using per-query token counts and score plots. It generally improves on Self-Refine at comparable or lower output cost, but its extra policy-generation tokens versus Self-StreamICL help mainly on task types where gains are observed.

  • Table 4 reports per-query input and output token counts for all 42 model–dataset configurations, with DRPG values including both policy-generation and answer calls.
  • DRPG generally achieves higher scores at comparable or lower output-token cost than Self-Refine across the evaluated configurations.Figure 4 plots scores against per-query completion tokens, counting all LLM calls for each method; up and left is better.
  • Compared with Self-StreamICL, DRPG spends additional output tokens mainly for policy generation, and the resulting gains depend on task type.The cost comparison covers all 42 model–dataset configurations and counts DRPG’s policy-generation and answer calls.

D Robustness to Retrieval Strategy

DRPG’s policy-generation performance is robust to whether retrieved examples are correct, incorrect, or mixed, and it does not consistently benefit from carrying the previous policy across steps. These findings support generating policies from retrieved experiences without requiring policy continuity.

  • All three retrieval compositions achieve comparable performance across benchmarks and models, showing robustness to correct-only, wrong-only, and contrastive inputs.The ablation varies the composition of retrieved cases, not the retriever’s relevance computation or embedding configuration.
  • Neither including nor omitting the previous policy consistently outperforms the other across models and tasks.The stateless design can synthesize strategies from retrieved examples alone while avoiding inherited outdated or overly specific rules.

F Effect of Environment Feedback: A No-Feedback Ablation

Environment feedback materially contributes to DRPG’s gains: feedback-conditioned policy generation outperforms a no-feedback variant across all six datasets. The broader comparison also shows that improvements concentrate on text-to-SQL and HotpotQA rather than instance-specific tasks.

  • DRPG outperforms its no-feedback variant on all six datasets, indicating that feedback-conditioned policy generation drives the ablation’s gains.The comparison uses llama-3.3-70b across all six benchmarks, with other components held identical.
  • On DDXPlus and DS-1000, feedback keeps DRPG close to Self-StreamICL, whereas no-feedback policies fall far below that baseline.
  • Across 42 model–dataset configurations, DRPG significantly outperforms Self-StreamICL with one-sided p = 0.005.Per-dataset tests find significant improvements on Spider and CoSQL, while differences are not significant on DDXPlus and DS-1000.
  • The per-configuration gains concentrate on text-to-SQL benchmarks and HotpotQA, consistent with task-dependent policy benefits.Figure 5 groups Δ(DRPG − Self-StreamICL) by model family.

H Results on Additional Open-Weight Model Families: Qwen and Gemma

Additional evaluations show that DRPG generally improves performance for Qwen and Gemma under standard, non-reasoning inference, though gains vary by dataset. The results support a task-dependent benefit from policy-level guidance, especially when errors form recurring, generalizable patterns.

  • Qwen: DRPG outperforms Self-StreamICL on four Qwen benchmarks, matches it on CoSQL, and trails it on DDXPlus.The comparison covers Spider, BIRD, HotpotQA, CoSQL, DDXPlus, and DS-1000 under the same evaluation protocol summarized in Table 10.
  • Task-dependent effects: Policy-level guidance helps most when task errors share recurring, generalizable patterns, whereas narrow instance-specific policies limit gains on DDXPlus.Both models were evaluated with thinking mode disabled, matching the paper’s stated scope of standard non-reasoning inference.
  • Gemma: DRPG outperforms Self-StreamICL on five of six Gemma datasets and is comparable on HotpotQA.Unlike Self-StreamICL, DRPG remains above Zero-shot on BIRD and recovers most of the Zero-shot gap on DS-1000.
Loading 2609.16800v1…