Source-linked AI summary
Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning
Yanxi Chen, Weijie Shi, Yuexiang Xie, Boyi Hu, Yaliang Li, Bolin Ding, Jingren Zhou
TL;DR
Long-lifecycle agents must learn from experience and update context across related tasks, but standard task-by-task training does not target this meta-capability. The paper introduces end-to-end CoD reinforcement learning and finds that later-task success improves substantially, with gains extending across domains and settings.
Problem
Long-lifecycle agents need to explore, learn, and update context across related tasks, while standard task-by-task RL does not explicitly train this cross-task meta-capability.
Method
The CoD framework trains agents with end-to-end RL over rollouts interleaving task-solving and context-updating episodes.
Results
The fourth-task success rate on FrozenLake-Obscure increased from 28% to 76%, with improvements also observed for in-domain, cross-domain, and Ralph-loop evaluations.
Takeaways & Limitations
CoD training can elicit a meta-capability that improves performance from self-updated context and shows potential for generalization across domains and deployment settings.
Takeaways & Limitations
When tasks can be solved from scratch, CoD-Train may degenerate into standard task-by-task RL, making acquisition of CoD uncertain.
Abstract
from arXiv · showhide
This work presents a general framework for training large language models (LLMs) to "Connect the Dots" (CoD), a meta-capability required by long-lifecycle agents: as an LLM-based AI agent gets deployed in an environment, it solves a long sequence of tasks while continuously exploring the environment, learning from its own experiences, and iteratively self-updating its context about the environment, thereby achieving progressively better performance on future tasks conditioned on the updated context. Major components of the CoD framework include: (1) algorithm design and infrastructure for end-to-end reinforcement learning (RL) with long rollout sequences interleaving solve-task and update-context episodes; (2) tasks and environments for incentivizing and eliciting the targeted meta-capability in LLMs during training, as well as for faithfully measuring progress during evaluation. We present proof-of-concept implementations of the CoD framework, including a GRPO-style RL algorithm with fine-grained credit assignment, as well as tasks and environments tailored to the targeted meta-capability (rather than domain-specific LLM capabilities or standard task-by-task RL). Empirical results validate the efficacy of end-to-end RL training in the CoD setting, and demonstrate the potential for out-of-distribution generalization -- within the training domains, across different domains, and from CoD to Ralph-loop settings -- of the elicited meta-capability. Our investigation of CoD connects several lines of prior works, and opens up new opportunities for advancing LLMs and AI agents. To facilitate further research and applications, we release our implementations at \url{https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod}.
1 Introduction
The paper defines “Connect the Dots” as a meta-capability for long-lifecycle agents that continually solve related tasks, explore environments, and update context, and proposes end-to-end RL to train it. It introduces a framework interleaving task-solving and context-updating episodes, with proof-of-concept implementations and promising empirical validation.
- Motivation: CoD enables an agent to solve related tasks over long-lifecycle deployment while deliberately exploring and self-updating environmental context for future tasks.The capability is distinguished from domain-specific capabilities and standard task-by-task learning.
- Framework: The proposed framework interleaves solve-task and update-context episodes in CoD-Deploy, while CoD-Train matches this rollout pattern during RL post-training.This provides a unified abstraction for deployment and training across long state-action sequences.
- Challenges: Key challenges include credit assignment across long sequences of solve-task and update-context episodes and designing tasks and environments that properly elicit and measure CoD.The approach applies dynamic programming so each episode accounts for both immediate and future rewards.
- Implementations and results: The concrete implementations combine RL algorithms and infrastructure with CoD-tailored tasks and environments, and empirical results validate end-to-end RL for eliciting the meta-capability.The paper reports proof-of-concept implementations and experiments applying CoD-Train to Qwen3-8B-Instruct on FrozenLake-Obscure environments with initially hidden necessary clues.
- Open questions: The work identifies open directions including refining CoD-specific algorithms and environments, scaling CoD, and integrating it into existing LLM post-training pipelines.These directions remain open despite the framework’s proof-of-concept status and promising empirical results.
2 The CoD framework
The CoD framework trains and evaluates LLMs on long-lifecycle behavior by interleaving task-solving with context updates across environments. Its design extends RL to longer multi-episode trajectories, requiring credit assignment that accounts for both immediate and future rewards, while task and environment choices determine whether context transfer is incentivized and measured faithfully.
- CoD-Deploy: CoD-Deploy models deployment as alternating task-solving and context-update episodes in an environment, constituting gradient-free online learning through trial and error.The agent solves tasks x_i^M while updating its context from z_i^M to z_{i+1}^M.
- CoD-Train: CoD-Train uses the same rollout pattern for RL post-training across diverse environments, targeting generalization beyond standard task-by-task RL.From a hierarchical-RL perspective, it raises the trajectory level from tokens, to turns, to sequences of solve-task and update-context episodes.
- End-to-end reinforcement learning: Credit assignment in CoD-Train must span long state-action sequences containing multiple solve-task and update-context episodes, each potentially involving many turns and tokens.The framework applies a dynamic-programming principle: each episode should optimize immediate reward together with future rewards.
- End-to-end reinforcement learning: During solve-task episodes, the agent balances exploitation with exploration, using its updated context while gathering information and skills for future context updates.Update-context episodes deliberately condense prior experience into context intended to improve subsequent task performance.
- Tasks and environments for training and evaluation: Training tasks should require environment information unavailable before deployment, while evaluation tasks should reward proactive context maintenance and solving new tasks conditioned on that context.If tasks can be solved from scratch for maximum reward, CoD-Train may degenerate into standard task-by-task RL; existing benchmarks can also be converted into CoD counterparts.
3 Proof-of-concept implementations and experiments
The proof-of-concept CoD implementations combine long-horizon RL with dedicated environments and context updates to test meta-capability learning. Results show improved within-sequence performance, in-domain OOD generalization, and cross-domain transfer, with some instability in mixed-domain training.
- Reward design: CoD-Train assigns task-wise outcome rewards to solve-task episodes and small format rewards to update-context episodes.The current implementation uses outcome rewards for simplicity, while update-context episodes lack another direct reward signal.
- RL implementation: CoD-Train uses critic-free GRPO with episode returns combining current rewards and future solve-task rewards across multi-episode trajectories.The algorithm adapts GRPO’s single-trajectory reward assumption to sequences containing multiple episode rewards.
- Experimental settings: The study evaluates dedicated CoD environments—FrozenLake-Obscure, Alchemy-Random, and TerminalSimulator—under task sequences of length 4 in both single-domain and mixed-domain training.Setting A trains on FrozenLake-Obscure; Setting B mixes FrozenLake-Obscure and Alchemy-Random.
- Empirical results: Evaluation on harder same-domain instances confirms in-domain OOD generalization, while Alchemy-Random and TerminalSimulator improve in CoD-Deploy and Ralph-loop settings.These findings support transfer of the elicited CoD capability beyond the training task instances and domains.
- Empirical results: In setting A, mean reward at position 0 rises from 0.18 to 0.45, while position 3 rises from 0.28 to 0.76 during CoD-Train.Later positions improve more substantially, indicating better use of accumulated context within task sequences.
- Empirical results: Mixed-domain setting B produces less stable training, including minor Alchemy-Random degradation and later evaluation fluctuations after rapid early gains.The instability contrasts with the otherwise positive results reported for setting B.
4 Related works
The work connects CoD to lifelong agents, meta reinforcement learning, and test-time inference scaling while distinguishing its cross-task, self-updating context setting from prior approaches. It also cautions that TerminalSimulator’s unrelated task sequences limit observed later-task gains.
- Lifelong agents: Lifelong-agent research emphasizes context management and long-lifecycle deployment in which agents update their own context.The area includes both theoretical work on computational expressivity and practical work on deployed LLM-based agents.
- Evaluation caveat: TerminalSimulator shows no later-task gain in CoD-Deploy because different tasks within a sequence lack close relations, warranting caution in interpreting those evaluations.This contrasts with the repeated-attempt structure of the Ralph-loop setting.
- Meta reinforcement learning: CoD relates to meta reinforcement learning and RL2, but transfers context through an LLM’s updated context rather than an RNN hidden state.Classic RL2 concatenates multiple solve-task episodes into one rollout and transfers context via recurrent state.
- Meta reinforcement learning: LLM adaptations of RL2 generally repeat episodes or attempts for the same task, aligning more closely with sequential test-time scaling or Ralph loops than classic meta RL.The cited works include LaMer, MAGE, and Orbit, and differ from CoD in both problem settings and RL algorithms.
- LLM inference scaling: CoD-Deploy is related to inference scaling, where repeated LLM calls recursively improve performance through sequential, parallel, or combined organizations.Examples include Ralph loops and auto-research systems.
- LLM inference scaling: Inference-scaling methods can improve rollout trajectories with reflection, retry, or skills, while still training the policy to solve tasks from scratch after removing that added context.These approaches incorporate inference scaling during RL rollouts, but the extra context is not retained for policy execution.
5 Discussions
The discussion identifies open directions for making CoD more principled, diverse, broadly validated, and flexible. It also frames CoD as complementary to task-by-task RL and considers integrating it into existing LLM post-training pipelines.
- Limitations and future work: The current GRPO-style RL algorithm includes heuristic augmentations and potential caveats, motivating more principled and theoretically grounded alternatives.The paper presents this as an open direction for future work.
- Limitations and future work: Future work should design more diverse and challenging CoD environments beyond FrozenLake-Obscure and Alchemy-Random for training and evaluation.The proposed expansion targets environments that better test the CoD meta-capability.
- Limitations and future work: CoD should be validated with longer task sequences, potentially nonstationary environments, and broader out-of-distribution generalization across environments.These settings would test whether the framework remains effective beyond the current evaluation scope.
- Limitations and future work: The framework could support more powerful context management beyond rewriting a hint and more flexible rollout patterns than those shown in Figure 1.This extends both how context is updated and how solve-task and update-context episodes are organized.
- Integration into LLM post-training pipelines?: CoD is complementary rather than alternative to task-by-task RL, and one straightforward integration strategy is adding CoD-Train as an extra sequential post-training stage.The discussion presents integration into existing LLM post-training pipelines as a natural question.
A The complete RL algorithm
The complete RL algorithm assigns credit across interleaved solve-task and update-context episodes, then applies token-mean policy-gradient updates with adaptive re-weighting to stabilize CoD-Train. Experiments on Alchemy-Random show the final method is more stable than vanilla GRPO and REC-OneSide-NoIS, while the return and baseline designs remain open to improvement.
- Credit assignment: For each trajectory and task position, the method computes episode returns, position-specific baselines, and advantages to assign credit across the long rollout.The rollout contains S tasks and G end-to-end trajectories, with episode-wise rewards for solving tasks and updating contexts.
- Gradient calculation: The gradient update uses token-mean aggregation over a batch, importance-sampling weights, and one-side clipping, following the GRPO-style formulation.The batch contains N tokens, each with an advantage and probabilities under the rollout and current policies; clipping uses ϵhigh and ϵlow.
- Gradient calculation: A negative mean advantage triggers token-level re-weighting, with temperature T selected by bisection to make the weighted mean advantage approximately zero while enforcing T > 0.8.This heuristic addresses suspected positive–negative gradient imbalance associated with decreasing mean advantage and training instability.
- Experiments: On Alchemy-Random, the final algorithm produces the most stable training among vanilla GRPO, REC-OneSide-NoIS, and the re-weighted method.Vanilla GRPO has unstable reward curves and decaying mean advantage, while REC-OneSide-NoIS exhibits more severe issues.
- Remarks: The algorithm may need revised returns for longer task sequences and more principled baselines, because finite-sample baseline choices can affect practical policy-gradient estimates.Suggested return improvements include discounting or a finite sliding window; future work could develop a theoretically grounded algorithm tailored to CoD-Train.
B Hyperparameters and configurations
This section specifies the hyperparameters used for CoD training and evaluation and the configurations of its tasks and environments. The configurations include variant-sharing rules, a cross-domain-only TerminalSimulator setting, and fixed dataset sizes.
- Table 1 lists the hyperparameters for CoD training and evaluation experiments.
- Table 2 summarizes the configurations of tasks and environments, with values spanning both columns shared by easy and hard variants.
- TerminalSimulator is used only for cross-domain evaluation with a single configuration, and each dataset has 50,000 training instances and 4,000 test instances.
C Prompt templates
The experiments use runtime-filled prompt templates for update-context and solve-task episodes across FrozenLake-Obscure, Alchemy-Random, and TerminalSimulator. The solve-task templates instruct agents to act incrementally, reason under uncertainty, and balance short-term task completion with information gathering.
- Update-context episodes: Update-context prompts are generic and designed to apply across diverse environments, with double-braced fields filled at runtime.The prompt templates are presented in Tables 3–6, and runtime substitution is explicitly noted.
- FrozenLake-Obscure: FrozenLake-Obscure prompts require reaching G from P while discovering an unknown action-to-direction mapping through trial-and-error and contextual hints.Agents choose among Direction 1–4, avoid holes, seek short paths, and balance exploration with exploitation.
- Shared solve-task structure: Across solve-task prompts, agents take one action per step or round, provide a tagged answer, operate under response or horizon limits, and receive runtime state feedback.The Alchemy template exposes round counts, action feedback, and discovered recipes, while TerminalSimulator exposes current-step counts and command output.
- Alchemy-Random: Alchemy-Random prompts require synthesizing a target element by combining inventory elements while discovering unknown recipes through experimentation or contextual hints.Recipes use two inputs, produce a higher-tier output, and do not consume inventory elements; agents are encouraged to pursue short paths while balancing exploration and exploitation.
D Examples of hints
The examples show update-context hints that recover environment-specific structure and reusable procedures. They include action-to-direction mappings, crafting recipes with tiered synthesis plans, and reusable command sequences.
- Hints for FrozenLake-Obscure recover the environment’s action-to-direction mapping.
- The Alchemy-Random hint records discovered crafting recipes and an ascending-tier strategy for synthesizing the target element fhsj.The plan derives hjklt from lqlrr and gwzzz, nlji from lqlrr and vbpzh, joouk from hjklt and nlji, and fhsj from joouk and joouk.
- The TerminalSimulator hint distills reusable command sequences.