Source-linked AI summary
Online Experiential Learning for Language Models
Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei
TL;DR
Offline training leaves deployment experience underused, especially when server-side training cannot access user environments or scalar rewards are unavailable. OEL extracts experiential knowledge from user-side trajectories and distills it into model weights, yielding iterative gains in task accuracy and efficiency while retaining out-of-distribution performance.
Problem
Offline training relies on human annotations or simulated environments, while deployment environments are inaccessible to server-side training and often provide textual rather than scalar feedback.
Method
OEL extracts and accumulates transferable experiential knowledge from user-side trajectories, then consolidates it into model parameters through on-policy context distillation without user-side environment access.
Results
Across text-based game environments, model scales, and thinking variants, OEL consistently improves task accuracy and inference efficiency while preserving out-of-distribution performance.
Takeaways & Limitations
Extracted experiential knowledge is more effective than raw trajectories, and on-policy consistency is critical for effective learning.
Takeaways & Limitations
The extraction process does not use ground-truth labels and retrieves knowledge at a fixed accumulation step rather than selecting optimal knowledge with scalar rewards.
Abstract
from arXiv · showhide
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.
1 Introduction
OEL addresses the limits of offline training by enabling language models to learn from deployment experience despite inaccessible user environments and textual rather than scalar feedback. It extracts transferable knowledge from trajectories and consolidates it into model parameters, producing iterative improvements in accuracy and efficiency.
- Offline training depends on pre-collected human annotations or simulated environments, limiting improvement to data and environments curated before deployment.
- Real-world deployment creates textual feedback, while server-side training typically cannot access the user-side environments that produced it.
- OEL extracts transferable experiential knowledge from deployment trajectories and consolidates it into model parameters through on-policy context distillation without reward models, verifiable rewards, or human annotations.
- Across two environments, model scales, and thinking variants, OEL consistently improves task accuracy and inference efficiency while preserving out-of-distribution performance.
2 Preliminary: Online Learning
Online experiential learning uses deployment interactions as training signal instead of relying solely on pre-constructed supervision. The approach supports a feedback cycle in which real-environment experience updates the model without annotations, verifiable rewards, or simulated environments on the server side.
- Offline learning trains on pre-constructed data using human annotations or simulated environments with verifiable rewards.
- The offline paradigm faces a coverage ceiling because increasingly costly annotations or faithful simulations cannot fully represent deployment diversity.
- Online experiential learning feeds test-time experience from real user environments back into server-side model updates.
- The paradigm relies on textual environment feedback and requires no human annotations, verifiable reward functions, or simulated environments on the server side.
3 Online Experiential Learning
OEL collects user-side interaction trajectories, extracts accumulated experiential knowledge, and consolidates it into model weights on the server side. Iterating these stages lets improved models generate higher-quality trajectories while training remains independent of user-side environment access.
- 3.1 Extract Experiential Knowledge from User Trajectories: OEL collects multi-turn trajectories from a user-side environment and extracts transferable experiential knowledge sequentially while conditioning on previously accumulated knowledge.
- 3.1 Extract Experiential Knowledge from User Trajectories: The extraction stage uses interaction trajectories and does not require ground-truth labels.
- 3.2 Consolidate Experiential Knowledge into Model Weights: Extracted knowledge is consolidated into model parameters through on-policy context distillation using partial rollout prefixes and a knowledge-conditioned teacher.
- 3.2 Consolidate Experiential Knowledge into Model Weights: Single-turn server-side rollouts provide token-level training signals from textual feedback without user-side environment access, reward models, or verifiable rewards.
- 3.3 Online Learning Process: After consolidation, the updated model is redeployed to collect higher-quality trajectories and richer knowledge for subsequent rounds.
4 Experiments
OEL is evaluated in text-based games across model variants and scales, with iterative extraction and consolidation improving performance and efficiency. Experiments also examine on-policy distillation, model size, and the value of extracted knowledge over raw trajectories.
- Experimental Setup: Experiments use Frozen Lake and Sokoban with thinking and non-thinking Qwen3 models, including Qwen3-1.7B and Qwen3-4B-Instruct-2507.
- OEL Enables Online Learning: Iterating experiential knowledge extraction and consolidation progressively improves task performance across both game settings.The loop is demonstrated on Frozen Lake with Qwen3-1.7B and Sokoban with Qwen3-4B-Instruct-2507.
- Token Efficiency: Roughly 70% of initial response length by the third iteration is reached by Qwen3-1.7B on Frozen Lake while pass rate concurrently improves.The reduction is attributed to experiential knowledge helping the model reach correct answers faster, with the pattern retained after consolidation.
- Distillation Strategy: On-policy context distillation achieves higher in-distribution performance and better preserves out-of-distribution performance than off-policy context distillation.Figure 6 compares game pass rate with IF-Eval accuracy using Qwen3-1.7B on FrozenLake.
- Effect of Model Size: OEL improves pass rates across Qwen3-1.7B, 4B, and 8B, with gains from Round 1 to Round 2 consistent across model sizes.Larger models generally achieve higher pass rates and generate higher-quality trajectories for knowledge extraction.
- Experiential Knowledge versus Raw Experience: Extracted experiential knowledge substantially improves pass rate before and after consolidation, whereas raw interaction trajectories yield only modest improvement.Table 1 evaluates Qwen3-4B-Instruct-2507 on Sokoban and supports extraction as an essential OEL stage.
5 Related Work
Related work spans on-policy distillation, context distillation, and learning from interaction experience, while OEL's evaluation highlights the importance of on-policy knowledge matching.
- Qwen3-1.7B benefits more from on-policy experiential knowledge derived from its own trajectories than from off-policy knowledge produced by Qwen3-4B on Frozen Lake.The comparison is reported in Table 2.
- On-policy distillation trains students on their own trajectories to reduce train-inference mismatch, whereas context distillation transfers in-context knowledge into model parameters.OEL uses on-policy context distillation as its consolidation mechanism.
- Prior experience-based methods use reflection, external memory, or self-play to improve subsequent behavior from interaction histories.These approaches include reflecting on failures and extracting trajectory insights for retrieval.
6 Conclusion
OEL is presented as a reward-free online learning loop that extracts deployment experience into knowledge and consolidates it into model parameters. Experiments report iterative gains in accuracy and efficiency while preserving out-of-distribution performance.
- OEL extracts transferable experiential knowledge from deployment trajectories and consolidates it into model parameters through on-policy context distillation.The framework requires no human annotations, reward models, or server-side access to user environments.
- Experiments across model scales and thinking variants show consistent improvements in task accuracy and inference efficiency over successive iterations.Response lengths decrease as experiential knowledge is internalized.
- OEL preserves out-of-distribution performance, while analysis confirms that knowledge extraction and on-policy consistency are important components.
A Implementation of On-Policy Context Distillation
On-policy context distillation trains a student on its own sampled responses to match a knowledge-conditioned teacher, using reverse KL and trajectory-derived training contexts. The implementation supports cross-trajectory and self-trajectory variants.
- The student model matches a teacher conditioned on guiding context by minimizing reverse KL divergence on responses sampled from the student policy.The guiding context is prepended to the input for the teacher but not required by the student at inference.
- The loss decomposes sequence-level divergence into token-position terms, with the vocabulary summation approximated using the student’s top-k tokens and k = 256.
- Cross-Trajectory Variant: In the Cross-Trajectory variant, knowledge accumulated from trajectories T is applied to partial rollout prefixes from a separate set T′.This decouples knowledge extraction from training-data construction and supports generalization across trajectories.
- Self-Trajectory Variant: In the Self-Trajectory variant, knowledge extracted from each trajectory is paired only with prefixes derived from that same trajectory.This creates tighter coupling between extracted knowledge and its training context.
- Online Experiential Learning Procedure: The algorithm collects user-side trajectories, extracts experiential knowledge server-side, constructs partial-rollout training prefixes, and consolidates knowledge into model weights.
C.1 Dataset Details
The dataset uses Frozen Lake and Sokoban text games in which models discover rules through textual feedback rather than explicit instructions. Prompt templates support structured or unstructured knowledge extraction and knowledge-guided problem solving.
- Environments: Frozen Lake requires navigating a grid to a goal while avoiding holes, whereas Sokoban requires pushing a box to a target without becoming stuck.The experiments use a 3 × 3 Frozen Lake grid with two holes and a 6 × 6 Sokoban grid with one box.
- Environments: Neither game provides explicit rules, so models must discover them through exploration under minimal prior knowledge.The setup replaces TextArena’s original rules with general task descriptions.
- Environment Feedback: Each turn returns textual feedback about the action outcome and an updated map, enabling multi-turn interaction with the environment.
- Prompt Templates: The prompt templates cover structured extraction, unstructured extraction, and embedding accumulated experiential knowledge into new problem solving.The unstructured and new-problem templates are identified in Figures 11 and 12.
- Prompt Templates: The interaction-history template presents prior experience as the game input and the model’s response and action output.
- Prompt Templates: The agent is framed as a grid-game reasoning engine whose learned rules or strategies may remain incomplete or incorrect.
C.3 Extraction Stage
The extraction stage accumulates transferable experiential knowledge from deployment trajectories, using distinct structured and unstructured extraction settings and a fixed accumulation step across rounds.
- Extraction formats: Structured extraction prompts the model to output transferable knowledge items with a required “– EXPERIENCE ITEM:” prefix.Entries that do not conform to the format are discarded.
- Extraction settings: Structured extraction uses n = 25 or n = 50 trajectories with Lmax = 8192, whereas unstructured extraction uses n = 15 and Lmax = 2048.The settings define different accumulation sizes and maximum generation lengths for the two formats.
- Configuration: Tables 3 and 4 document the extraction and consolidation hyperparameters, with final configurations fixed for each model-task pair across OEL rounds.The supplied passages identify these tables as the references for search ranges and final configurations.
- Knowledge accumulation: The extraction process is repeated K = 10 times with different random seeds for both formats, producing accumulated experiential knowledge C.The resulting knowledge set is detailed in Tables 3 and 4.
- Knowledge selection: Because extraction is performed server-side without scalar environment rewards, OEL retrieves knowledge at a fixed accumulation step rather than selecting the optimal knowledge.This retrieval rule is applied across OEL rounds.
C.4 Consolidation Stage
The consolidation stage applies on-policy context distillation using sampled experiential knowledge, with fixed training configurations across OEL rounds and knowledge examples illustrating the resulting content.
- Distillation procedure: On-policy context distillation runs for 20 or 100 steps per OEL round, using 64 game samples per step.This requires 1280 or 6400 trajectory samples per training round.
- Distillation procedure: Each game interaction spans up to 5 turns, with a maximum response length of 1024 tokens per turn.For each training prefix, experiential knowledge e is randomly sampled from C.
- Optimization: The reverse KL divergence uses the top 256 vocabulary tokens ranked by student-model probability, while learning rates are searched in [1e-6, 5e-6].The learning rate remains fixed across OEL rounds, and sampling temperature is 0.7.
- Configuration: Hyperparameters are fixed across OEL rounds for each model-task pair, and Qwen3-1.7B excludes previously accumulated knowledge from extraction context.The exclusion is motivated by the finding that smaller models lack sufficient capacity to leverage long contextual information.
- Knowledge examples: A Sokoban experiential-knowledge example describes “axis-aligned convergence,” directing moves to reduce Manhattan distance toward the goal.The example frames directional progress as the game’s core mechanic and links it to shortest-path behavior in static obstacle-filled environments.