Source-linked AI summary
Joint Agent Memory and Exploration Learning via Novelty Signals
Shizuo Tian, Xiaohong Weng, Rui Kong, Yuxuan Chen, Guohong Liu, Yuebing Song, Jiacheng Liu, Yuchen Li, Dawei Yin, Ting Cao, Yunxin Liu, Yuanchun Li
TL;DR
Open-ended exploration is fundamental but difficult for language-model agents, and long histories make memory costly while latent memory lacks reliable supervision. JAMEL jointly trains memory and exploration through novelty-driven interaction, using persistent signals such as GUI code coverage. It generalizes to unseen environments, outperforming open-weight baselines and rivaling a closed-source model’s exploration depth while reducing token consumption.
Problem
Language-model agents struggle with open-ended exploration, while retaining long interaction histories is costly and latent memory lacks reliable step-level supervision.
Method
JAMEL jointly trains latent agent memory and exploration policy through novelty-driven interaction, using deterministic GUI code coverage as a persistent novelty signal.
Results
JAMEL generalizes to unseen environments, outperforms open-weight baselines, and rivals a closed-source model’s exploration depth while reducing token consumption.
Takeaways & Limitations
Novelty-seeking exploration can provide supervision for agentic memory while simultaneously driving discovery of unexplored behaviors.
Takeaways & Limitations
Dense interfaces with persistent modal overlays can occasionally challenge JAMEL’s compressed representation.
Abstract
from arXiv · showhide
In open-ended environments, exploration is fundamental for autonomous agents, yet current language model agents struggle with this. Effective exploration requires memory, but retaining raw interaction histories is computationally expensive over long trajectories. While latent memory offers a solution to compress interaction histories, its training lacks reliable supervisory signals. We introduce \textbf{J}oint \textbf{A}gent \textbf{M}emory and \textbf{E}xploration \textbf{L}earning (\textbf{JAMEL}), a framework that trains agentic memory and exploration policy together through novelty-driven interaction. We observe that memory and exploration form a mutually dependent loop: sustained exploration requires memory to distinguish exhausted behaviors from unseen ones, while novelty-seeking interaction provides the supervision needed to make memory useful for future exploration. By utilizing deterministic and persistent novelty signals such as code coverage in the GUI domain, we provide natural, annotation-free supervision for the memory module. Empirical evaluations demonstrate that \ours successfully generalizes to unseen environments. Its exploration capability outperforms open-weight baselines and rivals the exploration depth of a closed-source model while reducing token consumption. Our code and model are open-sourced at https://github.com/MobileLLM/JAMEL.
1 INTRODUCTION
JAMEL jointly trains latent agent memory and exploration through novelty-driven interaction. It addresses the computational cost of long histories and the lack of reliable memory supervision by using exploration-generated novelty signals.
- Exploration is difficult for language-model agents in open-ended environments, where extrinsic rewards are sparse or absent.
- Long interaction histories are expensive to retain, while latent memory lacks reliable step-level supervision.
- JAMEL treats memory and exploration as mutually dependent: memory avoids repetition, while novelty-seeking exploration supervises useful memory representations.
- Novelty rewards create a natural curriculum because increasingly familiar interactions leave deeper multi-step sequences as the remaining sources of novelty.
- JAMEL compresses histories into memory tokens, trains through rejection fine-tuning on 24k samples from 86 web applications, and generalizes to 10 held-out apps.
2 RELATED WORK
Related work situates JAMEL among approaches to efficient agent memory and exploration. The supplied passages describe context compression, retrieval, recurrent summaries, intrinsic novelty signals, and archive-based search methods.
- Agent Memory: Fixed context windows and external retrieval are early approaches to organizing interaction histories, but they scale poorly with interaction length.
- Agent Memory: Prompt tuning introduces trainable soft prefixes for parameter-efficient adaptation.
- Agent Memory: Recurrent memory transformers propagate summary states across interaction segments to model longer dependencies.
- Exploration Policy: Intrinsic-motivation methods such as ICM and RND encourage novelty through prediction or random-network errors in sparse-reward environments.
- Exploration Policy: Archive-based methods such as Go-Explore maintain frontier states for targeted exploration, while IGE incorporates LLM-based similarity judgments.
3 METHODOLOGY
JAMEL models exploration in partially observable environments using latent memory and persistent novelty rewards. It compresses prior observation-action history into memory tokens and trains the policy and memory aligner on novelty-driven trajectories, with GUI code coverage providing deterministic supervision.
- 3.1 EXPLORATION PROBLEM: JAMEL formulates exploration as a finite-horizon partially observable process in which actions are conditioned on the current observation and interaction history.The objective rewards behavior that has not appeared earlier in the same session rather than reaching a task-specific goal state.
- 3.2 MODEL ARCHITECTURE OF JAMEL: JAMEL compresses each historical observation-action pair into one latent memory token using a frozen vision-language model and an EOS hidden state.A learned linear aligner projects the tokens into the policy embedding space, where the projected memory is prepended to the input sequence.
- 3.3 NOVELTY-BASED INTRINSIC REWARD: Novelty is a binary intrinsic reward for discovering behavior absent from the accumulated history, and persistent signals prevent repeated states from generating reward.Previously visited states must never register as novel again, preventing cycles through a small state set from producing spurious exploration reward.
- 3.3 NOVELTY-BASED INTRINSIC REWARD: In the GUI domain, code coverage provides deterministic novelty supervision: a step is novel exactly when it executes a previously unexecuted code path.The cumulative coverage score sums executed lines, branches, statements, and functions; coverage is maintained across episodes so explored paths do not receive later reward.
- 3.3 NOVELTY-BASED INTRINSIC REWARD: The data-collection pipeline deploys a general-purpose LLM across GUI applications and yields 24k training samples from 86 web applications.Sessions contain multiple reset episodes, while the shared coverage baseline makes rewards progressively sparser and creates a natural curriculum.
- 3.3 NOVELTY-BASED INTRINSIC REWARD: Training retains prefixes through the last positive-reward step, then maximizes action likelihood conditioned on the current observation and precomputed memory tokens while jointly updating the aligner.Episodes without positive reward are discarded, and every retained step belongs to a trajectory that eventually produces novelty reward.
4 EXPERIMENTS
JAMEL is evaluated on unseen GUI applications using cumulative code-coverage reward, token consumption, reward trajectories, and per-app exploration patterns. It achieves strong exploration while compressing history into latent memory and reducing context cost.
- Experimental Setup: Evaluation uses 86 training apps and 10 unseen test apps, with 50 interaction steps per application.Agents are assessed on ScaleWoB using cumulative coverage reward as the exploration metric.
- Experimental Setup: JAMEL is compared with ReAct-text, ReAct-vision, MAI-UI-8B, and Mobile-Agent-v3.5 under the same environment and step budget.The baselines differ in explicit-history retention, screenshots, or hierarchical context compression.
- Main Results: JAMEL achieves the highest reward among small GUI agents, exceeds ReAct-text, and trails ReAct-vision by only 0.2 reward.JAMEL uses a 2B memory compressor and a 7B decoder while remaining competitive with the Gemini-based ReAct baselines.
- Reward Accumulation: JAMEL maintains a steady upward reward trajectory, whereas local baselines plateau early as exploration stagnates.JAMEL compresses historical information into latent memory tokens rather than truncating context, enabling continued discovery of application states.
- Reward Accumulation: Reward accumulates rapidly during shallow interactions and slows as deeper multi-step paths must be discovered.This progressively harder signal forms a natural curriculum for richer exploration strategies.
- Token Efficiency: Mobile-Agent-v3.5 and MAI-UI consume 2.76× and 2.81× more tokens per corresponding evaluation than JAMEL, respectively.Latent memory keeps the context compact compared with methods retaining longer explicit histories or screenshots.
- Exploration Patterns: Per-app trajectories vary with application structure: JAMEL explores deeply in some commerce and travel apps but plateaus early in media and lifestyle apps.Stepwise surges appear in Alibaba and Taobao, while Pinduoduo exposes prolonged plateaus linked to dense interfaces.
- Case Study: Persistent modal overlays in Pinduoduo can make visually available background elements functionally unresponsive, hindering interaction.Cleaner layouts such as Expedia allow JAMEL to use persistent navigation structures to move across modules.
5 DISCUSSION
The discussion identifies scaling novelty-driven memory architectures and using their latent memories for later task execution as future directions.
- Scaling Laws of Exploration: Future work should investigate scaling laws for novelty-driven memory architectures such as JAMEL.The discussion also identifies reinforcement learning as a natural direction because novelty rewards provide a progressively harder curriculum.
- Memory-Conditioned Task Execution and Continual Learning: An explore-then-execute paradigm could first build structural memory through autonomous exploration and then use it to execute user instructions.This direction connects exploration-generated latent memory with downstream task execution and continual learning.
6 CONCLUSION
JAMEL trains latent memory and exploration together with persistent novelty signals, generalizes to unseen GUI environments, and reduces token consumption. Dense interfaces with persistent modal overlays remain a challenge for its compressed representation.
- Conclusion: JAMEL jointly trains agentic latent memory and exploration through novelty-driven interaction.Persistent signals such as application code coverage supervise both capabilities by rewarding discovery of previously unexplored behaviors.
- Conclusion: JAMEL generalizes to unseen environments, outperforms current open-weight baselines, and rivals a closed-source model’s exploration depth while reducing token consumption.The conclusion presents memory and exploration as mutually reinforcing capabilities rather than separate modules.
- Conclusion: Dense interfaces with persistent modal overlays can occasionally challenge JAMEL’s compressed representation.This limitation constrains continuous state discovery in some interaction settings.