Source-linked AI summary
ExpSeek: Self-Triggered Experience Seeking for Web Agents
Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, Yongbin Li
TL;DR
Web agents can struggle with inefficient exploration and premature responses, while passive global experience injection does not adapt well to changing interaction contexts. ExpSeek therefore uses model entropy to trigger step-level experience seeking and generates guidance tailored to the ongoing interaction. Across four challenging web-agent benchmarks, it achieves 9.3% and 7.5% absolute improvements for Qwen3-8B and Qwen3-32B agents, respectively.
Problem
Web agents may explore inefficiently or respond prematurely, and passively injected experience can be difficult to align with changing step-level decisions.
Method
ExpSeek estimates step-level entropy thresholds for intervention timing and uses experience triplets plus historical context to generate tailored guidance.
Results
Across four challenging web-agent benchmarks, ExpSeek achieves 9.3% and 7.5% absolute improvements over Qwen3-8B and Qwen3-32B base models, respectively.
Takeaways & Limitations
Entropy can serve as an intrinsic trigger for proactive experience seeking, and a small-scale 4B experience model can improve a larger 32B agent.
Takeaways & Limitations
Threshold estimation relies on the training set and the tool model’s assessment of step quality, and extension beyond web domains remains unexplored.
Abstract
from arXiv · showhide
Experience intervention in web agents emerges as a promising technical paradigm, enhancing agent interaction capabilities by providing valuable insights from accumulated experiences. However, existing methods predominantly inject experience passively as global context before task execution, struggling to adapt to dynamically changing contextual observations during agent-environment interaction. We propose ExpSeek, which shifts experience toward step-level proactive seeking: (1) estimating step-level entropy thresholds to determine intervention timing using the model's intrinsic signals; (2) designing step-level tailored experience content. Experiments on Qwen3-8B and 32B models across four challenging web agent benchmarks demonstrate that ExpSeek achieves absolute improvements of 9.3% and 7.5%, respectively. Our experiments validate the feasibility and advantages of entropy as a self-triggering signal, reveal that even a small-scale 4B experience model can significantly boost the performance of larger agent models. The code is released at https://github.com/WYRipple/ExpSeek.
1 Introduction
ExpSeek shifts web-agent experience use from passive global injection to proactive, step-level seeking guided by the agent’s entropy and current interaction context. Across four benchmarks, it reports substantial gains for Qwen3-8B and Qwen3-32B agents.
- Motivation: Agents often explore inefficiently or respond prematurely in multi-turn environments, making reliable web interaction difficult.The problem is especially noted for small-scale, cost-effective language models.
- Framework: ExpSeek introduces a self-triggered framework in which agents proactively seek step-level experience guidance rather than relying on passive global injection.The framework addresses both when to seek experience and which guidance to seek during interaction.
- Method: ExpSeek estimates step-specific entropy threshold intervals with logistic regression and bootstrap resampling to control intervention timing.The intended timing is when the agent becomes confused and genuinely needs guidance.
- Method: The experience model retrieves relevant triplets from successful and failed trajectories and generates guidance tailored to the current historical context.This provides concrete content after the framework determines that intervention is needed.
- Results: 9.3% and 7.5% absolute improvements are achieved over Qwen3-8B and Qwen3-32B base models, respectively, across four challenging web-agent benchmarks.The experiments evaluate agents powered by both model sizes.
- Analysis: Self-triggered guidance increases entropy during intermediate interaction steps while decreasing it at the final answer step.The reported pattern is associated with broader exploration during processing and more confident convergence at answer time.
2 Related Work
Related work treats experience as accumulated memory for agents and includes offline refinement and online self-evolution. ExpSeek addresses the difficulty of aligning passively injected experience with decisions at individual steps, while extending entropy’s role to multi-step interaction.
- Experience-based agents: Experience functions as long-term memory that helps agents prevent repeated mistakes and accumulate insights, distinct from short-term contextual memory.
- Experience-based agents: Prior experience methods mainly follow offline refinement or online self-evolution paradigms.Offline methods process trajectories into reusable patterns, whereas online methods accumulate experience through iterative interaction and feedback.
- Experience-based agents: Passive experience injection is difficult to align with decisions made at individual interaction steps.
- Entropy in reasoning: Entropy has been used for static reasoning evaluation, including answer selection, confidence calibration, and error detection, and its role extends to multi-step reasoning and training.
3 Preliminaries
The paper models web-agent problem solving as interleaved reasoning and acting, then quantifies confidence at each interaction step and formalizes how experience is constructed and used. Unlike traditional methods that provide static experience at the beginning, ExpSeek can obtain guidance at any step from the current context.
- Agent Framework: Web-agent problem solving is represented as a ReAct trajectory alternating agent responses and environment observations.A trajectory contains the query, responses R_t, observations O_t, and a terminal response.
- Agent Framework: Process steps combine reasoning thoughts and tool actions followed by observations, whereas the answer step produces the terminal answer.Process steps occur before T; the terminal step contains the final answer y.
- Step Entropy: Step entropy measures agent confidence by averaging token entropy across each response R_t.Token entropy is based on the model’s predicted vocabulary distribution conditioned on preceding context.
- Experience Intervention: Experience intervention consists of constructing an experience base from training trajectories and using it as additional input during reasoning.The construction phase extracts E from D_train, while the utilization phase obtains applicable experience e_t from E and the current context.
- Experience Intervention: Traditional methods provide static experience at the beginning, while ExpSeek invokes the mapping function at any step to seek context-appropriate guidance.Traditional utilization uses e = G(E, q); ExpSeek uses G(E, h_t) during interaction.
4 Methodology
ExpSeek builds structured experience from successful and failed trajectories, then self-triggers tailored guidance at interaction steps using entropy-based thresholds. It distinguishes process and answer steps, estimates uncertainty-aware intervention ranges, and injects guidance according to step type.
- Experience Base Construction: ExpSeek constructs experience triplets describing behavior, mistakes, and directional guidance from contrasting successful and failed trajectories.A tool model labels step correctness and generates triplets for incorrect steps, which are organized into topics.
- Entropy Analysis for Step Correctness: Step entropy is used to test whether correct and incorrect process and answer steps are distinguishable without costly reward-model analysis.Correct steps have lower entropy; separability is statistically significant for both types, but discrimination is weaker for process steps than answer steps.
- Threshold Estimation: The method fits separate logistic regressions that model incorrectness from entropy and derive decision thresholds for process and answer steps.The probability of incorrectness increases with entropy, and the P = 0.5 boundary defines each threshold.
- Threshold Estimation: Bootstrap resampling converts fitted thresholds into 95% confidence intervals, providing separate threshold ranges for process and answer steps.Each resampled step collection preserves its original size before logistic regression is refit.
- Threshold-Based Triggering: During inference, low-entropy steps receive no intervention, high-entropy steps always receive guidance, and intermediate cases are handled probabilistically.The intervention probability is computed from the applicable threshold interval for each step type.
- Guided Intervention at Inference: When triggered, an experience model selects relevant topics, adapts guidance to the current context, injects it by step type, and disables the next-step intervention.Process guidance is appended to the observation, while answer guidance enables continued refinement or tool use.
5 Experiments
ExpSeek is evaluated across four challenging web-agent benchmarks with Qwen3-8B and Qwen3-32B, using accuracy, efficiency, and generalization analyses. It substantially outperforms vanilla ReAct and global intervention baselines while retaining cross-task performance.
- Main Results: ExpSeek achieves average absolute improvements of 9.3% and 7.5% over vanilla ReAct on Qwen3-8B and Qwen3-32B, respectively.The comparison spans the evaluated benchmarks and reports average absolute gains for both agent sizes.
- Main Results: Global intervention baselines improve by under 3% or degrade performance on challenging open-web reasoning tasks.The paper attributes this limited effectiveness to poor adaptation to noisy web environments and added reasoning burden for smaller agents.
- Main Results: ExpSeek maintains robust performance across three out-of-distribution benchmarks despite deriving its experience entirely from WebWalkerQA training data.This demonstrates cross-task generalization beyond the experience source domain.
- Ablation Study: Guiding only process or answer steps falls below full intervention by 4.91% and 2.44% on 8B, and 4.51% and 4.12% on 32B.Answer-only guidance remains closer to the full method than process-only guidance.
6 Experimental Analyses
The analyses examine ExpSeek’s internal behavior, trigger design, intervention intensity, retrieval and generation, model scaling, repository size, and efficiency. Results support entropy-triggered, generative guidance with adaptive intervention and show gains from smaller experience models and compact repositories.
- Internal Mechanisms: ExpSeek increases entropy during process steps but shifts answer-step entropy lower with a sharper peak.The resulting diverge-then-converge behavior supports broader exploration followed by higher-confidence answer convergence.
- Triggering and Efficiency: Rule-based triggering incurs 1.7× step and 2.6× time overhead on GAIA, while entropy-based triggering maintains similar accuracy with lower overhead.On xbench, rule-based overhead is 1.5× in steps and 2.1× in time; RM also increases time overhead despite reducing step overhead.
- Triggering and Efficiency: Entropy-based triggering raises its trigger rate by 25.6% on xbench versus GAIA while maintaining similar accuracy.This pattern is reported as evidence that intervention timing adapts to problem difficulty.
- Guidance Design: Generative guidance yields substantially higher accuracy than retrieval-based guidance in the reported GAIA and xbench comparison.The analysis identifies generative guidance as necessary for the evaluated setting.
- Scaling and Transfer: A 4B experience model improves the 32B agent by 5.2 and 9.7 percentage points on GAIA and xbench, respectively.The three tested experience-model sizes exhibit scaling on both benchmarks, while repository swapping retains transfer value for abstract guidance knowledge.
- Repository Size: ExpSeek maintains high accuracy with one experience per topic, whereas removing the repository degrades performance.The remaining accuracy after repository removal indicates that the experience model retains inherent world knowledge, while topic coverage remains important.
- Intervention Intensity: With approximately two interventions, accuracy reaches 43.01%; beyond six interventions, additional performance gains are minimal.Lower trigger thresholds rapidly increase reasoning steps, while small threshold fluctuations leave performance stable.
7 Conclusion
ExpSeek enables web agents to seek step-level guidance using step entropy. On complex real-world web reasoning tasks, it improves performance and generalizes across settings with small open-source agents.
- Conclusion: ExpSeek enables web agents to actively seek step-level guidance based on step entropy.The conclusion presents entropy-triggered, step-level guidance as the framework’s central capability.
- Conclusion: ExpSeek demonstrates significant performance improvements and strong generalization on complex real-world web reasoning tasks, including with small open-source agents.The conclusion characterizes these results as indicating potential for future development.
Limitations
The paper identifies limitations in threshold estimation, domain and tool coverage, and possible use for agentic reinforcement-learning rollouts.
- Limitations: Threshold estimation currently relies on the training set and a tool model’s assessment of step quality.The paper calls for more accurate threshold-estimation strategies.
- Limitations: Whether ExpSeek extends to nonweb domains or integrates more tools remains unexplored.This limits the demonstrated scope to the evaluated web-agent setting.
- Limitations: The paper has not studied whether ExpSeek can enhance agentic reinforcement-learning rollouts to improve convergence speed and sampling quality.This question is motivated by ExpSeek’s reported pass@k improvements.
Ethical Considerations
The method is intended for academic research only and excludes applications involving risks, religion, racial discrimination, or ethical violations.
- The authors restrict the method to academic research applications.They explicitly exclude applications involving risks, religion, racial discrimination, or ethical violations.
A Details of Method
ExpSeek estimates entropy-based intervention thresholds through logistic regression and bootstrap resampling, then uses them to control experience-seeking timing. Its analyses connect uncertainty patterns and targeted guidance to performance and reasoning behavior.
- Threshold estimation: Bootstrap resampling generates N = 1000 datasets for estimating a distribution of logistic-regression decision boundaries.Each resampled dataset trains the threshold-estimation procedure.
- Threshold estimation: The logistic-regression decision boundary θ = −b/w identifies the entropy value where predicted correctness reaches 0.5.This boundary is interpreted as the point of maximum model uncertainty and a natural intervention threshold.
- Threshold estimation: The 95% confidence interval [θlower, θupper] uses the 2.5th and 97.5th percentiles of the bootstrap distribution.The quantile function supplies the interval endpoints.
- Computational efficiency: Threshold estimation for one step type completes within seconds, making the total offline computation negligible relative to online inference costs.Process and answer steps are estimated independently.
- Performance analysis: ExpSeek improves sampling diversity over vanilla, with absolute pass@3 gains of 12.9% and 8.8%.The two baselines do not show better diversity and fall below vanilla on multiple datasets.
- Guidance mechanism: ExpSeek provides real-time guidance from an experience model, keeping experience connected to the agent’s current reasoning context.The framework is designed to address difficulties in acquiring and using experience during multi-turn interactions.
B.3 Case Studies
The case studies show that incomplete evidence can lead agents to incorrect entity-attribute inferences, while targeted guidance redirects them toward authoritative sources and explicit verification.
- B.3 Case Studies: The unguided agent incorrectly counted films as Universal releases after relying on snippets that omitted distributor information.“Spider-Man: No Way Home” was distributed by Sony Pictures, illustrating distributor-attribute confusion.
- B.3 Case Studies: ExpSeek’s extended results include Pass@3 accuracy for baselines and ExpSeek, together with absolute differences from experience-free ReAct.The accompanying case-study tables document interactions with and without guiding experience.
- B.3 Case Studies: Guidance redirected the agent to authoritative box-office sources and reinforced the need to satisfy both revenue and distributor constraints.The guidance emphasized verification rather than directly supplying the answer.
- B.3 Case Studies: The experience triplet design targets process and answer steps by pairing behavior, mistakes, and directional guidance.Process-step experience focuses on source-selection decisions and extending trajectories for thorough verification.
- B.3 Case Studies: The evaluation uses Qwen3-235B-A22B-Instruct-2507 as an LLM judge and employs search and visit tools for web interaction.The four benchmarks are described as recognized in web-agent research and permitted for academic studies.
- B.3 Case Studies: The prompt suite covers triplet generation, topic assignment, topic selection, guidance generation, intervention triggering, and direct experience retrieval.These prompts support both constructing and using the experience base during inference.