Source-linked AI summary
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, Aviral Kumar
TL;DR
Interactive tasks can require agents to acquire new information and adapt, which longer reasoning before acting does not provide. The paper scales test-time interaction and trains TTI with a curriculum that adjusts rollout lengths. It reports improved performance on WebVoyager and WebArena, along with adaptive exploration and exploitation.
Problem
Longer per-step reasoning does not let interactive agents acquire new environmental information or adapt their behavior over time, despite those needs in uncertain and evolving tasks.
Source: §1 · pp. 1–2Method
The paper scales interaction horizons and introduces TTI, a curriculum-based online RL approach that adaptively adjusts rollout lengths.
Source: §1 · p. 2Results
TTI achieves state-of-the-art performance among open-source, open-data agents on WebVoyager and WebArena, improving over the non-fine-tuned agent by 9% and 8%, respectively.
Source: §1 · p. 2Takeaways & Limitations
The results present interaction scaling as a complementary test-time scaling dimension that supports agents exploring and adapting dynamically.
Source: §1 · p. 2, §7 · p. 14WebArena learning can be slower because its tasks are more complex, and some otherwise valid real-world actions fail in its counterparts due to environment bugs.
Source: §6.3 · p. 13
Abstract
from arXiv · showhide
The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction, an untapped dimension of test-time scaling that increases the agent's interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents.
1. Introduction
Reactive policies and longer per-step reasoning can fall short when web tasks require new information or adaptation. The paper proposes scaling interaction, then trains TTI with a curriculum; it reports gains from both prompting and training.
1. Introduction
- Reactive policies and longer reasoning from the agent’s internal state can be suboptimal when tasks are partially observable or non-stationary and require information-seeking behavior.Examples include incomplete page details and fluctuating flight prices.Source: §1 · pp. 1–2
- Interaction scaling increases the number of steps at deployment, allowing agents to gather environmental information and try behaviors that per-step chain-of-thought alone cannot support.The paper illustrates this with browsing hotel listings to compare reviews and availability before choosing.Source: §1 · p. 2
- TTI uses online RL and a curriculum that first teaches effective exploitation, then gradually extends the interaction horizon to encourage exploration.The curriculum addresses the risk that longer rollouts produce aimless exploratory actions without progress toward the task.Source: §1 · p. 2
- Prompted interaction scaling raises WebArena task success from 23% to ≥28%, while equivalent compute spent on longer thinking or best-of-n yields less than a 3% gain.The prompt asks the agent to think and act again after it decides to terminate.Source: §1 · p. 2
- TTI achieves state-of-the-art performance among open-source agents trained on open data on WebVoyager and WebArena, improving over the non-fine-tuned agent by 9% and 8%, respectively, with Gemma 3 12B.The training pipeline scales to more than 100K tasks across approximately 20 domains using synthetic tasks for online data collection.Source: §1 · p. 2
- Curriculum training supports adaptive exploration: agents initiate searches or backtrack in complex tasks while taking efficient paths in simpler ones.The paper presents interaction scaling as complementary to scaling per-step compute.Source: §1 · p. 2
2. Related Work
Prior work scaffolds foundation models, trains agents through interaction, or scales test-time computation. This paper distinguishes its approach by scaling the interaction horizon so agents can gather information and adapt over time.
Scaffolded foundation models as web agents
RL training for foundation model agents
- Prior RL agents can improve through interaction, but most lack test-time exploration for adapting over long horizons under partial observability.AutoWebGLM [65] and WebRL [23] base curricula on estimated task difficulty, whereas this approach gradually increases the allowed interaction horizon.Source: §2 · p. 3
Scaling test-time compute
- Existing test-time methods scale per-action search, verifiers, agent count, or per-step reasoning, while this work scales the time horizon for exploration, backtracking, and information gathering.The related-work discussion names EXACT [75], GenRM-CoT [76], and Jin et al. [77] as examples of other scaling targets.Source: §2 · p. 3
3. Problem Setup
The setup models web tasks as finite-horizon sequential decisions: an agent acts from observation and action histories until stopping or reaching the interaction limit, with success receiving reward. It specifies the observations and discrete actions available to the agent.
3. Problem Setup
- The paper centers its study on web agents but states that the insights should generalize to other agent domains and expresses hope that future work extends them beyond web navigation.Source: §3 · p. 3
- A web task is a finite-horizon decision process with an evolving environment; the policy maps observation and action histories to the next action, and task success receives reward 1 versus 0 otherwise.The process ends when the agent stops or reaches horizon h; h_stop denotes the number of steps actually taken.Source: §3 · pp. 3–4
Observation space design
- Observations include the task goal, URL, accessibility tree, and a screenshot with identifiers overlaid on interactive elements; only the most recent three observations are retained, while past actions remain accessible.The stated reason for truncating observation history is that retaining all observations quickly exhausts the context window in practice.Source: §3 · p. 4
Action space parameterization
- The discrete action space comprises click, type, scroll, go back, search, and stop with an answer; compound actions such as goto[url] are excluded.The passage says complex action spaces can hinder performance.Source: §3 · p. 4
4. Scaling Test-Time Interaction: A New Dimension of Agent Scaling
The section compares extending interaction with increasing per-step reasoning under test-time compute. It finds interaction scaling more effective in WebArena, while prompting alone can confuse agents or fail to support nuanced behavior.
4. Scaling Test-Time Interaction: A New Dimension of Agent Scaling
- The controlled WebArena study uses 62 test tasks and ℎ=30; its Gemma 3 12B CoT baseline reaches 23.81% success.The study is for analysis rather than benchmarking; the prompting agent uses a single model call without retrieval, verifiers, or other external modules.Source: §4 · p. 4
4.1. Scaling Test-Time Interaction by Acting Longer
- Re-checking raises success on most WebArena domains and changes the agent’s action about 25% of the time after double-checking.Agents may reaffirm their decision or revise it after reflection.Source: §4.1 · p. 5, Figure 2 · p. 5
4.2. Scaling Test-Time Interaction vs. Per-Step Test-Time Compute
- At matched token budgets, interaction scaling achieves the highest success rate; best-of-n incurs the highest cost for the least improvement.Budget forcing plateaus around 0.26.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
- Prompting alone cannot support mid-rollout switching between exploration and exploitation, and repeated re-checking or thinking can confuse agents and degrade performance.Repeated re-checks can turn correct answers into incorrect ones.Source: §4.1 · p. 5, §4.2 · p. 6
- Across WebArena tasks, interaction scaling outperforms longer per-step reasoning by spreading compute across more, lower-cost steps that gather new environmental information.Forced waits changed actions only 15% of the time, mostly by reordering subtasks rather than exploring alternatives or correcting errors.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
5. TTI: Curriculum-Based Online RL for Scaling Interaction
Section 5 examines whether training at longer horizons is enough, then presents TTI’s curriculum over interaction horizon. Fixed horizons create learning trade-offs, while curriculum-based horizon growth outperforms fixed-horizon training.
5.1. Challenges of Training Agents with Fixed, Long Horizons
- h=5 learns quickly but overfits and later terminates prematurely; h=20 learns more slowly under noisy credit assignment, yet helps more on exploratory CMS than simpler Reddit.Increasing interaction length alone is insufficient to learn reliable interaction scaling.Source: §5.1 · p. 7, Figure 4 · p. 8
5.2. Our Approach: Curriculum Over Interaction Horizon
- TTI starts with short trajectories and gradually raises the maximum interaction steps per trajectory, without requiring explicit task-complexity labels.Interaction length serves as a rough proxy for task difficulty.Source: §5.2 · p. 8
- The curriculum contrasts conservative additive horizon growth, which allows core skills to solidify, with more aggressive multiplicative growth that exposes agents to exploration earlier.The multiplicative schedule assumes agents can acquire basic skills quickly and benefit from earlier exploratory behavior.Source: §5.2 · p. 8
- Nearly 3%: with about 700 training tasks, TTI outperforms fixed h=20 and uses 40% fewer training steps over 10 iterations.The multiplicative curriculum also outperforms the additive curriculum and is adopted as TTI’s default.Source: Table 2 · p. 8, §5.2 · p. 8, Figure 4 · p. 8
6. Experiments: Scaling Up to Realistic Benchmarks
The experiments evaluate TTI on WebVoyager and WebArena, finding strong open-data results and gains over fixed-horizon training. They also show adaptive exploration in complex tasks, efficient exploitation in simpler tasks, and remaining failure modes.
6. Experiments: Scaling Up to Realistic Benchmarks
- Training uses Gemma 3 12B with 128K synthetic WebVoyager tasks and 11K WebArena tasks, training separate agents for the two benchmarks.Rollouts use a multiplicative curriculum with minimum horizon 10 and maximum horizon 30.Source: §6 · p. 9
6.1. WebVoyager Results and Analysis
- TTI reaches 64.8% average success on WebVoyager and leads open-source, open-data agents; on full WebArena it leads open-source self-improvement agents and improves 7.8% over zero-shot.The WebVoyager result is the best aggregate result among open-weight agents trained on open-source synthetic data; WebArena evaluation uses the benchmark’s original evaluators.Source: §6.1 · pp. 9–10, Table 3 · p. 10, §6.3 · pp. 12–13
- TTI beats fixed h=10 by 5.7% and fixed h=30 by 19.6% in WebVoyager average accuracy; its trajectories first shrink, then expand as the allowed horizon grows.As the horizon expands, GoBack and Bing actions and trajectory length rise while per-step token use falls; fixed h=30 instead drifts into aimless exploration and hurts performance.Source: Figure 5 · p. 9, §6.1 · pp. 10–11, Figure 6 · p. 11
- TTI improves success by 31.4% on Allrecipes and 15.6% on Cambridge, but underperforms baselines on Amazon and GitHub, where searches can distract from built-in filters.The authors suggest information-dense tasks benefit from extended exploration, while noisy external information can reduce success in domains where the base model already performs strongly.Source: §6.1 · p. 10
6.2. Case Studies: Strengths and Failure Modes
- Case studies show TTI explores and backtracks to compare options on complex tasks, yet completes simpler tasks efficiently without over-exploration.Longer training horizons precede the emergence of exploratory behavior in the apple-pie recipe example.Source: §6.2 · p. 12
- Observed failures include repeated Bing resets instead of domain-specific recovery and submitting answers despite recognizing they fail the task criteria.The authors identify more systematic exploration and stronger self-verification as areas for improvement.Source: §6.2 · p. 12
6.3. WebArena Results and Analysis
- The authors attribute WebArena’s smaller gains partly to task complexity, fewer successful rollouts, and actions that fail in benchmark environments such as Postmill search.They note that some actions valid in real-world domains are invalid in WebArena counterparts because of environment bugs.Source: §6.3 · p. 13
- Re-checking improves success across TTI training stages, with clearer benefits early and modest gains persisting later.As TTI training progresses, longer interaction traces naturally incorporate behaviors like re-checking, reducing the added benefit of explicit re-checks.Source: §6.3 · p. 13, Figure 7 · p. 13
7. Conclusion and Future Work
The paper presents interaction scaling as a test-time scaling dimension for interactive agents and reports that it supports dynamic exploration and adaptation. It identifies broader domains and better ways to balance interaction, reasoning, and RL training as future directions.
7. Conclusion and Future Work
- Web-only experiments show interaction scaling can support dynamic exploration and adaptation while improving task performance.The authors propose extending the approach to more stochastic and uncertain settings, including robotic control and open-world compute use.Source: §7 · p. 14
- The authors leave open how RL training should balance longer interaction against per-step reasoning, despite observing that reasoning tokens decreased as agents acted longer.They report that this shift made better use of total test-time compute in their experiments, but say no general rule or training procedure is clear.Source: §7 · p. 14
- The current online filtered behavior cloning approach is described as simple, with value-function RL and memory or context limits identified as future challenges as horizons grow.The authors suggest value functions may help with credit assignment in stochastic, non-stationary environments.Source: §7 · p. 14
Appendices
The appendices discuss deployment considerations and explain the observation inputs used for web agents. The observation design pairs screenshots with accessibility trees, whose richness trades off against context length.
A. Broader Impact
- The authors caution that increased autonomy could amplify benefits and risks in sensitive real-world applications, while their experiments use simulated environments.They call for safeguards, interpretability tools, and human-in-the-loop designs in future deployments.Source: A. Broader Impact · p. 24
B. Observation Space Design
- Agents receive screenshots alongside accessibility trees; the rich tree includes more HTML tags and attributes than the simple tree but is longer and harder to optimize because of context length.The appendix describes the rich tree as modified from WebArena code and the simple tree as modified from PAE code.Source: B. Observation Space Design · p. 24
C. TTI Implementation
The implementation describes TTI as filtered behavior cloning with interaction scheduling. It uses a replay buffer that favors recent rollouts when sampling training examples.
C. TTI Implementation
- TTI is implemented as filtered behavior cloning with interaction scheduling, initialized from a pretrained policy and configured with an evaluator, environment, learning rate, replay buffer, and horizon bounds.The algorithm pseudocode identifies the policy, evaluator, environment, and scheduler hyperparameters as inputs.Source: C. TTI Implementation · p. 25
- Rollouts are collected for replay-buffer updates, and sampling weights favor more recently added trajectories.The text says recency weights are used to encourage learning from recent examples.Source: C. TTI Implementation · pp. 24–25
D. WebArena Experiments
The WebArena appendix gives the agent a structured prompt for reasoning and action selection, then adds domain-specific navigation hints. It also reports that a more complex thought prompt did not significantly improve accuracy and could increase costs.
D.1. WebArena Agent Prompt
- A more complex thought prompt did not significantly improve downstream accuracy and could increase training and inference cost, so the authors did not use it.The reported accuracy comparison refers to Table 5.Source: §D.1 · pp. 26–27
- The agent prompt asks for page summarization, planning, verification, backtracking when needed, and comparison of candidate actions before selecting one action and continuing after a new observation.The prompt specifies action formats including clicking, typing, scrolling, going back, and answering only after task completion.Source: §D.1 · pp. 26–27
D.2. WebArena Domain-Specific Prompts
- For some tasks, the hints prescribe multi-page or item-by-item exploration, such as checking all forum pages or inspecting orders individually within a date range.Other examples include inspecting product reviews and stopping at the item page when the task only asks to find an item.Source: §D.2 · p. 28
- Domain-specific hints cover common interaction rules such as saving changes, using dropdowns, and clearing filters, alongside search and navigation advice tailored to particular sites.Examples include formatting dates, iterating through orders, and adjusting location searches or transportation selections.Source: §D.2 · pp. 27–28
D.3. CoT Experiments for Base Agent
The base-agent CoT experiments analyze 62 randomly sampled WebArena tasks and report results averaged over three runs. The supplied passages identify the prompting variants and experimental setup but do not provide outcome values.
D.3. CoT Experiments for Base Agent
- Rollouts use multiple Docker containers on one GPU with vLLM and the listed inference settings.The settings include max_new_tokens 1024, temperature 1, and top_p 0.95.Source: §D.3 · p. 29
- The analysis randomly subsamples 62 WebArena test tasks and compares the zero-shot agent with CoT and Complex CoT prompting.CoT uses the General Prompt and Complex CoT uses the Complex Prompt.Source: §D.3 · p. 29
- The base-agent results in Table 5 are averaged over three runs on a WebArena subset.The supplied table passage gives its scope but no cell values.Source: Table 5 · p. 29
D.4. Scaling Trade-off Experiments
The section compares prompting strategies for scaling interaction and per-step reasoning, then records training schedules and the task-generation, evaluation, and agent prompts used in the experiments.
D.4. Scaling Trade-off Experiments
- Check-again prompting appends a request after the task-stop signal and varies its wording to prompt answer re-evaluation.The prompts ask the agent to reconsider the current page and whether evidence supports its answer.Source: §D.4 · p. 29
- Per-step budget forcing prompts longer reasoning with successive phrases such as “Wait, let me think deeper” and “But hold on.”The phrases change across repeated waits.Source: §D.4 · pp. 29–30
- Per-step best-of-n selection by majority voting performed slightly better than selection by log likelihood.The passage does not report the size of the difference.Source: §D.4 · p. 30
- The experiments also tested combining methods that scale along different axes.The supplied passage does not report the combined methods’ outcomes.Source: §D.4 · p. 30
D.5. TTI and Online Filtered BC Hyperparameters for Preliminary Experiments
- Preliminary experiments freeze Gemma 3’s vision tower and compare multiplicative horizons of 10, 20, 30 with additive horizons of 10, 11, 12, 13.The listed schedule continues at 30 for the multiplicative curriculum.Source: §D.5 · p. 30
D.6. TTI Hyperparameters for Full WebArena Experiments
- Full WebArena experiments use a horizon schedule of 10, 20, 20, 30, 30, 30, with evaluation horizon 30.The listed schedule continues at 30.Source: §D.6 · pp. 30–31
E.1. Task Generator & Evaluator Prompt
- The prompt asks for realistic, clear, specific, achievable, self-contained tasks requiring 3–20 steps, and the evaluator labels a task SUCCESS only if all three criteria are met.Source: §E.1 · pp. 31–33
E.2. Agent Prompt
- The agent prompt supplies a screenshot and accessibility tree, asks for a plan and possible actions, and requires one formatted action per turn.Available actions include clicking, typing, scrolling, going back, using Bing, and answering.Source: §E.2 · pp. 33–35
E.3. Experiment Details
The experiment details specify WebVoyager training settings and explain a Figure 6 annotation. The case studies contrast early-stage exploitation with fully trained exploration and list tasks examined for strengths and failure modes.
E.3. Experiment Details
- WebVoyager experiments use a horizon schedule beginning 10, 20, 20, 30 and keep the evaluation horizon at 30.Figure 6’s green area begins at iteration 6, when all sampled training trajectories can have horizon H=30.Source: §E.3 · p. 35
E.4. Case Studies: Strengths 1
- The first strengths case study contrasts a fully trained agent that explores with an early-stage agent that prefers exploitation.The task asks the agent to find a qualifying apple-pie recipe and report its maximum baking temperature.Source: §E.4 · pp. 36–37
E.5. Case Studies: Strengths 2
- The second strengths case study asks for the latest trending open-source Machine Learning project on GitHub and its star count.Source: §E.5 · p. 38
E.6. Case Studies: Fail Modes 1
- One fail-mode case examines keyboard options for a 14-inch MacBook Pro; another asks about Cornell and its undergraduate enrollment via arXiv.These passages state the case-study tasks, not their outcomes.Source: §E.6 · pp. 39–40
E.7. Case Studies: Fail Modes 2
- The second fail-mode case asks for a 2022 GitHub project related to AI agriculture, its main programming language, and its description.Source: §E.7 · p. 41