DynaRev: Benchmarking Web Agents under Dynamic Goal Revision

1 Introduction

• Most web-agent tasks fix goals before execution, unlike real interactions where requirements can be added, overridden, or conditioned on observed web state.The paper names this setting dynamic goal revision.Source: §1 · p. 1
• Existing web settings leave gaps in task-semantic revision timing, runtime-conditioned revisions, and analysis of constraint satisfaction, responsiveness, and cost–quality trade-offs.Trajectory-progress triggers can deliver revisions at different task stages for agents following different paths.Source: §1 · p. 2
• DynaRev evaluates web agents under dynamic goal revisions, using semantic milestones to trigger additive, unconditional-override, and conditional-override revisions independently of execution path, and measures correctness, responsiveness, and cost-aware efficiency.Source: §1 · p. 2
• Agents often satisfy only part of task requirements, with outcomes varying by task source, revision type, and model.Instruction rewriting while preserving browser state improves quality and efficiency, while browser resets degrade performance.Source: §1 · p. 2

GUI Agents and Web Agents

• GUI agents act through graphical interfaces, and web agents use browser environments that may also expose structured page representations such as HTML and accessibility trees.The paper focuses on web agents while noting that dynamic goal revision is relevant to GUI agents more broadly.Source: §2 · p. 2

Web Agent Evaluation Benchmarks

• Web-agent benchmarks range from offline trajectory evaluation to interactive sandbox, enterprise, and live-web tasks, but generally keep the objective fixed throughout execution.Examples include Mind2Web, WebArena, WorkArena, WebVoyager, Online-Mind2Web, and Mind2Web 2.Source: §2 · p. 2

Multi-turn Interaction and Dynamic Goal Revision

• Dynamic-goal settings study shifts, interruptions, additions, revisions, and retractions, while related web and GUI work does not align delivery to task-semantic progress or include web-state-conditioned revisions.The paper identifies InterruptBench as the closest related setting for long-horizon web navigation.Source: §2 · pp. 2–3

Fine-grained and Process-aware Agent Evaluation

• Prior evaluations assess fine-grained criteria or intermediate processes, while DynaRev binds atomic constraints to revisions and evaluates final responses, trajectories, post-revision responsiveness, and cost-aware efficiency.The related-work discussion cites Mind2Web 2 and ProBench as examples of fine-grained and process-aware evaluation.Source: §2 · p. 3

3.1 Web Agent Interaction

• A web-agent interaction proceeds through browser observations and actions, with a trajectory recording the preceding state-action sequence and latest observed state.The formulation introduces this interaction as the setting for dynamic goal revision.Source: §3.1 · p. 3

3.2 Dynamic Goal Revision

• A dynamic goal revision task consists of an initial instruction followed by execution-time revisions that update the active constraints.Each instruction specifies one or more task constraints.Source: §3.2 · p. 3
• Fixed-step or trajectory-position triggers can expose agents following different paths to revisions at substantially different stages of task progress.Successful web trajectories may vary in length and structure.Source: §3.2 · p. 3
• A milestone is an inevitable semantic stage of the previously specified goal, intended to be reached by any successful execution regardless of its UI-action sequence.It is derived from the goal before the upcoming revision, not from that revision.Source: §3.2 · p. 3
• A milestone trigger determines whether the stage has been reached, and the corresponding revision is delivered before the next action.The flight-search example delivers a nonstop-only requirement after search results have been observed.Source: §3.2 · p. 3

3.3 Revision Types

• Additive revisions preserve active constraints, unconditional overrides replace one, and conditional overrides replace one only when a runtime condition holds.For conditional overrides, the agent determines the applicable branch and active constraints from the web state.Source: §3.3 · pp. 3–4

4 Benchmark Construction

• Dynamic tasks are constructed by transforming static goals into instruction and revision sequences, assigning semantic milestones, and validating task quality and naturalness.The construction process covers static-to-dynamic transformation, milestone construction, and quality control.Source: §4 · p. 4

4.1 Static-to-Dynamic Task Transformation

• Original goals are decomposed into independently verifiable atomic constraints, with an executable initial instruction that leaves some constraints for later revisions.Remaining constraints are ordered naturally, and related constraints may be introduced together.Source: §4.1 · p. 4

4.2 Milestone Construction

• Each revision’s milestone is derived from the cumulative goal before that revision and identifies a completed semantic stage without relying on UI actions, reference trajectories, or upcoming-revision information.A completed search can qualify even when it returns no matching candidates.Source: §4.2 · p. 4

4.3 Quality Control

• Quality checks require coherent executable initial goals, meaningful revisions, task-implied milestones independent of reference paths, observable conditional triggers, and consistent constraints.Tasks failing the checks are regenerated or excluded.Source: §4.3 · p. 4

5.1 Constraint Verification

• A constraint is satisfied only when all its independently verifiable claims are verified; page claims require evidence from interaction states, and undelivered revisions count as unsatisfied.Simple claims can be checked directly from the final response.Source: §5.1 · p. 4

5.2 Post-Revision Behavioral Assessment

• For each delivered revision, an LLM judge checks the subsequent local action window for behavior consistent with the revised intent, separately from final task outcomes.This yields a binary revision-response judgment.Source: §5.2 · p. 4

5.3 Evaluation Metrics

• The metrics report dataset-level mean constraint satisfaction and task success, revision-level satisfaction, Mean First Failure Round, and Window Response Accuracy.Higher Mean First Failure Round means the agent satisfies revised requirements for more rounds; Window Response Accuracy assesses immediate behavioral adjustment independently of eventual outcome.Source: §5.3 · p. 5
• Cost-aware efficiency uses action steps or token usage to define Cumulative Task Success and Cumulative Constraint Satisfaction across cost thresholds, while cost efficiency combines task quality with execution efficiency.Cumulative Task Success counts fully successful tasks; Cumulative Constraint Satisfaction also credits partial constraint satisfaction.Source: §5.3 · p. 5

6.1 Experimental Setup

• The benchmark uses 181 tasks from Online-Mind2Web and Mind2Web 2, evaluating five models with a 100-step budget in BrowserGym.Tasks cover three revision types; the dataset was constructed from 70 seed tasks per source and manually reviewed.Source: §6.1 · p. 5, Table 1 · p. 5

6.2 Main Results

• Gemini has the highest mean constraint satisfaction (0.735), while DeepSeek has the highest task success rate (43.1%); Qwen’s lower Score than Claude (0.604 vs. 0.681) comes with higher success (39.2% vs. 36.5%).Gemini maintains revised requirements for the most rounds before first failure (2.28), while GPT fails earliest on average (1.91).Source: §6.2 · pp. 5–6, Table 2 · p. 6
• Claude performs better at low and intermediate constraint-satisfaction thresholds, while Qwen retains more tasks at the highest thresholds.Figure 3 reports the proportion of tasks reaching each score threshold.Source: §6.2 · p. 6, Figure 3 · p. 6
• M2W2 scores are lower for every model and revision type than Online-Mind2Web, while revision-type effects vary across sources and agents.M2W2 generally requires more action steps and has higher constraint counts and longer revision sequences; conditional tasks are the lowest-scoring M2W2 type for three of five agents, including Qwen (0.275).Source: §6.2 · p. 6, Table 3 · p. 6
• WRA ranges from 66.8% to 69.4% across models, but immediate behavioral adjustment does not consistently translate into satisfied revisions.Revisions with WRA = 1 have higher final satisfaction for all five agents, although the gap is modest for most models and many responsive revisions remain unsatisfied.Source: §6.2 · p. 6, Table 4 · p. 6

Models exhibit distinct cost–quality trade-offs.

• Gemini achieves the highest cost efficiency under both action-step and token costs, while GPT’s performance plateaus early and Gemini and DeepSeek continue improving with larger budgets.GPT uses the fewest action steps (24.9) but has the lowest overall Score (0.598); their step-based CE is nearly identical for GPT and DeepSeek (0.477 vs. 0.482).Source: Models exhibit distinct cost–quality… · pp. 6–7, Figure 4 · p. 7

7.1 Static vs. Dynamic Delivery

• 32.0% of model–task pairs improve, 28.0% worsen, and 40.0% remain unchanged under dynamic delivery, which typically requires more actions.Correctness trends differ by model and task source; trajectory inspection identifies both intermediate progress and rework, interface loops, or budget exhaustion.Source: §7.1 · p. 7, Table 5 · p. 7

7.2 Revision Handling Strategies

• Score rises from 0.647 to 0.718 with instruction rewriting and preserved execution, while action steps fall from 50.3 to 44.9 and token usage from 806.7k to 658.8k.Browser resets lower Score to 0.586 and raise steps to 59.7; Reset + Rewrite lowers Score further to 0.525 and requires the most steps.Source: §7.2 · pp. 7–8, Table 6 · p. 7

7.3 Revision Injection Timing

• 56.8% mean relative injection position has a 28.5-percentage-point standard deviation, and 70.0% of tasks show cross-agent timing ranges above 20.A fixed trajectory percentage may therefore deliver a revision at substantially different task stages across agents; semantic milestones instead anchor delivery to a task-defined stage.Source: §7.3 · p. 8, Figure 6 · p. 8

7.4 Failure Mode Case Studies

• Three representative failures are discarded progress after browser resets, immediate adaptation without final constraint satisfaction, and correct condition resolution without the corresponding action.In the last case, an agent selects a more expensive option despite identifying a newly eligible cheaper one.Source: §7.4 · p. 8, Figure 5 · p. 8

• DynaRev evaluates web agents under dynamic goal revisions using semantic milestones, runtime-conditioned overrides, and fine-grained measures of correctness, responsiveness, and efficiency.Source: §8 · p. 8
• Across five models, immediate adaptation does not ensure final constraint satisfaction, while instruction rewriting with preserved browser state improves task quality and efficiency.Source: §8 · p. 8
• The benchmark provides a foundation for studying web agents’ ability to handle evolving user goals.Source: §8 · p. 9

• DynaRev is built from static web tasks and may not capture the full diversity of real-world goal changes; its LLM-based judgments are imperfect, and some tasks may not have identifiable semantic milestones.Source: Limitations · p. 9

A.1 Constraint Decomposition

• Tasks with too few independent constraints are excluded to avoid artificial multi-turn revisions; the remaining requirements are decomposed into independently verifiable atomic constraints.Source: §A.1 · p. 11

A.2 Initial Instruction Construction

• The initial instruction includes requirements needed for a coherent executable request, with later requirements added only when needed to make the goal sufficiently clear.Tasks with persistent access failures or abnormal redirections that prevent meaningful execution are excluded.Source: §A.2 · p. 11

A.3 Revision and Milestone Organization

• Revisions follow natural interaction order, while each milestone marks a completed semantic stage derived from the goals already specified, not a particular UI action or upcoming requirement.Overrides change preference constraints; conditional overrides retain the original preference unless an observable runtime condition activates an alternative.Source: §A.3 · pp. 11–12

A.4 Quality Control

• Quality checks require executable initial goals, meaningful revisions, milestones independent of the upcoming revision, and observable, contextually meaningful conditions for conditional overrides.Invalid instances are regenerated or excluded after the prescribed retries.Source: §A.4 · p. 12

A.5 Constraint Binding

• Each atomic constraint is bound to the instruction that introduces or modifies it, in a separate post-processing step that preserves natural revision structure.Source: §A.5 · p. 12

B.1 Interaction Context

• Execution preserves complete instruction and action histories but retains detailed web observations only for the five most recent interaction steps.Source: §B.1 · p. 12

B.2 Revision Delivery and Termination

• Revisions arrive at their semantic milestones; premature completion is intercepted while revisions and budget remain, and runs end after all revisions or at the step limit.Source: §B.2 · p. 12

B.3 Environment Configuration

• All agents run in BrowserGym with Playwright-controlled headless Chromium, viewport-restricted AXTree observations, and the same execution configuration.Source: §B.3 · p. 12

C Evaluation Details

• The evaluation details cover claim construction, trajectory evidence retrieval, post-revision behavioral assessment, and the relationship between cost-aware efficiency and CCS AUC.These procedures implement the constraint-verification and behavioral evaluation framework described in the section.Source: C Evaluation Details · p. 12

C.1 Claim Construction and Verification

• Constraints requiring multiple checks are decomposed into independently verifiable claims, with simple claims checked against the final response and page claims sent to trajectory retrieval.For relative-time constraints, the task execution date is also supplied to resolve the reference; claims can be generated separately for each relevant item.Source: §C.1 · pp. 12–13

C.2 Trajectory Evidence Retrieval

• Page claims use the final web state first; otherwise, subject-matched candidate states after the bound instruction are examined in trajectory order for sufficient evidence.Lexical matching retrieves candidates but is not evidence itself; a claim with no sufficient retrieved evidence is unverified.Source: §C.2 · p. 13

C.3 Post-Revision Behavioral Assessment

• After each revision, a behavioral judge assesses at most the next three agent actions, stopping earlier if another revision arrives or the task ends.Source: §C.3 · p. 13

D Equivalence between Cost-aware Efficiency and CCS AUC

• Cost-aware efficiency is equivalent to the normalized area under the CCS curve.The section states that the expression for the curve’s area is exactly the cost-aware efficiency score.Source: D Equivalence between Cost-aware… · p. 13

E Dataset Details

• 99 of the 140 sampled seed tasks contribute at least one retained dynamic task to the benchmark.Source: E Dataset Details · p. 13
• Each retained seed task is assigned to one scenario category.Source: E Dataset Details · p. 13
• Figure 7 presents the scenario distribution of retained seed tasks.Source: Figure 7 · p. 13, E Dataset Details · p. 13

F Static vs. Dynamic Delivery

• Table 7 is identified as reporting pairwise score changes under dynamic delivery.Source: Table 7 · p. 13

G Score Threshold Results

• Table 8 reports task-level percentages reaching corresponding constraint-satisfaction score thresholds and provides detailed values underlying Figure 3.Source: Table 8 · p. 14, G Score Threshold Results · p. 14

F Static vs. Dynamic Delivery

• Table 9 reports full results for static and dynamic additive delivery on paired tasks.The comparison uses 30 paired additive tasks, 15 from each source; Static is the original single-turn task and Dynamic its progressively delivered additive counterpart.Source: F Static vs. Dynamic Delivery · p. 13, Table 9 · p. 14

H Round-wise Results

• The analysis groups constraint satisfaction and behavioral responsiveness by instruction round.Source: H Round-wise Results · p. 14
• CSS_k measures satisfaction of constraints introduced or modified by revision k, while CSS_0 corresponds to the initial instruction.Source: H Round-wise Results · p. 14
• For later rounds, rounds 4 and above are pooled as I4+ because they contain fewer observations.Source: H Round-wise Results · p. 14

I Milestone Trigger Validation

• Timing labels classify triggers as early, aligned, or late according to milestone satisfaction at the trigger state and the preceding state.Annotators inspected 30 sampled cases, considering each milestone description, trigger state, and immediately preceding state.Source: I Milestone Trigger Validation · p. 14
• For sampled triggers, two annotators agreed on 25 of 30 three-way timing labels (83.3%; Cohen’s κ = 0.438), while agreement on each underlying binary judgment was 90.0%.Source: I Milestone Trigger Validation · p. 14

J.1.1 Constraint Extraction

• Constraint extraction turns instructions into individually checkable requirements while preserving their scope and logical structure and excluding subjective wording or invented requirements.The prompt keeps OR conditions as one composite constraint and retains concrete requirements when mixed with subjective language.Source: §J.1.1 · p. 15

J.1.2 Initial Instruction Construction

• Initial-instruction construction classifies constraints as required by homepage inputs or later refinements, then forms a natural, minimally complete request that states the task’s core subject.The prompt permits adding later constraints only when needed to clarify the core intent, and advises against including every constraint.Source: §J.1.2 · p. 16

J.1.3 Additive Revision and Milestone Construction

• The additive-revision procedure keeps I0 unchanged and arranges remaining constraints into subsequent instructions, each with a milestone for injection.The supplied passage introduces this sequence-design procedure but does not provide further construction rules.Source: §J.1.3 · p. 17
• Additive turns introduce remaining constraints once in a natural sequence, without repeating requirements, inventing new ones, or splitting composite logic.Constraints should be grouped as a user would naturally express them, while preserving AND/OR relationships.Source: §J.1.3 · pp. 17–18
• Each later-turn milestone describes an earlier, reachable, branch-robust semantic state established by prior instructions, where the next revision would feel natural.Milestones describe user-visible progress, remain valid across empty-result branches, and must not reveal the upcoming instruction.Source: §J.1.3 · p. 18

J.1.4 Override Revision and Milestone Construction

• Override milestones follow the same rule: describe a branch-robust semantic state derived only from prior turns, favoring an earlier reachable point over the final goal.They should remain valid when prior execution yields no results.Source: §J.1.4 · pp. 19–20
• Override sequences use one turn to switch a concrete preference to a different value; unconditional overrides have no if-clause, while other turns remain additive.When no candidate preference exists, select an existing constraint on the same dimension and do not change the task subject.Source: §J.1.4 · pp. 19–20
• Conditional overrides keep the default preference unless a page-observable price, rating, time, or count condition permits a uniform task-level change.Conditions should express a realistic tradeoff or shortfall using a concrete task- and page-based threshold, not restate the replacement preference.Source: §J.1.4 · p. 21

J.1.5 Milestone Review

• Milestones are valid when they follow from the cumulative prior goal, mark an inevitable path-independent semantic stage, remain page-observable, and do not presuppose a conditional revision’s outcome.Reaching the relevant search stage counts even when no matching results appear.Source: §J.1.5 · pp. 22–23

J.1.6 Instruction Review

• Instruction review checks that each turn is executable and understandable, preserves original task requirements unless explicitly overridden, and forms a natural, coherent progression.Milestone timing and validity are reviewed separately from instruction evolution.Source: §J.1.6 · pp. 24–25
• Instruction review checks that overrides replace the declared requirement without changing the underlying task, and that conditional branches are explicit, decidable, and task-wide.It also checks that conditional defaults precede the revision and that meaningful thresholds and inclusive boundaries are clear when relevant.Source: §J.1.6 · p. 25

J.1.7 Constraint Synchronization

• Constraint synchronization propagates unconditional replacements to related constraint descriptions and splits condition-dependent constraints into separate default and alternative outcomes.The prompt requires edits to related wording so all constraints reflect the updated final goal.Source: §J.1.7 · p. 26

J.1.8 Constraint Binding

• Constraint binding maps each constraint to its most relevant instruction, audits missing or extra coverage, and flags serious gaps to determine whether to retry I0 design or instruction arrangement.For conditional overrides, the target default and alternative constraints bind to the revision; other default constraints bind to the instruction that introduced them.Source: §J.1.8 · p. 27

J.1.9 Construction Retry and Reflection

• Construction retries require diagnosing recurring critique patterns, dimensions already changed, and possible text-swap loops before choosing and justifying the next design change.The reflection is required before generating the revised instructions.Source: §J.1.9 · p. 28

J.2.1 Policy Agent

• The policy agent must return one executable function call in a markdown action fence and use numeric element bids, rather than visible labels, for interactions.The action contract lists click, fill, press, and scrolling as available operations.Source: §J.2.1 · p. 28
• The policy agent must follow all active instructions, inspect page evidence for conditions, explore reasonable paths, and report completion with concrete evidence.Conditional completion messages should state the observed condition and branch taken.Source: §J.2.1 · pp. 28–29

J.2.2 Per-step Input Template

• Each step supplies the agent with active instructions, the current URL, a completion checkpoint, the last error, and visible page content.Source: §J.2.2 · p. 29

J.2.3 Milestone Triggering

• Milestones fire when page evidence shows the semantic checkpoint, regardless of navigation path, with junctions triggering before terminal completion and confidence below 0.8 preventing a trigger.Lists may include nonmatching items, and quantitative states can qualify when the specified number of matching items is observable without an explicit filter marker.Source: §J.2.3 · p. 29

J.2.4 Instruction Rewriting

• Instruction rewriting preserves active requirements, replaces or relaxes only conflicting earlier requirements, and retains conditional branches without deciding them.Source: §J.2.4 · pp. 29–30

J.3.1 Claim Generation

• Claim generation decomposes each constraint into exhaustive, nonduplicative sub-claims, preferring answer-text checks when sufficient and scoping list checks to the answer section for that constraint.Page sub-claims are used when trajectory page content is needed; later overrides supersede earlier requirements on the same dimension.Source: §J.3.1 · p. 30

J.3.2 Simple Claim Verification

• Simple claim verification judges binary predicates from the final answer and logic, requires explicit supporting text, and applies list-related checks only to the answer section for that constraint.Source: §J.3.2 · pp. 32–33
• Simple claim verification asks whether the done_message addresses a predicate and requires the judgment to cite specific text from that message.The prompt supplies the task execution date, predicate, and agent message as inputs.Source: §J.3.2 · p. 33

J.3.3 Conditional Branch Resolution

• Conditional branch resolution marks a condition true or false only when the done_message reports a concrete observation; otherwise it defaults to not_addressed.The judgment uses what the agent said, requires concrete values, and must cite specific message text.Source: §J.3.3 · p. 33

J.3.4 Web Evidence Verification

• Web evidence verification checks subject presence and assertion support against the same GUI snapshot, using the AXTree and page URL rather than world knowledge.It distinguishes pages centered on a subject or showing it as a result from incidental mentions, and requires direct page evidence for asserted properties.Source: §J.3.4 · pp. 33–34

J.3.5 Post-Revision Behavioral Assessment (WRA)

• Post-revision behavioral assessment independently labels whether actions advance the injected instruction and whether adaptation is overwrite, additive, or unrelated.Partial exploratory progress can count as responsive, but irrelevant actions do not; additive adaptation preserves prior progress, while overwrite abandons it.Source: §J.3.5 · pp. 34–35