CLIFT: Conformal Self-Verification for Web Agent Training and Test-Time Scaling

1 Introduction

• Binary success is too sparse for credit assignment, while frontier-model judges are costly during training and cannot be assumed available at test time.The introduction notes that a shared outcome across GRPO rollouts can collapse the advantage estimate.Source: §1 · p. 1
• CLIFT certifies natural-language verification questions by URL state, then reuses the bank for denser per-step training rewards and test-time trajectory selection.At test time, the trained agent generates a greedy trajectory and diverse retries, and a conservative selector chooses one trajectory per task.Source: §1 · pp. 1–2
• Across WAI, VWA, and OM2W, CLIFT supports open-agent training, transfer to GPT-5.5 at state-of-the-art canonical-harness performance, and improved live-web zero-shot evaluation.No agent is trained on OM2W; the certified bank is translated and recalibrated for that setting.Source: §1 · p. 2

2 CLIFT: Conformal Lift-calibrated Introspective Filtering of Trajectories

• CLIFT uses one certified, URL-aware question bank to support both per-step training rewards and test-time trajectory selection.The bank comprises questions with URL scopes, polarity signs, and certified trust weights.Source: §2 · pp. 2–3, §2.3 · p. 5

2.1 The Conformal Verifier

• The CCC certifies question weights separately by URL tier using polarity-aware agreement with judge scores, while neutral questions do not certify.The method queries the most-specific certified tier, falling back from URL-path to host to global; new questions enter uncertified, and observed lift can expose polarity mistakes.Source: §2.1 · pp. 3–4

2.2 Training: Per-step Rewards and URL-stratified Advantages

• Training combines judge scores with certified verifier evidence, then converts discounted step rewards into URL-stratified token advantages.The verifier contribution is clipped at zero, so certified evidence can strengthen but not reduce the judge signal; advantages apply to tool-call spans, while tokens outside them receive zero gradient.Source: §2.2 · p. 4

2.3 Test Time: CTS (Conformal Trajectory Selection)

• At test time, CTS compares a greedy incumbent with diverse retries using structured evidence gathered at each distinct URL state.The self-verifier compares evidence sheets, action traces, and final states rather than relying on a hand-designed URL-trace scalar.Source: §2.3 · p. 4
• CTS swaps to a retry only when a conservative repeated-vote rule favors it; ties and uncertainty preserve the incumbent.The certified bank is frozen after training and reused as evidence in the comparative selector.Source: §2.3 · pp. 4–5

3.1 Task synthesis and bank construction

• Training tasks are synthesized without reading evaluation tasks, then filtered for malformed, duplicate, hardcoded, pure-lookup, and trivial-navigation cases.The reported checkpoint uses 1,886 WAI tasks and 1,272 VWA tasks across 16 LoRA shards; OM2W provides no policy-training data.Source: §3.1 · p. 5, Table 2 · p. 5

3 Experimental Design

• The three benchmarks test whether the verifier bank can train an open agent, transfer to a closed backbone, and support live-web zero-shot selection.WAI is the primary training setting, VWA the controlled transfer setting, and OM2W the live-web transfer setting.Source: §3.2 · p. 5, §3 · p. 5

3.2 Per-benchmark protocols

• WAI trains Gemma 4 31B + LoRA with the full URL-path / host / global CCC hierarchy, comparing the backbone, vanilla GRPO, and CLIFT before adding CTS.The public leaderboard slice covers success rates on nine apps; its CLIFT +CTS column uses K=3 and one selected trajectory per task.Source: §3.2 · p. 5, Table 3 · p. 6
• VWA applies the bank learned during open-model training to unmodified GPT-5.5, adding CTS over completed trajectories to test transfer beyond the Gemma-4 policy.The completed-task interface is held constant, and GPT-5.5 is not updated.Source: §3.2 · p. 5, §3 · p. 5
• OM2W translates and recalibrates the WAI/VWA bank for live-web domains, then applies CTS without OM2W-specific policy training.The 300 public-domain tasks are judged by an LLM rubric; recalibration uses a held-out OM2W rollout sample and adds live-web failure-mode questions.Source: §3.2 · p. 5, §3 · p. 5, Table 2 · p. 5

4.1 Training with certified feedback

• CLIFT-trained Gemma-4 with K=3 CTS reaches the highest published 9-app WAI success rate in the comparison and improves the same backbone by 12.8 percentage points overall.CLIFT separates from vanilla GRPO during training before selection is used; training supplies the large WAI gain, while CTS preserves it and adds a small safe lift.Source: §4.1 · pp. 6–7, Figure 3 · p. 6

4.2 Bank transfer to a closed backbone

• With GPT-5.5 fixed, VWA CTS at K=4 raises weighted average success to 53.7, narrowly above WALT’s 52.9, with gains on all three environments.The certified bank was created during Gemma-4 training; the selected-output rows emit one trajectory per task.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Cross-benchmark transfer

• On OM2W, the transferred verifier bank certifies 100 of 132 questions without policy finetuning on benchmark data, supporting reuse beyond the training benchmarks.The recalibrated bank contains 75 positive-lift and 25 negative-lift questions, with mean lift 0.34.Source: §4.3 · p. 7, Table 5 · p. 7
• The transferable artifact is the certified question taxonomy and calibration recipe, not the policy; recalibration on a new domain uses judge-scored rollouts without a policy update.The questions are URL-scoped and polarity-tagged, with calibrated trust weights.Source: §4.3 · p. 8
• At rollout budget K=4, GPT-5.5 high-effort OM2W success rises from 49.7 to 61.0 (+11.3pp), while transferred Gemma-4 rises from 40.0 to 45.0.Each selected-output row emits one trajectory per task, not an oracle union over attempts.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4 Ablation study

• In WAI ablations, training accounts for most gains; CTS improves five apps, ties four, and never lowers the single-rollout CLIFT anchor.Vanilla GRPO improves the base model by 2.9 points with 18.5 average steps on successful trajectories; CLIFT improves it by 10.8 points with 16.0 steps, and CTS adds 2.0 points at 15.5 steps.Source: §4.4 · p. 8, Table 6 · p. 8

• CLIFT withholds unreliable verifier evidence from both training rewards and test-time selection when judge, state partition, or question-bank evidence is insufficient.The refusal is a diagnostic of unsupported distillation, not a silent collapse.Source: §5 · p. 8
• The study trains one open-weight backbone, relies on a frontier judge, and does not establish that one bank is universal across web domains.Calibration detects systematic judge bias only through observed data; deployment should retain rate limits and domain-specific safeguards given misuse risk.Source: §5 · p. 8
• Future work proposes executable browser-state assertions that can be reused during rollouts and test-time selection, reducing dependence on a separate self-verification bank while preserving auditability.Source: §5 · p. 9

• CLIFT connects web-agent benchmarks, RL credit assignment, test-time search, and verifier-based selection by carrying judge feedback from online RL into trajectory selection.Source: §6 · p. 9
• Unlike prompt-level search, CLIFT reuses a certified per-URL bank learned during RL training rather than relying only on more samples.Source: §6 · p. 9
• CLIFT retains GRPO but augments judge feedback with a certified self-verifier and normalizes rewards by URL state, rather than changing the policy-gradient family.Source: §6 · p. 9
• Where prior conformal methods calibrate outputs, policies, or judgments, CLIFT calibrates verification questions using URL scope, polarity, and a Mondrian-ACI trust weight.A CEGAR-style loop turns judge rationales into specifications for training and test-time selection.Source: §6 · p. 9

• Across WAI, VWA, and OM2W, the same bank helps train an open model, transfers to a stronger closed model, and recalibrates for live-web selection without policy fine-tuning.Source: §7 · p. 9
• Each certified question carries a URL scope, polarity sign, trust weight from observed lift, and provenance to its introducing judge rationale.Source: §7 · p. 9
• When a question’s lift collapses on a new distribution, the certifier withholds trust rather than amplifying noise, allowing the bank to be inspected and recalibrated.Source: §7 · p. 9

• The appendix collects evaluation prompts, the CLIFT-only training setup, detailed OM2W bank-transfer accounting, and evaluation-log case studies.Source: A Appendix · p. 12

A.1 Evaluation Prompts

• WAI agents receive accessibility snapshots and must issue exactly one browser tool call per turn, using element IDs and the current task and URL.The prompt lists actions including clicking, typing, scrolling, navigation, and stopping.Source: §A.1 · p. 13
• VWA agents act on screenshot-based numbered elements, issue one valid tool call at a time, verify completion, and avoid constructing internal URLs.The prompt also provides the objective, URL, observation, and previous action.Source: §A.1 · p. 13
• OM2W prompts target live websites and require one tool call per turn, while WebJudge assesses completion using the task, action history, key points, and important pages.WebJudge returns a success or failure status after considering task requirements such as filters and submission actions.Source: §A.1 · p. 13

A.2 CLIFT Training Setup

• Gemma-4-31B-it is trained with rank-8 LoRA across 16 shards, whose adapters are averaged and merged for evaluation; OM2W is excluded from policy training.OM2W is used only for bank transfer and CTS evaluation.Source: §A.2 · p. 14
• The training pool contains 1,886 WAI and 1,272 VWA tasks, synthesized from app documentation or environment exploration and filtered for validity, duplication, hardcoding, and trivial navigation.Synthesis and filtering do not read evaluation task descriptions or evaluation reward signals.Source: §A.2 · p. 14
• Each prompt yields eight browser rollouts scored by a comparative batch judge, while the base Gemma-4 answers verification questions with its task adapter detached.Training ran on shard-parallel H200 jobs, with up to eight nodes of eight GPUs each.Source: §A.2 · p. 14
• URL-tiered Mondrian-ACI trackers certify question evidence using target α⋆=0.10, nmin=50, and liftmin=0.02; the asymmetric reward blend adds support but cannot subtract from judge reward.URL-stratified returns become token advantages over tool-call spans, and the frozen certified bank is reused by CTS.Source: §A.2 · p. 14

A.3 Reported CLIFT Hyperparameters

• The setup uses judge threshold θj=0.5, GRPO group size G=8, certification target α⋆=0.10, minimum certification count nmin=50, and minimum lift liftmin=0.02.CEGAR runs every M=60 training steps and proposes batches of 3 to 5 questions.Source: §A.3 · pp. 14–15
• The reward blend uses β=0.15 and return discount γ=0.95; CTS uses m=3 verifier votes at temperature 0.7.The greedy incumbent temperature is 0, while diverse retries use temperature 1.0.Source: §A.3 · p. 15
• The seed banks contain 281 WAI questions and 89 VWA questions before filtering and certification.Source: §A.3 · p. 15

A.4 Bank Details and OM2W Transfer

• After CEGAR growth and certification, the WAI bank has 44 nonzero-weight questions from 992 candidates, and the VWA bank has 21 certified questions from 1,565 candidates.For OM2W reuse, 36 WAI and 48 VWA questions are retained or rewritten; app- or environment-specific URL patterns and DOM references do not transfer.Source: §A.4 · p. 15
• Representative WAI and VWA questions assess reaching relevant content, avoiding unintended destructive actions, acting on search results, and navigating to target detail pages.The examples include positive trust weights of +0.54 for relevant content and constructive non-destructive behavior, +0.20 for active search, and +0.13 for navigation depth.Source: §A.4 · pp. 15–16

A.5 OM2W Transfer Performance Details

• The OM2W performance table reports that the transferred selector improves Gemma-4, while GPT-5.5 high-effort gains further as K rises from 2 to 4.Each row with K>1 emits one CTS-selected trajectory, not an oracle union; the canonical judge column matches the OM2W protocol.Source: Table 8 · p. 17, §A.5 · p. 17

A.4 Bank Details and OM2W Transfer

• The OM2W setting trains no policy on that benchmark; its translated 132-question bank adds live-web failure-mode predicates and retains the training-bank schema for CTS.The transfer process filters and rewrites general predicates, adds questions for failures such as bot blocks and hallucinated extraction, and recalibrates them.Source: §A.4 · p. 17
• Among 100 certified OM2W questions, 75 have positive lift and 25 negative lift, providing both progress and failure-mode evidence.Questions below the discrimination floor remain at zero weight and may be recertified on another rollout sample.Source: §A.4 · p. 17

A.6 Case Studies

• On WAI Gmail, CTS replaces an 87-step failing greedy rollout with a 25-step successful diverse rollout completing the profile and contact-label task.The selector assigns the greedy and diverse trajectories scores of -0.5 and +0.5, respectively, and swaps to the successful trajectory.Source: §A.6 · p. 18
• On WAI Handshake, the diverse rollout completes a message, event, and job chain that the greedy rollout leaves unfinished.CTS swaps to the diverse trajectory, which receives a positive score while the greedy trajectory receives a negative score.Source: §A.6 · p. 18
• CTS keeps a successful greedy security-change trajectory over a failing retry, while an OM2W Indeed trajectory recovers from blank results and verification by using native search.The live-web trajectory ends with visible listings and a WebJudge success label.Source: §A.6 · p. 18