PaperlayerView exact revision on arXiv

Source-linked AI summary

WaitLens: Temporal VLM-Based Adaptive Waiting for Industrial GUI Testing Agents

Shika Ikeura, Xuan Wang, Ziyi Kuang, Jianping Xu, Chao Kang, Yongxiang Hu, Xiaoyu Wang, Juxing Yuan, Yangfan Zhou

arXiv:2610.32482v1cs.SE

TL;DR

Prior adaptive waiting methods rely on app-specific data or platform-specific information, while VLM readiness prediction requires temporal observations and general-purpose models may struggle to identify completed interfaces. WAITLENS fine-tunes a lightweight VLM on annotated GUI loading trajectories, using temporal resampling and temporal screenshots to predict readiness and end waiting. In industrial replay evaluation, WAITLENS reduced mean post-Ready waiting by 94.98% compared with the existing fixed waiting strategy.

  • Problem

    Prior adaptive waiting methods rely on app-specific data or platform-specific information, while VLM readiness prediction requires temporal observations and general-purpose models may struggle to identify completed interfaces.

    Source: §I · p. 1

  • Method

    WAITLENS fine-tunes a lightweight VLM on annotated GUI loading trajectories, using temporal resampling and temporal screenshots to predict readiness and end waiting.

    Source: §I · pp. 1–2

  • Results

    In industrial replay evaluation, WAITLENS reduced mean post-Ready waiting by 94.98% compared with the existing fixed waiting strategy.

    Source: §I · p. 2

  • Takeaways & Limitations

    The Meituan industrial evaluation demonstrates practical potential for reducing unnecessary waiting in GUI testing.

    Source: §I · p. 2

    The findings may not generalize to all applications, interaction types, or device conditions, and fixed-sequence video replay cannot establish effects on task success or overall execution efficiency.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Improving the efficiency of GUI testing agents is important for industrial deployment. Beyond LLM inference, fixed waiting between consecutive interactions introduces substantial overhead. Our industrial study reveals that fixed waiting accounts for 42.3% of the time within the sampled interaction intervals, comparable to the 54.8% spent on model calls. Moreover, 62.9% of the waiting time is spent after the GUI has already finished loading. Although prior studies have proposed adaptive waiting strategies, their reliance on app-specific data or platform-specific information limits their applicability across diverse industrial environments. To this end, we propose WaitLens, which fine-tunes a lightweight VLM on GUI loading trajectories annotated with waiting end times. We introduce temporal resampling to construct diverse training examples, enabling the model to learn varying GUI loading behaviors from temporal observations. Experiments show that WaitLens achieves 86.08% accuracy in deciding whether to keep waiting, compared with 54.13% for the backbone VLM. Our evaluation using recordings from Meituan's industrial testing pipeline shows that WaitLens reduces mean extra waiting after the interface is ready by 94.98% compared with a 20-second fixed wait.

I. INTRODUCTION

GUI agents must wait after each action, but fixed waits leave substantial idle time after interfaces are ready. WAITLENS uses temporal GUI observations to adapt waiting, addressing limits in prior methods and VLMs.

I. INTRODUCTION

  • 62.9% of waiting time is spent idling after the GUI is ready, motivating adaptive waiting instead of fixed post-action delays.Agents risk incorrect decisions if they observe too early, while long fixed waits leave them idle after readiness.Source: §I · p. 1
  • Prior adaptive methods rely on app-specific training data or platform-specific information, while general-purpose VLMs face temporal-input and readiness-classification challenges.Feeding all historical screenshots into a VLM can increase input size and inference latency as loading continues.Source: §I · p. 1
  • WAITLENS periodically checks screenshots and uses temporal resampling during lightweight VLM fine-tuning to predict readiness across diverse loading conditions.It uses a sliding window to keep temporal visual input bounded.Source: §I · pp. 1–2
  • In industrial replay evaluation on Meituan GUI testing recordings, WAITLENS reduced mean post-Ready waiting by 94.98% relative to the existing fixed waiting strategy.Source: §I · p. 2

II. BACKGROUND

The background describes industrial GUI testing's fixed-wait practice and a Meituan study quantifying its overhead. The study finds waiting substantial and often continues well after readiness.

A. Industrial Practice

  • Agents across diverse devices and network conditions use configurable fixed waits because observing too early can capture intermediate states and mislead subsequent actions.At Meituan, the studied configuration waits 20 seconds after each interaction before capturing the next screenshot.Source: A. Industrial Practice · p. 2

B. Motivation Study

  • Ready latency measures time from action completion to readiness, while post-Ready waiting measures the remaining time before the fixed interval ends.The study defines Ready as the point when the primary response has appeared and provides sufficient information for the next observation.Source: B. Motivation Study · p. 3
  • 42.3% of measured execution time is fixed waiting, compared with 54.8% for model calls and 2.9% for other time.This breakdown identifies waiting as a substantial execution overhead.Source: B. Motivation Study · p. 3, Figure 2 · p. 3
  • 12.58 seconds of post-Ready waiting occur per click on average, accounting for 62.9% of fixed waiting time.The study reports 463 of 468 clicks reach Ready within 20 seconds; configured waits total 156 minutes, with approximately 98 minutes spent idling after readiness.Source: B. Motivation Study · p. 3, Figure 3 · p. 3
  • Shorter waits reduce redundancy but increase premature-observation risk, whereas longer waits accommodate slow responses at the cost of more waiting.The section concludes that adaptive waiting is needed to balance testing effectiveness and efficiency across diverse interactions.Source: B. Motivation Study · p. 3

III. METHOD

WAITLENS replaces fixed post-action waiting with periodic VLM checks of temporal screenshots. It is trained on labeled interaction recordings augmented through temporal resampling.

A. Overview

  • WAITLENS periodically checks screenshots after an action and ends waiting after the required number of consecutive ready predictions.It returns the latest checked screenshot, or the latest available screenshot if the original waiting budget expires first.Source: A. Overview · p. 3
  • The trained VLM combines the pre-interaction screenshot with recent historical screenshots to predict whether the GUI is Ready.Online checks continue with subsequent screenshots unless the stopping condition is met.Source: A. Overview · p. 3
  • Labeled visual sequences from annotated recordings are augmented through temporal resampling to fine-tune a lightweight pretrained VLM.The approach uses Qwen3-VL-2B as the pretrained model.Source: A. Overview · p. 3

B. Bounded Temporal Visual Representation

WAITLENS combines a pre-interaction reference, recent historical screenshots, and a current screenshot so the VLM can judge GUI readiness from temporal changes. Temporal resampling varies observation times and history lengths to broaden training examples across loading phases.

B. Bounded Temporal Visual Representation

  • The input combines a pre-interaction reference, historical screenshots, and the current screenshot to capture GUI changes over time.The reference identifies action-triggered changes, historical frames show interface evolution, and the current screenshot is the one judged for readiness.Source: B. Bounded Temporal Visual… · pp. 3–4
  • The sliding window retains at most K recent preceding screenshots, bounding history size as waiting continues.History grows as frames become available, then advances by dropping older frames once it reaches K images.Source: B. Bounded Temporal Visual… · p. 4

C. VLM Fine-Tuning with Temporal Resampling

  • Fixed-interval sampling can couple short histories with not_ready labels, potentially misleading the model when loading speeds vary.History length grows with elapsed time until it reaches K, while GUIs typically become ready later in loading.Source: C. VLM Fine-Tuning with Temporal… · p. 4
  • Temporal resampling supplements regularly sampled examples with observations designed to vary current times and historical frame spacing.Current times are drawn from three segments of each label interval, while histories are sampled from equal-duration strata preceding the selected time.Source: C. VLM Fine-Tuning with Temporal… · pp. 4–5
  • The supplemented training set targets balanced coverage of ready and not_ready examples with short or full histories.Selection also balances individual history lengths within the short-history groups.Source: C. VLM Fine-Tuning with Temporal… · p. 5

D. Integration with GUI Agents

WAITLENS is designed as an agent-agnostic plugin that replaces fixed waiting between an action and the next observation. It checks screenshots over time and returns early after consecutive ready predictions, retaining the original waiting budget as a fallback.

D. Integration with GUI Agents

  • WAITLENS replaces the fixed wait between action execution and the next observation without changing the agent’s reasoning or action logic.The plugin receives the pre-interaction screenshot, screenshot access, and the original waiting budget.Source: D. Integration with GUI Agents · pp. 5–6
  • The plugin operates entirely on screenshots, requiring no app instrumentation, source-code modification, or view-hierarchy access.Source: D. Integration with GUI Agents · p. 6
  • WAITLENS returns the latest checked screenshot after n consecutive ready predictions, or the latest available screenshot when the original budget expires.The waiting budget sets a deadline, and consecutive predictions reduce fluctuations before early return.Source: D. Integration with GUI Agents · p. 6

IV. EVALUATION

The evaluation uses Meituan GUI interaction recordings to assess readiness prediction and waiting-policy performance through offline replay. Its datasets separate prediction testing from waiting-policy evaluation, with no overlapping recordings.

IV. EVALUATION

  • The evaluation examines readiness prediction and waiting-policy performance through offline replay of Meituan industrial GUI recordings.Source: §IV · p. 6

1) Datasets:

  • 102 annotated recordings cover 12 tasks for RQ1, split by task into 7 training, 2 validation, and 3 test tasks.The resulting observations include 424 regularly sampled and 424 temporally resampled test examples.Source: 1) Datasets · p. 6
  • The RQ2 dataset contains 1,101 recordings and 3,868 post-action response sequences, with Ready times annotated by Meituan testing personnel.The RQ1 and RQ2 datasets contain no overlapping recordings.Source: 1) Datasets · p. 6

2) Baselines and Variants:

The evaluation compares WAITLENS with VLM and rendering-state baselines, reports readiness and waiting-policy metrics, and specifies its model and inference configuration.

2) Baselines and Variants:

  • RQ1 compares WAITLENS with pretrained Qwen3-VL-2B-Instruct, Qwen3.8-Max, AdaT, and a WAITLENS variant trained without temporal resampling.The two VLM baselines receive the same temporal visual input; the ablation uses DG alone, while the complete model uses DG ∪ C⋆.Source: 2) Baselines and Variants · pp. 6–7

3) Metrics:

  • RQ1 reports accuracy, false positive rate, false negative rate, and median and P95 model-call latency; RQ2 reports premature return rate and mean and P95 actual and post-Ready waiting times.Premature returns classify an unready interface as ready; post-Ready waiting is zero for premature returns.Source: 3) Metrics · p. 7

4) Implementation and Configuration:

  • The model uses Qwen3-VL-2B-Instruct with LoRA fine-tuning, while local inference runs on one A100 GPU with bfloat16 precision and greedy decoding.The default sampling interval is 0.25 s and maximum history length is four; Qwen3.8-Max is accessed by API at temperature zero.Source: 4) Implementation and Configuration · p. 7

B. RQ1: Ready State Prediction and Overhead

WAITLENS improves GUI readiness prediction and reduces waiting in industrial replay, while exposing a trade-off between faster returns and premature returns. Its prediction accuracy and efficiency also compare favorably with the supplied baselines.

B. RQ1: Ready State Prediction and Overhead

  • 86.08% accuracy and 24.59% false positive rate were the best among evaluated methods; WAITLENS improved accuracy by 31.95 and 26.53 percentage points over Qwen3-VL-2B and Qwen3.8-Max.Both general-purpose VLMs had high false positive rates: 75.41% and 74.94%, respectively.Source: B. RQ1: Ready State Prediction and… · p. 7, Table I · p. 8
  • Temporal resampling improved overall accuracy by 2.12 percentage points and reduced false positive rate from 29.47% to 24.59%, with false negative rate increasing from 2.16% to 2.88%.False positives can cause premature returns, while false negatives lead to extra checks and longer waits.Source: B. RQ1: Ready State Prediction and… · p. 7
  • 0.243 s median and 0.786 s P95 model-call latencies were reported for WAITLENS, versus 1.607 s median and 2.839 s P95 for Qwen3.8-Max.The paper notes that model calls may recur during the waiting policy.Source: B. RQ1: Ready State Prediction and… · p. 7, Table I · p. 8

C. RQ2: Waiting Policies in Industrial Scenarios

  • 94.98% lower mean extra waiting (17.957 s to 0.901 s) came with a 15.64% premature return rate, versus 0.13% for Meituan’s 20 s fixed wait.The comparison covers 3,868 interactions from Meituan’s daily regression pipeline.Source: C. RQ2: Waiting Policies in Industrial… · p. 7, Table II · p. 8
  • 10 s and 5 s fixed waits kept premature return rates relatively low at 2.25% and 8.04%, but mean extra waiting remained 8.027 s and 3.268 s, respectively.Both extra-wait values were reported as more than three to eight times WAITLENS’s.Source: C. RQ2: Waiting Policies in Industrial… · pp. 7–8, Table II · p. 8
  • 2 s and 1 s fixed waits raised premature return rates to 23.60% and 71.79%; against a 3 s wait, WAITLENS cut mean actual and extra waiting by 25.9% and 38.8%, with a slightly higher 15.64% rate.The paper notes that a fixed threshold selected from historical data may not generalize to changing device and network conditions.Source: C. RQ2: Waiting Policies in Industrial… · p. 8, Table II · p. 8
  • WAITLENS reduced AdaT’s premature return rate by 75.16 percentage points, from 90.80% to 15.64%, and had rates of 12.87% on internal and 17.74% on external camera recordings.The external-camera recordings were an unseen recording modality during training.Source: C. RQ2: Waiting Policies in Industrial… · p. 8

D. Case Study

The case study identifies two ways WAITLENS can return before the interface is ready: visible page structure with incomplete content, and apparently complete content that later updates. These examples illustrate the limits of visual-only readiness judgments.

D. Case Study

  • A structured page may still lack its main content when WAITLENS predicts readiness, as in the example where the merchant header appears before the package list and purchase buttons.Figure 7 pairs screenshots judged Ready with later screenshots from the same interaction.Source: D. Case Study · p. 8, Figure 7 · p. 9
  • Missing merchant details and deals can appear after the map and merchant title, and without loading indicators those empty regions may resemble intentional page content.This is another instance of incomplete content on an otherwise structured page.Source: D. Case Study · p. 9
  • Apparently complete pages can still receive asynchronous updates, such as changes to merchant status, voucher price, or a service item.The paper identifies missing expected content and later updates to important business information as risks of relying solely on visual observations.Source: D. Case Study · p. 9

V. DISCUSSION

The discussion limits WAITLENS’s claims to visual evidence and recorded replay, and notes that adaptive waiting’s efficiency depends on avoiding more waiting than its checks cost. It frames waiting overhead as distinct from GUI agents’ test generation, exploration, and verification goals.

A. Limitations

  • Visual-only predictions can confuse intermediate and completed GUI states; even a deadline fallback may return a screenshot before Ready.The efficiency benefit also depends on avoidable waiting exceeding screenshot sampling, input preparation, and model execution costs; fast interactions with short fixed waits leave little room for improvement.Source: A. Limitations · p. 9

B. Threats to Validity

  • Annotators may differ on the earliest Ready time, so the study used an explicit criterion, two annotators, a third reviewer for disagreements, and task-group partitioning to limit train-test overlap.Derived observations remained within their source recording’s partition.Source: B. Threats to Validity · pp. 9–10
  • The findings may not generalize across all applications, interaction types, or device conditions; fixed-sequence video replay also cannot establish effects on task success or overall execution efficiency.The paper says closed-loop agent evaluation is needed for those effects.Source: B. Threats to Validity · p. 10

A. GUI Testing Agents

  • Unlike prior work focused on test generation, exploration, and verification, this study addresses waiting overhead between consecutive GUI-agent interactions.The paper presents this as an orthogonal problem for GUI testing agents.Source: A. GUI Testing Agents · p. 10

B. Waiting Strategies for GUI Testing

Prior GUI-testing work uses browser changes, expected DOM changes, rendering-state inference, and runtime monitoring to schedule waits. The conclusion reports that WAITLENS improves readiness-classification accuracy over its pretrained backbone and reduces redundant waiting in Meituan’s production pipeline.

B. Waiting Strategies for GUI Testing

  • Web-GUI approaches analyze browser or DOM changes, while mobile approaches infer rendering states or monitor widget readiness,,,,.The paper positions its focus as adaptive waiting for GUI testing agents in industrial practice, where diverse GUI states and heterogeneous devices demand generalizable strategies.Source: B. Waiting Strategies for GUI Testing · p. 10

C. VLMs for GUI Understanding

  • Static-screen VLM work targets GUI recognition and grounding –, while dynamic-GUI work includes screencast loading detection; WAITLENS predicts readiness at runtime.Liang et al. investigate VLM interpretation of UI animations and report remaining challenges in understanding their meanings.Source: C. VLMs for GUI Understanding · p. 10

VII. CONCLUSION

  • 86.08% accuracy, compared with 54.13% for its pretrained backbone, with a median latency of 0.243 s per model call.The industrial evaluation in Meituan’s production testing pipeline demonstrates real-life usability in substantially reducing redundant waiting while maintaining high observation reliability.Source: §VII · p. 10

Related papers

Loading 2610.32482v1…