Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking

• Search engines primarily rank “How-to” pages by query similarity, although executable solutions vary with users’ devices, operating systems, and application versions.Source: §1 · p. 1
• Automated verification is motivated by users’ time-consuming attempts to follow instructions across multiple pages that may not solve their problems.Source: §1 · p. 1
• The proposed pipeline extracts step-by-step instructions, simulates their execution on devices, and reranks pages using the resulting execution information.Source: §1 · p. 1
• The system adds verification and reranking to technical search, using generative-AI extraction and a generic action agent for Android applications.Source: §1 · p. 1
• The authors developed MagicWand and the WeWeb dataset to support Android “How-to” verification and reranking research.Source: §1 · p. 2
• The evaluation reports significant improvement over Google, while the approach is intended to support future platform-agnostic adaptations.Source: §1 · pp. 1–2

2 RELATED WORK

• No prior work known to the authors verifies “How-to” search results through simulated execution.Source: §2 · p. 2

2.1 Information Extraction

• Information-extraction research spans heuristic, machine-learning, and LLM-based methods, with this work inspired by zero-shot GPT3 plan extraction.Source: §2.1 · p. 2

2.2 Acting Agent

• Android acting-agent research progressed from fixed action prediction toward transformer and multimodal systems, but ambiguous instructions remain challenging without UI hierarchy knowledge.Source: §2.2 · pp. 2–3
• The proposed Android action model dynamically prompts GPT4-V with fine-grained UI control information distilled from runtime metadata.Source: §2.2 · p. 3
• Accessibility Service API was selected over ADB because it can support standalone mobile applications without additional device-computer connection settings.Source: §2.2 · p. 3

2.3 Reranking

• Reranking research evolved from classic learning-to-rank methods toward neural objectives such as NeuralNDCG, which motivates the paper’s ranking choices.Source: §2.3 · p. 3

• The system incorporates device, operating-system, and application-version information alongside client-side verification outcomes to address limits of traditional ranking.Source: §3 · p. 3
• Stage 1 uses a generative model and HTML grounding to generate and improve step-by-step instructions from the top N retrieved pages.Source: §3 · p. 3
• Stage 2 predicts device actions from extracted instructions and iteratively executes them until the task is completed or the agent cannot proceed.Source: §3 · p. 3
• Stage 3 extracts representation features from execution information and assigns higher ranking priority to pages producing better execution results.Source: §3 · p. 3
• Pre-trained GPT models provide a rapid implementation covering instruction extraction, on-device execution, and reranking.Source: §3 · p. 3

• The extraction stage separates instruction generation from grounding and requires no collected training data through zero-shot learning.Source: §3.1 · p. 3
• For each query and candidate page, the generative model receives the query, page title, and cleaned HTML with extraneous information removed while preserving hierarchy.Source: §3.1 · p. 3
• Grounding matches generated instructions to HTML snippets using semantic and ROUGE similarity, then constrains later matches by XPath similarity to preserve sequence.Source: §3.1 · p. 3
• Only instructions grounded in the original HTML proceed to the execution stage, reducing unsupported generated content.Source: §3.1 · p. 3

3.2 On-Device Execution

• The module executes extracted instructions on an Android device to assess whether they can be completed, while collecting execution information throughout the process.The loop ends when all instructions finish, no next action can be determined, or the maximum step count is reached.Source: §3.2 · pp. 3–4

3.2.1 Runtime UI Context Detection.

• Runtime UI Context Detection represents the current screen with a screenshot, actionable-region metadata, UI controls, potential actions, and runtime information for prediction and reranking.Actionable regions are hierarchically organized and may include textual descriptions, coordinates, and scrolling properties.Source: §3.2.1 · p. 4, §3.2 · p. 4
• Control distillation filters invisible or non-actionable regions and hierarchically consolidates descriptions from nearby child regions when parent controls lack useful text.The procedure addresses missing textual information needed for accurate action prediction.Source: §3.2.1 · p. 4

3.2.2 Multimodal Action Prediction Model.

• The multimodal Action Prediction Model uses GPT4-V with current UI context, prior screenshots and actions, candidate controls, and required external context to select action types and parameters.Candidate actions include clicks, swipes, inputs, and back actions; the execution proxy then locates and performs the predicted action, using validation and fallback alternatives when needed.Source: §3.2.2 · p. 4, §3.2.3 · p. 4

3.2.3 Execution Proxy.

• The execution and auxiliary service modules synchronize textual and visual trajectories with external analysis plugins that summarize and evaluate execution status.The execution proxy can adjust actions for different application settings, while the auxiliary service monitors progress.Source: §3.2.3 · pp. 4–5

3.3 Reranking

• Execution information is converted into a feature vector containing textual, visual, UI, action, and runtime signals for each retrieved page.Collected fields include the query, goal, page, instructions, mobile screens, control lists, predicted actions and parameters, action attribution, and runtime information.Source: §3.3 · p. 5

3.3.1 Feature Design.

• The reranking features are normalized to float values between 0 and 1, while F17 and F18 capture execution progress and action coverage across extracted instructions.F17 measures the furthest matched instruction position, and F18 measures the distance between the first and last matched instruction terms.Source: §3.3.1 · p. 5
• Textual features measure query-term frequency, page-query relevance, and common keyword frequency in extracted instructions.GPT generates the page-query relevance score over [0, 1], while the query-term ratio uses query tokens without stopwords.Source: §3.3.1 · p. 5
• Visual and execution features quantify instruction completion, action-term overlap, visible UI-term alignment, distilled-control alignment, and how much of each step was executed.These features use averages, minima, maxima, and variance where applicable; visible UI-term alignment reflects correspondence between screen controls and instruction terms.Source: §3.3.1 · p. 5

3.3.2 Reranking Models.

• A page is verified only if it reaches on-device execution, successfully executes at least one step, and has non-zero predicted instruction completion; unverified pages retain relative order below verified pages.Pages without extracted instructions or instructions to execute do not advance to on-device execution.Source: §3.3.2 · pp. 5–6
• Three learning-to-rank models rerank verified pages: pairwise logistic regression, TMLP with LambdaLoss, and TMLP with NeuralNDCG loss.LambdaLoss weights pairwise errors by their effect on the ranking metric, whereas NeuralNDCG uses differentiable sorting to optimize NDCG directly.Source: §3.3.2 · p. 6

4 EXPERIMENTS

• The experiments use MagicWand to standardize training, execution, and evaluation for Android agents, with Google as the original search engine.The platform can run tasks on Android emulators or connected devices through web or ADB-equivalent interfaces.Source: §4 · p. 6

4.1 Test Data: "How-to" WeWeb dataset

• The WeWeb dataset covers 17 apps across 9 Google Play categories plus a System domain, using top-20 Google results and human-verified instruction pages.Workers extracted instructions and manually executed them; 168 pages were successful and 64 queries had at least one successful page.Source: §4.1 · p. 6, Table 3 · p. 6
• The dataset records worker execution evidence, including screenshots, action histories, screen recordings, UI context, and binary task-completion labels.A label is 1 when an annotator can follow a page’s instructions and complete the task, and 0 otherwise.Source: §4.1 · p. 6

4.2 Evaluation Metrics

• Evaluation uses MRR@1, P@1, P@5, and NDCG@5 to measure the quality of reranked search results.These metrics are computed for each query using the human-derived relevance score y.Source: §4.2 · p. 7, §4.4 · p. 7

4.3.1 Extraction.

• GPT4 extracts step-by-step instructions from candidate pages, while ROUGE and FAISS thresholds ground generated steps in webpage text.GPT4 runs at temperature 0, and Cohere produced no noticeable performance difference in the reported comparison.Source: §4.3.1 · p. 7

4.3.2 On-Device Execution.

• The on-device execution stage predicts Android actions from instructions and UI context, iterating until the task is completed or the agent cannot proceed.Execution information and status are stored centrally, and a GPT4V plugin generates a task completion rate for each execution.Source: §4.3.2 · p. 7

4.3.3 Pre-training Reranking Models.

• Because WeWeb is small, pairwise LTR models are pre-trained on synthetic META-GUI data and then applied to WeWeb in a zero-shot setting.Synthetic positives come from generated trajectories and query swaps, while negatives come from failed or modified trajectories; the authors note that the synthetic data differs significantly from the test set.Source: §4.3.3 · p. 7

4.4 Methods compared

• The comparison includes Google ordering, an oracle, a GPT4V completion-degree feature, logistic regression, NeuralNDCG + TMLP, and LambdaLoss + TMLP.All methods are evaluated with MRR, P@1, P@5, and NDCG@5 using human labels as relevance scores.Source: §4.4 · pp. 7–8

5.1 Comparison with Baselines

• The baseline comparison shows that all evaluated reranking models improve Google, with statistically significant gains for LR, NeuralNDCG + TMLP, and LambdaLoss + TMLP.The results use zero-shot learning on the How-to WeWeb dataset.Source: §5.1 · p. 8, Table 5 · p. 8
• The F4-based rank underperforms other reranking models, while TMLP models slightly outperform LR by capturing feature interrelationships.The small neural-model difference may reflect low-quality synthetic pre-training data.Source: §5.1 · p. 8

5.2 Further Analysis and Ablation Studies

• Instruction extraction produces instructions for 398 of 3321 pages, but only 75 extracted pages contain relevant instructions because LLM hallucinations introduce noise.The extracted pages cover 117 of 167 queries, while 323 extracted pages are negative samples.Source: §5.2 · p. 8
• Execution filters 119 pages, while about 75% of pages with instructions support completing more than 75% of instructions; false F4 = 1 assignments still add reranking noise.The analysis motivates more robust execution-derived signals.Source: §5.2 · p. 8, Table 6 · p. 8

Q4. Which features contribute more to the overall perfor-

• Reranking models learn useful nonlinear relationships among features such as F4 and F2, but limited negative examples and features cause abnormal low-valued inputs to be undervalued.Execution cycles can also produce incorrect rankings despite high visual-feature scores.Source: §5.2 · p. 8, Q4. Which features contribute more to… · p. 9

5.2 Further Analysis and Ablation Studies

• HTML grounding reduces false positives from 953 to 303 but also reduces true positives from 142 to 96, indicating a precision–recall trade-off.Grounding adds missing steps and removes hallucinated instructions using matching HTML DOM locations.Source: §5.2 · p. 9, Table 7 · p. 9
• A rule that promotes pages with F4 = 1 reaches P@1 = 0.1317, underperforming the proposed LTR approaches and showing that one strong feature is insufficient.The rule is not significantly better than the baseline.Source: §5.2 · p. 9

5.3 Discussion and Future Work

• The study remains preliminary because results are Android-centered and domain-specific, while the reranker lacks real-world training data and the system needs safeguards against harmful actions.Future work targets cross-platform execution, richer data and representations, and safer verification environments.Source: §5.3 · pp. 9–10

• Verified executable instructions add instruction completeness to textual relevance and authority when reranking “How-to” search results.The experimental results further improve a strong Google baseline.Source: §6 · p. 10
• The workflow is presented as a pioneering approach to improving the reliability and usefulness of online software-task help resources.Source: §6 · p. 10
• The current system remains limited in supporting a wider range of instructions, motivating future improvements to the software agent.Source: §6 · p. 10