PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid Security Framework for Mini-Programs: Visual UI Compliance and Network Risk Assessment

Panpan Shen, Lei Xie, Xiaoqi Li

arXiv:2608.25877v1cs.CR

TL;DR

Mini-Programs present UI-compliance and operational-safety concerns, while existing testing lacks dedicated visual-interactive inspection integrated with network-risk assessment. The paper develops a hybrid YOLOv8–mitmproxy framework that checks interface elements and redirected traffic, delivering a standardised testing scheme for Mini-Program security.

  • Problem

    Non-standard interactive controls can cause inadvertent clicks, payments, and questionable redirections, while existing methods lack integrated visual-oriented, user-interactive, and network-risk inspection.

  • Method

    The framework combines YOLOv8-based visual element recognition with mitmproxy traffic analysis to perform UI compliance checking and dynamic behavioural risk assessment.

  • Results

    The prototype accurately locates, measures, and archives non-compliant interactive widgets while cross-verifying element geometry with URL security features.

  • Takeaways & Limitations

    The system delivers a standardised testing scheme intended to regulate Mini-Program UI design and protect users’ property interests.

  • Takeaways & Limitations

    The combined approach failed to achieve satisfactory detection accuracy, and the prototype motivates further refinement for diversified threats and full-lifecycle monitoring.

Abstract

from arXiv · show

With the continuous development of the WeChat ecosystem, WeChat Mini Programs, due to their advantages of not requiring installation, using little memory, and being ready to use instantly, have seen a surge in user numbers and have now become an indispensable service carrier in mobile internet. However, as Mini Programs rapidly became popular, issues regarding the compliance of their interface interaction design and the safety of operational behavior have become increasingly apparent. Many Mini Programs have problems such as clickable buttons and icons not being standard in size, or ad pop-ups and payment entrances being placed in a way that is easy to misclick. The close or cancel buttons are often too small or hidden, making it easy to accidentally click on ads or payment content, and difficult to accurately click the cancel button. This can result in involuntary payments or being redirected to illegal pages, causing unnecessary financial losses and seriously harming users' property security and legal rights. To address the above issues, this article develops a detection program to check the position and size of various icons and buttons in Mini Programs, and analyze whether redirected links fall within a safe range. YOLOv8 is used to identify various buttons in images, displaying the corresponding icon and its data based on the mouse click position. Violations are flagged and recorded. At the same time, mitmproxy is used to capture relevant data requests generated during clicks, analyzing the safety of redirections, and presenting key information for user observation.

1 Introduction

WeChat Mini-Programs have created growing interface-compliance and operational-security concerns, while existing security tools leave visual, spatial, and interactive gaps. The paper addresses this gap with a hybrid framework combining YOLOv8-based UI inspection, threshold-based compliance checking, evidence retention, and mitmproxy-based network-risk assessment.

  • 450 million daily active users and trillions of yuan in annual transactions illustrate the scale of the WeChat Mini-Program ecosystem by 2022.
  • Undersized controls and poorly placed advertisements or payment entrances can cause accidental clicks, unwanted payments, and questionable redirections.
  • Existing approaches address architectural, code-level, static-taint, and data-transmission risks but do not provide dedicated visual-interactive inspection.
  • The proposed framework combines YOLOv8 visual recognition with mitmproxy traffic analysis for UI compliance checking and dynamic behavioural risk prediction.
  • A real-time screen-capture and preview module removes the detector window so users can verify image clarity, occlusion, and positioning.
  • YOLOv8 detects interactive elements after mouse clicks, while a 44 × 44 CSS-pixel rule identifies non-compliant controls.
  • Non-compliant elements are highlighted, and timestamped screenshots are archived as evidence while mitmproxy captures click-triggered requests for risk analysis.

2 Background

The framework combines Mini-Program architecture analysis, YOLOv8-based element detection, event-driven screen interaction, evidence retention, and URL-risk assessment. Its pipeline links user clicks to cleaned screenshots, localized UI components, compliance judgments, and network inspection.

  • 2 Background: Mini-Programs separate rendering and logic layers, creating distinct risks involving UI manipulation, insecure API calls, and improperly validated data.
  • 2.2 YOLOv8 for UI-Element Detection: YOLOv8 uses hierarchical visual features to detect UI components despite variation in scale, lighting, appearance, and partial occlusion.
  • 2.2 YOLOv8 for UI-Element Detection: YOLOv8 predicts bounding-box coordinates and category probabilities, then retains the candidate with the maximum composite t value to suppress false positives.
  • 2.2 YOLOv8 for UI-Element Detection: Coordinate-based regression adapts bounding boxes to different screen resolutions and aspect ratios without manually changing anchor parameters.
  • 2.2 YOLOv8 for UI-Element Detection: The compound loss combines CIoU, Distribution Focal, and binary cross-entropy losses to optimize localization and classification.
  • 2.3 Screen Capture and Display: PyAutoGUI captures RGB screen arrays, while Win32gui masks the detector window to provide a clean Mini-Program view.
  • 2.4 Event-Driven Interaction with Pynput: Pynput activates detection on mouse events; the system then selects the UI component nearest the click and displays its category, size, and compliance state.

3 System Design and Methodology

The framework separates data, business logic, and interaction responsibilities into a layered architecture, integrating event-driven visual inspection with optional network-risk analysis. Its interface presents previews, assessments, logs, packet data, and summary statistics for observable detection workflows.

  • Layered Architecture: The three-tier architecture divides the system into data, business logic, and interaction layers according to functional responsibilities and data transmission sequence.This separation is intended to improve maintainability and scalability.
  • Layered Architecture: The business-logic layer coordinates screen capture, YOLOv8 recognition, mouse-event monitoring, and data processing through event-driven module collaboration.YOLOv8 outputs element categories, confidence scores, and bounding-box coordinates for subsequent compliance checking.
  • Interface Design: The graphical interface contains dedicated regions for preview, risk prediction, operation logs, parsed JSON, and summary statistics.These regions display processed screens, traffic assessments, recent detection records, core packet fields, and aggregate violation and risk information.
  • Detection Modes: Precise detection compares the clicked element with a 44 × 44 CSS-pixel threshold, whereas full-scene recognition marks all detectable UI elements on the current screen.The visual and network modules can also be activated independently or together.
  • Architecture Benefits: The modular architecture maintains low coupling, allowing YOLOv8 optimization without changing storage mechanisms and alternative storage backends without modifying the business logic.The design also supports development, debugging, later expansion, and performance optimization.
  • Network Risk Assessment: For network assessment, mitmproxy forwards outbound HTTP and HTTPS requests, filters traffic using a trusted-domain whitelist, and feeds extracted features to a classifier for risk scoring.Risk scores and UI compliance labels are displayed synchronously in the interface.

4.1 Experimental Environment

The experimental environment combines Windows and Python software with GPU-equipped hardware and four datasets spanning self-built and public interface images. The datasets cover office, web, game, and payment-oriented Mini-Program screens.

  • Hardware and Software: The system ran on an Intel Core i7-12500H, 16 GB DDR4 memory, an NVIDIA GeForce GPU, a 512 GB SSD, and a 1920 × 1080 display.The software environment used Windows and Python with YOLOv8, PyAutoGUI, Pynput, Win32gui, tkinter, Pillow, and mitmproxy.
  • Datasets: The YOLO training data comprised three self-built datasets and one public dataset containing 100, 150, 150, and 500 images, respectively.The image corpus covered office applications, web pages, games, and payment-oriented Mini-Program screens.

4.2 Testing Procedure and Error Analysis

Testing compared screen-capture and recognition approaches, then analyzed failure modes affecting text extraction, semantic classification, spatial heuristics, and cross-platform integration. YOLOv8 was selected because the earlier OpenCV–PyAutoGUI combination did not provide satisfactory detection accuracy.

  • Approach Comparison: The initial OpenCV–PyAutoGUI combination failed to achieve satisfactory detection accuracy, motivating evaluation of alternative visual-detection approaches.The testing procedure examined screen capture, visual recognition, and mouse-event monitoring under diverse scenarios.
  • Error Analysis: Text extraction was unstable because capture-window size, cursor occlusion, bounding-box scaling, and background graphics degraded recognition robustness.Advertising graphics were frequently misclassified as text blocks.
  • Error Analysis: The low-level classification pipeline used contour shape, component size, and pixel colour without semantic reasoning, producing coarse labels for complex interactive widgets.It could not identify the functional semantics of each element.
  • Error Analysis: Hard-coded spatial assumptions, such as predefined close-button, commodity-panel, and navigation-bar regions, generated false detection records when those regions contained no interactive elements.The method therefore produced invalid records even in the absence of relevant controls.
  • Tool Selection: Airtest supported mobile screen capture and multi-platform deployment but produced coordinate outputs that could not be stably parsed when integrated into the framework.The extracted positions originated from the underlying PyAutoGUI interface rather than Airtest itself.
  • Tool Selection: YOLOv8 was adopted because it provides competitive graphic classification and precise prediction of UI-element coordinates and dimensions, despite lacking direct text recognition.The study compared multiple datasets and hyperparameter combinations to determine training configurations.

4.3 Comparative Performance

Comparative experiments examined capture performance, YOLO-model metrics, and dataset suitability. YOLOv8 outperformed YOLOv7 across all reported metrics, while the balanced Custom Set 3 achieved the best practical detection performance and was used thereafter.

  • Screen Capture: BitBlt acquired screenshots faster and consumed fewer resources than PyAutoGUI screenshot() under 1920×1080 resolution.Display clarity and interface stuttering differed little in practical testing, while BitBlt was limited to Windows and screenshot() supported Windows, macOS, and Linux.
  • Object-Detection Models: YOLOv8 outperformed YOLOv7 across all reported metrics under identical hardware and training-data conditions.The comparison found substantial improvements in precision and recall.
  • Object-Detection Models: Training performance depended strongly on batch size and epoch count, with insufficient epochs producing near-zero accuracy and excessive iterations potentially degrading performance.Under CPU-only training, 100 images over 100 epochs required approximately 1.5 hours, compared with nearly 4 hours for 500 images.
  • Dataset Comparison: Custom Set 3 achieved the best practical detection performance because its element categories were balanced, except for rare maximise and share controls.Custom Set 1 had weak generalisation, Custom Set 2 was biased toward product widgets, and Public Set 4 was not tailored to Mini-Program interfaces.
  • Dataset Comparison: Custom Set 3 was selected for all subsequent experiments after the four datasets were evaluated for real-world detection capability.The selection followed comparison of dataset label distributions, statistical indicators, and practical detection behaviour.

4.4 Comparison of Packet-Capture Tools

The comparison evaluates Fiddler, Wireshark, and mitmproxy across integration, HTTPS decryption, filtering, and output readability. Mitmproxy was adopted for its Python integration and flexible traffic filtering despite requiring system-proxy configuration.

  • Fiddler parses Mini-Program domains and JSON payloads effectively, but HTTPS decryption requires manual third-party certificate installation.
  • Wireshark captures and classifies all network traffic but cannot decrypt HTTPS by default, requiring private-key configuration.
  • Wireshark filtering is time-consuming, and its output mainly contains domain names and ciphertext rather than structured application-layer information.
  • Mitmproxy offers analytical power comparable to Fiddler, flexible filtering, and seamless integration with the Python-based framework.
  • The framework adopts mitmproxy because its native programmability and analytical capacity support the network-risk assessment module.

4.5 Experimental Findings and Optimisation Directions

Testing identifies hardware, dataset, filtering, and parameter consistency as determinants of detection performance and stability. The paper proposes dataset expansion, runtime optimisation, functional extensions, and usability improvements to address current limitations.

  • Experimental Findings: GPU performance and library versions directly affect object-detection latency, while frequent full-screen capture can severely stall the system.
  • Experimental Findings: Dataset quality and domain-whitelist rules dominate accuracy: better annotations improve detection, while refined filtering removes irrelevant network-traffic noise.
  • Experimental Findings: Parameter mismatches between capture and detection modules cause stuttering, crashes, or functional failure, making consistent tuning essential for stability.
  • Experimental Findings: Bounding boxes can deviate from UI boundaries when background colours resemble target widgets, and large panels can occlude tiny close buttons.
  • Optimisation Directions: The proposed improvements include expanding annotated data, tuning YOLOv8 with augmentation, and adjusting thresholds to improve small-object detection.
  • Optimisation Directions: Faster capture APIs, multithreading, resource allocation, batch detection, result export, compression, and multi-window inspection are proposed for efficiency and scale.
  • Optimisation Directions: Actionable feedback, problem-report channels, and manually labelled misdetections are proposed to support usability and iterative model fine-tuning.
  • Optimisation Directions: Implementing these optimisations is expected to enhance the tool’s accuracy, speed, usability, and extensibility in complex testing scenarios.

5 Conclusion

The paper concludes with an integrated framework combining visual UI inspection and network-traffic analysis for WeChat Mini-Programs. It also identifies broader directions for diversified detection, lifecycle monitoring, and cross-platform iteration.

  • The prototype combines YOLOv8 visual inspection with mitmproxy traffic analysis to assess UI compliance and predict network risks.
  • The framework cross-verifies UI geometry and URL security features, while configurable filtering balances data completeness with processing efficiency.
  • With expanded training data and parameter tuning, the system generalises across diverse Mini-Program interfaces and reveals threats behind standard-looking UI components.
  • The authors present the system as a standardised scheme for regulating Mini-Program UI design and protecting users’ property interests.
  • Future work targets Mini-Program-specific threats, full-lifecycle monitoring, real-time alerts, cross-platform deployment, adaptive optimisation, and cloud synchronisation.
Loading 2608.25877v1…