PaperlayerView exact revision on arXiv

Source-linked AI summary

5uGUE: Agentic Hardware-in-the-Loop Fuzzing for 5G Devices and beyond

Guoqiang Hua, Sakshi Udeshi, Matheus Eduardo Garbelini, Daisuke Mashima, Sudipta Chattopadhyay

arXiv:2610.29155v1cs.NI

TL;DR

Proprietary, black-box 5G devices process untrusted wireless signaling at enormous scale, but LLM agents have largely been designed for software-in-the-loop fuzzing. 5UGUE applies an LLM strategist to OTA hardware-in-the-loop fuzzing and adapts its tests from runtime feedback. The paper reports substantially more crashes than the state-of-the-art baseline and extends the approach to Bluetooth Classic.

  • Problem

    LLM-based agents are predominantly designed for software-in-the-loop settings, while proprietary, black-box wireless devices remain difficult to analyze at scale despite their growing use.

    Source: §I · p. 1

  • Method

    5UGUE uses an LLM as a strategist that selects and executes OTA tests on COTS devices, then replans from structured runtime feedback.

    Source: §IV · p. 4

  • Results

    5UGUE discovers 17.5× as many crashes as the state-of-the-art baseline, and the paper also demonstrates adaptation to Bluetooth Classic.

    Source: §IX · p. 13

  • Takeaways & Limitations

    The authors conclude that LLM agents can autonomously fuzz hardware-in-the-loop systems, including proprietary, black-box devices receiving untrusted over-the-air signals.

    Source: §IX · p. 14

    The evaluation focuses on 5G NR; although Bluetooth Classic is demonstrated, detailed multi-protocol evaluations are deferred to future work.

    Source: §VIII · p. 13

Abstract

from arXiv · show

In this paper, we present 5uGUE, a large language model (LLM)-driven hardware-in-the-loop agent, which enables the wireless fuzzing of 5G-enabled devices (such as mobile phones). 5uGUE combines multi-step reasoning with tool-assisted interaction and integrates into the fuzzing pipeline by leveraging specialized tools for internet search, 5G protocol analysis, memory management, and test-case execution. The agent iteratively generates test cases, executes them on targets, collects runtime feedback, and adaptively refines the fuzzing strategy. We evaluate 5uGUE on five commercial off-the-shelf (COTS) 5G devices across six LLMs. The best performing runs for 5uGUE trigger 167 crashes in total (141 novel and 31 unique) within 60 interaction turns (49 to 133 minutes total runtime). In contrast, the best runs for the state-of-the-art baseline (5GHOUL) discovers only 13 crashes in total (10 novel and 6 unique) within 150 minutes. Overall, by integrating LLM-driven agents, 5uGUE demonstrates significantly better crash discovery performance (12.8x total, 5.2x unique improvement). Notably, among the crashes discovered by 5uGUE, MediaTek has confirmed two new high-severity vulnerabilities, with one CVE assigned (CVE-2026-20525), while UNISOC confirmed seven crashes. Finally, to demonstrate that 5uGUE is agnostic to wireless protocols, we also extend it to Bluetooth Classic with only minimal code changes. In our evaluations, 5uGUE finds an average of 6.4 novel crashes and 1.6 unique crashes. In a nutshell, 5uGUE highlights that LLM agents are capable to operate effectively beyond software-in-the-loop settings, extending autonomous vulnerability discovery to the proprietary, hardware-bound systems that affect billions of end-user devices across multiple wireless protocols.

I. INTRODUCTION

5G devices expose a rapidly growing but comparatively underexplored attack surface because proprietary, black-box hardware processes untrusted over-the-air signaling. 5UGUE addresses the gap with an LLM strategist that autonomously runs and adapts a hardware-in-the-loop fuzzing loop, with an architecture that also extends to Bluetooth Classic.

I. INTRODUCTION

  • Black-box, hardware-bound 5G devices process untrusted over-the-air signaling at enormous scale, yet remain comparatively underexplored in offensive security.The passage estimates 2.9 billion subscriptions by the end of 2025 and projects 6.4 billion by 2031.Source: §I · p. 1
  • Existing agentic systems mainly target software-in-the-loop settings, while 5UGUE fuzzes commercial 5G devices over the air and extends to Bluetooth Classic with minimal procedural code changes.Software-in-the-loop agents often use source access, coverage, and feedback that are unavailable for proprietary wireless hardware.Source: §I · p. 1
  • 5UGUE makes the LLM strategist responsible for the full fuzzing loop, from sourcing seeds and planning tests to executing them on devices, reading feedback, and adapting strategy.The agent orchestrates this loop without human intervention and can use modular tools for its actions.Source: §I · p. 2
  • 5UGUE is presented as the first agentic hardware-in-the-loop fuzzing framework for COTS 5G devices, with modular tools that can be added or removed with minimal effort.The listed tools include exploit execution, memory, and web or literature search.Source: §I · p. 2

II. BACKGROUND

  • The threat model assumes a rogue gNB can connect over the air and eavesdrop, modify, or inject messages, but cannot decrypt encrypted messages without the key.The experiments focus mainly on non-physical protocol layers from MAC upward, particularly pre-authentication messages.Source: §II · p. 2
  • A typical 5G NR test case selects packets with a Wireshark filter and mutates payload bytes through offset-value pairs.The example selects RRCSetup packets and changes a chosen payload byte.Source: §II · p. 2

III. RELATED WORK

Prior fuzzing work applies LLMs to parts of the software fuzzing pipeline or targets selected wireless protocol layers. 5UGUE instead performs OTA tests on physical COTS devices and uses an adaptive agent loop, with persistent strategy, feedback processing, and reflection guiding continued exploration.

III. RELATED WORK

  • Earlier LLM fuzzers use models for protocol extraction, seed diversification, or test generation, while 5UGUE makes the LLM the strategist for the full loop.Conventional and LLM-assisted approaches are described as human-directed or limited to strengthening individual pipeline elements.Source: §III · p. 3
  • 5UGUE performs OTA fuzzing against real silicon in COTS 5G devices and replans using the vendor-reported crash oracle.The agent plans tests, delivers mutated messages over a radio link to a physical handset, then replans from results.Source: §III · p. 3
  • Unlike CONSET, which targets messages containing ASN.1 fields, 5UGUE generates invalid messages across 5G protocol layers and can extend to other protocols such as Bluetooth.The paper also notes that CONSET does not provide its OTA exploits for direct comparison.Source: §III · p. 3

IV. METHODOLOGY

  • 5UGUE selects concrete packet filters, offsets, and mutation values, then replans after each test using structured feedback such as crash identifiers, mutation counts, and state-machine differences.Its initial seeding strategy is also adjustable rather than fixed.Source: §IV · p. 4

A. System Architecture

  • A persistent strategy memo preserves plans across turns, while context compaction summarizes older messages and retains test results and key findings.The memo is re-injected into each system prompt, so tested offset-value combinations can survive conversation compaction.Source: A. System Architecture · p. 4
  • After five consecutive compiled tests with neither a crash nor a new state, 5UGUE prompts the agent to inspect its history and revise its strategy.Each test is also tagged with a mutation category, with no more than three consecutive tests in one category.Source: A. System Architecture · pp. 4–5
  • The default seeding strategy uses external information to find an initial corpus, but the agent can instead sweep candidate packets when those sources are unavailable.Both strategies are prompt-level choices and switching between them requires no code change.Source: A. System Architecture · p. 5

B. Tool Interface

5UGUE gives the agent tools for inspecting packets and devices, researching prior work, executing exploits, and retaining findings. Its versions progressively add GitHub-aware retrieval and batched execution, with a filtered variant restricting external knowledge.

B. Tool Interface

  • 5UGUE’s tools span packet inspection, exploit execution, crash and state analysis, device inspection, memory, and external research.Together, these tools let the agent move from protocol observations and research to tests and post-test diagnostics.Source: B. Tool Interface · p. 5
  • Exploit tools validate required function signatures and return structured execution results; batch execution runs up to five tests sequentially in one call.Crash and state tools parse crash identifiers and logs and build state-transition graphs, while device tools expose logs and device status.Source: B. Tool Interface · pp. 5–6
  • Memory tools preserve findings and a strategy memo across turns, while web and local-paper search provide prior vulnerability and research information.The agent can save observations and retrieve them by tag or keyword; web retrieval can extract exploit repositories and local paper text.Source: B. Tool Interface · p. 6

C. Agent Versioning

  • V1 uses basic search and single-test execution; V2 adds GitHub-aware retrieval, and V3 adds batch testing with a larger context-summarization threshold.V3 raises that threshold from 50,000 to 200,000 tokens to avoid prematurely summarizing batch results.Source: C. Agent Versioning · p. 6
  • V3-filtered restricts external knowledge by disabling paper tools and blocking specified web domains, and adds a broad sweep-based seeding strategy.It is a restricted variant of V3 rather than another progressively expanded configuration.Source: C. Agent Versioning · p. 6

D. Execution and Oracle

5UGUE turns protocol-level hypotheses into OTA tests and uses device logs and other signals to detect crashes. Its case study illustrates a research-to-execution loop that records findings, updates strategy, and shifts toward novel crashes after the first crash; proprietary firmware limits root-cause analysis.

D. Execution and Oracle

  • 5UGUE generates packet- and byte-level exploit tests through an OTA framework, then uses device logs or other device-specific signals to detect crashes.Phone crashes are identified from log indicators such as “Modem Resets!” and “modem_failure”; Bluetooth devices may report “Guru Meditation Error.”Source: D. Execution and Oracle · pp. 6–7
  • A crash is unique within a session when its identifier string has not previously appeared in the session’s external knowledge base.That database is used only for a deterministic presence check and is not accessible to the agent.Source: D. Execution and Oracle · p. 7

E. Crash Root Cause Analysis and Mitigation

  • Because tested devices run proprietary firmware, available diagnostics localize failures but cannot establish the underlying flaw or support validation of exact source-level mitigations.The authors say crashes may involve insufficient validation or improper handling of unexpected values, but cannot determine the precise failure type.Source: E. Crash Root Cause Analysis and… · p. 7

A. Case Study: An Actual 5UGUE Run

  • In the case study, the agent researched during Turns 1–14, then generated and executed exploits from Turn 15, recording outcomes and refining its strategy after tests.After its first crash, it changed its objective to prioritize novel crashes for subsequent turns.Source: A. Case Study: An Actual 5UGUE Run · pp. 7–8

B. Case Study: Building the 5UGUE Agent

The agent’s development combined iterative tool and prompt refinement with an evaluation setup spanning 5G and Bluetooth devices. The authors also define a research-quality diagnostic, while noting that it can be objectively assessed only on the MediaTek target.

B. Case Study: Building the 5UGUE Agent

  • The GitHub fetch tool was optimized to return repository structure as clean JSON, while batch execution lets one call run multiple exploits sequentially.These changes addressed redundant HTML/token use and the overhead of making separate calls for candidate exploits.Source: B. Case Study: Building the 5UGUE Agent · pp. 8–9
  • Research quality scores how well retrieved information supports effective fuzzing, using predefined rules on a 0–5 scale from coarse knowledge to technical detail.A score of 5 requires a crash and demonstrated score-4-level knowledge; scores are not strictly cumulative.Source: B. Case Study: Building the 5UGUE Agent · p. 9
  • Research quality is evaluated only on the MediaTek Dimensity 900 target because other targets lack the ground truth needed to apply the criteria.The metric is diagnostic rather than strictly cumulative.Source: B. Case Study: Building the 5UGUE Agent · p. 9

VI. EVALUATION SETUP

  • The evaluation setup covers five COTS 5G phones from three manufacturers and two modem vendors, plus Bluetooth Classic using an ESP32 host and ESP-WROOM-32 target.The 5G devices were selected to represent differing modem implementations and smartphone brands.Source: §VI · pp. 9–10
  • The evaluation includes six LLMs—four commercial and two open-source models—and repeats each experiment five times under identical configurations.The model selection balances capability, provider diversity, and open-source practicality.Source: §VI · p. 10

VII. EVALUATION RESULTS

The configuration results show that adding GitHub-aware retrieval and batch testing improves crash discovery, with V3 strongest against V1 on total and unique crashes. The reported research-quality score also rises from V1 to V3, though it peaks at V2.

A. How effectively does 5UGUE discover crashes?

  • Crashes absent from publicly accessible sources are classified as novel, while crashes with public resources and available exploit scripts are classified as known.Unique crashes are counted by identifier, as described in the evaluation setup passage.Source: A. How effectively does 5UGUE discover… · p. 10
  • 32.2 average total crashes in V3, compared with 7.2 in V1 and 12.8 in V2; average unique crashes rose from 3.6 to 7.0 and 10.0, respectively.V2’s novel-crash count was 0.4 below V1, while its known-crash count reached 10.4, a 2.36× improvement over V1.Source: A. How effectively does 5UGUE discover… · pp. 10–11
  • V3 added 10 average novel crashes and 15 average known crashes compared with V1, while average research quality scored 3.14 in V1, 4.20 in V2, and 4.14 in V3.The authors attribute the known-crash increase in V2 likely to GitHub fetch and report V3’s batch testing and larger context window as contributing to its gains.Source: A. How effectively does 5UGUE discover… · p. 11
  • Compared to V1, V3 achieved a 4.47× improvement in average total crashes, a 2.78× improvement in average unique crashes, and a 31.8% increase in average research quality score.Source: A. How effectively does 5UGUE discover… · p. 11

B. How effectively does 5UGUE discover crashes across different modems?

5UGUE discovers crashes across five modem targets, extends to Bluetooth Classic, and shows substantial performance differences across LLMs. Its results also show that a refined seeding strategy can discover crashes with limited external knowledge.

B. How effectively does 5UGUE discover crashes across different modems?

  • Under V3, average runtime remained below 105 minutes per target within 60 turns, with variation across runs attributed to the agent selecting different fuzzing paths.Source: B. How effectively does 5UGUE discover… · p. 11
  • 21 total crashes and 4.48 unique crashes per target on average; 5UGUE found crashes on all five targets, ranging from 11.6 total on unisoc_ud710 to 32.2 on mediatek.The mediatek target also averaged 10 unique crashes; publicly available exploit-related information was substantially greater for that target.Source: B. How effectively does 5UGUE discover… · p. 11, Figure 5 · p. 11

C. Does 5UGUE extend to other protocols?

  • 6.4 novel crashes and 1.6 unique crashes were found on esp32 across five runs after adapting 5UGUE to Bluetooth Classic with minimal changes and no protocol-specific optimization.The evaluation also reports 8 total crashes on average; three of eight unique BT crashes could not be reproduced.Source: C. Does 5UGUE extend to other protocols? · pp. 11–12
  • Despite limited access to external knowledge sources, 5UGUE V3-filtered discovered an average of 17.4 total crashes, including 2.8 unique, on mediatek, and 22.2 total, including 3.4 unique, on unisoc_ud710, averaged over five runs.V3-filtered uses a refined prompt to bootstrap exploration without target-specific packets or offsets; non-filtered versions derive seeds and mutation targets mainly from external knowledge.Source: D. How does 5UGUE perform with… · p. 12, Figure 6 · p. 12

E. How does the choice of LLM affect the performance of 5UGUE?

  • 35.8 total crashes per run made Kimi K2.6 the strongest model, 6.2× Gemini 3.1 Pro’s 5.8 and 11.2% above Claude Sonnet 4.6.Kimi averaged 13.4 known, 22.2 novel, and 8.6 unique crashes; research quality and total crashes had a moderate positive correlation (r = 0.662), but DeepSeek V4 Pro’s high quality score did not correspond to high crash discovery.Source: E. How does the choice of LLM affect… · pp. 12–13, Figure 7 · p. 12

F. How does 5UGUE compare to the state-of-the-art 5G NR fuzzing?

Across five 5G targets, 5UGUE outperforms the non-agentic 5GHOUL baseline in crash discovery. The paper also reports scope limits, responsible-disclosure practices, and broader conclusions about hardware-in-the-loop fuzzing.

F. How does 5UGUE compare to the state-of-the-art 5G NR fuzzing?

  • 5UGUE found its first crash faster and with fewer iterations across all five targets; on mediatek, mean TTFC fell from 88 to 4.5 minutes and mean ITFC from 1381 to 3.5UGUE also used less execution time than 5GHOUL’s fixed 150-minute duration across the evaluated targets.Source: F. How does 5UGUE compare to the… · p. 13
  • 21 crashes per target on average versus 1.2 for 5GHOUL, a 17.5× difference; 5UGUE found crashes on every target, while 5GHOUL found none on two.On mediatek, 5UGUE averaged 32.2 crashes and 10 unique crashes, compared with 5GHOUL’s 1.4 crashes and 0.6 unique crashes.Source: F. How does 5UGUE compare to the… · p. 13, §IX · p. 13

VIII. THREATS TO VALIDITY

  • The evaluation focuses on 5G NR, with BT as an extension; detailed multi-protocol evaluations are deferred, and LLM latency, possible code errors, and deployment costs may constrain use.Privacy, connectivity, and resource requirements can also constrain LLM interaction; local open-weight models may require substantial computation.Source: §VIII · p. 13

IX. CONCLUSION; ETHICS CONSIDERATIONS

  • The conclusion presents 5UGUE as a strategist controlling the fuzzing loop and reports 4.47× improved crash discovery from V1 to V3, 21 crashes per 5G device on average, and adaptation to BT.The paper states that experiments were isolated from production systems, findings were disclosed to vendors, and exploit code or sensitive details were withheld to mitigate dual-use risks.Source: §IX · pp. 13–14, ETHICS CONSIDERATIONS · p. 14

APPENDIX A GENERATIVE AI USAGE CONSIDERATIONS

The appendix documents generative-AI use, detailed experiment results, the mutation taxonomy, publicly accessible crash identifiers, and first-crash metrics.

APPENDIX A GENERATIVE AI USAGE CONSIDERATIONS

  • The authors used Claude for editorial assistance and coding agents for implementation support, while manually reviewing model suggestions and verifying manuscript claims.They also report using open-weight models and releasing code to support future independent verification if proprietary models become unavailable.Source: Appendix A · pp. 15–16

APPENDIX B DETAILED RESULTS

  • Detailed results are provided for 5UGUE and 5GHOUL; crashes whose strings could not be recovered were detected but could not be classified or counted in category totals.An API interruption caused the final two turns of one run to fail, and the two MediaTek-confirmed vulnerabilities came from separate identified runs.Source: Appendix B · p. 16

APPENDIX C MUTATION TAXONOMY

  • Each test receives a category from an eight-class mutation taxonomy, which the authors hypothesize diversifies exploration and gives reflection a way to identify under-tested strategies.Source: Appendix C · p. 16

APPENDIX D PUBLICLY ACCESSIBLE CRASH IDENTIFIERS

  • Public crash identifiers were located for the MediaTek Dimensity 900 target, but not for the other evaluated targets.Source: Appendix D · p. 16

APPENDIX E METRICS TO THE FIRST CRASH

  • On the MediaTek Dimensity 900 target, V3’s TTFC and ITFC increased relative to V1 and V2, yet remained below five minutes and four iterations.Figure 8 reports first-crash metrics for different configurations.Source: Appendix E · p. 16, Figure 8 · p. 16
  • Most tested LLMs triggered a first crash within six minutes and four iterations; Qwen3.7 Plus took 9.2 minutes and six iterations, and first-crash metrics did not clearly track overall discovery.GPT-5.5 had the shortest TTFC at 2.9 minutes, while DeepSeek V4 Pro had the fewest iterations at 1.2.Source: Appendix E · p. 16, Figure 9 · p. 16

APPENDIX F HOW COST-EFFICIENT IS 5UGUE IN TERMS OF LLM

5uGUE’s LLM cost varies by configuration and model, and the most expensive model is not the strongest crash finder. Costs are calculated from average input and output token use and model-specific prices.

Cost calculation and comparison settings

  • Cost is computed from average input and output token counts across five runs, multiplied by their respective model-specific prices per million tokens.The comparisons use Claude Sonnet 4.6 for V1–V3 and V3 for the different-LLM comparison on the MediaTek Dimensity 900 target.Source: Appendix F · p. 18

Cost across versions

  • With Claude Sonnet 4.6 on the MediaTek Dimensity 900, cost rises from $8.59 in V1 to $9.29 in V2 and $21.98 in V3 as tools are added.The paper reports that V3 outperforms V1 and V2, suggesting its additional cost is justified by improved performance.Source: Appendix F · p. 18

Cost and performance across LLMs

  • Kimi K2.6 achieves the best overall crash discovery performance at $4.39, while GPT-5.5 costs $36.40 and Qwen3.7 Plus costs $2.15.Across the evaluated LLMs, higher cost does not necessarily translate to better performance.Source: Appendix F · pp. 18–19

APPENDIX G EXAMPLE EXPLOIT CODE

The appendix presents an example 5G exploit written in C++.

Example exploit code

APPENDIX H V3-FILTERED PROMPT

The appendix presents the refined prompt used for 5uGUE V3-filtered.

V3-filtered prompt

  • The refined 5uGUE V3-filtered prompt is split across two pages because of its length.Source: Appendix H · p. 19

APPENDIX I BT PROMPT

The appendix shows selected modifications to 5uGUE’s prompt for Bluetooth Classic fuzzing.

Bluetooth prompt

  • The Bluetooth prompt identifies the agent as a Bluetooth security researcher and sets bug discovery in the Bluetooth stack as its objective.Its briefing includes Bluetooth terms such as host, peripheral, and L2CAP, and provides Bluetooth-specific documentation to support concept understanding and exploit generation.Source: Appendix I · p. 19

APPENDIX J COMPARISON OF TTFC, ITFC AND SESSION TIME

Across five targets, 5UGUE V3 finds first crashes faster and in fewer iterations than 5GHOUL. The comparison defines how first-crash timing and iterations are counted and reports session runtimes.

APPENDIX J COMPARISON OF TTFC, ITFC AND SESSION TIME

  • TTFC measures minutes until the first crash, while ITFC counts iterations; each 5UGUE test run is treated as one iteration for comparison with 5GHOUL.A 5GHOUL iteration is a full communication period during which different packets may be fuzzed.Source: Appendix J · p. 19
  • When 5GHOUL finds no crash, its TTFC is set to 151 minutes to represent reaching the 150-minute timeout, and its ITFC is set to the maximum across the five devices.Source: Appendix J · p. 19
  • Across all five targets, 5UGUE V3 finds first crashes faster and in fewer iterations than 5GHOUL; on MediaTek Dimensity 900, mean TTFC is 88 minutes for 5GHOUL versus 4.5 for 5UGUE.On that target, mean ITFC is 1381 for 5GHOUL and 3 for 5UGUE.Source: Appendix J · p. 19, Figure 13 · p. 23
  • 5UGUE V3 requires less time than 5GHOUL across all five targets, while 5GHOUL runs for a fixed 150 minutes.The 5UGUE runtime results are shown across five runs.Source: Appendix J · p. 19, Figure 14 · p. 23

Related papers

Loading 2610.29155v1…