Source-linked AI summary
RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation
ZhuoXin Liu, Zhiming Ma, Ying Zhang, Mengzheng Yang, Yifan Wang, Zhengqi Huang, Yanhan Zhou, Zekun Lin, Jun Zhang, Shun Zhang, Yue Chen, Qiao Zhao, Peng Chen
TL;DR
Existing benchmarks evaluate obfuscated text and risky webpages separately, leaving the effect of target recovery on downstream investigation unresolved. RiskChainBench links restoration to evidence-grounded investigation, and its results show that leaders differ across recovery, reconstruction, website decisions, and violation typing.
Problem
Separate evaluations of obfuscated text and risky webpages do not measure how target recovery affects downstream evidence acquisition.
Source: §1 · p. 2Method
Results
Leaders differ across actionable-entry recovery, full reconstruction, website decisions, and fine-grained typing, while Entry Top-1 ranges from 35.19% to 95.22%.
Source: §4.2 · p. 8, §5 · p. 9Takeaways & Limitations
The offline entry gate exposes upstream loss without rerunning website investigation, while frequent execution failures show reliable exploration remains prerequisite to evidence-grounded judgment.
Source: §5 · p. 9Evaluation uses one trajectory per model–website pair, sparsely populates several violation types, and conditions evidence diagnostics on successful judge coverage.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Platform abuse campaigns conceal redirection instructions with emojis, homophones, character decomposition, and redundant symbols, then route users through disguised links to services associated with pornography, fraud, gambling, or illicit transactions. Existing benchmarks evaluate obfuscated text and risky webpages separately, obscuring how target recovery affects downstream evidence acquisition. We introduce RiskChainBench, pairing 3,600 synthetic token-text restoration inputs from 600 source sessions with 600 corresponding human-labeled local web environments. A model first restores the message, operational intent, and destination; the same underlying model then acts as a VLM-driven web agent that investigates the correctly associated website and produces a frozen, evidence-cited risk report without message-side semantics or domain-reputation cues. We score restoration and correct-routing web investigation separately and compose them offline by applying the frozen primary-entry prediction as a gate to the same Task 2 result. Human labels determine task correctness, while a fixed multimodal evidence judge assesses faithfulness, sufficiency, completeness, and consistency. Across ten models, Entry Top-1 ranges from 35.2% to 95.2% and web decision accuracy from 26.3% to 62.8%; the leading systems differ across entry recovery, full reconstruction, website decisions, and fine-grained typing. Execution failures account for 31.9% of web runs, whereas post-decision type errors account for only 0.9%, identifying stable exploration and risk judgment as the principal bottlenecks. We release the benchmark, protocol, and resettable local sandbox.
1 Introduction
The introduction frames cross-channel platform risk investigation as linking obfuscated-message restoration to destination-specific, evidence-grounded web assessment. RiskChainBench pairs synthetic message inputs with local web environments and separates task correctness from evidence diagnostics.
1 Introduction
- Separate text and web evaluations cannot expose how restoration errors change the destination investigated downstream.Source: §1 · p. 2
- Cross-channel platform risk investigation spans message restoration, target identification, web exploration, and evidence-grounded risk judgment.Source: §1 · pp. 1–2
- RiskChainBench pairs 3,600 synthetic token-text inputs with 600 human-labeled local web environments for safe, reproducible investigation.The benchmark links restoration and investigation by destination identity, and the web agent receives neither the source message nor its restoration.Source: §1 · p. 2
- The same ten underlying models are evaluated in text-only restoration and VLM-driven web-agent settings, with failures localized across recovery, execution, judgment, and typing.Human annotations establish website decisions and types, while a fixed multimodal evidence judge provides separate trajectory-evidence diagnostics.Source: §1 · p. 2
2 Related Work
Prior benchmarks study obfuscated-content restoration or classification and reproducible, risk-aware web interaction, but do not measure whether a recovered destination supports subsequent investigation. RiskChainBench links destination recovery to active investigation while separating human-labeled task correctness from evidence diagnostics.
2 Related Work
- Obfuscated-content work evaluates reconstruction, moderation robustness, restoration, and detection across character perturbations, emojis, Unicode homoglyphs, phonetic, visual, and semantic substitutions, and coded-language understanding in Chinese online reviews.Examples include TNT, Hatemoji (Kirk et al., 2022), OTH (Cooper, Surdeanu, and Blanco, 2023), ToxiCloakCN (Xiao et al., 2024), PCR-ToxiCN (Guo et al., 2025), HomoP-CN, and CodedLang (Wan, Li, and Huang, 2026). Jiang et al. (2026) introduce AD-VJARGON, an annotated dataset linking adversarial jargon variants to canonical forms, and JADE, a corresponding detection framework.Source: §2 · p. 2
- Web-agent benchmarks cover reproducible interaction, disguised malicious links, adversarial environments, and safe processing of web evidence.These include WebArena (Zhou et al., 2024), MalURLBench (Kong et al., 2026), SecureWebArena (Ying et al., 2026), and FraudSM-SWalker (Zhou et al., 2026).Source: §2 · pp. 2–3
- These lines of work leave unmeasured whether a recovered destination supports subsequent investigation; RiskChainBench links recovery to active investigation while isolating message cues and domain reputation.Prior trajectory-evaluation work also reports judge inconsistency, degradation on longer trajectories, and weaknesses in evidence verification: AgentRewardBench (Lù et al., 2025), Plan-RewardBench (Wang et al., 2026a), and REFLECT (Wang et al., 2026b).Source: §2 · pp. 2–3
3 Method
RISKCHAINBENCH treats the website as the primary unit, pairing message variants with local web environments and evaluating restoration, investigation, and trajectory evidence separately. A frozen primary-entry prediction gates the same web result offline, while the web agent investigates without message-side semantics or domain-reputation cues.
3 Method
- 600 website clusters contain 3,600 message variants, while website annotations apply to environments rather than message rhetoric.Each website has six variants; one fixed primary variant supplies the sole entry gate, and the remaining five are used only for restoration evaluation.Source: §3 · p. 3
3.1 Benchmark Construction
- Synthetic message variants combine obfuscations while preserving intended meaning and destination, and local environments reproduce relevant webpage interactions offline.The environments define observable states, allowlisted actions, transitions, and a bounded horizon, without encoding a risk label or unique valid evidence path.Source: §3.1 · pp. 3–4
3.2 Obfuscated Message Restoration
- Restoration requires a canonical message, operational intent, and ranked entry candidates, with the top-ranked entry controlling the end-to-end gate.Entry Recall@k credits any candidate that resolves to the website; a wrong top entry fails the gate, and lower-ranked candidates cannot repair it.Source: §3.2 · pp. 5–6
3.3 Evidence-Grounded Web Investigation
- The web agent investigates a local site without source-message or restoration context, then freezes its risk conclusion, rationale, and evidence references.Only content observed in the recorded trajectory is admissible as evidence; message-side cues therefore cannot directly determine website classification.Source: §3.3 · p. 6
3.4 Evaluation Protocol
- Web-only scoring evaluates investigation under correct routing, while gated end-to-end scoring reuses that frozen result and also retains entry-recovery failures.The protocol reports the decision-accuracy difference in percentage points as entry-recovery loss; invalid runs count as task failures, and the evidence judge separately scores faithfulness, sufficiency, completeness, and consistency.Source: §3.4 · pp. 6–7
4 Experiments
Experiments evaluate message restoration and correct-routing web investigation, then compose them with a frozen entry gate; results vary across tasks, and execution is the largest pooled failure source.
4.1 Experimental Setup
- Ten models are evaluated on 600 websites and 3,600 messages; web agents receive a 30-action, 600-second budget, and evidence scores are conditioned on judge coverage.Source: §4.1 · pp. 7–8
4.2 Main Results
- Entry Top-1 spans 35.19%–95.22%; GPT-5.4 leads entry recovery, while GPT-5.6 SOL leads full reconstruction and has the lowest CER.Source: §4.2 · p. 8, Table 1 · p. 9
- Task 2 decision accuracy ranges from 26.3% to 62.8%, and hierarchical exact match ranges from 26.2% to 61.0%; GPT-5.6 SOL leads both, while Kimi K2.5 leads type macro-F1.Source: §4.2 · p. 8, Table 1 · p. 9
- Gated accuracy ranges from 16.7% to 60.8%, with losses of 0.7–32.3 percentage points, using the same frozen web results without rerunning the browser.Source: §4.2 · p. 8
4.3 Diagnostic Analysis
- Execution is the largest pooled failure source at 31.9%; only 0.9% fail at typing after a correct decision.Source: §4.3 · p. 8, Figure 5 · p. 8
5 Conclusion
RISKCHAINBENCH links obfuscated-message restoration with evidence-grounded web investigation, and its conclusion highlights differing model rankings and exploration as a bottleneck.
5 Conclusion
- Model rankings differ across actionable-entry recovery, full reconstruction, website decisions, and fine-grained violation typing, while frequent execution failures make reliable exploration a prerequisite to evidence-grounded judgment.The offline entry gate exposes upstream loss without rerunning website investigation.Source: §5 · p. 9
- The results use one trajectory per model–website pair, sparsely populate several violation types, and condition evidence diagnostics on successful judge coverage.These constraints motivate measuring rerun variance, broadening underrepresented risk types and languages, and auditing the evidence rubric with frozen human trajectories.Source: §5 · p. 9
- Future work should measure rerun variance, broaden underrepresented risk types and languages, and validate the evidence rubric through a frozen human trajectory audit.Source: §5 · p. 9
6 Ethical Statement
The ethical statement describes isolated, non-routable environments and sanitized evidence, with local substitutes for sensitive interactions.
6 Ethical Statement
- Synthetic, non-routable entries and isolated environments avoid contact with live services.Source: §6 · p. 9
- Local substitutes replace personal identities, payment flows, account operations, and communication with third parties.Source: §6 · p. 9
- Evidence packages retain only sanitized observations needed for evaluation, and benchmark outputs do not establish legal attribution.Source: §6 · p. 9