PaperlayerView exact revision on arXiv

Source-linked AI summary

Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications

Yang Luo, Tiantian Ji, Zifeng Kang, Yucheng Wang, Yong Liu, Zhiyi Yin, Xinran Liu

arXiv:2610.18612v1cs.CRcs.AI

TL;DR

Existing evaluations leave the reachability of attacks and their downstream security consequences in real-world Jev applications underexplored. JEVPROBE traces attacker-controlled contexts through program-consumed decisions to potential security impacts. Across 228 applications, the study finds widespread decision flips and downstream impacts.

  • Problem

    Existing evaluations leave attack reachability, program-consumed decision changes, and downstream security consequences in real-world Jev applications underexplored.

    Source: §1 · p. 2

  • Method

    JEVPROBE discovers reachable attacker-controlled context surfaces, generates adversarial variants, and traces program-consumed decision changes through downstream effects.

    Source: §1 · p. 2

  • Results

    Across 2,273 test cases, 81.5% exhibit decision flips and 98.6% of those lead to potential downstream security impacts; JEVPROBE improves macro-average Decision Flip Rate by 50.4 percentage points over the strongest adapted baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    JPDH evaluation in real-world applications must connect attacker-controlled contexts to program decisions and downstream behavior.

    Source: §6 · p. 9

    The evaluation covers open-source Jev applications; closed-source applications are excluded because source code and decision logic are inaccessible.

    Source: Limitations · p. 10

Abstract

from arXiv · show

System One models such as Jev change how software makes decisions by mapping open-ended semantic contexts into structured judgments that directly drive program execution. However, this paradigm introduces a new security risk, which we term Jev Program Decision Hijacking (JPDH): attacker-controlled contexts can redirect program decisions through schema-valid Jev judgments. Existing evaluations focus primarily on model-level attacks or predefined injection channels, leaving attack reachability and downstream security consequences in real-world Jev applications underexplored. In this paper, we present JevProbe, the first automated framework to evaluate JPDH by tracing adversarial influence from reachable context surfaces through program-consumed decisions to downstream security consequences in real-world Jev applications. Our key insight is that application logic maps diverse Jev judgments to observable decisions, enabling systematic, program-grounded attack evaluation. Across 228 popular real-world Jev applications and 2,273 test cases, 81.5% exhibit decision flips, of which 98.6% lead to potential downstream security impacts. Compared with the strongest adapted baseline, JevProbe improves macro-average Decision Flip Rate (DFR) by 50.4 percentage points, while existing defenses offer limited protection. We further construct JevProbeBench to support systematic evaluation of JPDH in real-world Jev applications. We have responsibly disclosed our findings to Jev application maintainers, pending their responses. The source code of JevProbe and the JevProbeBench dataset are publicly available on GitHub (https://github.com/xiaoxiaohg8d/JevProbe-).

1 Introduction

Jev Program Decision Hijacking (JPDH) occurs when attacker-controlled contexts induce schema-valid judgments that redirect program decisions. The introduction presents JEVPROBE to trace these attacks in real-world applications and reports widespread decision flips and potential downstream impacts.

1 Introduction

  • JPDH occurs when attacker-controlled contexts induce schema-valid Jev judgments that redirect program-consumed decisions without changing application code or predefined decision logic.The threat arises because structural validity does not ensure semantic correctness.Source: §1 · p. 2
  • Existing evaluations leave attack reachability, program-consumed decision changes, and downstream security consequences in real-world Jev applications insufficiently explored.Traditional software testing and LLM-agent evaluations focus on different testing objectives or predefined environments.Source: §1 · p. 2
  • Systematic evaluation must validate application-specific context reachability, interpret heterogeneous Jev judgments, and assess their propagation through downstream logic.Manipulated decisions can also interact with other Jev judgments.Source: §1 · p. 2
  • JEVPROBE discovers reachable attacker-controlled context surfaces, generates application-aware adversarial variants, and uses paired executions to trace decision changes and downstream effects.Its key insight is that application logic maps diverse judgments to observable decisions.Source: §1 · p. 2
  • 81.5% of 2,273 program-grounded test cases exhibit decision flips, 98.6% of which lead to potential downstream security impacts; JEVPROBE improves macro-average Decision Flip Rate by 50.4 percentage points over the strongest adapted baseline.The study covers 228 real-world Jev applications; existing defenses offer limited protection.Source: §1 · p. 2
  • JEVPROBEBENCH comprises 2,273 program-grounded cases with application-specific decision and impact annotations.The paper also releases the framework and benchmark dataset publicly.Source: §1 · pp. 2–3

2 Background and Related Work

The background describes Jev judgments as structured outputs that directly govern program decisions, while distinguishing this interface from conventional software testing and agent-security evaluations.

2.1 Jev as a Semantic Decision Interface

  • Jev maps open-ended contexts to schema-constrained judgments, but schema compliance ensures structural validity rather than semantic correctness.These judgments can govern program branches and threshold-based decisions.Source: §2.1 · p. 3

2.2 Software Testing and Program Analysis

  • Software testing and fuzzing (Xia et al., 2024; She et al., 2024), static and dynamic analyses (Wang et al., 2025; Zhang et al., 2025b; Shi et al., 2024) explore inputs, paths, data flow, or reachability, but not application-specific semantic criteria for model-generated judgments.Source: §2.2 · p. 3

2.3 LLM Agent Security Evaluation

  • Agent-security evaluations such as InjecAgent (Zhan et al., 2024), AgentDojo (Debenedetti et al., 2024), Agent Security Bench (Zhang et al., 2025a), and PIArena (Geng et al., 2026) assess model- or agent-level outcomes rather than application-consumed judgment integrity.This work instead evaluates altered program-consumed decisions and downstream security consequences.Source: §2.3 · p. 3

3 Problem Formulation and Threat Model

This section defines JPDH as manipulation of reachable external context that redirects program-consumed decisions through schema-valid judgments, and sets out the attacker model and possible downstream consequences.

3.1 Jev Program Decision Hijacking

  • JPDH manipulates reachable, lower-trust context to induce schema-valid judgments that redirect program-consumed decisions without changing code or predefined decision logic.Source: §3.1 · p. 3
  • A decision flip changes the application-consumed decision; it is confirmed hijacking only when the baseline matches an independently established expected decision and the manipulated decision deviates.Source: §3.1 · pp. 3–4
  • In TradingAgents, attacker-controlled web content lowers confidence from 0.70 to 0.16 while preserving sentiment, causing a risk-reducing signal to be discarded despite schema-valid outputs.Source: Figure 2 · p. 4
  • A hijacked decision is security-impacting when it causes observable behavioral divergence that violates a predefined application-specific security or correctness rule.Source: §3.1 · p. 4
  • Jev applications can use single, sequential, joint, or feedback-dependent judgments, which may jointly affect decisions or propagate changes into later contexts and program states.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Threat Model

  • The attacker may modify external context fields that reach Jev judgments, but cannot change application code, Jev implementations, model parameters, internal prompts, or downstream decision logic.Source: §3.2 · p. 4
  • The assumed attacker knows the application's public functionality and controlled external content, but need not know its source code, internal prompts, or Jev decision logic.Source: §3.2 · p. 4
  • The attacker's objective is to induce unintended program-consumed decisions through manipulated external contexts, potentially causing downstream security violations.Source: §3.2 · p. 4

4 JevProbe

JevProbe evaluates program decision hijacking by tracing attacker-controlled contexts through application decisions and downstream execution. Its stages validate reachable surfaces, generate adversarial contexts, detect decision changes, and assess consequences.

4.1 Overview

  • JevProbe is an automated framework with four stages that grounds JPDH evaluation in program-consumed decisions and downstream execution semantics.Source: §4.1 · p. 5

4.2 Reachable Surface Discovery

  • JevProbe traces data dependencies backward from Jev calls, then validates candidate context surfaces by checking whether injected benign markers reach the targeted judgment context at runtime.Source: §4.2 · p. 5

4.3 Adversarial Context Generation

  • For each validated surface, JevProbe generates scenario-aware adversarial variants using complementary manipulation strategies while preserving constraints and task-relevant facts and refining variants with execution feedback.Source: §4.3 · pp. 5–6

4.4 Decision Hijacking Detection

  • JevProbe compares baseline and adversarial executions under the same scenario, state, and configuration, interpreting judgments through application-specific semantics and checking flips against an independently established expected decision.Source: §4.4 · p. 6

4.5 Security Impact Assessment

  • JevProbe assesses security consequences by tracing hijacked decisions through downstream execution and comparing traces for behavioral divergence and violations of application-specific security or correctness rules.Source: §4.5 · p. 6

5 Evaluation

JevProbe evaluates reachable-context attacks across screened real-world Jev applications, measuring program-consumed decision flips and downstream impacts. Across the evaluation, flips were widespread, frequently potentially unsafe, and JevProbe outperformed adapted baselines while tested defenses offered limited protection.

5.1 Experimental Setup

  • The study screened 356 of 1,711 collected repositories and identified 228 applications with reachable attack surfaces; RQ3–RQ5 also used 183 cases from 10 applications.The main evaluation used jev-1.13.0; adversarial variants were generated with Qwen3-32B, and Gemma-3-27B-IT assisted impact assessment.Source: §5.1 · p. 6

5.2 RQ1: Program Decision Hijacking

5.3 RQ2: Downstream Security Consequences

  • 98.6% of decision flips were potentially unsafe; Information Contamination accounted for 92.5% of unsafe-labeled cases across 213 applications.Other consequences involved targeted validation, data integrity, access control, and financial decision-making.Source: §5.3 · pp. 7–8, Table 3 · p. 7

5.4 RQ3: Effectiveness of JevProbe

  • 50.4 percentage points higher macro-average DFR than strongest baseline ASB (JevProbe 79.8%).JevProbe combines application-specific surface discovery with program-consumed decision analysis to evaluate heterogeneous Jev applications.Source: Table 4 · p. 8, §5.4 · p. 8

5.6 RQ5: Effectiveness of Existing Defenses

  • Spotlighting-Delimit reduced DFR from 78.7% to 68.9%, while paraphrasing increased it to 79.8%; evaluated defenses remained insufficient to preserve decision integrity.Other defenses produced smaller improvements.Source: Table 5 · p. 8, §5.6 · pp. 8–9

5.5 RQ4: Cross-Model Decision Robustness

  • Jev had 79.8% macro-average DFR versus 50.7%–66.8% for evaluated general-purpose LLMs, and led in nine of ten applications.The reported results indicate greater susceptibility under the evaluated Jev-targeted attacks, with susceptibility varying across applications and models.Source: §5.5 · p. 8

6 Conclusion

The paper identifies JPDH as a security threat and presents JEVPROBE to investigate it in real-world Jev applications. Experiments find widespread risks and limited effectiveness of existing defenses, motivating security for the context-to-decision boundary.

6 Conclusion

  • JEVPROBE is presented as the first automated framework for uncovering JPDH in real-world Jev applications.The paper identifies JPDH as a security threat arising from integrating System One models such as Jev into applications.Source: §6 · p. 9
  • Experiments reveal widespread JPDH risks and limited effectiveness of existing defenses, highlighting the need to secure the context-to-decision boundary.Source: §6 · p. 9

Limitations

The evaluation is limited to open-source Jev applications, and its defense evaluation excludes model-internal mechanisms because Jev’s architecture and parameters are not publicly accessible.

Limitations

  • The evaluation covers open-source Jev applications collected from GitHub; closed-source applications are excluded because source code and decision logic are difficult to access.Closed-source applications are left for future work.Source: Limitations · p. 10
  • The defense evaluation excludes model-internal mechanisms because Jev’s architecture and parameters are not publicly accessible.Investigating these defenses remains future work.Source: Limitations · p. 10

Ethical Considerations

Evaluations used isolated local test environments and did not target live production systems. JEVPROBE manipulates attacker-controlled context fields and categorizes potential impacts using application-specific decision semantics and execution evidence.

Ethical Considerations

  • All security evaluations used isolated local test environments and did not access real user data or execute real-world financial transactions.Adversarial manipulations were confined to controlled environments to avoid disrupting real-world services.Source: Ethical Considerations · p. 10

A Context Manipulation Strategies

  • JEVPROBE generates application-aware context variants by selecting strategies based on the input channel, controllable field, and application scenario.It modifies only attacker-controllable portions of the original context while preserving the surrounding scenario and input format; Table 6 summarizes its strategy taxonomy.Source: A Context Manipulation Strategies · p. 12

B Application-Level Impact Rules

  • For each unsafe-labeled case, JEVPROBE assigns one primary consequence category based on the affected decision contract and application-specific evidence.The analysis examines judgment instructions, decision criteria, program-consumed decisions, and observed behavioral changes; Table 7 summarizes six categories.Source: B Application-Level Impact Rules · p. 12

C Repository Collection Methodology

The repository collection combines community-curated lists, GitHub Repository Search API results, and repositories from prior source audits. After deduplication, it yields 1,711 unique candidate repositories.

C Repository Collection Methodology

  • Table 6’s strategy taxonomy covers instruction-based manipulation, authority and verification cues, structured representations, scenario and persona framing, semantic manipulation, and context transformation.Source: Table 6 · p. 13
  • The collection combines three community-curated repository lists, GitHub Repository Search API results, and repositories from previous commit-pinned source audits.The lists were snapshotted between September 18 and 30, 2026; the search queries returned 237 entries, and prior audits contributed 18 repositories.Source: C Repository Collection Methodology · p. 13
  • Repositories are deduplicated by owner/name identifiers, with source provenance retained.Source: C Repository Collection Methodology · p. 13
  • The three repository sources yielded 2,597 raw entries and 1,711 unique candidate repositories after deduplication.Of the unique candidates, 1,658 have recorded GitHub star counts.Source: C Repository Collection Methodology · p. 13

D.1 Benchmark Overview

JevProbeBench preserves the application context and decision path needed to evaluate attacks in real-world Jev applications. The framework comparison reports that JevProbe has the highest overall DFR across the evaluated applications, though some baselines match or outperform it on individual applications.

D.1 Benchmark Overview

  • JevProbeBench contains 2,273 cases from 228 open-source applications and preserves the chain from attacker-controlled context through consumed decisions to downstream behavior and impact.Its cases span multiple application scenarios and are tied to fixed source-code revisions.Source: §D.1 · p. 13

D.2 Benchmark Construction

  • Cases are generated by validating an attacker-controlled surface, replaying benign and manipulated contexts, and retaining executions, consumed decisions, and impact assessments.Table 8 summarizes the benchmark; its impact taxonomy classifies consequences using case-specific decision semantics and behavioral evidence.Source: §D.2 · pp. 13–14, Table 7 · p. 14, Table 8 · p. 14

D.3 Case Representation

  • Each case represents one application-specific context-to-decision evaluation, retaining the information needed to reproduce comparisons and interpret downstream consequences.The representation connects attacker-controlled context, structured judgment, program-consumed decision, and application impact.Source: §D.3 · p. 14, Table 9 · p. 15

D.4 Decision and Impact Labels

  • A decision flip requires a change in the program-consumed decision, including numerical values crossing application-defined thresholds; unsafe impact additionally requires downstream behavior violating an application-specific rule.Cases without confirmed flips cannot be labeled unsafe, and insufficient impact evidence remains distinguishable from safe cases.Source: §D.4 · p. 14

D.5 Benchmark Coverage

  • The benchmark spans heterogeneous context channels, judgment types, decision-consumption patterns, manipulation strategies, and consequences, including threshold-based scores and confidence values.Covered consequences include access control, financial decisions, content integrity, availability, policy enforcement, and difficult-to-reverse actions.Source: §D.5 · pp. 14–15, §D.6 · p. 15

D.6 Intended Use and Limitations

  • Unsafe outcomes are potential consequences established through controlled execution and application-specific rules, while recorded behavior reflects specific application revisions and model configurations.Later changes to application logic, Jev behavior, model configurations, or downstream policies may change decisions and impact labels.Source: §D.6 · p. 15

E Per-Application Framework Comparison

  • JevProbe has the highest overall DFR and produces decision flips in every evaluated application, despite some baselines matching or outperforming it on individual applications.The comparison uses 183 test cases across 10 applications, with matched attack budgets and payloads limited to 96 characters.Source: E Per-Application Framework Comparison · p. 15, Table 10 · p. 16

F.1 Model Adaptation Protocol; F.2 Per-Application Results

  • General-purpose LLMs are evaluated through an adapter that preserves application decision logic, using paired benign and adversarial contexts at the same decision boundaries.The cross-model results cover open- and closed-source models using Jev-targeted contexts and application-specific criteria; Qwen3 models are excluded because Qwen3-32B generates adversarial contexts.Source: §F.1 · pp. 15–16, §F.2 · p. 16

G Case Studies of Jev Judgment Dependencies

The case studies show how adversarial context can affect single, sequential, joint, and feedback-dependent Jev judgments. Their traces connect changed judgments to program-level decisions, while the feedback example does not establish a later-round decision flip.

Case Study Overview

  • The cases cover single, sequential, joint, and feedback-dependent judgment structures, with the table summarizing observed changes and their program-level effects.The cases illustrate different ways judgments can affect decisions or subsequent judgment contexts.Source: G Case Studies of Jev Judgment… · p. 16, Table 12 · p. 17

Single Judgment

  • A single manipulated confidence field drops from 0.65 to 0.27 while classification stays positive, switching the program decision from acceptance to rejection at the 0.5 gate.The example shows a program decision reversing without a change in the categorical output.Source: G Case Studies of Jev Judgment… · p. 16

Sequential Judgments

  • In sequential processing, KeepCall rises 0.26 to 0.73 and KeepResult 0.19 to 0.54, crossing 0.5 thresholds and changing the exchange from dropped to kept.The observed trace establishes the initial decision change in the sequential pipeline.Source: G Case Studies of Jev Judgment… · p. 16

Joint Judgments

  • For joint judgments, classification changes from positive to negative as confidence falls from 0.75 to 0.11, altering both decision fields consumed by the application.A single adversarial context influences multiple components of the program decision.Source: G Case Studies of Jev Judgment… · pp. 16–17

Feedback Dependencies

  • In a feedback-related case, relevance rises from 0.19 to 0.21 past the 0.2 threshold, changing context blanking from true to false and retaining content.Retained context could enter later requests, but the recorded result does not establish a later-round decision flip.Source: G Case Studies of Jev Judgment… · pp. 16–17

Related papers

Loading 2610.18612v1…