PaperlayerView exact revision on arXiv

Source-linked AI summary

Business Compromise Detection with Agentic AI and LLM-driven Knowledge Discovery

Diego Palma, Kyu Bin Kim, Zhen Han, Allbright Dsouza, Zhiyuan Liu

arXiv:2609.32643v1cs.AI

TL;DR

Compromised business ad accounts let attackers exploit established reputations to launch fraudulent campaigns, while LLM agents can make unreliable decisions on ambiguous evidence. The paper keeps the agent as an evidence extractor and delegates the verdict to a neuro-symbolic stage; this substitution raises MCC from 0.295 to 0.435 on realistic low-prevalence data (p = 0.018).

  • Problem

    Compromised ad accounts enable fraudulent campaigns using established business reputations, while ambiguous evidence makes reliable detection difficult.

    Source: §1 · p. 1

  • Method

    The system converts an LLM agent’s reported evidence into a structured signal vector and delegates the verdict to FOIL-IE rules, Naïve Bayes calibration, and contradiction resolution.

    Source: §1 · p. 1

  • Results

    MCC rises from 0.295 to 0.435 on realistic low-prevalence data (p = 0.018) when the neuro-symbolic stage replaces the agent as decision-maker.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The agent’s evidence can be retained while its final decision is replaced by a neuro-symbolic arbiter, improving precision at a recall cost.

    Source: §5 · p. 7

    The backbone-robustness claim remains unverified because the study lacks a controlled two-backbone comparison.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Detecting compromised business ad accounts is a challenge in digital advertising, as attackers exploit hijacked accounts to launch fraudulent campaigns. Large Language Model (LLM) agents show promise for integrity enforcement, but hallucinated mistakes on hard cases create business friction. In a study we find the autonomous agent is a strong, recall-heavy signal extractor but an unreliable final arbiter, conceding precision on ambiguous decisions. We therefore keep the agent as an investigator that emits a structured, interpretable signal vector, and delegate the verdict to a neuro-symbolic stage: symbolic rules discovered by Inductive Logic Programming (FOIL-IE), a Na\"ive Bayes calibration layer, and a data-tuned contradiction layer. Evaluating on a compromise-over-sampled population and a realistic low-prevalence sample with subject-matter-expert labels, this arbiter substitution raises MCC from 0.295 to 0.435 ({\Delta}MCC +0.139, 95% CI [+0.026, +0.245], p=0.018, paired bootstrap), lifting precision from 0.250 to 0.446 (1.8x) at a recall cost (0.920 to 0.660). Benchmarked under identical conditions, it also edge tree ensembles (0.386).The rules encode domain w labels while remaininginterpretable and auditable.

1 Introduction

Compromised ad accounts let attackers exploit trusted business histories to launch fraudulent campaigns, creating financial and brand harm. Existing detection approaches trade interpretability against generalization, while LLM agents can still make inconsistent decisions on ambiguous evidence.

1 Introduction

  • Compromised accounts let attackers bypass initial checks and use established reputations to run policy-violating ads, causing financial loss and brand damage.Detection is difficult as attackers evolve tactics, including synthetic profiles, IP obfuscation, and sock-puppet agency structures.Source: §1 · p. 1
  • Rule-based systems are interpretable but brittle and require ongoing SME maintenance, while statistical models generalize well but lack enforcement-ready interpretability.Source: §1 · p. 1
  • LLM agents can hallucinate and struggle to weigh subtle, contradictory business evidence consistently, leaving a precision gap on ambiguous production cases.Self-reflection reduces but does not eliminate this risk; wrong decisions can disrupt customer operations.Source: §1 · p. 1
  • The proposed architecture separates evidence extraction from the verdict: an agent emits a structured signal vector, then FOIL-IE rules, Naïve Bayes calibration, and contradiction resolution decide.The contradiction layer is designed to rescue false negatives and veto false positives.Source: §1 · p. 1

1. An empirical finding: an autonomous LLM

The paper positions the LLM as a recall-heavy evidence extractor rather than the final arbiter. Its neuro-symbolic decision stage has the strongest point estimates against evaluated baselines on realistic low-prevalence data, with significance only for the comparison against the agent.

1. An empirical finding: an autonomous LLM

  • The agent is an effective recall-heavy signal extractor but a poor final arbiter on ambiguous cases, motivating separation of extraction from decision.Source: §1 · p. 2
  • The evaluation covers an over-sampled general population and a realistic expert-labeled hard-case sample, with paired-bootstrap intervals, error analysis, and threshold sensitivity study.The analyses assess which design choices, including contradiction resolution, are supported by the data.Source: §1 · p. 2

2 Background

Business compromise is difficult to distinguish from abuse because both produce policy-violating ads, even though account control and enforcement consequences differ. The background motivates interpretable relational rules combined with probabilistic scoring for this setting.

2 Background

  • Compromise and abuse both produce fraudulent ads, but compromise involves third-party takeover and calls for account recovery and owner notification rather than owner-directed penalties.Reviewers often disagree when contradictory evidence makes legitimate agencies hard to distinguish from compromised accounts.Source: §2 · p. 2
  • Graph-based methods can use connection ages and shared business entities, but rarely expose those signals interpretably.Source: §2 · p. 2
  • The paper combines FOIL-IE rule generation with Naïve Bayes probabilistic scoring to pair human-readable rules with probabilistic robustness.Related work notes explainability can come at a scalability cost relative to decision trees.Source: §2 · p. 2

3 Methodology

The methodology separates agent evidence gathering from the final decision, using a structured expert-guided signal vector as input to a three-layer neuro-symbolic stage. The stages cover terminology and agent operation, interpretable rule learning, probabilistic calibration, and contradiction handling.

3.1 Terminology

  • Compromise means third-party takeover of a legitimate account, distinct from owner-operated abuse; both can yield fraudulent ads but trigger different enforcement paths.Confusing abusive accounts for compromised ones is the most frequent error across evaluated methods.Source: §3.1 · p. 2

3.2 LLM Agent for Business Compromise

  • The multi-turn agent gathers objective evidence through an SME-elicited checklist and tools, then emits categorical judgments, boolean signals, and bounded numerics as a structured vector.The vector makes inconsistencies visible and gives the downstream stage a stable feature source; hard-case errors include abusive, agency, benign-shift, and hallucination cases.Source: §3.2 · pp. 3–4

3.3 Layer 1: Symbolic Learning with FOIL-IE

  • FOIL-IE combines FOIL rule learning with Inverse Entailment to overcome FOIL’s connectivity limitation and produces compromise-confirmation and legitimate-account rejection rules.The learned rule set uses background knowledge and positive and negative examples.Source: §3.3 · p. 4

3.4 Layer 2: Probability Modeling

  • Naïve Bayes models class-conditional probabilities for rule firings, raw signals, rule combinations, and contradiction flags to handle borderline cases under low compromise prevalence.Feeding L1 rule-fires back as features lets L2 calibrate the discrete rules probabilistically.Source: §3.4 · p. 4

3.5 Layer 3: Contradiction Resolution

  • The five contradiction categories are expert priors, while their inputs are computed from discovered rules, deterministic signals, and the L2 posterior; swept thresholds govern vetoes and rescues.A low-posterior false-positive-leaning flag can veto a positive, while false-negative- or compromise-leaning flags can rescue a negative just below threshold; thresholds are selected on training folds.Source: §3.5 · p. 4
  • At the current label volume, L3’s resolution policy is static; its observed gain comes from contradiction flags as L2 features and the posterior floor, not veto/rescue actions.L3 is the most data-hungry component.Source: §3.5 · p. 4

4 Experiments and Results

Across class-rich and realistic low-prevalence evaluations, the paper tests a neuro-symbolic decision stage that rearbitrates the LLM agent’s structured evidence. Its clearest result is a significant MCC improvement over the agent on hard cases, while the stage’s contradiction-resolution actions add no measurable benefit and benign shifts remain difficult.

4.1 Experimental Setup

  • The evaluation uses a 975-case, roughly 70%-positive over-sampled population for cross-validation and a 397-account, 15.1%-positive SME-adjudicated hard-case population for operationally faithful comparison.The over-sampled set is a controlled higher-power comparison, not a production-prevalence estimate; scarce SME labels constrain dataset size, and human reviewers agree with SME adjudication 75–85% of the time.Source: §4 · p. 4, §4.1 · p. 5

4.2 Over-sampled General Population

  • On the over-sampled population, the random forest has the best MCC (0.823), slightly above the full stage (0.807), while gradient boosting is at 0.800.The gaps are within cross-validation and tuning noise; the single decision tree scores 0.733, so the ensemble’s accuracy comes without the same auditable-rule form.Source: §4.2 · p. 5

4.3 Low Prevalence Population

  • The full stage’s hard-case MCC of 0.435 is not significantly higher than the human reviewer’s 0.422 (ΔMCC +0.012, p = 0.881), so the paper does not claim superiority to humans.On realistic data, random forest scores 0.386 and gradient boosting 0.338; their ranking reverses from the class-rich population, but the proposed sample-efficiency explanation is unproven without a learning curve.Source: §4.3 · pp. 5–6
  • MCC rises from 0.295 to 0.435 when the knowledge stage replaces the agent’s verdict on 397 hard cases, with precision increasing from 0.250 to 0.446 and recall falling from 0.920 to 0.660.The paired-bootstrap difference is ΔMCC +0.139 (95% CI [+0.026, +0.245], p = 0.018); the stage removes 97 of 138 agent false positives while giving up 13 true positives.Source: §4.3 · p. 5, Table 2 · p. 6

4.4 What L3 Actually Contributes

  • In the L3 decomposition, IE-enhanced Naïve Bayes features add +0.025 MCC and the posterior floor adds +0.019, while veto and rescue actions leave all four metrics unchanged.The tuner selects rescue-off in all five folds; threshold sweeps yield MCC 0.435 for veto ≤0.4 and 0.420 for veto ≥0.5, with rescue thresholds making no difference. The authors call the policy static and not yet justified by the available labels.Source: §4.4 · p. 6

4.5 Per-Method Error Analysis

  • Benign shifts account for 109 of the agent’s 138 false positives and 32 of the full stage’s 41; the stage removes about two thirds of these cases but does not solve them.The knowledge stage reduces abusive-versus-compromised confusion from 10 cases to 4; gradient boosting has 20 false positives but misses 31 of 50 compromises, versus 17 misses for the full stage.Source: §4.5 · p. 7, Table 4 · p. 7

4.6 Operational Characteristics

  • The decision stage takes 21 ms on CPU (p99 25 ms), while reviews average 175 seconds over 330 cases, with sequential tool round-trips dominating agent-side time.Tool use has median 1 call, mean 3.6, and p99 16; routine execution is automatic, with human review retained only for a subset of managed accounts.Source: §4.6 · p. 7

5 Conclusion

The conclusion centers on replacing the LLM agent’s verdict while retaining its evidence, and reports a significant MCC gain on realistic low-prevalence data. It also emphasizes that the contradiction policy is not yet validated and the rules remain auditable.

5 Conclusion

  • On realistic low-prevalence data, replacing only the agent’s decision-maker raises MCC from 0.295 to 0.435 (p = 0.018), with precision 1.8× higher at a recall cost.The stage’s margins over human reviewers and tree ensembles are not significant, and ensembles lead on class-rich data.Source: §5 · p. 7
  • The contradiction categories contribute as probabilistic features, but the veto/rescue policy is static and the available label volume does not yet justify it.The authors retain it as architecture they expect to activate as SME labels accrue, rather than as a validated result.Source: §5 · p. 7
  • The discovered rules align with SME reasoning and remain auditable.The authors propose testing the approach on other integrity problems and distilling the agent into a smaller model for throughput.Source: §5 · p. 7

Limitations

The paper bounds its claims by small, costly single-platform data, unresolved comparisons, and untested generalization. It also details the hybrid FOIL-IE procedure for learning rules.

Limitations

  • The evidence comes from a small, costly, SME-adjudicated hard-case sample on one platform and one integrity problem, so absolute results have wide confidence intervals and broader generalization is unmeasured.Against humans and ensembles, only the margin over the agent is statistically significant; the sample-efficiency explanation lacks a learning curve, and L3’s veto/rescue policy contributes nothing measurable at 50 positives.Source: Limitations · pp. 7–8
  • The agent comparison does not test self-consistency, reflection, verifier-based reasoning, or majority voting, and the claimed advantage over better prompting remains untested.Raw tool outputs were not retained, preventing controlled frozen-evidence comparisons; the prompt is also not reproduced verbatim, although the decision stage and evaluation protocol can be reproduced with a matching signal vector.Source: Limitations · p. 8

A FOIL-IE Algorithm

  • The hybrid FOIL-IE procedure selects a remaining example, builds its bottom clause, reduces it to a high-gain rule, then adds the rule and removes examples it explains.The algorithm’s stated output is the set of learned rules H.Source: A FOIL-IE Algorithm · p. 10

C Worked Example: One Account End to End

A worked hard case shows how the agent’s structured signals pass through rules, an IE-enhanced posterior, and the final decision. The accompanying configuration section describes the prompt structure and signal-vector output.

C Worked Example: One Account End to End

  • In the de-identified hard case, the human reviewer labeled the account Not Compromise, the SME labeled it Compromise, and the full stage matched the SME.The rules combined a mid-band score with language/target-country shift and absent baseline ad history; the posterior cleared the threshold, while L3 passed the decision through unchanged.Source: C Worked Example: One Account End to End · p. 10
  • The case’s agent signals marked four components suspicious, with five suspicious checks counted overall and high confidence; its risk score was 0.39.The destination domain was new to the business despite being 416 days old, and there was no baseline ad history.Source: C Worked Example: One Account End to End · p. 10

D Agent Configuration, Prompt Sketch, and Output Schema

  • The agent prompt distinguishes compromise, abuse, and legitimate behavioral shifts, and instructs the model to default to Not Compromise absent clear, strong evidence.Its output is a JSON signal vector combining per-component verdicts and sub-signals with tool-derived numeric signals; the prompt is a structural sketch, not a verbatim reproducible prompt.Source: D Agent Configuration, Prompt Sketch,… · p. 11

E Operational Distributions

Table 7 reports measured operational characteristics, including per-review token use and latency. The agent’s JSON response is organized around independent checklist checks, each with a verdict and boolean sub-signals.

E Operational Distributions

  • Token consumption is described as modest in absolute terms, making the multi-turn design affordable.Table 7 reports operational characteristics over 330 samples; agent latency is end-to-end and stage latency covers the neuro-symbolic decision stage.Source: E Operational Distributions · p. 11, Table 7 · p. 12
  • Each checklist component receives its own verdict and boolean sub-signals in the agent’s JSON response.The response sketch uses dimensions for observable ad properties whose shifts reviewers track.Source: Figure 2 · p. 12
  • Tool calls are the logged proxy for reasoning steps and are bounded by a 25-step cap.The operational characteristics are measured over 330 samples.Source: Table 7 · p. 12

F Over-sampled General Population Results

Table 8 presents the five-fold cross-validation results discussed in Section 4.2.

F Over-sampled General Population Results

G Full Hard Cases Population

On the full SME-adjudicated population, the full stage, FOIL-IE alone, and random forest have closely grouped MCC results. Table 10 covers contradiction types and their resolution actions.

G Full Hard Cases Population

  • Table 9 evaluates all 397 SME-adjudicated accounts, including 60 positives (15.1%), using the same folds, gating, and per-fold threshold tuning as Table 1.Source: Table 9 · p. 12
  • The full stage, FOIL-IE alone, and random forest are separated by less than 0.005 MCC on the full hard-cases population.At this label volume, the methods are statistically indistinguishable.Source: G Full Hard Cases Population · p. 12

H Contradiction Types and Resolution

  • Table 10 lists contradiction types and how the resolution layer acts on them.Source: Table 10 · p. 13
Loading 2609.32643v1…