PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift

San Kim, JinYeong Bak

arXiv:2608.21043v1cs.CL

TL;DR

The paper addresses whether phishing detectors generalize from familiar scenarios to unseen ones without relying on scenario-specific cues. It evaluates this with SL-OOD detection and proposes ECoG, whose evidence-consistent training improves hard-case classification and generated-output behavior under social-engineering shift.

  • Problem

    Random-split evaluation can overestimate robustness when training and test data share recurring scenario-specific patterns, making unseen-scenario generalization important for SMS and voice phishing detection.

  • Method

    ECoG combines evidence-span supervision with a rationale–label consistency objective during training for decoder-based detection under held-out social-engineering scenarios.

  • Results

    ECoG consistently reduces prediction–rationale inconsistency across decoder backbones while providing backbone-dependent gains in hard-case classification and evidence-span overlap.

  • Takeaways & Limitations

    Evidence-consistent training improves compact generative phishing detection under social-engineering shift, while random-split evaluation can mislead deployment-oriented model selection.

  • Takeaways & Limitations

    The evaluation focuses on Korean SMS and voice phishing, and its explanation metrics do not establish causal faithfulness.

Abstract

from arXiv · show

Conventional in-distribution evaluation can overestimate robustness when training and test data share recurring task-specific patterns or surface cues. This risk is especially relevant in social-engineering fraud detection, where attackers can preserve malicious intent while changing the scenario, impersonated entity, or wording. We study this problem as scenario-level out-of-distribution (SL-OOD) detection for SMS and voice phishing, where entire attack scenarios are held out from training while the label space remains fixed. This setting tests whether models can generalize to unseen attack scenarios using decision-relevant evidence rather than familiar scenario-specific cues. Using this SL-OOD evaluation, we find that high in-distribution performance does not reliably predict held-out robustness across feature-, encoder-, and decoder-based baselines. We interpret this gap as scenario memorization: reliance on recurring scenario-specific lexical or entity cues rather than decision-relevant evidence. We propose ECoG, an evidence-consistent generative framework that combines evidence-span supervision with a rationale-label consistency objective during training. On the 0.5B decoder, relative to the same backbone trained without consistency regularization, ECoG raises Macro-F1 on OOD challenging instances by 3.22 points, reduces the share of predictions whose generated rationale supports the opposite label by 4.22 points, and increases token-level overlap with reference evidence spans by 8.38 points; the reduction in prediction-rationale inconsistency is consistent across four decoder backbones. These results suggest that compact generative detectors can benefit from evidence supervision and rationale-label consistency under social-engineering shift.

1 Introduction

The paper argues that random-split evaluation can overestimate phishing-detector robustness when models exploit recurring scenario-specific cues. It introduces SL-OOD evaluation and ECoG to test and improve evidence-grounded generalization under unseen scenarios.

  • Motivation and evaluation: SL-OOD evaluation holds out entire social-engineering scenarios while keeping the label space fixed, testing generalization beyond familiar scenario-specific cues.This setting is stricter than conventional random splitting because train and test examples no longer share the same scenario-specific signals.
  • Motivation and evaluation: The evaluation includes a challenging set for intent-level discrimination and measures both classification robustness and generated-output behavior.Output-side measures include evidence-span overlap and prediction–rationale consistency.
  • ECoG: ECoG combines evidence-span supervision with rationale–label consistency regularization during training for decoder-based detection under unseen scenarios.The framework encourages input-side grounding and label-consistent rationales without changing the inference-time decoding path.
  • Motivation and evaluation: High in-distribution performance is a weak predictor of scenario-level robustness across feature-, encoder-, and decoder-based baselines.The paper describes the associated failure mode as scenario memorization: reliance on recurring lexical or entity cues rather than decision-relevant evidence.
  • ECoG: Evidence supervision with rationale–label consistency improves hard-case classification and generated-output behavior under scenario shift.The paper reports consistent reductions in prediction–rationale inconsistency across decoder backbones, alongside improvements in hard-case classification and evidence-span overlap.

2 Related Work

Related work connects shortcut-based distribution-shift failures, evidence and rationale supervision, and phishing detection. The paper positions ECoG as combining these strands within an SL-OOD framework.

  • Distribution shift: Shortcut learning describes decision rules that perform well under standard evaluation but fail to transfer to shifted conditions.Prior NLP work links such failures to shallow lexical or syntactic heuristics.
  • Distribution shift: Scenario-level holdout changes the input distribution while keeping the labeling function fixed, and exposes reliance on recurring scenarios, entities, or templates.The paper describes this application-specific shortcut as scenario memorization.
  • Evidence and rationales: Rationale research distinguishes plausible human-readable explanations from explanations faithful to the model’s actual decision process.Evidence and rationale annotations support evaluation of whether explanations provide label-relevant information.
  • Evidence and rationales: Joint generation of labels and natural-language rationales does not ensure faithful or label-consistent rationales.Related work studies label-specific explanations, rationale association, simulatability, and counterfactual consistency.
  • Phishing detection: Phishing-detection research spans surface-form classifiers, neural sequence models, pretrained language models, and generative approaches for SMS and voice data.The paper extends this application line by evaluating Korean SMS messages and ASR-transcribed voice phishing.
  • Positioning: ECoG combines shortcut-aware SL-OOD evaluation with evidence supervision and rationale–label consistency regularization.This positioning joins research threads that prior work studied separately.

3 Dataset and Evaluation Protocol

The paper reconstructs Korean SMS and voice-phishing data for scenario-level OOD evaluation, annotating inputs with labels, scenarios, evidence spans, and rationales. Evaluation uses random splits, scenario holdouts, and a human-filtered challenging set.

  • Data construction: The dataset contains SMS messages and voice-call transcripts reconstructed through deduplication, scope filtering, scenario annotation, and challenging-set construction.The collection is not treated as an off-the-shelf benchmark.
  • Data construction: Phishing samples come from institutionally verified Korean cases, while benign samples come from DeepNatural and AIHub sources.Benign samples mentioning commonly impersonated entities are selected to reduce topic-level shortcuts and increase lexical similarity to phishing messages.
  • Data construction: Each instance contains input text, a binary phishing label, and a scenario label.The supplied passage also identifies Finance, Parcel, and other scenarios, though the excerpt truncates the full list.
  • Annotations: Evidence spans and rationale annotations are generated with GPT-4o-mini at temperature 0 for training supervision.Evidence spans must be copied verbatim from the input, with filtering and quality-audit procedures documented separately.
  • Evaluation protocol: The in-distribution setting uses a conventional random split in which training, validation, and test sets share the same scenario distribution.This provides the standard evaluation comparison for the scenario-held-out protocol.
  • Evaluation protocol: Scenario-level holdout excludes one or more phishing scenarios from training and evaluates recognition of phishing intent in unseen scenarios.SMS holds out Credit, Finance, and Parcel in turn; Voice holds out Government and Finance in turn, with scenario-balanced OOD averages within modality.
  • Evaluation protocol: The challenging set contains ambiguous or lexically confusable instances requiring intent-level discrimination.SMS candidates are mined with encoder baselines and retained after blind human difficulty filtering; the set is a stress test rather than an adversarial benchmark.

4 The ECoG Framework

ECoG is a decoder-based detector that jointly generates labels, evidence spans, and rationales while training auxiliary objectives on shared hidden states. Its weighted training loss combines native language-model supervision with prompt-side classification and evidence supervision and rationale-position consistency regularization.

  • Framework Overview: ECoG jointly generates a label, evidence spans, and a rationale from the prompted input.During training, each instance includes the input, binary label, reference evidence spans, and rationale; inference uses only the input.
  • Training Objectives: The native LM head supervises the label, evidence, and rationale fields, while auxiliary heads provide classification, evidence-span, and rationale–label consistency objectives.The native objectives comprise label-token prediction and next-token generation for the remaining serialized target fields.
  • Framework Overview: Position masks select prompt states for classification and evidence supervision and rationale states for the consistency branch.The auxiliary heads are training-only and operate on selected shared decoder states.
  • Training Objectives: Evidence-span supervision marks prompt tokens overlapping annotated evidence spans and uses masked token-level binary cross-entropy.A sampled subset of non-evidence tokens reduces imbalance, and evidence-span loss is omitted when the evidence field is “none”.
  • Training Objectives: The consistency branch predicts the label from pooled rationale-position states while excluding label, delimiter, and evidence-span tokens from its pooling mask.Because the decoder is causal and teacher-forced, rationale states can still be conditioned on preceding target tokens.
  • Optimization and Inference: ECoG combines native LM supervision with auxiliary objectives in a weighted joint loss and uses single-stage full fine-tuning.The default consistency weight is λ= 0.1 across backbones and scenarios, rather than being selected on held-out OOD performance; auxiliary heads add fewer than 0.1% of backbone parameters.

5 Experiments

The experiments evaluate scenario-level robustness across model families, decoder scales, and evidence-generation objectives. Results show that held-out scenario performance and rationale quality reveal weaknesses hidden by in-distribution evaluation, while ECoG improves hard-case and output-side behavior.

  • Experimental Setup: Scenario-balanced Macro-F1 is reported across ID/OOD and Test/Challenging settings, with prediction–rationale inconsistency, Span F1, and BERTScore evaluating generated outputs.The study compares feature-based, neural, encoder, proprietary LLM, and decoder-based baselines, using compact HyperCLOVA X SEED and Qwen3 backbones.
  • RQ1: Does ID Predict SL-OOD Robustness?: 32.20 points: KoBERT Voice Test Macro-F1 drops from 77.77 ID to 45.57 OOD, illustrating that high in-distribution performance can conceal severe scenario-shift degradation.TF-IDF + SVM also drops from 97.29 to 74.92 on SMS Test and from 99.13 to 80.33 on Voice Test.
  • RQ1: Does ID Predict SL-OOD Robustness?: Comparable ID Macro-F1 values produce substantially different ID–OOD gaps, while ECoG variants cluster in the high-ID, low-gap region across model-family panels.The figure excludes proprietary LLMs because they are not fit on the ID training distribution, making their gap less directly interpretable as robustness degradation.
  • RQ2: Which Model Families Generalize?: ECoG is the most consistent trainable decoder family across modalities, difficulty settings, and backbones, winning three of four OOD columns; GPT-5.4 remains strongest on Voice Challenging OOD at 90.42.HCX-0.5B ST-SFT reaches 84.98 on SMS Challenging OOD, while rationale-only variants transfer inconsistently across modalities.
  • RQ4: Grounded and Consistent Outputs?: +8.38 points: ECoG increases Span F1 from 33.85 to 42.23, raises BERTScore F1 by 4.98 points, and reduces inconsistency by 4.22 points on HCX-0.5B.Across four backbone configurations, the default consistency weight reduces inconsistency, although Span F1 and BERTScore are more backbone-dependent.

6 Discussion

Random-split performance is a weak proxy for deployment robustness under unseen phishing scenarios. Evidence supervision and rationale–label consistency play complementary roles, but consistency regularization does not establish causal faithfulness.

  • Random-split performance can hide large differences in scenario-level degradation because recurring scenario cues reward memorization rather than intent-level generalization.Scenario-level holdout, especially with challenging instances, provides a stricter deployment-oriented test.
  • Evidence supervision anchors training to input-side spans, whereas consistency regularization encourages generated rationales to remain directionally consistent with predicted labels.The two objectives therefore target distinct aspects of evidence-grounded detection.
  • Consistency regularization primarily improves hard-case and generated-output behavior: Cls F1 changes modestly while Hard F1, Span F1, and BERTScore improve and Inc. decreases.The authors interpret this branch as a hard-case training-time regularizer rather than a general-purpose OOD accuracy booster.
  • The consistency branch biases rationale representations toward label-discriminative information without changing inference-time decoding, but it is not a post-hoc verifier of rationale sufficiency.Thus, improved consistency should not be read as proof that generated rationales faithfully explain model decisions.

7 Conclusion and Future Work

The paper frames Korean SMS and voice phishing as scenario-level OOD generalization and proposes ECoG for compact generative detection. It concludes that random-split selection can mislead deployment-oriented decisions, while evidence-consistent training improves detection under social-engineering shift, within a limited evaluation scope.

  • The study formulates Korean SMS and voice phishing detection as scenario-level OOD generalization and constructs a holdout protocol with a challenging set.
  • ECoG combines evidence-span supervision with rationale–label consistency regularization in a decoder-based generative framework.
  • Across two backbone families and four model sizes, ECoG consistently reduces prediction–rationale inconsistency, with backbone-dependent gains in hard-case classification and evidence-span overlap.
  • The evaluation focuses on Korean SMS and voice phishing, while other languages, channels, temporal updates, and end-to-end audio–text inputs remain future work.Voice results may also inherit errors from whisper-small ASR transcripts.
  • Adversarial within-scenario perturbations, temporal concept drift, and stronger causal-rationale tests are identified as directions for stronger deployment-oriented assessment.

A Annotation Details

The annotation pipeline uses GPT-4o-mini to create extractive evidence spans and Korean rationales, filters invalid spans, de-identifies PII, and audits annotation quality. The challenging set is evaluation-only and combines error-driven or verified phishing cases with similar benign examples.

  • Annotation procedure: GPT-4o-mini constructs evidence-span and Korean rationale annotations from each input and gold label, targeting functional phishing cues such as impersonation and urgency.
  • Filtering and reproducibility: Invalid evidence spans are discarded when they are not exact input substrings, duplicate another span, or contain only trivial formatting artifacts.
  • PII de-identification: The dataset is de-identified for names, addresses, phone numbers, and account numbers before training and evaluation; a 10% manual audit found no remaining sensitive PII.Conservative over-redaction affected 0.874% of spans, typically for non-identifying fields.
  • Evidence–rationale audit: A 1,916-instance evidence–rationale audit yielded a 96.87% usable annotation rate, with Fleiss’ κ=0.8047 and Krippendorff’s nominal α=0.8047.Majority votes were 69.89% Pass, 26.98% Acceptable, and 3.13% Invalid.
  • Challenging-set construction: The challenging set is evaluation-only and stresses ambiguous or lexically confusable instances requiring intent-level discrimination.
  • SMS challenging set: SMS challenging cases come from error-driven mining and human-difficulty filtering, with paired benign examples retrieved by lexical similarity.
  • Voice challenging set: Voice challenging cases use institutionally verified victim transcripts paired with the most semantically similar benign transcript.

C Training Configuration

Experiments use fixed loss weights and sweep the consistency coefficient, while parser behavior is checked against a blinded human audit. ECoG generally lowers inconsistency across backbones and scenarios, but the effect varies by scenario.

  • All results come from a single training run, with Δ values used for sensitivity analyses to mitigate single-seed noise.
  • The default consistency weight is λ=0.1, selected from a sweep over {0.0, 0.05, 0.1, 0.2}; other loss weights remain fixed across experiments.
  • Figure 4 compares per-scenario inconsistency reduction by ECoG against Joint w/o C across decoder backbones, with positive values indicating lower inconsistency.SMS and voice scenarios are distinguished by background colors.
  • The Inc. parser is validated through a blinded audit of 1,916 SMS and voice challenging cases labeled as Consistent, Unclear/Mixed, or Inconsistent.
  • 95.0% precision, 94.2% recall, and 94.6% F1 characterize parser detection of human-identified inconsistencies, with 99.3% overall parser–human agreement.The parser is treated as a conservative proxy for prediction–rationale direction mismatch, not causal faithfulness.

E Per-Scenario Inconsistency

ECoG generally reduces prediction–rationale inconsistency across held-out scenarios and backbones, while its effects vary by scenario. On HCX-0.5B, its gains over Joint w/o C increase with human-judged difficulty.

  • Per-Scenario Inconsistency: ECoG generally reduces prediction–rationale inconsistency across held-out scenarios and backbone configurations, with heterogeneous effects across settings.The largest reported reduction is on SMS Finance for HCX-0.5B (+25.5 points), while several voice-finance settings show smaller or negative reductions.
  • Per-Scenario Inconsistency: +25.5 points is the largest single reduction reported, occurring on SMS Finance for HCX-0.5B.Other backbones show consistently positive but smaller reductions on the same scenario.
  • Human-Difficulty Validation: ECoG’s improvement over Joint w/o C grows monotonically with human-judged difficulty for HCX-0.5B.The analysis uses retained SMS challenging-set instances stratified by disagreement among five blind annotators, with verified labels retained as ground truth.
  • Human-Difficulty Validation: Qwen3-0.6B shows a nearly flat difficulty pattern, consistent with backbone-dependent sensitivity.The reported scores are instance-level Macro-F1 rather than scenario-balanced averages.

F Reproducibility and Supplementary Materials

The supplementary repository provides datasets, annotation materials, training and evaluation resources, parser code, and per-fold OOD results. The paper also documents the roles of generative AI assistants and author-written code.

  • Supplementary Materials: The supplementary repository includes de-identified SMS and voice datasets with scenario labels and challenging-set splits.It also includes annotation prompts covering evidence, rationale, and PII redaction.
  • Supplementary Materials: The repository provides class weights, optimization hyperparameters, training scripts, model configurations, and evaluation scripts.These materials support reproduction of training and evaluation procedures.
  • Supplementary Materials: The repository includes parser source code with cue-pattern rules and per-fold OOD Macro-F1 breakdowns.The listed materials also include model configurations and evaluation scripts.
  • AI and Code Disclosure: GPT-4o-mini was used for evidence-span and rationale annotations and PII redaction, while no other generative AI was used for data generation, augmentation, or filtering.Generative AI assistants supported plotting scripts, debugging, and refactoring; the authors wrote the training, evaluation, and metric code.
Loading 2608.21043v1…