FRAUDSkill: Structured Frozen-Weight Skill Optimization for Audio Anti-Fraud Detection

• Audio anti-fraud requires service-scenario identification, fraud detection, and conditional fraud-type classification within a predefined label ontology.Because decisions are sequentially dependent, an invalid early prediction can propagate and invalidate the decision process.Source: Introduction · p. 1
• Fine-tuning and manually designed prompts do not jointly provide adaptable, maintainable protocol enforcement while keeping the underlying model unchanged.Fine-tuning encodes task knowledge and policies in weights; prompt methods offer flexibility but limited guarantees for structured output and consistency.Source: Introduction · pp. 1–2
• FRAUDSkill learns a modular external skill layer representing fraud knowledge, label constraints, routing policies, and validation-derived decision preferences without changing model parameters.The artifacts are inspectable and replaceable, allowing policies to evolve without retraining the underlying model.Source: Introduction · p. 2
• Route-aware skill programs capture evidence, policies, and label constraints, while structured output control and validation-guided multi-path inference enforce valid labels and consistent decision sequences.Source: Introduction · p. 2
• 73.50% Macro-F1, 31.96% above the shared frozen-model baseline, with a 1.94% invalid-output rate on TeleAntiFraud.Source: Introduction · p. 2

Related Work

• Prior audio anti-fraud adaptation primarily updates model parameters, while general skill methods lack explicit support for closed-set labels and conditional decision chains.FRAUDSkill instead targets route-aware skills, official label constraints, and cross-route consistency with the audio-language model frozen.Source: Related Work · p. 2

Problem Setting and Framework Overview

• FRAUDSkill leaves the audio actor unchanged and optimizes external instructions, anti-fraud skills, and routing policies for three ordered decisions.The external artifacts are selected during offline optimization, then frozen alongside label maps, route rules, and the selector for deployment.Source: Problem Setting and Framework Overview · pp. 2–3

External Skill Program

• The skill program separates global instructions, localized fraud knowledge, and route control, which can be revised while actor parameters remain unchanged.The route policy selects an ordered skill list based on the current decision route and normalized history.Source: External Skill Program · p. 3

Offline Skill Optimization

• Offline optimization diagnoses complete route trajectories, edits programs using textual critique, and selects candidates on held-out validation data.Error diagnosis distinguishes label mistakes, unmappable outputs, skipped type decisions, cross-route inconsistencies, and minority-class errors; retained programs are chosen using a route-aware validation criterion.Source: Offline Skill Optimization · pp. 3–4

Fixed Structured Deployment

• At deployment, retained programs independently guide the frozen actor; label projection and route normalization produce canonical decisions, and a validation-fitted selector aggregates trajectories.The selector is calibrated apart from test data and the examples used for textual error feedback; class balancing affects only the fixed aggregation rule.Source: Fixed Structured Deployment · pp. 4–5

Experimental Setup

• The evaluation uses 10,711 training and 2,677 test recordings after audio-level deduplication; programs and selectors are fixed before testing, with no test labels used in their construction.The 1,453 test recordings with fraud-type annotations are evaluated under the sequential protocol, where an upstream error can prevent a downstream route from running.Source: Experimental Setup · p. 5
• All directly compared methods use deterministic decoding with Qwen2-Audio-7B-Instruct, whose parameters remain fixed during adaptation and evaluation.Differences therefore arise from the representation, optimization, and application of external task knowledge rather than changes to the audio model.Source: Experimental Setup · p. 5
• The comparison progresses from shared task knowledge and editable or modular skill artifacts to route-aware text search and FRAUDSkill’s structured decision control.FRAUDSkill-Text uses seeds 42, 43, and 44; its reported comparison is the arithmetic mean, with sample standard deviation measuring variation.Source: Experimental Setup · p. 5, Main Results · p. 5

Main Results

• SkillOpt and EvoSkill reduce invalid outputs relative to the shared baseline but do not improve Macro-F1, illustrating that better-formed instructions do not ensure better closed-set classification.The baseline has 41.54% Macro-F1 and 36.04% Invalid Rate; SkillOpt has 37.67% and 29.00%, while EvoSkill has 39.07% and 34.19%, respectively.Source: Main Results · p. 5
• FRAUDSkill-Text reaches 42.42% average Macro-F1, 0.88 percentage points above the shared baseline, but its Invalid Rate remains 34.48%.Its Joint Accuracy rises from 7.10% to 8.18%; route-aware textual organization improves discrimination to some extent, but still combines open-ended generation with decision control.Source: Main Results · p. 5
• 73.50% Macro-F1, a 31.96-percentage-point gain over the shared baseline, accompanies 79.40% W-F1, 78.72% Accuracy, 58.87% Joint Accuracy, and a 1.94% Invalid Rate.Table 1 reports audio-level results on the complete test set; SFT and SFT+Memory are parameter-training references, not frozen-skill baselines.Source: Table 1 · p. 6, Main Results · p. 6

Ablation Studies

• 22.55 percentage points of Macro-F1 gain come from closed-set projection and route normalization, which raise results from 43.66% to 54.47% and then 66.21%.The analysis starts from the best textual program and attributes the gains to label projection and decision-chain consistency.Source: Ablation Studies · p. 6, Figure 3 · p. 6
• Complementary multi-path inference raises Macro-F1 from 66.21% to 70.31%; reliability weighting reaches 70.84%, and class-balanced selection adds 2.66 points to reach 73.50%.The structured components contribute 29.84 Macro-F1 points over the best textual program; the selection-stage benefit is attributed mainly to protecting minority classes (Han et al. 2025).Source: Ablation Studies · p. 6

Error Analysis

• Scene-level errors reach 74.2%, with missing or off-ontology outputs accounting for 91.1% of those errors.Table 2 reports FRAUDSkill-Text test-set error counts.Source: Error Analysis · p. 6
• The type route remains the most difficult, with an 83.1% error rate on annotated examples, motivating label projection, route normalization, and complementary multi-path inference to improve label validity and cross-route consistency.Source: Error Analysis · p. 7

Case Study

• Without FRAUDSkill, the actor labels the scene “Loan service” and the fraud type “fake loan fraud,” which fall outside the official label set.Only the fraud judgment is valid, yielding one valid prediction out of three.Source: Case Study · p. 7
• With FRAUDSkill, the same call is labeled customer consultation, fraud, and bank fraud, and all three predictions are valid.The rationales cite identity and bank-card information requests and urgency from a limited quota; decisions and reasons are inspectable step by step.Source: Case Study · p. 7
• The case-level change is consistent with invalid outputs falling from 36.04% to 1.94% overall, without model-weight updates or task-specific tuning.The authors characterize this as a training-free, plug-and-play deployment approach for frozen audio-language models.Source: Case Study · p. 7

Conclusion

• FRAUDSkill extends prompt-style search into a routed skill program and pairs it with normalized complementary frozen-actor trajectories and class-balanced reliability selection.It adapts external skill artifacts rather than audio-model parameters.Source: Conclusion · p. 7
• Under the studied protocol, the complete system performs clearly above the plain SFT reference and slightly below SFT+Memory without updating the audio actor.The authors attribute its competitive performance to validity control, complementary trajectories, and class-aware selection.Source: Conclusion · p. 7
• FRAUDSkill-Text marginally improves mean Macro-F1 over the shared baseline, but variation exceeds the mean gain.The authors characterize text-layer skill search alone as useful but unstable in this setting.Source: Conclusion · p. 7

Ethical Statement

• FRAUDSkill is intended only for defensive research on telecom-fraud detection.Source: Ethical Statement · p. 8
• Its synthetic, non-private artifacts must support defensive analysis and never deception, impersonation, or scam training.Source: Ethical Statement · p. 8

Experimental Protocol Details

• Each example follows the ordered route sequence scene, fraud, then type; type is evaluated only when fraud is predicted or annotated as fraud, otherwise it is NA.Source: Experimental Protocol Details · p. 9
• A required missing, malformed, or unrecoverable non-official label output maps to INV, distinguishing unusable output from a valid negative decision.Source: Experimental Protocol Details · p. 9
• The scene route uses six official labels, the fraud route uses normal and fraud, and the type route uses eight official fraud-type labels.Source: Experimental Protocol Details · p. 9
• Task-averaged Macro-F1 over the three routes is primary; reported secondary metrics include weighted F1, accuracy, joint accuracy, and invalid rate.Joint accuracy requires every applicable route to be correct; invalid rate counts malformed, missing, or out-of-ontology route outputs after parsing and normalization.Source: Experimental Protocol Details · p. 9
• Textual skill search uses training examples for candidate generation and held-out validation for program selection, while selector calibration is disjoint from test data and textual error-feedback examples.Test labels are not used for skill editing, label projection, route normalization, or selector fitting.Source: Experimental Protocol Details · p. 9

External Skill Search Settings

• All frozen-weight methods use the same fixed audio-language actor, while FRAUDSkill changes external instructions, skills, route policies, label maps, normalizers, and selector configurations.Source: External Skill Search Settings · p. 9
• The released search uses seed 42, beam width 3, branch factor 3, five rounds, and batches of 64 examples, scoring candidates on held-out validation data.The reported paper metric remains task-averaged Macro-F1.Source: External Skill Search Settings · p. 9
• Rollouts record raw, parsed, and normalized route decisions, histories, and errors; a textual critic summarizes recurring failures for a typed editor to revise external programs.Editable components include the root instruction, named skills, local skill content, and route policy; audio-model parameters remain unchanged.Source: External Skill Search Settings · p. 9
• FRAUDSkill-Text is the best single selected external program, whereas full FRAUDSkill retains complementary programs and fixes projection, route normalization, and class-balanced reliability selection before test evaluation.Source: External Skill Search Settings · p. 9

Text-Search Stability and Repair Results

• 42.42% mean Macro-F1 with a 1.39-point sample standard deviation across three runs characterizes FRAUDSkill-Text’s test-set search stability.The standard deviation is the sample standard deviation.Source: Text-Search Stability and Repair Results · p. 9, Table S1 · p. 9
• Text-level output repairs improve validity-oriented metrics, especially invalid rate and joint accuracy, but do not produce the final structured-inference gains.The reported repair results are on the validation set.Source: Text-Search Stability and Repair Results · p. 9, Table S2 · p. 9

Reproducibility Details

• The supplement includes evaluation scripts, configurations, the official label ontology, result tables, prediction-trace schemas, selected skill programs, selector configuration, and replay scripts.The smallest verification path regenerates paper-table CSV files from released JSON results and checks trace-schema conformance.Source: Reproducibility Details · pp. 9–10
• Local replay checks the search and evaluation pipeline without private API credentials or the full audio benchmark.It verifies route sequence, trace schema, program-selection bookkeeping, deterministic normalization, and output-file structure.Source: Reproducibility Details · p. 10
• A full rerun requires access to the TeleAntiFraud audio split, frozen audio-language model, and critic/editor interfaces; raw audio, model weights, and credentials are not included in the supplement.Audio and model access follow the source dataset’s conditions and upstream provider’s license and distribution channel.Source: Reproducibility Details · p. 10
• Text-level search is stochastic, is reported over seeds 42, 43, and 44, and separates frozen-actor inference, critic/editor calls, and deterministic deployment post-processing in its compute disclosure.The complete system fixes retained programs and selector configurations before test evaluation.Source: Reproducibility Details · p. 10