PaperlayerView exact revision on arXiv

Source-linked AI summary

SIRF: A Spec-Internalized Risk Foundation Model for Industrial Content Risk Control

Suwan Wu, Yumeng Lin, Pengcheng Yuan, Xiaolong Jiang

arXiv:2609.11752v1cs.AIcs.CLcs.LG

TL;DR

Industrial content risk control requires high-precision, low-latency decisions over complex policies rather than average accuracy alone. SIRF internalizes synthesized policies into model weights through CPT, followed by domain SFT and verdict-only selective deployment. It reaches 71.3 Black Recall@P95, 15.1pp above the same-source baseline, with deployment and transfer evidence, while evaluation remains bounded by single-platform, proprietary-data, and interface constraints.

  • Problem

    Industrial risk control must maximize auto-handled risk at high precision and second-level latency despite complex policies, heterogeneous account data, and changing strategies.

  • Method

    SIRF synthesizes policy knowledge and account-level reasoning, internalizes it through policy-grounded CPT, then applies domain SFT with verdict-only selective deployment.

  • Results

    71.3 Black Recall@P95 is 15.1pp above Qwen3-8B-SFT in the controlled same-source comparison, with the gain attributed to policy content rather than extra in-domain tokens.

  • Takeaways & Limitations

    SIRF supports high-precision risk handling under low latency, with production adjudication and low-cost transfer to another risk domain.

  • Takeaways & Limitations

    Experiments use one Chinese-language platform with proprietary data, comparison coverage is interface-limited, and the foundation-model claim rests on a first transfer scenario.

Abstract

from arXiv · show

For industrial content risk control, the real deployment constraint is not average accuracy but how much risk can be auto-handled under high precision and second-level latency. We present SIRF (Spec-Internalized Risk Foundation Model), which internalizes a platform's complex policies, synthesized without additional human annotation via EntiGraph, MAGA rewriting and account-level chain-of-thought (CoT), into the weights via continued pretraining (CPT), so rules are applied at high precision under an ultra-low-latency, verdict-only deployment. A controlled same-source comparison (Qwen3-8B-SFT vs. SIRF-8B-SFT, identical policy injection and verdict-only output form, differing only in policy-grounded CPT) attributes the gain to internalization: SIRF-8B-SFT reaches 71.3% Black Recall@P95, +15.1pp over the baseline, using only ~70M CPT tokens without harming general ability, and among included, logprob-available models under this interface it matches or exceeds far larger systems. SIRF is deployed as a tree-model adjudication layer (20% more mis-penalized samples recovered) and transfers to a freezing scenario at low cost (~70% relative mis-penalization reduction).

1 Introduction

Industrial risk control must optimize high-precision auto-handling under second-level latency, not average accuracy. Complex policies, heterogeneous account features, and frequently changing strategies make this difficult, while inference-time policy injection and pure classification each fail deployment or coverage requirements.

  • High-precision auto-handling matters because false positives harm innocent users while false negatives leave safety hazards.
  • Account-level decisions combine around a hundred policies, heterogeneous multisource features, second-level latency, verdict-only output, and tunable thresholds.
  • Inference-time policy injection adds context or retrieval overhead, whereas pure classifiers require impractical coverage of long-tail policies and trigger–exemption branches.
  • SIRF internalizes synthesized policy specifications into model weights through policy-grounded CPT, then applies them with domain SFT in low-latency verdict-only deployment.

1. SIRF, a paradigm that internalizes risk-control

SIRF frames risk control as spec internalization rather than generic scaling or average-accuracy optimization. Its reported contributions combine a controlled attribution study, transfer evidence, and deployment-oriented efficiency results.

  • Among included logprob-available models, SIRF-8B is above compared open-weight and closed-source systems at the high-precision operating point.
  • SIRF uses 70M synthesized tokens to internalize platform policies, improving high-precision Black Recall@P95 while preserving general knowledge.
  • A controlled same-source comparison and component ablation isolate the gain to policy content rather than additional in-domain tokens.
  • Approximately 70% relative mis-penalization reduction demonstrates transfer to a new risk domain after light SFT.
  • A 13% prompt reduction and 20% more release provide deployment evidence under second-level latency, alongside preserved general knowledge.

2 Related Work

Related work spans synthetic domain internalization, online policy-conditioned moderation, training-mixture studies, selective classification, and confidence calibration. SIRF combines these themes by internalizing risk policies into weights and deploying a re-thresholdable confidence score without retrieval or reasoning traces.

  • Synthetic CPT and spec internalization: SIRF differs from alignment-oriented spec midtraining by internalizing industrial risk-control policies for deployment rather than general alignment.
  • LLM data synthesis: MAGA expands synthetic corpora through multi-genre, multiaudience rewriting, which SIRF specializes for policy internalization with label-leakage control.
  • Content moderation: Unlike guard models and RAG systems that supply policies online, SIRF places them in weights and targets verdict-only deployment with adjudication and transfer evidence.
  • Midtraining, data mixing, and schedule: Prior midtraining work motivates checking mixture and schedule sensitivity rather than assuming token volume alone determines gains.
  • Selective classification: Recall@P fixes risky-class precision and measures how much risk can be auto-handled, expressing the operator-facing error–reject trade-off.
  • SIRF positioning: SIRF’s pipeline is Spec-Internalized CPT followed by Domain SFT and Selective Risk Deployment, using a first-token score rather than retrieval-time policy supply.

3 Method

SIRF builds a policy-grounded CPT corpus from structured specifications and synthetic account reasoning, then applies domain SFT and selective deployment using a tunable first-token confidence score. Controlled training and deployment choices separate stable rules from changing strategies while preserving verdict-only serving.

  • SIRF trains a base with policy-grounded CPT, domain SFT, and second-level verdict-only deployment.
  • Corpus synthesis: The corpus is synthesized from roughly a hundred trigger–exemption policies without additional human annotation.
  • Rule-strategy decomposition: Stable judgment rules are internalized in CPT, while volatile activation and threshold strategies remain adjustable through dynamic thresholds and light SFT.
  • Corpus synthesis: EntiGraph extracts entities and relations, MAGA rewrites policies across genres and audiences, and account-level CoT distillation produces feature–clause–decision chains.
  • Account-level CoT distillation: SEADA filters account-level chains using clause-faithfulness judging, trainability, self-consistency, and confidence gates while excluding ground-truth dispositions.
  • Training corpus: The CPT corpus contains approximately 70M tokens, including risk reasoning, structured policy knowledge, anti-forgetting data, and fraud hard negatives.
  • Controlled comparison: SIRF and its baseline share the checkpoint, policy injection, verdict-only form, and SFT data; CPT is the added training stage.
  • Selective deployment: Deployment uses first-token verbalizer probability as a monotone, re-thresholdable score rather than requiring calibrated probabilities or reasoning traces.

4 Experiments and Results

On the production-distribution evaluation, SIRF improves high-precision Black recall over the same-source baseline while preserving broad classification and general ability, with ablations attributing the gain to policy-bearing CPT. The experiments also examine confidence behavior, deployment efficiency, class-level robustness, and evaluation boundaries.

  • 4.2 Main Results: 71.3% Black Recall@P95 is 15.1pp above Qwen3-8B-SFT, with SIRF exceeding the included logprob-available comparators.The same-source comparison is central; cross-model values are corroboration under the shared interface.
  • 4.2 Main Results: SIRF’s advantage emerges beyond approximately P93 and peaks at P95 because its high-confidence samples remain purer under stricter precision constraints.Figure 3 reports the Black-recall versus precision sweep for the three same-source 8B arms.
  • 4.2 Main Results: At matched White precision, SIRF recalls 68.2% at P85 and 54.5% at P90 versus 59.1% and 46.8% for the baseline.The White Recall@P95 decrease is confined to the steepest cutoff, while Macro-F1 and Accuracy remain on par.
  • 4.3 Where Does the Gain Come From? Per-Component CPT Ablation: The CPT ablation reaches 71.3 Black Recall@P95 with the full corpus, versus 56.2 without CPT and 55.7 with domain-only tokens, isolating policy content rather than token volume.Policy-carrying EntiGraph+MAGA and CoT arms improve over no CPT, while the full corpus is highest; the ablation does not order the two carriers.
  • 4.4 Inference Efficiency: 13% policy-prompt trimming leaves B@P90 nearly unchanged at 84.6 versus 84.7 but reduces P95 Black recall from 71.3 to 66.9 while improving latency, QPS, KV use, and maximum concurrency.Median latency falls 8.8%/12.6%/18.2% at concurrency 16/32/100, and maximum concurrency rises from 381 to 436.
  • 4.6 General Ability: General ability before and after CPT almost fully overlaps across 8B, 14B, and 32B, although text-only CPT weakens instruction-following.At 8B, AGIEval-CN, GSM8K, and HellaSwag rise while MMLU-Pro drops 3.6 points.
  • 4.5 Per-Subclass Behavior: Across 24 fine classes, gains concentrate in high-risk categories while drops are mostly low-risk, but roughly 200 samples per class limit per-class significance.Reported high-risk gains include +8.8 for infant conditioning, +6.6 for qualification documents, and +6.5 for debt relief and fund recovery.

5 Production Deployment Evidence

SIRF is deployed as a conservative adjudication layer for flow control and transfers to freezing with light SFT, recovering mis-penalized users while preserving high-precision gains.

  • 5.1 Flow Control: Adjudication Layer: 20% more of the tree model’s original false positives are recovered through SIRF’s high-confidence White release arm.SIRF adjudicates tree-model hits using separate class-specific cutoffs, while Gray cases remain in the original flow.
  • 5.1 Flow Control: Adjudication Layer: About 10% of all tree-model hits are auto-escalated, while the remaining cases stay under the original flow-control policy.The routing separates high-confidence Black escalation, high-confidence White release, and unresolved cases.
  • 5.2 Freezing: Cross-Domain Transfer: SIRF reduces freezing mis-penalization by about 70% relatively after transfer with only light in-domain SFT and no CPT rerun.The transfer released users at cumulative million-user scale online.
  • 5.2 Freezing: Cross-Domain Transfer: An online randomized A/B test finds significantly better weekly active penetration for treatment users receiving SIRF adjudication and high-confidence releases.The experiment changed no other model or policy and used the same population and time window.
  • 5.2 Freezing: Cross-Domain Transfer: The deployment validates SIRF as a risk foundation model, while multi-domain transfer beyond the first freezing scenario remains future work.The authors connect fewer false positives with improved normal-user experience and activity.

6 Conclusion

SIRF internalizes platform policies into model weights with a small synthesized corpus, improving high-precision risk recall while preserving general knowledge and supporting low-latency deployment.

  • 6 Conclusion: SIRF reaches +15.1pp Black Recall@P95 over the same-source baseline using only 70M synthesized CPT tokens.The controlled comparison attributes the gain to policy content rather than extra in-domain tokens, while general knowledge remains preserved.

Limitations

The evaluation supports SIRF’s reported gains but is bounded by operating-point sensitivity, same-set threshold fitting, weak supervision, deployment trade-offs, and limited comparison and reproducibility scope.

  • Limitations: Black Recall@P95 is sensitive to boundary samples, so the authors rely on sweeps, replication, confidence intervals, and multi-month deployment evidence.A larger, multi-period evaluation set remains future work.
  • Limitations: CPT uses no additional human annotation, but its verdict-terminating chains provide self-generated, quality-gated weak supervision.Ground-truth disposition is excluded from both the training data and teacher context.
  • Limitations: Text-only CPT weakens instruction-following, while the reduced policy trades −4.4pp at P95 for a 13% prompt reduction.The trade-off is accepted because deployment emits a single verdict.
  • Limitations: The study uses one Chinese-language platform’s proprietary data, excludes models without token logprobs, and releases neither code nor model.The foundation-model claim is based on a first transfer scenario.

Ethical Considerations

SIRF uses ultra-conservative thresholds to reduce false-positive risk in decisions affecting users’ speech and account rights, while recognizing that platform-specific policies may behave unevenly across populations and content types.

  • Ethical Considerations: Ultra-conservative thresholds concentrate automatic execution on the model’s most confident samples to reduce false-positive risk.The system must balance over-enforcement against legitimate users with under-blocking that harms the ecosystem.
  • Ethical Considerations: Policies and labels may encode one platform’s value judgments, so fairness should be re-examined and policies recalibrated when transferring SIRF.The authors specifically caution that performance may vary across populations or content types.

A Evaluation Protocol and Set Composition

The evaluation uses class-specific precision-constrained recall, production and balanced datasets, calibration diagnostics, confidence analyses, and deployment-oriented efficiency measurements. The protocol emphasizes threshold behavior, distributional robustness, and high-concurrency serving trade-offs.

  • Evaluation metric: The main evaluation uses per-class Recall@P with the smallest feasible threshold, independently fitted for each class and model.This maximizes the accepted set subject to the target precision constraint without sharing thresholds across classes or models.
  • Set composition: The production-distribution set contains 1,000 accounts: 596 Black, 250 Gray, and 154 White, with live outcomes plus random human re-checks as ground truth.The long-tailed distribution limits fine-class conclusions, so subclass analysis uses the balanced set instead.
  • Calibration: SIRF is more over-confident than its baseline, with ECE increasing from 0.128 to 0.176 while accuracy stays flat.Strictly monotone recalibration improves ECE but leaves Recall@P unchanged because the ranking is preserved.
  • Robustness evaluation: The balanced set reproduces the SIRF-over-baseline direction, with transferred thresholds retaining Black precision ≥95% and SIRF reaching 95.7%.On this distribution, Recall@P90 equals Recall@P95 for every model, so the precision constraint is not binding.
  • Confidence analysis: SIRF’s top-1 confidence shifts upward, with mean probability increasing from 0.788 to 0.814 and median probability from 0.851 to 0.900.SIRF exceeds the baseline on 662 of 1,000 samples, with the empirical confidence distribution shifted right.
  • Serving efficiency: Reducing the policy prompt by 13% yields stable high-concurrency efficiency gains, including approximately 18% lower p50 end-to-end latency at concurrency 100.The reduced tier also provides +14%–+23% QPS and higher concurrency headroom, with a quantified 4.4pp Black Recall@P95 trade-off while preserving Black Recall@P90 and accuracy.

F Interpretability via Logit Lens

Logit Lens probes show that CPT strengthens policy-aligned decision signals in intermediate-to-late representations, while SFT supplies the output routing needed to emit decision labels. Their combination produces the strongest and cleanest correct-label signal, with interpretability claims limited by the probe’s approximations.

  • Probe method: Logit Lens projects each layer’s residual stream through the model’s final LayerNorm and lm_head to inspect decision-label first-character probabilities.The probe uses the deployment-aligned dialogue template and reads layer-wise vocabulary distributions at the first decision-label token.
  • Layer-wise signal emergence: CPT strengthens the correct-label signal after emergence rather than making it appear earlier, with SIRF final-layer probabilities of 0.66 versus 0.40 for Gray and 0.50 versus 0.35 for Black.For Gray and Black, both SFT and SIRF signals emerge near L30; SIRF’s signal is stronger thereafter.
  • Routing versus knowledge: SFT builds the structural routing to emit decision labels, while CPT alone changes knowledge and expression preferences without reliably emitting those labels.The CPT model has near-zero correct-label probability across layers, whereas SFT produces the correct label near the end.
  • Combined mechanism: CPT plus SFT yields the highest and cleanest correct-label signal across White, Gray, and Black, matching the paper’s high-precision recall improvement.The mechanism is described as SFT routing over CPT-injected, policy-aligned representations.
  • Caveat: The interpretation is evidence of layer-wise emergence rather than strict feature localization because the probe uses vocabulary projection and approximate first-character signals.Larger-scale validation and stronger causal methods remain future work.

G Per-Component CPT Ablation

The CPT ablation varies policy and domain corpus components under a shared training recipe and fixed evaluation setup. The design tests whether gains come from policy-bearing content rather than simply adding in-domain tokens.

  • Ablation design: The ablation holds the 8B base, SFT data and schedule, and evaluation set fixed while varying the CPT corpus.Table 6 reports the component comparison and shared training recipe.
  • Shared recipe: All ablation arms use full-parameter next-token CPT for one epoch followed by full-parameter SFT for three epochs on the same 40,989-sample set.Both stages use a 1e-5 learning rate, with the specified cutoffs, packing, cosine schedule, and effective batch size of 512.
  • CPT components: The CPT corpus combines account-level CoT, EntiGraph+MAGA policy text, a public fraud corpus, and a general anti-forgetting corpus.The components contain approximately 51M, 17.5M, 7.1M, and 10.5M characters respectively.
  • Input control: The account-level feature representation serializes heterogeneous signals into fixed sections, preserves empty domains explicitly, and uses a fixed recent-content window.Identifiers are removed or pseudonymized, and internal-ecosystem labels are stripped before serialization.
Loading 2609.11752v1…