SIRF: A Spec-Internalized Risk Foundation Model for Industrial Content Risk Control

• High-precision auto-handling matters because false positives harm innocent users while false negatives leave safety hazards.Source: §1 · p. 1
• Account-level decisions combine around a hundred policies, heterogeneous multisource features, second-level latency, verdict-only output, and tunable thresholds.Source: §1 · p. 1
• Inference-time policy injection adds context or retrieval overhead, whereas pure classifiers require impractical coverage of long-tail policies and trigger–exemption branches.Source: §1 · p. 1
• SIRF internalizes synthesized policy specifications into model weights through policy-grounded CPT, then applies them with domain SFT in low-latency verdict-only deployment.Source: §1 · p. 1

• Among included logprob-available models, SIRF-8B is above compared open-weight and closed-source systems at the high-precision operating point.Source: Figure 1 · p. 2
• SIRF uses 70M synthesized tokens to internalize platform policies, improving high-precision Black Recall@P95 while preserving general knowledge.Source: §1 · p. 2
• A controlled same-source comparison and component ablation isolate the gain to policy content rather than additional in-domain tokens.Source: §1 · p. 2
• Approximately 70% relative mis-penalization reduction demonstrates transfer to a new risk domain after light SFT.Source: §1 · p. 2
• A 13% prompt reduction and 20% more release provide deployment evidence under second-level latency, alongside preserved general knowledge.Source: §1 · p. 2

Synthetic CPT and spec internalization

• SIRF differs from alignment-oriented spec midtraining by internalizing industrial risk-control policies for deployment rather than general alignment.Source: §2 · p. 2

LLM data synthesis

• MAGA expands synthetic corpora through multi-genre, multiaudience rewriting, which SIRF specializes for policy internalization with label-leakage control.Source: §2 · p. 2

Content moderation

• Unlike guard models and RAG systems that supply policies online, SIRF places them in weights and targets verdict-only deployment with adjudication and transfer evidence.Source: §2 · p. 2

Midtraining, data mixing, and schedule

• Prior midtraining work motivates checking mixture and schedule sensitivity rather than assuming token volume alone determines gains.Source: §2 · p. 2

Selective classification

• Recall@P fixes risky-class precision and measures how much risk can be auto-handled, expressing the operator-facing error–reject trade-off.Source: §2 · p. 2

SIRF positioning

• SIRF’s pipeline is Spec-Internalized CPT followed by Domain SFT and Selective Risk Deployment, using a first-token score rather than retrieval-time policy supply.Source: Figure 2 · p. 3, §2 · p. 3

• SIRF trains a base with policy-grounded CPT, domain SFT, and second-level verdict-only deployment.Source: §3 · p. 3

Corpus synthesis

• The corpus is synthesized from roughly a hundred trigger–exemption policies without additional human annotation.Source: §3.1 · p. 3

Rule-strategy decomposition

• Stable judgment rules are internalized in CPT, while volatile activation and threshold strategies remain adjustable through dynamic thresholds and light SFT.Source: §3.1 · p. 3

Corpus synthesis

• EntiGraph extracts entities and relations, MAGA rewrites policies across genres and audiences, and account-level CoT distillation produces feature–clause–decision chains.Source: §3.1 · p. 3

Account-level CoT distillation

• SEADA filters account-level chains using clause-faithfulness judging, trainability, self-consistency, and confidence gates while excluding ground-truth dispositions.Source: §3.1 · p. 3

Training corpus

• The CPT corpus contains approximately 70M tokens, including risk reasoning, structured policy knowledge, anti-forgetting data, and fraud hard negatives.Source: §3.2 · p. 3

Controlled comparison

• SIRF and its baseline share the checkpoint, policy injection, verdict-only form, and SFT data; CPT is the added training stage.Source: §3.2 · p. 4

Selective deployment

• Deployment uses first-token verbalizer probability as a monotone, re-thresholdable score rather than requiring calibrated probabilities or reasoning traces.Source: §3.3 · p. 4

4.2 Main Results

• 71.3% Black Recall@P95 is 15.1pp above Qwen3-8B-SFT, with SIRF exceeding the included logprob-available comparators.The same-source comparison is central; cross-model values are corroboration under the shared interface.Source: §4.2 · p. 5, Table 1 · p. 6
• SIRF’s advantage emerges beyond approximately P93 and peaks at P95 because its high-confidence samples remain purer under stricter precision constraints.Figure 3 reports the Black-recall versus precision sweep for the three same-source 8B arms.Source: §4.2 · p. 5, Figure 3 · p. 5
• At matched White precision, SIRF recalls 68.2% at P85 and 54.5% at P90 versus 59.1% and 46.8% for the baseline.The White Recall@P95 decrease is confined to the steepest cutoff, while Macro-F1 and Accuracy remain on par.Source: §4.2 · p. 5

4.3 Where Does the Gain Come From? Per-Component CPT Ablation

• The CPT ablation reaches 71.3 Black Recall@P95 with the full corpus, versus 56.2 without CPT and 55.7 with domain-only tokens, isolating policy content rather than token volume.Policy-carrying EntiGraph+MAGA and CoT arms improve over no CPT, while the full corpus is highest; the ablation does not order the two carriers.Source: §4.3 · pp. 5–6

4.4 Inference Efficiency

• 13% policy-prompt trimming leaves B@P90 nearly unchanged at 84.6 versus 84.7 but reduces P95 Black recall from 71.3 to 66.9 while improving latency, QPS, KV use, and maximum concurrency.Median latency falls 8.8%/12.6%/18.2% at concurrency 16/32/100, and maximum concurrency rises from 381 to 436.Source: §4.4 · p. 6

4.6 General Ability

• General ability before and after CPT almost fully overlaps across 8B, 14B, and 32B, although text-only CPT weakens instruction-following.At 8B, AGIEval-CN, GSM8K, and HellaSwag rise while MMLU-Pro drops 3.6 points.Source: Figure 4 · p. 6, §4.6 · pp. 6–7

4.5 Per-Subclass Behavior

• Across 24 fine classes, gains concentrate in high-risk categories while drops are mostly low-risk, but roughly 200 samples per class limit per-class significance.Reported high-risk gains include +8.8 for infant conditioning, +6.6 for qualification documents, and +6.5 for debt relief and fund recovery.Source: §4.5 · p. 6

5.1 Flow Control: Adjudication Layer

• 20% more of the tree model’s original false positives are recovered through SIRF’s high-confidence White release arm.SIRF adjudicates tree-model hits using separate class-specific cutoffs, while Gray cases remain in the original flow.Source: §5.1 · p. 7
• About 10% of all tree-model hits are auto-escalated, while the remaining cases stay under the original flow-control policy.The routing separates high-confidence Black escalation, high-confidence White release, and unresolved cases.Source: §5.1 · p. 7

5.2 Freezing: Cross-Domain Transfer

• SIRF reduces freezing mis-penalization by about 70% relatively after transfer with only light in-domain SFT and no CPT rerun.The transfer released users at cumulative million-user scale online.Source: §5.2 · p. 7
• An online randomized A/B test finds significantly better weekly active penetration for treatment users receiving SIRF adjudication and high-confidence releases.The experiment changed no other model or policy and used the same population and time window.Source: §5.2 · p. 7
• The deployment validates SIRF as a risk foundation model, while multi-domain transfer beyond the first freezing scenario remains future work.The authors connect fewer false positives with improved normal-user experience and activity.Source: §5.2 · p. 7

6 Conclusion

• SIRF reaches +15.1pp Black Recall@P95 over the same-source baseline using only 70M synthesized CPT tokens.The controlled comparison attributes the gain to policy content rather than extra in-domain tokens, while general knowledge remains preserved.Source: §6 · p. 7

Limitations

• Black Recall@P95 is sensitive to boundary samples, so the authors rely on sweeps, replication, confidence intervals, and multi-month deployment evidence.A larger, multi-period evaluation set remains future work.Source: Limitations · pp. 7–8
• CPT uses no additional human annotation, but its verdict-terminating chains provide self-generated, quality-gated weak supervision.Ground-truth disposition is excluded from both the training data and teacher context.Source: Limitations · p. 8
• Text-only CPT weakens instruction-following, while the reduced policy trades −4.4pp at P95 for a 13% prompt reduction.The trade-off is accepted because deployment emits a single verdict.Source: Limitations · p. 8
• The study uses one Chinese-language platform’s proprietary data, excludes models without token logprobs, and releases neither code nor model.The foundation-model claim is based on a first transfer scenario.Source: Limitations · p. 8

Ethical Considerations

• Ultra-conservative thresholds concentrate automatic execution on the model’s most confident samples to reduce false-positive risk.The system must balance over-enforcement against legitimate users with under-blocking that harms the ecosystem.Source: Ethical Considerations · p. 8
• Policies and labels may encode one platform’s value judgments, so fairness should be re-examined and policies recalibrated when transferring SIRF.The authors specifically caution that performance may vary across populations or content types.Source: Ethical Considerations · p. 8

Evaluation metric

• The main evaluation uses per-class Recall@P with the smallest feasible threshold, independently fitted for each class and model.This maximizes the accepted set subject to the target precision constraint without sharing thresholds across classes or models.Source: A Evaluation Protocol and Set… · p. 11

Set composition

• The production-distribution set contains 1,000 accounts: 596 Black, 250 Gray, and 154 White, with live outcomes plus random human re-checks as ground truth.The long-tailed distribution limits fine-class conclusions, so subclass analysis uses the balanced set instead.Source: A Evaluation Protocol and Set… · p. 11

Calibration

• SIRF is more over-confident than its baseline, with ECE increasing from 0.128 to 0.176 while accuracy stays flat.Strictly monotone recalibration improves ECE but leaves Recall@P unchanged because the ranking is preserved.Source: B Calibration and Confidence Variants · p. 11, Table 2 · p. 11

Robustness evaluation

• The balanced set reproduces the SIRF-over-baseline direction, with transferred thresholds retaining Black precision ≥95% and SIRF reaching 95.7%.On this distribution, Recall@P90 equals Recall@P95 for every model, so the precision constraint is not binding.Source: C Balanced-Set Replication · p. 11, Table 3 · p. 12

Confidence analysis

• SIRF’s top-1 confidence shifts upward, with mean probability increasing from 0.788 to 0.814 and median probability from 0.851 to 0.900.SIRF exceeds the baseline on 662 of 1,000 samples, with the empirical confidence distribution shifted right.Source: D Confidence-Reliability Analysis · p. 12, Figure 5 · p. 12, Figure 6 · p. 12

Serving efficiency

• Reducing the policy prompt by 13% yields stable high-concurrency efficiency gains, including approximately 18% lower p50 end-to-end latency at concurrency 100.The reduced tier also provides +14%–+23% QPS and higher concurrency headroom, with a quantified 4.4pp Black Recall@P95 trade-off while preserving Black Recall@P90 and accuracy.Source: E Efficiency Curves · pp. 12–13, Table 5 · p. 12, Figure 8 · p. 13

Probe method

• Logit Lens projects each layer’s residual stream through the model’s final LayerNorm and lm_head to inspect decision-label first-character probabilities.The probe uses the deployment-aligned dialogue template and reads layer-wise vocabulary distributions at the first decision-label token.Source: F Interpretability via Logit Lens · p. 13

Layer-wise signal emergence

• CPT strengthens the correct-label signal after emergence rather than making it appear earlier, with SIRF final-layer probabilities of 0.66 versus 0.40 for Gray and 0.50 versus 0.35 for Black.For Gray and Black, both SFT and SIRF signals emerge near L30; SIRF’s signal is stronger thereafter.Source: F Interpretability via Logit Lens · p. 13, Figure 10 · p. 13

Routing versus knowledge

• SFT builds the structural routing to emit decision labels, while CPT alone changes knowledge and expression preferences without reliably emitting those labels.The CPT model has near-zero correct-label probability across layers, whereas SFT produces the correct label near the end.Source: F Interpretability via Logit Lens · pp. 13–14

Combined mechanism

• CPT plus SFT yields the highest and cleanest correct-label signal across White, Gray, and Black, matching the paper’s high-precision recall improvement.The mechanism is described as SFT routing over CPT-injected, policy-aligned representations.Source: F Interpretability via Logit Lens · pp. 13–14, Figure 10 · p. 13

Caveat

• The interpretation is evidence of layer-wise emergence rather than strict feature localization because the probe uses vocabulary projection and approximate first-character signals.Larger-scale validation and stronger causal methods remain future work.Source: F Interpretability via Logit Lens · p. 14

Ablation design

• The ablation holds the 8B base, SFT data and schedule, and evaluation set fixed while varying the CPT corpus.Table 6 reports the component comparison and shared training recipe.Source: Table 6 · p. 14, G Per-Component CPT Ablation · p. 14

Shared recipe

• All ablation arms use full-parameter next-token CPT for one epoch followed by full-parameter SFT for three epochs on the same 40,989-sample set.Both stages use a 1e-5 learning rate, with the specified cutoffs, packing, cosine schedule, and effective batch size of 512.Source: G Per-Component CPT Ablation · p. 14

CPT components

• The CPT corpus combines account-level CoT, EntiGraph+MAGA policy text, a public fraud corpus, and a general anti-forgetting corpus.The components contain approximately 51M, 17.5M, 7.1M, and 10.5M characters respectively.Source: G Per-Component CPT Ablation · p. 14

Input control

• The account-level feature representation serializes heterogeneous signals into fixed sections, preserves empty domains explicitly, and uses a fixed recent-content window.Identifiers are removed or pseudonymized, and internal-ecosystem labels are stripped before serialization.Source: H Prompt and Feature Serialization · p. 14