PaperlayerView exact revision on arXiv

Source-linked AI summary

HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation

Nikita Oblakov, Sabrina Sadiekh, Evgeniy Kokuykin

arXiv:2609.01046v1cs.CRcs.AI

TL;DR

Existing guardrail reports provide limited evidence on Russian prompt injection and surface obfuscation. HiveTraceGuard-Pro addresses this gap with a compact bilingual generative guardrail, achieving leading results on selected Russian robustness measures while retaining important evaluation limitations.

  • Problem

    Published evaluations provide limited evidence about Russian prompt injection and Russian surface obfuscation.

  • Method

    HiveTraceGuard-Pro is a LoRA-tuned 0.6B generative guardrail trained on Russian and English with paired benign-harmful examples and shared obfuscation transforms.

  • Results

    HiveTraceGuard-Pro has the highest clean Russian robustness combined-F1 at 0.88 and Russian prompt-injection recall at 0.999 among fifteen compared models.

  • Takeaways & Limitations

    The model provides a compact guardrail evaluated extensively on Russian robustness, with its binary verdict serving as input to an application moderation policy.

  • Takeaways & Limitations

    Response results use legacy standalone-reply serialization rather than the natural assistant-role path, and the reserved Russian confirmation set is small and distributionally narrow.

Abstract

from arXiv · show

Production LLMs must handle inputs that attempt to override system instructions, bypass safety policies or elicit harmful responses. A common mitigation is a separate guardrail model. Existing reports, however, provide little evidence on Russian prompt injection or Russian surface obfuscation. We present HiveTraceGuard-Pro, a 0.6B generative guardrail LoRA-tuned from Qwen3-0.6B. It is trained on Russian and English and uses one binary scoring rule (safe/unsafe) for the final target turn. Its training corpus pairs harmful examples, where a counterpart exists, with benign examples from the same domain and applies eight obfuscation transforms to both labels. In one harness, we compare HiveTraceGuard-Pro with thirty-four other guards on nineteen benchmark groups, sixteen of which are public. Its aggregate key is 0.7432, behind 0.7641 and 0.7552 for the two higher-scoring guards. Over the sixteen public groups alone, its key is 0.7153 and four of the thirty-four other suite guards score higher. In a fifteen-model comparison, HiveTraceGuard-Pro has the highest clean Russian robustness combined-F1 (0.88) and Russian prompt-injection recall (0.999). Both results use Russian sets assembled by our team, and at least 27.1% of the prompt-injection set overlaps the training corpus. Its 14.3 ms median latency is the lowest among those fifteen models in that run. Across the suite, FPR is 0.268 and FNR is 0.156. All reported response results use a legacy standalone-reply serialization rather than the natural assistant-role path of the shipped chat template. We release the merged weights on Hugging Face under Apache-2.0. The corpus, evaluation sets and evaluation code remain internal.

1 Introduction

The paper addresses limited evidence on Russian prompt injection and surface obfuscation by proposing a compact bilingual guardrail trained to preserve harmful-versus-benign distinctions under transformed inputs.

  • Published evaluations provide limited evidence on Russian prompt injection and Russian surface obfuscation.
  • HiveTraceGuard-Pro targets Russian traffic under tight latency and memory budgets with a 0.6B base model.
  • Its training corpus uses a structured harm taxonomy, paired benign and harmful examples, and obfuscation applied to both labels.
  • The model is trained on Russian and English and uses one binary decision rule for requests and responses.
  • A reserved confirmation set and training-overlap audit support evaluation after development and contamination-aware rescoring.

2 Related Work

Prior guardrails cover multiple domains, languages, adversarial prompts, jailbreaks, and multilingual moderation, but reported results provide little Russian-specific evidence.

  • Existing guardrails differ in their moderation targets and policy interfaces across safety domains, languages, adversarial prompts, jailbreaks, and multilingual settings.
  • Published reports do not specifically evaluate Russian prompt injection or Russian surface obfuscation, even for models supporting Russian.
  • In the benchmark run, 8B and 12B guards took 63 to 243 ms, compared with 14.3 ms for HiveTraceGuard-Pro.
  • The evaluation treats larger guards as accuracy references and includes sub-1B variants where available because serving path also affects latency.

3 HiveTraceGuard-Pro Model

HiveTraceGuard-Pro classifies the final moderated turn with a binary generative verdict, while its threat model covers specified single-turn injected instructions and fixed surface transformations.

  • Threat model: Prompt injection overrides or ignores system instructions, whereas jailbreaks preserve malicious intent through semantic reframing and surface obfuscation changes textual form without changing meaning.
  • Guard placement: The natural deployment path judges the final user turn before generation or the final assistant turn before delivery.
  • Guard placement: Reported response benchmarks pass each reply as a standalone user turn rather than exercising the assistant-target branch.
  • Threat model: The threat model covers injected instructions in the moderated turn but excludes unseen retrieved-document instructions, longer dialogue histories, and adaptive attacks against released weights.
  • Model: The model is a LoRA-finetuned Qwen3-0.6B classifier that emits one binary verdict without explanation or category labels.
  • Training: Training combines moderation data, translated resources, synthetic adversarial examples, and internal robustness augmentations in Russian and English dialogue roles.

4 Evaluation

HiveTraceGuard-Pro performs strongly on clean Russian robustness and Russian prompt-injection detection while maintaining low latency, but performance varies by domain and evaluation harness. Reserved-set over-blocking, response-path constraints, training overlap, and non-independent comparisons limit interpretation.

  • Head-to-head comparison: 0.88 clean Russian robustness combined-F1 is highest among fifteen models, while Russian prompt-injection recall reaches 0.999.The Russian prompt-injection set contains 4 misses among 4,417 attacks.
  • Head-to-head comparison: 14.3 ms median latency is lowest among the fifteen models in the benchmark run.The two guards with higher aggregate keys take 53.35 and 71.79 ms in that run.
  • Evaluation scope: The comparison is constrained by development-influenced internal sets, overlap with some evaluation data, and separate harnesses whose values are not interchangeable.The head-to-head comparison has not been reproduced by a third party, and latency values are comparable only within the same benchmark run.
  • Internal held-out evaluation: 0.852 robustness-augmented harm-F1 is the primary internal augmented result, with 0.898 macro-F1 when clean and augmented conditions are pooled.Across the combined robustness test, the model correctly labels 8,110 benign turns, flags 607 benign turns, detects 2,798 harmful turns, and misses 364.
  • Reserved confirmation set: 0.563 response-side FPR on the reserved set exceeds 0.043 on robustness-test, showing that development-set over-blocking does not transfer to the reserved response distribution.The reported response results use a legacy standalone-reply serialization rather than the natural assistant-role branch.

5 Intended Use and Scope

HiveTraceGuard-Pro is intended as a Russian-English request and response filter whose binary verdict supports, but does not replace, a moderation policy. Its operating characteristics require distribution-specific calibration, validated prompt-length handling, and explicit treatment of failure modes.

  • Intended use and scope: HiveTraceGuard-Pro supplies a binary verdict for one final request or response, not categories, explanations, policy reasoning, or multi-step agent monitoring.It is text-only and does not provide legal or compliance determinations.
  • Operating point: 0.985 AUROC on RU-categories and 0.973 on pooled robustness-test rows exceed 0.888 on Aegis-2.0.The English shortfall reflects both ranking quality and threshold placement.
  • Operating point: τ = 0.532 changes mean macro-F1 by less than 0.001 globally, while a robustness-test response sweep improves macro-F1 by 0.061.Operators serving substantial English traffic should calibrate on their own distribution.
  • Prompt handling: 40 960 positions were verified, but behavior beyond that range is unspecified and an exploratory roughly 80 000-token harmful request was labeled safe.The tokenizer applies no automatic truncation, so application-side caps must preserve the policy prefix and target suffix.
  • Failure behavior: At the shipped operating point, the Russian robustness set has 7.0% FPR on benign turns and 11.5% FNR on harmful turns.Observed slices range from FPR 0.016 on clean requests to 0.563 on reserved-set responses.

6 Limitations

The reported limitations center on over-blocking, response-side distribution shift, obfuscation trade-offs, context handling, and evaluation uncertainty. These boundaries make the strongest results less representative of all deployments and conditions.

  • Error profile: 0.268 pooled FPR exceeds 0.156 FNR, while the ordering reverses in benchmark-group averages only in magnitude, not direction.Twenty-five guards have lower FPR and eight have lower FNR than HiveTraceGuard-Pro.
  • Response-side limitations: 0.563 response-side FPR on the reserved set contrasts with 0.043 on robustness-test, showing that development-set response FPR does not transfer.All reported response results use legacy standalone-reply serialization rather than the natural assistant-role branch.
  • Obfuscation: Obfuscation experiments repeatedly traded improved harmful-request recall for increased benign over-blocking, or the reverse, without removing the trade-off.Because every experiment kept the model at 0.6B, the data and capacity sources of this limitation remain unresolved.
  • Prompt length: 40 960 positions define the verified context range; deployments should enforce a validated limit while preserving both the policy prefix and target suffix.Behavior beyond the verified range is unspecified and the tokenizer does not truncate automatically.

7 Conclusion

The paper presents HiveTraceGuard-Pro as a compact Russian-English guardrail and evaluates it against a broad guardrail suite with confirmation and overlap audits. Its strongest results occur on in-domain Russian tests, while English, strong obfuscation, and response-side precision remain weaker.

  • Conclusion: 0.6B HiveTraceGuard-Pro performs request and response moderation in Russian and English, with weights released under Apache-2.0.The evaluation compares thirty-five guards across nineteen benchmark groups and includes a reserved confirmation set and training-overlap audit.
  • Conclusion: 0.7432 aggregate key trails the 8B guard at 0.7641 and 2B guard at 0.7552; on sixteen public groups, its key is 0.7153 with four guards ahead.These results come from the reported suite ordering.
  • Conclusion: 14.3 ms is the lowest median latency in the comparison, while unsafe-class F1 is at least 0.90 in 25 of 32 reported category cells.The latency margin over the next model is 0.41 ms, although absolute latency depends on serving setup.
  • Conclusion: The strongest results are on in-domain Russian tests, with weaker performance under strong obfuscation, on English general-moderation benchmarks, and in response-side precision.Three guards score higher on augmented requests and three on the legacy response evaluation.

8 Model Availability

The merged HiveTraceGuard-Pro weights are publicly available under Apache-2.0, while the corpus, evaluation datasets, and evaluation code remain internal. Reported latency and memory figures are tied to particular serving and context configurations.

  • Release: The single merged model is released on Hugging Face under Apache-2.0, but the training corpus, evaluation datasets, and evaluation code remain internal.The repository documents serialization, verdict-token IDs, truncation behavior, and a reproduction of one reported result.
  • Resource footprint: 1.11 GiB weights and 1.92 GiB allocator peak were measured at a 2048-token context, or 1.39 GiB with logits restricted to the final position.These are measured inference-footprint configurations.
  • Latency: 10.7× faster than Qwen3Guard-Gen-8B was observed at 152.7 ms in the comparison run.The ratio applies only to that run.

A Dataset Construction

The corpus combines licensed public resources, translated and internally curated Russian data, synthetic adversarial examples, and labeled augmentations. It is weighted toward Russian, preserves binary labels across obfuscations, and uses paired benign examples where possible, with several provenance and policy limitations.

  • Sources and Admission: The corpus combines public safety datasets, translated moderation resources, internally curated Russian datasets, and synthetic adversarial examples admitted through family-level license rules.Rows inherit existing labels rather than being re-annotated, and no regex, keyword, or length filters infer safety labels.
  • Limitations: The corpus has no internally measured inter-annotator agreement, so label reliability rests on upstream source annotations.The paper reports agreement statistics for peer corpora but not for this corpus.
  • Synthetic Data Provenance: Labels are fixed by generator cells before generation, while validators independently reject empty outputs, harmful-cell refusals, and other invalid generations.Roleplay wrappers are applied after labels are fixed, and 78 of 84 wrappers appear with both labels.
  • Augmentation: 47 421 rows are mechanical obfuscations and 19 398 rows wrap harmful seeds in jailbreak templates, with inherited source labels.The obfuscation transforms include keyboard swaps, transliteration, symbol changes, leetspeak, zero-width insertion, and phonetic substitution.
  • Policy Scope: The taxonomy encodes one deployment policy rather than a general theory of harm, so operators with different policies must re-label and re-tune.The bundled template supplies the policy and silently drops caller-provided system turns.

A.4 Contamination audit

The contamination audit finds measurable overlap in several evaluation sets and shows that performance changes are usually modest after removal, but Russian prompt-injection recall remains non-independent. The audit itself is limited by language-specific matching and nonuniform removal rules.

  • Audit Scope: The final removal rule varies across assembly stages, and stricter near-match filtering was rejected because it removed too many paired benign Russian examples.Some stages use a 50% threshold, while the dominant stage uses exact whole-message matches of at least 40 characters and audits near matches.
  • Residual Overlap: 27.1% of the Russian prompt-injection set overlaps training, including 200 content-bearing verbatim matches, while Aegis-2.0 has 21.3% overlap.The Russian set’s upstream repository was used as a training source and some rows were reused as obfuscation seeds.
  • Audit Scope: The audit uses word-level n-grams within one language and therefore cannot detect overlap preserved through translation or surface transforms.Measured overlap is 0.07% for the English prompt-injection set, but that does not establish independence from training.
  • Sensitivity: Removing audited Aegis-2.0 exact matches changes prompt-side unsafe-class F1 from 0.8229 to 0.8230, while an aggressive 50%-containment cut yields 0.809.FPR remains between 0.158 and 0.161, and the response result is unchanged.
  • Sensitivity: S-Eval recall falls from 0.806 to 0.784 after removing the contaminated instruction_jailbreak category, which changes task composition by dropping one attack family.The removed category originally scores 0.996, and two YuFeng-XGuard-Reason guards remain ahead.
  • Sensitivity: 0.999 Russian prompt-injection recall changes by less than 0.001 after removal, but it is not an independent generalization estimate because part of the set was seen during training.The attack-only set contains four false negatives.

B.1 Inference contract

The shipped guardrail classifies a selected dialogue turn with a binary safe/unsafe verdict under a fixed policy template. Reported response results use a standalone-reply serialization rather than the natural assistant-role path, and inference behavior has explicit length and serving constraints.

  • Template: The bundled chat template prepends a fixed English policy message, wraps dialogue turns, and appends a Target line naming the turn to judge.Caller-supplied system turns are silently dropped by the template.
  • Response Serialization: Reported response results score each reply as a second user turn after an empty first user turn, omitting the original request.The measured path ends with an assistant generation prefix and does not rely on add_generation_prompt.
  • Response Serialization: Using the natural assistant-role form on Aegis-2.0 changes response macro-F1 from 0.807 to 0.799 and overall macro-F1 from 0.813 to 0.811.This is a sensitivity check on one English benchmark, not a full-suite evaluation of the natural path.
  • Decision Rule: The model selects the larger safe or unsafe logit, equivalent to a fixed 0.5 boundary under a two-way softmax.The rule uses one final-position forward pass and matched greedy one-token generation in evaluation.
  • Length Handling: Forward passes were verified through 40 960 positions, while behavior beyond that range is unspecified; an approximately 80 000-token padded harmful request was labeled safe.The 2048-token value is a training cap, not an inference limit.
  • Length Handling: A length-capping wrapper must shorten the conversation while preserving both the policy block and the target turn with its Target marker.Right-side truncation can remove the target, whereas left-side truncation can remove the policy.

F Additional Results

Additional results show strong Russian performance but lower performance on augmented or English settings, with measurable variation across seeds and transform-specific trade-offs. The robustness-test set contains no English partition.

  • Aggregate Results: 0.898 is macro-F1 on pooled clean and augmented Russian robustness-test rows, compared with 0.866 on augmented rows alone.Russian RU-categories macro-F1 is 0.937, while English nvidia/Aegis-2.0 macro-F1 is 0.813.
  • Seed Variation: 0.938 ± 0.008 is synthetic-clean-subset harm-F1 across three recipe seeds, while robustness-augmented harm-F1 is 0.844 ± 0.011.The released checkpoint is the first recipe run, not a selection from the three.
  • Seed Variation: 0.860 ± 0.008 is robustness-augmented macro-F1, 0.939 ± 0.004 is RU-categories macro-F1, and 0.801 ± 0.010 is Aegis-2.0 macro-F1 across three seeds.These bands measure seed variation rather than evaluation uncertainty.
  • Evaluation Design: The clean request row uses 345 harmful and 4 234 benign examples, whereas each transformed request has 342–345 harmful and 449–450 benign examples.FNR is therefore more directly comparable across conditions than FPR or harm-F1.
  • Evaluation Design: RU-categories reports 16 of 17 topic pairs, while its aggregate includes all 17 pairs and one policy category is not broken out per category.The reported pairs contribute 3 520 of the 3 740 rows per class behind the aggregate.

G Ablations and Development

Development diagnostics report gains that often trade one obfuscation error rate against the other, while the released generative setup outperforms a classification-head variant. These comparisons mix builds and corpora, limiting direct attribution.

  • Evaluation scope: Development diagnostics were run on non-identical builds and were not a controlled ranking of data against optimization strategy.The later additive rounds used a paired held-out no-regression gate.
  • Corpus composition: 0.591 to 0.153: benign-English FPR fell after replacing the baseline corpus composition with a curated source-and-pair mix in one development round.The released checkpoint was later measured at 0.159 on a different set, while the earlier figure plotted only two development compositions.
  • Obfuscation augmentation: 0.836 to 0.735: removing keyboard-layout augmentation reduced development macro-F1 and increased wrong_layout request-FNR from 0.357 to 0.626.The model did not generalize to the unseen Russian↔English layout mapping in that experiment.
  • Obfuscation augmentation: 7 of 8 augmented-benign conditions regressed when assistant-side obfuscation was removed, without improving obfuscated-harm recall.This was a within-round comparison against a two-sided augmentation baseline.
  • Model objective: 0.822 versus 0.898 macro-F1: the sequence-classification head underperformed the released generative model on the same 10 934 held-out requests.This is the only reported ablation using the released corpus and protocol.
  • Development trajectory: 0.061 to 0.125 wrong_layout harm-FNR and 0.358 to 0.262 FPR: rebalancing predecessor augmentation produced mixed changes, while robustness-augmented harm-F1 rose from 0.849 to 0.852.Across six later additive rounds and one objective change, none of the tested mixtures removed the obfuscation trade-off.
Loading 2609.01046v1…