perturber: A Stateless Service for Semantics-Preserving Text Perturbations

1 Introduction

• Clean-text evaluations can overstate robustness and even rank a more fragile model above a more stable one when meaning-preserving input changes move answers.Real inputs include typos, inconsistent casing, ad hoc formatting, and reworded phrasing.Source: §1 · p. 1
• Existing tools do not provide a standalone transform layer that applies perturbations one at a time, reproducibly, and identically across library, shell, and network use.BrittleBench couples perturbations to model inference and scoring, while study-specific reimplementations make robustness numbers less directly comparable.Source: §1 · p. 1
• perturber provides a stateless service and Python library with a first-party catalog of 68 perturbations across 11 families.Its four contributions include standardized definitions, a single-primitive design, an introspectable catalog, and a reproducible fidelity harness that measures meaning-preservation per perturbation.Source: §1 · p. 1

2 System Overview

• The deterministic primitive applies one perturbation per call and is a pure function of text, name, parameters, and seed, reproducing output exactly.It reseeds pseudo-random generators under a lock, defaults the seed to zero, and excludes wall-clock or process randomness; the model-backed tier is non-deterministic and excluded from determinism and latency claims.Source: §2 · p. 2
• The library, CLI, and HTTP service forward through the same primitive, so chained calls produce byte-identical output across interfaces.The library serves in-process harnesses, the CLI reads arguments or standard input, and the HTTP service provides typed routes.Source: §2 · p. 2, Figure 1 · p. 2
• A single registry records each perturbation's provenance, scope, family, typed parameters, seed-sensitivity, and optional system prompt, generating all three interfaces.This registry is the boundary between the uniform primitive and the individual perturbation definitions.Source: §2 · p. 2
• The catalog can be enumerated and filtered by a harness, and the stateless service runs from one command without configuration or data-migration burden.Benchmark selection remains with the downstream harness rather than being fixed by perturber.Source: §2 · p. 2

3 The Perturbation Catalog

• The first-party catalog contains 68 perturbations across 11 families, each changing a human-readable surface form while moving the token sequence consumed by the model.The families are organized by the surface they act on and the token-level mechanism by which they can change model output.Source: §3 · p. 2, Table 1 · p. 3
• The optional BrittleBench compatibility tier wraps ten external perturbations, each of which also has a first-party native implementation.The two implementations can be compared byte-for-byte for a given seed, providing a check against an independent reference.Source: §3 · pp. 2–3
• The public release ships 68 perturbations; persona_instruction is withheld because its PersonaHub dependency is licensed for research use only.The restriction is flagged in the module, documentation, and playground.Source: §3 · p. 2
• Sixteen model-backed rewrites span the paraphrasing, context, and reasoning families, but are best-effort and non-deterministic.They are fenced off from the determinism contract.Source: §3 · p. 3
• Downstream harnesses select benchmarks by querying and filtering the catalog, which ships without a fixed benchmark manifest.Catalog scope distinguishes natural meaning-preserving surface changes from entries that inject hidden instructions or otherwise change the task; out-of-scope entries remain callable for defensive testing but are filtered from robustness harnesses.Source: §3 · p. 3

4 Evaluation

• The evaluation measures tool correctness, transport, latency, and fidelity; only the fidelity stage calls a judge, and the other stages run no model inference.The authors reserve perturbation-induced model brittleness measurements for a separate ongoing study.Source: §4 · p. 3

4.1 Correctness, Parity, and Cost

• 378 library-versus-HTTP comparisons are byte-identical; composition remains caller-side and order-dependent, and the deterministic tier’s median is 23.8 microseconds per call.Every family is under 100 microseconds, excluding model-backed rewrites and the optional external engine.Source: Figure 2 · p. 4, §4.1 · p. 4
• All 63 deterministic perturbations reproduce identical outputs over 20 repeats across five seeds, behave correctly on empty input, and match their declared seed sensitivity.The 16 model-backed perturbations are excluded because they are non-deterministic by design.Source: §4.1 · p. 4

4.2 Validating Semantics-Preservation

• Normalization and padding score 1.00, while word_level scores 0.74; surface-adversarial and prepend-a-frame context families are low by design, while schema is low because only one or two corpus inputs are JSON.Out-of-scope, content-changing perturbations score a mean same-meaning rate of 0.15, versus 0.67 for the in-scope mean.Source: §4.2 · p. 5
• Embedding cosine tracks judge rates (Spearman ρ = 0.57), but edit distance is nearly orthogonal (ρ=−0.37), supporting its independent surface-change role.The judge scores 313 pairs at temperature zero with three samples per pair; median edit distance is 0.13 for character perturbations and reaches 0.52 for model-backed rewrites, while judges still read them as the same questions.Source: Figure 4 · p. 4, §4.2 · p. 4

4.3 Meaning-Preserving Edits Can Flip Answers

• 12 ARC-Challenge examples flip correct answers to wrong after plausible, independently judged meaning-equivalent stem edits, with choices and gold answers fixed.The examples come from an ongoing robustness study and are presented as a preview, not a full robustness evaluation.Source: §4.3 · p. 5
• Near-zero net accuracy can hide item-level instability: confusable_words changes around fifty answers per model despite netting near zero.Some items break while others are accidentally repaired, so mean accuracy alone can make the edit appear ineffective.Source: §4.3 · p. 5

• The playground accepts text and a selected perturbation, then displays the transformed result with per-item timing.Source: §5 · p. 5
• Toggling the seed demonstrates determinism on the deterministic tier, while a model-backed rewrite shows the best-effort tier.Source: §5 · p. 5
• Chaining two perturbations makes order-dependence and transport identity visible, and the catalog lists family, seed sensitivity, parameters, and in-scope status for benchmark selection.The same catalog view is available programmatically through GET /perturbations.Source: §5 · p. 5

• Table 2 presents one in-scope example per family: a median-surface meaning-preserving input/output pair with a judge verdict, with strings clipped.Source: Table 2 · p. 6
• GeCo applies seeded, parameter-controlled corruption to personal records for entity resolution, whereas perturber varies LLM prompts while aiming to preserve meaning.Source: §6 · p. 6
• Prior prompt-robustness studies report formatting effects, unreliable single-prompt evaluation, adversarial degradation, and broad brittleness benchmarks (Sclar et al., 2024; Mizrahi et al., 2024; Zhu et al., 2023).Perturber complements these findings papers with a reusable transform layer.Source: §6 · p. 6
• Related tools include multimodal and textual augmenters, adversarial transformation frameworks, community transformations, behavioral suites, contrast sets, and Polyjuice (Papakipos and Bitton, 2022; Ma, 2019; Morris et al., 2020; Jin et al., 2020; Dhole et al., 2021; Ribeiro et al., 2020; Gardner et al., 2020; Wu et al., 2021).Source: §6 · p. 6
• Perturber distinguishes itself through caller-side chaining of one primitive, provenance tracking, seeded determinism and transport identity, schema perturbations, and live-service delivery.Source: §6 · p. 6

• Perturber applies exactly one semantics-preserving perturbation per call through a stateless service and Python library.Source: §7 · p. 6
• Its single-primitive design makes library and service outputs byte-identical, while callers compose perturbations by chaining calls.Source: §7 · p. 6
• The first-party catalog contains 68 perturbations, allowing downstream harnesses to select their own benchmarks.Source: §7 · p. 6
• Naming, scoping, and typing each transform once makes it a citable object that robustness studies can share.Source: §7 · p. 6

• Semantics-preservation differs across perturbation families, so the paper claims no single threshold that applies to all of them.Character edits and paraphrases preserve meaning in different senses, which the reported fidelity metrics distinguish.Source: Limitations · p. 7
• Fidelity metrics use a hand-curated smoke corpus of six inputs, demonstrating the mechanism but not providing a validated per-perturbation leaderboard.Source: Limitations · p. 7
• The model-backed tier is best-effort and non-deterministic, and is excluded from determinism and latency claims; reported latency is host-specific.Source: Limitations · p. 7
• Eight perturbations are outside the project’s robustness-perturbation definition but remain callable and flagged for harness exclusion.Source: Limitations · p. 7

Ethics Statement

• persona_instruction is withheld from the public release because its PersonaHub source is licensed for research use only.The restriction is flagged in the module, documentation, and playground.Source: Ethics Statement · p. 7
• The catalog includes prompt-injection probes for defensive evaluation.Examples include invisible_instructions and unicode_tag_smuggle.Source: Ethics Statement · p. 7
• Prompt-injection probes are flagged out of scope and hidden from the playground’s default view.Source: Ethics Statement · p. 7

A Reproducing the Evaluation

• Running python eval/run_all.py regenerates the Section 4 statistics, correctness and determinism matrix, transport comparison, latency table, and fidelity metrics as machine-readable artifacts.Correctness, transport, and latency stages run without model inference; only fidelity calls an embedding model and an LLM judge.Source: A Reproducing the Evaluation · p. 8

B The Full Catalog

• Tables 3 through 12 list the in-scope catalog by family and name, describing each transformation and how it can change a model’s output.Source: B The Full Catalog · p. 8
• Ten catalog behaviors are also available as BrittleBench variants, while the eight out-of-scope perturbations are omitted from the tables.The omissions match the playground.Source: B The Full Catalog · p. 8

C All Twelve Answer Flips

• All twelve listed cases are plausible edits in which a correct answer turns wrong, and an independent judge rated every pair meaning-equivalent.The table gives three cases per perturbation; only the stem is perturbed, with choices and gold held fixed.Source: C All Twelve Answer Flips · p. 8, Table 13 · p. 14
• −2.30 pp for realistic_typos, −1.88 pp for repeated_chars, −0.34 pp for extra_whitespace, and −0.85 pp for confusable_words (mean ∆acc over 1,172 ARC-Challenge items).Source: Table 13 · p. 14
• A doubled space can lengthen the token sequence while a confusable-word swap can flip the answer at an unchanged token count.The examples show that token-count growth is not required for a flip.Source: Table 13 · p. 14