Source-linked AI summary
The Signal in the Noise: An Auditable Reliability Layer for Biomedical Text Classification
Moustafa Yehia Hassan, Sharon Wong, Woh Kai Xuan
TL;DR
Biomedical NLP text can contain extraction artifacts that damage lexical evidence, creating a need for safer preprocessing. The paper presents a deterministic, auditable correction layer with bounded candidate generation, n-gram scoring, abstention, and biomedical safety gates. It reports strong synthetic-error recovery with zero harmful edits and downstream recovery of noise-induced performance degradation while preserving near-clean behavior.
Problem
Automated PDF and OCR artifacts corrupt biomedical text, while indiscriminate correction can rewrite valid domain entities and cause semantic distortions.
Method
The paper develops a deterministic reliability layer combining bounded spell correction, corpus-derived n-gram scoring, explicit abstention, and biomedical safety gates.
Results
94.61% recall on synthetic errors and zero harmful edits on negative controls were achieved, while downstream evaluation recovered approximately 80.45% of noise-induced performance degradation.
Takeaways & Limitations
The system provides a reproducible and auditable preprocessing pathway that prioritizes safe failure and supports deployment through traceability and edit-rate monitoring.
Takeaways & Limitations
The synthetic corruption model may not capture complex real-world PDF parser artifacts, limiting generalizability to highly complex document layouts.
Abstract
from arXiv · showhide
Biomedical NLP pipelines routinely presuppose clean input text, yet large-scale corpora assembled through automated PDF parsing harbour pervasive OCR-like artifacts, token splits and merges, hyphenation remnants, and character-level corruption, that systematically erode lexical evidence and degrade downstream classifiers. We introduce a conservative, fully auditable spell-correction reliability layer conceived as a safety-oriented preprocessing module rather than a maximal-accuracy corrector: under conditions of uncertainty, the system abstains from editing, in accordance with a medical do-no-harm philosophy. The deterministic architecture couples bounded edit-distance candidate generation with corpus-derived n-gram scoring and a suite of biomedical safety gates that protect domain-critical terminology. We evaluate the layer both intrinsically, on a manually curated benchmark of 2,104 token-level cases, and extrinsically, on a tri-class CORD-19 topic classifier (Prevention, Treatment, Epidemiology) spanning 10,000 examples under a principled four-run protocol (Clean, Noisy, Restored, Safety). Intrinsically, the layer attains 94.61% error-fix recall on synthetic errors with zero harmful edits on negative controls. Downstream, it recovers approximately 80.45% of the noise-induced macro-F1 degradation, elevating macro-F1 from 0.7654 (Noisy) to 0.7717 (Restored) while preserving near-clean performance (Safety: 0.7721). A supplementary case study on 103 real-world OCR-extracted abstracts classified with BioBERT confirms that transformer encoders appeared relatively robust to mild noise, motivating a future grey-box architecture that integrates bounded neural signals and UMLS lexicons without compromising auditability. The system is fully deterministic, artifact-driven, and designed with deployment and auditability in mind.
I. INTRODUCTION
The paper introduces a conservative biomedical text-correction layer designed to improve robustness while prioritizing abstention, safety, and auditability.
- Biomedical NLP pipelines face OCR and PDF-extraction artifacts that corrupt lexical evidence, especially in sparse biomedical terminology.
- The reliability layer uses bounded spell correction with explicit margin-based abstention and biomedical safety gates.
- The deterministic implementation persists vocabulary, n-gram counts, and configuration while logging token-level decision traces.
- The four-run Clean, Noisy, Restored, Safety protocol measures both robustness recovery and potential harm on pristine inputs.
B. Extraction noise and robustness
The paper treats OCR and PDF-extraction artifacts as a robustness problem for biomedical NLP and proposes conservative, auditable correction that edits only when evidence is sufficient.
- OCR and PDF-extraction artifacts can corrupt sparse biomedical terminology and degrade downstream NLP performance.
- Candidates are ranked with unigram and contextual bigram evidence, and edits require a margin over the original token.A stricter margin δ_rw is applied to tokens already present in the vocabulary.
- Biomedical safety gates protect short tokens, identifiers, abbreviations, numeric-heavy tokens, and in-vocabulary words from harmful rewrites.Gate activations are recorded as transparent rules in the decision trace.
- Every candidate decision is recorded with its context, candidates, scores, margin, threshold, gates, and apply-or-abstain outcome.These traces support post-hoc inspection, error analysis, and reproducibility.
IV. IMPLEMENTATION AND REPRODUCIBILITY
The implementation separates persisted learned artifacts from runtime correction logic, enabling deterministic behavior, audit traces, and controlled deployment monitoring.
- IV. IMPLEMENTATION AND REPRODUCIBILITY: The system separates vocabulary and n-gram artifacts from tokenization, gates, scoring, and trace emission.Given identical artifacts and text, the system produces identical corrections and audit traces.
- IV. IMPLEMENTATION AND REPRODUCIBILITY: Parameters are selected by constrained grid search to maximize recall and downstream recovery subject to zero harmful edits on negative cases.
- IV. IMPLEMENTATION AND REPRODUCIBILITY: Deployment guidance includes source-selective activation, document-level edit-rate logging, abnormal-rate alerts, raw-text retention, and artifact versioning.An audit UI supports iterative rule adjustment, artifact rebuilding, and rerunning the four-run protocol.
V. EXPERIMENTAL SETUP
The experiments use CORD-19 title–abstract data with interpretable weak supervision, controlled synthetic OCR-like corruption, and a four-run protocol measuring recovery and clean-text safety.
- Data: CORD-19 title–abstract pairs are mapped to Prevention, Treatment, and Epidemiology using weighted keyword evidence and conservative tie handling.Low-evidence cases are excluded rather than forced, and the resulting labels are not claimed equivalent to expert ground truth.
- Corruption model: Synthetic corruption injects character substitutions, deletions, and insertions with weights 0.50, 0.25, and 0.25, respectively.The model provides a consistent controlled stress test but is not claimed to perfectly replicate real PDF parsers.
- Four-run protocol and metrics: The four-run protocol evaluates Clean, Noisy, Restored, and Safety conditions using a TF–IDF plus logistic-regression classifier trained on clean text.
- Four-run protocol and metrics: Macro-averaged F1 is reported alongside Error Reduction Rate, which measures the fraction of noise-induced degradation recovered by the layer.
- Four-run protocol and metrics: Bootstrap 95% confidence intervals for ΔF1 are computed over 5,000 resampling iterations to assess statistical stability.
VI. RESULTS
The reliability layer achieves high intrinsic correction recall without harmful edits on negative controls, while the downstream evaluation measures macro-F1 across the four-run protocol.
- A. Intrinsic correction quality: 94.61% recall on synthetic errors was achieved with zero harmful edits on negative controls.This result operationalizes the conservative operating point in which safety takes precedence over maximal recall.
- B. Downstream topic classification: Macro-F1 scores are reported across the four-run evaluation protocol for downstream topic classification.The supplied passages identify the four-run macro-F1 evaluation but do not include the individual score values.
C. Qualitative correction examples
Representative cases show that the layer applies strongly supported corrections while abstaining when biomedical identifiers are protected or evidence is insufficient.
- C. Qualitative correction examples: Examples include correcting misspellings, repairing a split biomedical term, and abstaining on the protected identifier “IL-6”.The cases illustrate unigram and contextual evidence, pattern-gate behavior, biomedical-identifier protection, and asymmetric-risk abstention.
VII. ABLATIONS
Ablations identify contextual bigram evidence and margin-based abstention as indispensable components of the reliability layer.
- VII. ABLATIONS: Removing bigram context elevates the rate of false-positive edits.The ablation supports retaining contextual evidence in candidate disambiguation.
- VII. ABLATIONS: Relaxing the abstention threshold improves recall but can cause harmful rewrites of abbreviations and biomedical identifiers.The trade-off motivates conservative margin-based abstention and domain-specific protection.
- VII. ABLATIONS: Safety gates preserve accuracy on Clean inputs with negligible overhead.The gates provide a computationally inexpensive safeguard against risky edits.
VIII. DISCUSSION
The layer is designed for auditable, safe failure through deterministic traces and clean-text evaluation, while its scope is limited by weak labels, synthetic noise, and shallow count-based semantics.
- A. Interpretability and audit trails: Deterministic rules and logged scores let practitioners trace corrections to candidates, margins, and gate activations.This supports post-hoc inspection, reproducible debugging, and iterative refinement of conservative rules.
- B. Do-no-harm behavior: The Safety run yields near-clean macro-F1 with a minimal drop and aligns with zero harmful edits on negative controls.The paper presents this as evidence that the conservative policy achieves its primary safety objective.
- C. Threats to validity: Weakly supervised labels are imperfect, and synthetic corruption may not capture layout-induced token merges or complex table-extraction errors.These constraints limit interpretation of generalizability to highly complex document layouts.
- D. Limitations and future work: Count-based contextual evidence may be sparse for rare biomedical entities, creating semantic shallowness on long-tail terminology.The proposed future direction is a grey-box architecture that retains auditability while using bounded neural assistance.
- D. Limitations and future work: The proposed hybrid framework keeps the deterministic core as the primary decision engine while admitting bounded neural signals and UMLS-based protective shields.It prioritizes abstention over potentially harmful edits while addressing shallow context models.
E. Performance with Neural Encoders (Case Study)
The BioBERT case study found only marginal or statistically indistinguishable differences across pipelines, suggesting robustness to mild extraction noise and greatest benefit for lexical or hybrid systems.
- BioBERT appeared intrinsically robust to mild extraction artifacts through WordPiece tokenization and contextual representations.Hyphenation fragments, single-character substitutions, and minor merges could be decomposed into semantically coherent sub-units.
- The null result suggests the reliability layer provides greatest benefit in lexical or hybrid pipelines.In fully neural settings, the layer may still serve as a lightweight sanity-check preprocessing step.
- Corpus-derived scoring priors should be rebuilt for each target corpus rather than reused blindly across biomedical subdomains.A mismatched vocabulary prior may produce systematic abstention on valid terms and signal the need for domain-specific artifacts or external lexicons.
G. Safety considerations
The paper frames the layer as a safety-oriented preprocessing component that prioritizes avoiding harmful edits, uses abstention as a fallback, and supports deployment through deterministic auditing and monitoring.
- The system is a preprocessing component for reducing text-pipeline brittleness, not a clinical decision-making system.Its design assumes harmful edits are costlier than missed fixes.
- Approximately 80.45% of OCR-like noise-induced performance degradation was recovered while near-clean behavior and zero harmful edits on negative controls were preserved.The reported system is deterministic and artifact-driven, separating learned artifacts from runtime logic.
- The layer supports deployment through edit-rate monitoring, raw-text retention, and artifact versioning.These practices accompany exact reproducibility and preserve a clear boundary between learned artifacts and runtime logic.
- A grey-box roadmap would add UMLS lexicons and bounded neural signals without compromising auditability.The supplementary BioBERT case study motivates this direction by suggesting relative robustness to mild noise.
- The source code, persistent artifacts, and experimental scripts are openly available under an MIT license.The repository is identified in the paper as the project’s reproducibility resource.