PaperlayerView exact revision on arXiv

Source-linked AI summary

DualGuard: Dual-Mode Quality Control for Logic-Preserving Data Augmentation

Shenghao Li, Lin Zhao

arXiv:2609.32431v1cs.CL

TL;DR

Logical-reasoning augmentation can expand training data, but generated examples must preserve semantic, label, and logical reliability; after quality is assessed, systems still need to decide whether to retain, filter, or repair candidates. DualGuard uses current-batch control and action-specific cross-instance history, with shared semantic verification and symbolic verification when reliable logical representations are available. In Two-Stage Transfer, it achieves the highest Accuracy on five of seven tasks and outperforms the no-augmentation BERT baseline on all seven.

  • Problem

    After candidate quality is assessed, deciding whether an augmented example should be retained, filtered, or repaired remains an important control problem.

    Source: §1 · p. 1

  • Method

    DualGuard combines current-batch retention, filtering, attribution, and feedback with action-specific historical anomaly control, using semantic verification and symbolic verification when reliable logical forms are available.

    Source: §1 · p. 1, §7 · p. 11

  • Results

    In Two-Stage Transfer, DualGuard achieves the highest Accuracy on five of seven downstream tasks and outperforms the no-augmentation BERT baseline on all seven.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    Controlled ablations indicate complementary roles for Memory, Z3, and historical anomaly control within the framework.

    Source: §7 · p. 11

    Conditional symbolic verification depends on reliably converting natural-language instances into logical representations, a process affected by linguistic complexity and semantic ambiguity.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models provide a practical way to generate augmented data for logical reasoning at scale, but a larger generation volume does not guarantee semantic, label, or logical reliability. Existing work has improved generation quality through generation constraints, candidate validation, filtering, and feedback-based revision; however, once a quality judgment is available, deciding whether a candidate should be retained, filtered, or repaired remains an important control problem. We propose DualGuard, a dual-mode quality-control framework for logic-preserving data augmentation. The first mode uses the current instance and candidate batch for selective retention, filtering, attribution, and targeted feedback. The second mode accumulates cross-instance execution records of augmentation actions on top of per-sample diagnosis and attribution, compares new executions against each action's own historical behavior, and supports retrospective anomaly inspection, targeted rollback, and bounded repair. Both modes share semantic verification and additionally use symbolic verification when a reliable logical form is available. Across seven downstream tasks in the Two-Stage Transfer setting, DualGuard achieves the highest Accuracy on five tasks and outperforms the no-augmentation BERT baseline on all seven. Controlled ablations further show complementary roles for Memory, Z3, and history-aware anomaly control.

1 Introduction

DualGuard addresses the post-validation decision of whether augmented candidates should be retained, filtered, or repaired, using current-batch and cross-instance history signals. In Two-Stage Transfer, it achieves the highest Accuracy on five of seven downstream tasks and exceeds the no-augmentation BERT baseline on all seven.

1 Introduction

  • Logical-reasoning augmentation must preserve natural language quality alongside the inference relation and target label, which small changes can alter.Changes to quantifiers, negation, implication direction, or premise content can alter the reasoning relation or flip the label.Source: §1 · p. 1
  • The framework addresses the post-validation control problem of deciding whether each candidate should be retained, filtered, or repaired.Current-instance batches describe the present generation round, while cross-instance action histories provide behavioral references.Source: §1 · p. 1
  • The current-instance mode uses validation outcomes, candidate scores, risk information, and within-batch stability for selective retention, filtering, attribution, and targeted feedback.Feedback is used when a diagnosable issue is present.Source: §1 · pp. 1–2
  • DualGuard achieves the highest Accuracy on five of seven downstream tasks and exceeds the no-augmentation BERT baseline on all seven in Two-Stage Transfer.Controlled ablations, direct generation-quality evaluation, and process-level analyses examine component behavior.Source: §1 · p. 2
  • For diagnosable failures, DualGuard localizes likely failure sources and routes corrective feedback to the relevant processing node instead of restarting generation.The framework refines post-verification processing into retention, filtering, and repair, with task-configured current-batch or history-aware control.Source: §1 · p. 2
  • The history-aware mode accumulates action-level execution outcomes and diagnostics across samples, then uses each action’s historical behavior as a dynamic reference for anomaly detection.Detected anomalies can prompt targeted rollback and bounded repair.Source: §1 · p. 2

2 Related Work

Prior work spans language-model-based augmentation, logic-aware generation and verification, and feedback or memory for repeated generation. DualGuard focuses on decisions after candidate quality is assessed, distinguishing current-batch information from action-specific cross-instance history.

Language-Model-Based Data Augmentation

  • Language-model augmentation ranges from token perturbations and contextual replacement to instruction-driven, attribute-controlled, and chain-of-thought-guided generation.Examples include EDA, Contextual Augmentation, GPT3Mix, FlipDA, AttrPrompt, CoTAM, AugGPT, and Self-LLMDA.Source: §2 · p. 2

Logic-Aware Augmentation and Verification

  • Logic-aware methods use logical structure or structured semantic representations, while Logical DA combines multi-agent generation, label verification, and feedback-based reflection.Examples include logically related variants, AMR-guided transformations, and explicit logical constraints.Source: §2 · p. 2

Post-generation Feedback and Memory

  • Prior work uses validation signals for rejection, feedback, or revision, and uses feedback or memory to guide later generation and decision making.Examples include Self-Refine, CRITIC, LLM2LLM, R-GDA, Reflexion, ExpeL, and MemoryBank.Source: §2 · pp. 2–3

DualGuard’s Position

  • DualGuard focuses on post-assessment control, using the current candidate batch for retention and filtering and action-specific history for anomaly detection and targeted repair.The history is accumulated on top of per-sample diagnosis and attribution.Source: §2 · p. 3

3 Method

DualGuard preconfigures either current-batch or history-aware quality control over a shared generation and validation workflow. Difficult Mode uses the present batch, while Variance Mode also uses action-level execution history.

3.1 Overall Framework

  • The two modes are task-level alternatives sharing generation and validation, not sequential stages every candidate must pass through.Difficult Mode relies mainly on the current instance and batch; Variance Mode additionally uses cross-instance action history.Source: §3.1 · p. 3

3.2 Shared Candidate Validation

  • Both modes validate semantic and label consistency, enabling Z3 symbolic checks only when the input has a reliable logical representation.Candidates that require Z3 but fail its symbolic gate are discarded; the validation stage also produces quality scores and diagnostics.Source: §3.2 · p. 4

3.3 Difficult Mode: Current-Batch Quality Control

  • Difficult Mode evaluates candidates using individual quality, risk, and current-batch signals, retaining reliable candidates and filtering invalid ones.Batch variation is an auxiliary quality signal, not by itself evidence of a logical error.Source: §3.3 · pp. 4–5
  • Explicitly diagnosable failures receive per-candidate attribution, and the system regenerates only missing candidates before a final batch review.Attribution can occur even when batch scores appear stable; the system routes corrective feedback to the likely source node.Source: §3.3 · p. 5

3.4 Variance Mode: History-Aware Quality Control

  • Variance Mode records action-level rewards and diagnostics after per-sample diagnosis, using each action’s own recent behavior as its reference.Its history statistics describe the same action on compatible samples, rather than comparing absolute rewards across different actions.Source: §3.4 · pp. 5–6
  • Action rewards combine Solver confusion with fluency and novelty penalties, while task-specific dimensions and thresholds apply to candidates already meeting basic validity constraints.For a reused action, its prior records are excluded from the similarity reference, and reuse frequency is penalized separately.Source: §3.4 · pp. 5–6
  • A low historical anomaly flags an execution substantially below its action’s recent behavior; it is a routing signal, not proof of logical error.Anomaly judgments wait until enough history is available; new actions initially accumulate rewards and diagnostics without history-based judgments.Source: §3.4 · p. 7
  • For detected anomalies, existing diagnosis and attribution guide targeted rollback and bounded repair, with repaired candidates revalidated before acceptance.A sliding window also supports retrospective inspection of recent executions and their diagnostics.Source: §3.4 · p. 7

4 Experimental Setup

The experiments compare Two-Stage Transfer and Direct Augmentation across logical reasoning and inference tasks, using shared data, generation budgets, and downstream training procedures. They evaluate downstream performance primarily with Accuracy.

Datasets and Task Protocols

  • Two-Stage Transfer trains an intermediate model on augmentations from 150 logical inference seeds, then transfers it to seven downstream tasks.The tasks are FOILO, ProofWriter, LogicQA, ReClor, RTE, CB, and LogDed.Source: §4 · p. 7
  • Direct Augmentation trains each downstream classifier on original plus augmented target-task data, covering CB, RTE, FOILO, and ProofWriter.The augmentation sources contain 48, 100, 150, and 150 original examples, respectively.Source: §4 · p. 7

Baselines and Generation

  • The comparisons use BERT-base without augmentation and four augmentation baselines, with the same original examples, generator, temperature, and generation budget.Candidates failing quality checks are filtered rather than replaced with duplicates or accepted by relaxing quality standards.Source: §4 · pp. 7–8

Training and Evaluation

  • Two-Stage Transfer reports mean Accuracy across three learning-rate configurations, while Direct Augmentation reports mean Accuracy across three fixed cross-validation folds.Methods share task-level random seeds and training settings; Direct Augmentation keeps source examples and derived augmentations within the same partition.Source: §4 · p. 8

5 Results and Analysis

DualGuard leads the Two-Stage Transfer comparison on five of seven tasks and exceeds the no-augmentation BERT baseline on all seven. Direct augmentation, controlled ablations, a sample-quality audit, and process traces examine performance and control behavior under distinct settings.

Two-Stage Transfer Results

  • DualGuard has the highest Accuracy on five of seven Two-Stage Transfer tasks and beats no-augmentation BERT on all seven; its mean gain over BERT is 4.75 points.Gains are 8.34 points on LogDed, 8.15 on ProofWriter, and 5.58 on FOILO.Source: §5 · p. 8, Table 1 · p. 9, Figure 3 · p. 9

Direct Augmentation Results

  • In Direct Augmentation, DualGuard ranks first on ProofWriter (54.67%) and CB (79.17%), ties on FOILO (44.67%), and trails CoTAM and FlipDA on RTE.This protocol reports three-fold mean Accuracy on four target tasks.Source: §5 · p. 8, Table 2 · p. 10

Controlled Ablation

  • Removing Variance lowers Accuracy on all seven tasks by 0.17–4.76 points, averaging 2.74; removing Memory or Z3 yields mean drops of 2.07 and 2.37 points.The ablation patterns indicate complementary roles; the study lacks a standalone w/o Difficult Mode variant to identify its independent Accuracy contribution.Source: §5 · p. 9, Table 3 · p. 10

Generated-Data Quality

  • All 125 audited retained examples pass validity, logic-and-label correctness, and transformation checks; language-quality scores average 4.88–4.96/5 and meaningful-difference scores 3.35–4.10.The small audit describes selected outputs, not all-candidate acceptance or a cross-method quality ranking.Source: §5 · p. 9

Dual-Mode Process Analysis

  • In eight FOILO runs with low-reward outliers, refill candidates had a 91.43% logic-pass rate and 8.57% high-risk rate, versus 50.00% for each measure among outliers.The candidate sets are unpaired, so their mean-reward difference is not a per-candidate repair gain, and this trace does not separately validate End-stage review.Source: §5 · pp. 9–10
  • Across 30 trials initialized from one action-history snapshot but isolated runtime memory, three repairs met the strict criterion; these counts do not estimate ordinary-generation anomaly or repair rates.One additional repair was valid but did not improve reward.Source: §5 · p. 10

6 Limitations and Discussion

DualGuard’s quality-control process adds inference and validation overhead, especially as candidate generation or repair increases. Its symbolic and history-aware controls also depend on reliable formalization and sufficiently representative action histories.

6 Limitations and Discussion

  • Additional inference and validation overhead grows with more candidates or when replenishment and repair are triggered.The process includes multi-candidate generation, shared validation, attribution, replenishment, and possible repair and revalidation.Source: §6 · p. 10
  • Conditional symbolic verification depends on reliable natural-language formalization, which linguistic complexity and semantic ambiguity can affect.When reliable formalization is unavailable, the system skips symbolic verification and retains semantic validation.Source: §6 · p. 10
  • Sparse action histories may not provide stable statistical references, and histories can become less representative when later inputs differ substantially.The system defers anomaly judgments until sufficient history is available; robust cold-start and adaptive strategies remain open directions.Source: §6 · p. 10

7 Conclusion

DualGuard combines current-batch control with action-specific historical anomaly control, using semantic validation and symbolic verification when reliable logical representations are available. In Two-Stage Transfer, it achieves the highest Accuracy on five of seven tasks and beats the no-augmentation BERT baseline on all seven.

7 Conclusion

  • Difficult Mode controls the current instance and candidate batch, while Variance Mode uses action-specific cross-instance history for anomaly detection, rollback, and bounded repair.Both modes share LLM-based semantic validation and enable Z3 verification when a reliable logical representation is available.Source: §7 · p. 11
  • DualGuard achieves the highest Accuracy on five of seven tasks and outperforms the no-augmentation BERT baseline on all seven.Controlled ablations show complementary roles for Memory, Z3, and historical anomaly control.Source: §7 · p. 11
  • Direct Augmentation, generated-data quality evaluation, and process-level analyses provide additional evidence about effectiveness and stability.The process-level analyses cover both modes.Source: §7 · p. 11
  • Future work targets formalization reliability, multi-candidate and multi-round validation overhead, and more robust action-history modeling under sparse history or distribution shifts.Proposed directions include cold-start mechanisms and adaptive update strategies for Variance Mode.Source: §7 · p. 11
Loading 2609.32431v1…