Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

1 Introduction

• Comparison questions require inferential reasoning across context, making large-scale annotation challenging and leaving these datasets smaller than standard MRC datasets.Source: §1 · p. 1
• Models can exploit statistical patterns or annotation artifacts in comparison datasets, while QA consistency had not been addressed despite inconsistent model predictions.Source: §1 · p. 1
• The method augments labeled data with logical and linguistic knowledge, then applies symbolic-logic consistency regularization as additional supervision.Source: §1 · p. 1
• The approach is designed for varied question formats and improves results across causal classification, qualitative multiple-choice, and extractive comparison MRC tasks.Source: §1 · p. 2
• RoBERTa-based models improve by 1.0% on WIQA, 5.0% on QuaRel, and 2.5% on HotpotQA; the approach advances state of the art by 4.7% and 8.4% on WIQA and QuaRel.With 20% of the original labeled data, the method matches a competitive baseline trained on the full data.Source: §1 · p. 2

2 Related Work

• NLP augmentation has used backtranslation and dictionary-based word replacement, while Kang et al. (2018) study logic- and knowledge-based augmentation specifically for NLI.Source: §2 · p. 2
• The augmentation does not rely on task-specific assumptions and is presented as adaptable to different QA formats.Source: §2 · p. 2
• Concurrent visual-QA work studies logically composed questions and reports effective logic-guided augmentation.Source: §2 · p. 2
• Unlike prior NLI consistency methods that are model-dependent or rely on NLI-specific rules, this approach is model-agnostic and combines regularization with data augmentation.Source: §2 · p. 2
• Prior structured-learning work regularizes output constraints in structured prediction models, whereas this work targets global consistency among logically constrained augmented examples.Source: §2 · p. 2

3.1 Consistent Question Answering

• Symmetric consistency pairs questions with antonym or negation changes whose answers should be opposites; transitive consistency links cause-effect questions through a shared event.The method describes transitive examples for causal reasoning, where one question’s effect is the next question’s cause.Source: §3.1 · p. 2

3.2 Logic-guided Data Augmentation

• The method augments training data with examples generated by symmetry and transitivity rules so both kinds of cases appear during training.Table 1 illustrates transitive WIQA examples and symmetric examples for QuaRel and HotpotQA, marking shared events, antonyms, and added negation.Source: §3.2 · p. 3, Table 1 · p. 3
• Symmetric examples are generated by replacing polarity words with antonyms or adding or removing negation, then relabeling the answer as its opposite.The antonym dictionary is built from frequent polarity adjectives and verbs, with antonyms supplied by expert annotators.Source: §3.2 · p. 3
• In WIQA and QuaRel, antonym replacement is more effective than adding or removing words, while adding words contributes as much as antonym replacement on HotpotQA.The cited comparison concerns the relative effectiveness of augmentation operations.Source: §3.2 · p. 3
• Augmented examples are randomly sampled once at training start and reused across epochs; regularization is switched on after τ epochs to allow initial predictions to improve.The method notes that adding all consistent examples may alter the data distribution, and anneals the consistency weights from zero.Source: §3.2 · p. 3, §3.3 · p. 4

3 Method

• The method combines data augmentation with a regularizer that penalizes predictions violating symmetric and transitive rules.The task loss penalizes incorrect predictions, while the consistency loss adds supervision for rule agreement.Source: §3.1 · p. 2, §3.3 · p. 3

3.3 Logic-guided Consistency Regularization

• The consistency regularizer uses paired augmented examples to adapt consistency training to QA, where premise–hypothesis swapping from prior NLI methods does not directly apply.This pairing enables the approach to be applied across a variety of QA tasks.Source: §3.3 · p. 4

Main Results

• DA and DA + Reg improve RoBERTa QA performance by 1–5 points on all three datasets, advancing state-of-the-art scores on WIQA and QuaRel by 4.7% and 8.4%, respectively.The three datasets are WIQA, QuaRel, and HotpotQA.Source: §4 · p. 4
• On WIQA, the approach reduces violations of symmetric consistency from 13.9% to 8.3% and of transitive consistency from 10.0% to 2.5%, respectively.Source: §4 · p. 5

Results with limited training data

• When only 20% of labeled data is available, DA and Reg together yield more than 12% and 14% absolute accuracy improvements over RoBERTa baselines on WIQA and QuaRel, respectively.Source: §4 · p. 5

Ablation study

• Data augmentation and regularization each improve baselines, and their combination performs best on WIQA and QuaRel; standard WordNet augmentation alone gives no notable accuracy or consistency improvement.Regularization alone improves WIQA less than the combination with data augmentation.Source: Table 3 · p. 4, §4 · p. 5

Qualitative Analysis

• Qualitative examples show DA + Reg reducing confusion between opposite choices and assigning greater probability to ground-truth labels when DA alone yields small probability differences.On a HotpotQA example, RoBERTa selects the same answer for a question and its symmetric counterpart, whereas DA answers both correctly.Source: §4 · p. 5, Table 4 · p. 5

• The framework combines logic-guided data augmentation and consistency-based regularization for accurate, globally consistent QA, especially with limited training data.Source: §5 · p. 5
• The approach significantly improves state-of-the-art models across three substantially different QA datasets.Source: §5 · p. 5
• The method advances state-of-the-art results on QuaRel and WIQA and learns effectively from extremely limited training data.Source: §5 · p. 5

• Annotators selected polarity-bearing words among the 500 most frequent verbs, 50 verb phrases, and 500 adjectives sampled from WIQA and QuaRel training data.Source: A Details of Human Annotations · p. 7
• Human annotators supplied antonyms for selected verbs and adjectives, producing 64 antonym pairs mined from a comparison QA dataset.Source: A Details of Human Annotations · p. 7
• The same antonym dictionary was reused for all three datasets.Table 5 presents ten annotated antonym examples.Source: A Details of Human Annotations · p. 7, Table 5 · p. 7

Answer re-labeling in WIQA (symmetric)

• For symmetric WIQA examples, applying one operation to either cause or effect reverses “more” or “less” but leaves “no effects” unchanged; applying an operation to both leaves the answer unchanged.Source: B Details of answer re-labeling on… · p. 7

Answer re-labeling in WIQA (transitive)

• For transitive WIQA examples, if the first causal answer is “more,” the relabeled answer for the combined cause-to-effect question is the second question’s answer.This relabeling assumes the first effect equals the second cause.Source: B Details of answer re-labeling on… · p. 8

Answer re-labeling in HotpotQA

• For HotpotQA, the method extracts two question entities matching paragraph titles and treats them as possible answer candidates when both appear in gold paragraphs.The new symmetric answer is the candidate that differs from the original answer.Source: B Details of answer re-labeling on… · p. 8

C Details of Baseline Models

• WIQA is modeled as classification over the answer classes more, less, and no effects, using the final [CLS] representation of the paragraph-question input.The model predicts probabilities for each class.Source: C Details of Baseline Models · p. 8
• QuaRel is modeled as multiple-choice QA, with each answer choice paired with the paragraph and question in a separate input.The final [CLS] representation is used to predict probabilities for the answer choices.Source: C Details of Baseline Models · p. 8
• HotpotQA uses a RoBERTa span-QA model, without modifications, for questions whose answers can be extracted from the paragraph.The passage notes that some previous HotpotQA studies used modifications.Source: C Details of Baseline Models · p. 8

D Details of Implementations and Experiments

• The classification and span models use PyTorch and pytorch-transformers, while the multiple-choice model uses fairseq.The passage identifies PyTorch as the basis for all implementations.Source: D Details of Implementations and… · p. 8
• HotpotQA models train for six epochs; augmented models use three epochs on original data followed by three with augmented data and regularization.Training includes bridge and comparison questions, while evaluation is on extractable-answer comparison questions.Source: D Details of Implementations and… · p. 8
• WIQA and QuaRel use maximum training durations of 15 and 150 epochs, respectively, with early stopping after five and ten unimproved development epochs.The passage motivates these settings by high early-training variance on the small datasets.Source: D Details of Implementations and… · p. 8
• Table 6 contains ten examples of annotated antonyms for comparison questions.The passage also notes that surface matching can answer without understanding rich linguistic phenomena or world knowledge.Source: Table 6 · p. 9, D Details of Implementations and… · p. 9

E Qualitative Examples on HotpotQA

• RoBERTa produces many consistency violations, and examples show the baseline answering incorrectly where RoBERTa+DA answers correctly.The examples are presented in Table 2.Source: E Qualitative Examples on HotpotQA · p. 9
• The model answers a “more scope of profession” question correctly but fails on its semantically equivalent “less” version.The example compares B. Reeves Eason and Albert S. Rogell.Source: E Qualitative Examples on HotpotQA · p. 9
• The model struggles with negation, predicting the same answer for a question and its negated version about which species is native to Asia.The example concerns Corokia and Rhodotypos scandens.Source: E Qualitative Examples on HotpotQA · p. 9