Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment

1 Introduction

• FEEDBACKQA addresses the lack of an RQA dataset with interactive feedback, collecting ratings and natural-language explanations about answer correctness.Users provide feedback after interacting with a deployed QA system.Source: §1 · p. 1
• The paper investigates whether interactive feedback can improve RQA answer accuracy and generate explanations that help people distinguish correct from incorrect answers.The authors focus on post-deployment interaction, an underused source of user data.Source: §1 · pp. 1–2
• Feedback improves the deployed QA model and stronger models without collected feedback, while generated explanations help users make informed, accurate accept-or-reject decisions.The paper reports these outcomes for retrieval-based QA systems.Source: §1 · p. 2
• The authors train models on feedback to rerank original answers and provide explanations for them.Feedback includes structured ratings and unstructured natural-language explanations.Source: §1 · pp. 1–2

2 FEEDBACKQA Dataset

• Unlike explanation datasets based on ground-truth predictions sampled from training data, FEEDBACKQA collects user feedback after deployment for both correct and incorrect predictions.The paper contrasts its interactive feedback with prior NLU datasets that lack user-system interaction.Source: §2 · p. 2

2.1 Dataset collection

• The two-stage collection trains and deploys an RQA model, then shows crowdworkers questions and top-two predictions for feedback on each question-answer pair.Workers provide a rating and a natural-language explanation; Canada receives no collected feedback for the out-of-distribution test.Source: §2.1 · pp. 2–3

2.2 FEEDBACKQA analysis

• Feedback explanations can review answer quality and identify useful, irrelevant, or missing content; their patterns vary with rating labels.Explanations for excellent or acceptable answers predominantly identify useful content, while bad-answer explanations often flag incorrect or missing content.Source: §2.2 · p. 3

3 Experimental Setup

• Feedback-enhanced RQA models use feedback ratings to score and rerank answer candidates, with explanations generated for human review.The reranker is trained separately from the RQA model and can be reused across RQA models.Source: §3 · p. 4, §3.2 · p. 5

3.1 RQA Models (Pre-deployment)

• Dense passage retrievers rank candidate passages using question–passage similarity, with BERT or BART embeddings and question-sensitive poly-encoder representations.Training maximizes the log-likelihood of the correct answer, using in-batch negatives to avoid computing over all passages.Source: §3.1 · p. 4

3.2 Feedback-enhanced RQA models (Post-deployment)

• The post-deployment reranker uses BART to generate explanations and predict ratings from concatenated question and answer inputs.For reranking, it scores the top five candidates by summing the RQA score and reranker score.Source: §3.2 · p. 5

4.1 RQ1: Does feedback data improve the base RQA model?

• 1.84 points: feedback-trained reranking significantly improves the deployed BERT RQA model, while combining feedback and pre-deployment signals performs best.Performance also improves on Canada, where no feedback was collected. The reranker comparison includes a pre-deployment-trained model and a combined model.Source: §4.1 · pp. 5–6, Table 4 · p. 6

4.2 RQ2: Does feedback data improve the accuracy of RQA models that are stronger than the base model?

• Feedback reranking improves stronger BART RQA too; feedback-trained reranking beats vanilla reranking despite fewer samples, and the combined reranker performs best.The feedback data was not collected for the BART model.Source: Table 5 · p. 6, §4.2 · p. 6

4.3 RQ3: Do explanations aid humans in discerning between correct and incorrect answers?

• Human-written explanations had the highest utility and agreement; feedback-generated explanations exceeded summaries on both measures, while summaries lowered agreement.All explanation types improved answer-judgment accuracy over no explanation.Source: Table 6 · p. 7, §4.3 · p. 8
• The human evaluation tested 60 feedback samples across four explanation conditions, with two workers per condition rating answer quality.It measured judgment accuracy against the original annotation and inter-worker agreement using Spearman correlation.Source: §4.3 · p. 7

5 Related work

• Prior retrieval-based QA work targets pre-deployment improvements; this paper instead studies architecture-agnostic methods using post-deployment interactive feedback.Earlier approaches modify neural architectures, add external knowledge, or change retrieval strategies.Source: §5 · p. 8
• Unlike earlier chatbot and classification studies using automatically generated feedback, this work collects natural feedback from users for retrieval QA (Li et al., 2016; Weston, 2016; Campos et al., 2020).Related chatbot work collected suggested responses from users.Source: §5 · p. 8
• FEEDBACKQA combines positive and negative examples, structured and unstructured feedback, and post-deployment collection; it tests whether explanations improve system utility.This contrasts with prior datasets focused mainly on positive examples, one feedback form, or helping users interpret model decisions.Source: §5 · p. 8

6 Conclusion

• FEEDBACKQA contains interactive user feedback in the form of ratings and natural-language explanations.The paper introduces the dataset for retrieval-based question answering.Source: §6 · p. 8
• The proposed method trains a reranker to select answer candidates and generate explanations from feedback data.Source: §6 · p. 8
• Feedback-based reranking improves the deployed model and stronger models without collected feedback, while human-written and generated explanations help users judge whether to accept answers.Source: §6 · p. 8

7 Limitations and Ethical consideration

• Feedback-based reranker training and inference increase computational-resource use.The paper identifies this as a cost of the approach.Source: §7 · p. 9
• The feedback collection setup is a simulation of a deployed model, not collection from a real-world system.The paper distinguishes the study setup from real-world feedback conditions.Source: §7 · p. 9
• Real-world feedback may contain sensitive information that requires careful handling.The paper flags privacy-sensitive information as a consideration for deployment.Source: §7 · p. 9
• Real-world feedback may be noisy and vulnerable to adversarial attacks.These are identified as risks of feedback outside the simulated collection setup.Source: §7 · p. 9

A Details of Data Collection

• For pages without explicit questions and answers, the team extracts text as passages for question collection.Pages with FAQ questions and answers are collected directly.Source: A Details of Data Collection · p. 11
• Crowdworkers recruited from English-speaking countries write questions conditioned on passages and are instructed to avoid generic or copied questions.Workers are recruited through Amazon MTurk.Source: A Details of Data Collection · p. 11
• Question writers pass a two-part qualification test, have their questions manually reviewed and scored, and receive $0.20 per question.The test assesses question quality judgments and question writing from a passage.Source: A Details of Data Collection · p. 11

B Details of Feedback Collection

• Feedback workers provide ratings and natural-language feedback for question-answer pairs, after qualifying on rating-label accuracy, and receive $0.40 per feedback.The feedback qualification uses labeled question-answer pairs.Source: B Details of Feedback Collection · p. 11

C Details of Human Evaluation

• A worker is assigned to rate a given question-answer pair only once, reducing the risk of repeated identical judgments.The assignment rule is intended to prevent workers from blindly reusing the same judgment for a QA pair.Source: C Details of Human Evaluation · p. 11
• Workers qualify through a test similar to the feedback-collection test and are filtered if they fail to identify randomly sampled dummy answer candidates.Workers receive $0.30 per QA pair.Source: C Details of Human Evaluation · p. 11

D Implementation Details

• Experiments use four 32-GB Nvidia Tesla V100 GPUs.All experiment runs finished within 20 hours.Source: D Implementation Details · p. 11
• Table 8 presents hyperparameter settings for QA-model variants, EXPLAINRATE, and RATEONLY, with no pooling operation in the latter two models.Source: Table 8 · p. 11
• FEEDBACKRERANKER uses standard BART fine-tuning hyperparameters, batch size 16, 40 epochs, and truncation limits of 50 question tokens and 512 passage tokens.The best hyperparameter variant is averaged across three runs after a five-trial search.Source: D Implementation Details · p. 11

• Bi-encoder performance is consistently worse than Poly-encoder performance.Source: E Validation performance · p. 11
• Table 9 reports validation accuracy for different RQA models, averaged across three runs.Source: Table 9 · p. 12
• Table 10 reports validation accuracy for PIPELINE models trained with different feedback data, averaged across three runs.Source: Table 10 · p. 12