PaperlayerView exact revision on arXiv

Source-linked AI summary

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, Jing Xu

arXiv:2210.15893v1cs.CLcs.AI

TL;DR

Deployed dialogue agents may receive sparse explicit feedback, even as they need to learn from interactions. JUICER uses free-form feedback to label turns and correct replies, and model-corrected training data improves dialogue performance.

  • Problem

    Binary feedback and gold corrections can be sparse in deployment, while free-form feedback may be provided when users express dissatisfaction.

    Source: §1 · p. 1

  • Method

    JUICER trains a satisfaction classifier to label unannotated turns and a reply corrector to turn bad replies into good ones for dialogue-model retraining.

    Source: §1 · p. 1

  • Results

    Model-corrected replies improve overall performance, and the best JUICER models outperform baseline BlenderBot 2 and using DIRECTOR alone.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    JUICER provides a way to use limited organic feedback signals to augment final dialogue-model training when free-form textual feedback is provided.

    Source: §7 · p. 8

    The experiments assume dense free-form feedback after bad replies; its density in real deployments remains uncertain.

    Source: §8 · p. 9

Abstract

from arXiv · show

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

1 Introduction

Deployed dialogue agents need to learn from successes and failures, but binary feedback and gold corrections are often sparse. JUICER uses free-form feedback alongside these signals to label turns, correct bad replies, and retrain dialogue models.

1 Introduction

  • JUICER combines sparse binary feedback and gold corrections with free-form feedback to label unannotated turns, correct bad replies, and retrain dialogue models.The framework trains a satisfaction classifier and reply corrector on existing feedback, then uses their outputs to augment training data.Source: §1 · p. 1
  • The authors construct a sparse sampled version of FITS to evaluate JUICER with conversations involving BlenderBot 2 and SeeKeR.FITS contains fully annotated human-model conversations.Source: §1 · p. 1
  • Free-form textual feedback improves both satisfaction-classifier performance in identifying good and bad responses and reply-corrector performance in generating better corrections.Source: §1 · p. 2
  • The best JUICER models outperform baseline BlenderBot 2 and using DIRECTOR alone, while DIRECTOR uses both good and bad responses to further improve the final dialogue model.The paper also reports that augmenting training with reply-corrector-generated corrections works better than using only existing gold corrections.Source: §1 · p. 2

2 Related Work

Prior work studies human feedback for language models, output correction, and dialogue agents. This paper focuses on improving dialogue agents from binary, textual, and correction feedback.

2 Related Work

3 Human Feedback Setting

The setting combines sparse thumbs-up/down feedback and gold corrections with free-form textual feedback that accompanies mistakes. JUICER uses these signals to label turns, correct poor replies, and retrain the dialogue model.

3 Human Feedback Setting

  • Users may express dissatisfaction through free-form feedback, while thumbs-up/down signals and gold corrections are treated as sparse in deployment.The experimental setting assumes mistakes are followed by textual feedback.Source: §3 · p. 2
  • The setting includes binary thumbs-up/down, conversational explanations of what went wrong, and gold corrections suggesting an alternative reply.The figure illustrates textual feedback about a topic change and a proposed watermelon-related correction.Source: §3 · p. 2, Figure 1 · p. 3
  • JUICER trains a satisfaction classifier and reply corrector, labels unannotated turns, corrects bad replies, then retrains the dialogue model with good and bad replies.Source: Figure 1 · p. 3

4 The JUICER Method

JUICER uses sparse human feedback to label otherwise unlabeled replies, correct bad replies using textual feedback, and retrain the dialogue model with the resulting data. It also filters corrections for correctability and can use DIRECTOR to train with both positive and negative replies.

4.1 Step 1: Train satisfaction classifier and reply corrector on existing feedback

  • The satisfaction classifier uses the next human response as an optional quality cue, while the reply corrector learns from gold corrections or good subsequent replies.The classifier is fine-tuned from a 311M-parameter Reddit-pretrained transformer, and the corrector from a 3B-parameter R2C2 transformer.Source: §4.1 · p. 4

4.2 Step 2: Predict missing labels

  • The satisfaction classifier predicts binary labels for replies that lack human feedback, so each bot reply has a label.Human labels remain for replies that received them.Source: §4.2 · p. 4

4.3 Step 3: Convert lemons to cherries

  • JUICER corrects replies labeled bad, but uses only cases where feedback is judged useful for correction, based on similarity to the next bot reply.It generates 60 candidates and selects the one with the highest satisfaction-classifier score, skipping examples when every candidate is predicted bad.Source: §4.3 · p. 4

4.4 Step 4: Collect fruits and re-train

  • The final model is trained with positive replies and corrections; DIRECTOR also uses bad replies to penalize negative responses.The paper notes that other approaches using both positive and negative replies could replace DIRECTOR.Source: §4.4 · pp. 4–5

5 Experimental Setup

Experiments use BlenderBot 2 as the base dialogue model, evaluating JUICER with sparse feedback on FITS and testing component transfer on DEMO. The setup compares limited-feedback baselines, a prompt-based corrector, oracle methods, and JUICER variants.

5 Experimental Setup

  • BlenderBot 2 (BB2 3B) is the base dialogue model, which the experiments aim to improve using human feedback from deployment.Source: §5 · p. 5

5.1 Datasets: FITS and DEMO

  • FITS and DEMO provide the evaluation settings: FITS is sampled to mimic sparse feedback, while DEMO tests zero-shot transfer to deployment data.Source: §5.1 · p. 5, §5.1.1 · p. 5

5.1.2 DEMO

  • DEMO evaluates zero-shot transfer on 923 crowdworker-verified bot responses from 81 BlenderBot 3 conversations.Source: §5.1 · p. 5, §5.1.2 · p. 5

5.1.1 FITS

  • FITS retains free-form feedback but treats only 20% of training replies as binary- and gold-labeled; the rest are unlabeled.Evaluation uses perplexity, F1 overlap with gold responses, and crowdworker-rated good-response rate.Source: §5.1.1 · p. 5

5.2 Baselines

  • Baselines include fine-tuning on limited human labels, a prompt-based reply corrector, and oracle methods using all labeled feedback.The limited-feedback baselines use gold corrections or free-form textual feedback.Source: §5.2 · p. 5

5.3 JUICER models

  • JUICER augments training with predicted good replies and filtered corrections, while JUICER + DIRECTOR also trains with negative replies.Ablations remove predicted corrections or include all corrections without filtering for correctability.Source: §5.3 · p. 6

6 Results

JUICER improves final dialogue models by extending sparse feedback and using textual feedback to correct bad replies. Its component evaluations support those steps, while final evaluations show gains over unaugmented baselines and results comparable to oracle methods.

6.1 Satisfaction classifier

  • Adding the next human message raised satisfaction-classifier balanced accuracy on FITS from about 75% to about 95%, and zero-shot balanced F1 on DEMO from 64.77 to 71.24.The standard JUICER setup uses the variant that includes the next human response.Source: §6.1 · p. 6

6.2 Reply corrector

  • Reply-corrector performance improved with free-form textual feedback, reaching 23.39 F1 and 2.93 PPL versus 21.41 F1 and 3.07 PPL without the oracle’s 100% gold training data.The reported best results were relatively close to the oracle trained with 100% gold data.Source: §6.2 · pp. 6–7
  • Adding self-correction pairs raised reply-corrector F1 from 17.10 to 21.41; these pairs use good bot replies liked by humans or predicted good by the satisfaction classifier.The reply corrector can also improve through multitasking with dialogue tasks.Source: §6.2 · pp. 6–7

6.3 Final dialogue model evaluations

  • F1 rose from 15.3 to 18.5 on the unseen test set, while good responses rose from 33.2% to 41.9% in human evaluation with JUICER augmentation.JUICER also outperformed baselines without augmentation; Table 2 reports automatic evaluations and Table 3 reports human evaluations.Source: §6.3 · p. 7, Table 2 · p. 8, Table 3 · p. 8
  • JUICER’s supervised reply corrector outperformed a prompted corrector: downstream F1 was 16.7 versus 14.2, and perplexity was 8.5 versus 8.9.All JUICER models also performed better than the prompt-based reply-corrector baseline in automatic evaluations.Source: §6.3 · p. 7
  • Using predicted corrections improved test unseen F1 to 18.5 versus 17.9 without them; selecting about 62% correctable cases also improved valid and unseen test F1 to 18.5 versus 18.0, but not seen-test F1.The selection threshold was chosen on the validation set.Source: §6.3 · pp. 7–8
  • JUICER with DIRECTOR yielded 45.5% good responses versus 41.9% for JUICER alone; DIRECTOR also improved valid and test F1, but not unseen-test F1.Perplexity is not applicable to DIRECTOR because its classifier head alters the generation distribution.Source: §6.3 · p. 8
  • Test unseen F1 was 17.7 for the best JUICER model versus 17.6 for the best 100%-gold oracle method; human ratings were 3.34 versus 3.38.The authors describe these 100%-data methods as an upper bound and conclude JUICER extracts most of the signal from dialogue data without binary or gold feedback.Source: §6.3 · p. 8

7 Conclusion

JUICER uses limited organic feedback, including binary labels and gold corrections when available alongside textual feedback, to correct bad responses and augment final-model training. Experiments show that these predicted corrections can integrate human feedback and improve overall performance.

7 Conclusion

  • JUICER uses limited organic feedback to correct bad responses and augment final dialogue-model training, and experiments show improved overall performance.The framework uses binary labels and gold corrections when available alongside free-form textual feedback.Source: §7 · p. 8

8 Limitations and Discussions

The discussion flags reliance on dense, trustworthy feedback and an offline, iterative workflow. The appendix details FITS sampling and the roles, training, and evaluation conditions of JUICER’s helper models.

8 Limitations and Discussions

  • The experiments assume bad replies always receive free-form feedback and good intent, while the iterative training loop is long and improves models offline rather than correcting replies on the fly.The authors also note that feedback may be absent or incorrect, and that the best corrector may not yield the best final dialogue model.Source: §8 · p. 9

A.1 Details on FITS

  • FITS annotates bot messages with binary satisfaction feedback, free-form explanations for bad replies, and gold corrections.The setting also allows conversations to continue after feedback is integrated.Source: §A.1 · p. 11

A.2 Sampling FITS

  • The sampled FITS training set uses 20% of examples with binary and gold labels; 1,376 gold corrections represent 7% of bad responses, and 7,768 examples are sampled.The 7,768-example sample is similar in size to each validation and test set.Source: §A.2 · p. 11, Table 4 · p. 11

A.2.1 Varying the sampling rate

  • Perplexity is 9.52 at 5% sampling, 9.09 at 20%, 9.12 at 30%, and 8.80 at 50%; 20% is described as a balance of F1 and perplexity.The reported perplexity gains generally diminish as sampling increases.Source: §A.2.1 · p. 11, Table 5 · p. 12

A.3 Different modules in JUICER

  • JUICER uses a satisfaction classifier and reply corrector as helper modules; the classifier is evaluated on FITS and zero-shot DEMO, while the final dialogue model uses live search results.The reply corrector is evaluated using gold search results and textual feedback, unlike the final dialogue model.Source: §A.3 · pp. 11–12

A.4 Reply corrector

  • The reply corrector takes dialogue context, a bad reply, and the following human message as input, and outputs a correction; the main experiments fine-tune R2C2.Source: §A.4 · p. 12

A.4.1 Training the reply corrector with multiple tasks

  • Multi-task training for the reply corrector combines reply correction, feedback generation, and Wizard of Internet dialogue tasks.Task weights and other hyperparameters were tuned on the validation set.Source: §A.4.1 · p. 12

A.4.2 Evaluating the reply corrector

  • Reply-corrector and final-model results are not comparable: the corrector uses gold search results and textual feedback, while the final model uses live search without that feedback.Source: §A.4.2 · p. 12

A.4.3 Generating reply corrections

The appendix describes generating and reranking reply corrections, then reports differences among corrector variants. It cautions that a higher corrector F1 does not necessarily translate into better final-model perplexity.

A.4.3 Generating reply corrections

  • 99.96% (16,893 of 16,989) bad responses had at least one satisfactory-predicted correction among 60 generated candidates.The method generates multiple corrections and uses the satisfaction classifier to score and rerank them.Source: §A.4.3 · pp. 12–13

A.4.4 Using BB2 to train the reply corrector

  • R2C2 outperforms BB2 as a reply corrector because BB2 corrections resemble conversational replies rather than explicit corrections.Source: §A.4.4 · p. 13, Table 7 · p. 13

A.4.5 Using DIRECTOR in the reply corrector

  • DIRECTOR raises reply-corrector F1 to 22.81, but using its corrections slightly improves F1 and worsens perplexity in the final dialogue model.The authors say further analysis is needed to understand why.Source: §A.4.5 · p. 13, Table 7 · p. 13

A.5 Final dialogue model evaluation

  • The final dialogue model is evaluated on live search results rather than gold search results to better reflect performance with live users.Source: §A.5 · p. 13

A.5.1 Oracle performance using 100% feedback data

The 100% feedback methods trained on the full FITS dataset provide an upper-bound comparison for JUICER, which trains on 20%. JUICER performs comparably to the best oracle method on F1 and human evaluation.

A.5.1 Oracle performance using 100% feedback data

  • The 100% gold-correction method trains on context paired with gold corrections from all 6,601 FITS examples.Its setup is directly comparable to the 20%-data gold-correction condition in Table 2.Source: §A.5.1 · p. 13
  • The 100% free-form feedback method uses dialogue context as input and human-written free-form feedback as its target.It is compared with the 20%-data free-form feedback condition in Table 2.Source: §A.5.1 · p. 13
  • The 100% module-supervision method fine-tunes BlenderBot 2’s modules using human-written search queries, selected search documents, and gold corrections.BlenderBot 2 includes search and knowledge modules.Source: §A.5.1 · p. 13
  • The 100% reward-based method reranks BlenderBot 2 responses with a satisfaction classifier, then fine-tunes on high-reward responses.The 100% DIRECTOR method uses both binary satisfaction labels and textual feedback.Source: §A.5.1 · p. 14
  • F1 is 17.2 for both the best oracle method and JUICER; good-response rates are 47.0% and 45.5%, and human ratings are 3.38 and 3.34, respectively.The best oracle method fine-tunes individual modules; JUICER was not given the oracle methods’ 100% gold data.Source: §A.5.1 · pp. 13–14

A.5.2 DIRECTOR OVERLAP

DIRECTOR OVERLAP adapts DIRECTOR to bad-response and gold-correction pairs by treating shared tokens as positive. It improves reply-correction F1 and, in human evaluation, improves good-response rate and lowers search-result error over DIRECTOR.

A.5.2 DIRECTOR OVERLAP

  • In the bad-response and gold-correction pairs, 28.4% of bad-response tokens overlap; 21.9% are non-stop-word, non-punctuation tokens.The remaining 6.5% are stop words and punctuation.Source: §A.5.2 · p. 15
  • DIRECTOR OVERLAP reaches reply-correction F1 23.00, improving over DIRECTOR.The method assigns positive labels to tokens shared between a bad response and its gold correction.Source: §A.5.2 · pp. 14–15
  • DIRECTOR OVERLAP improves the final dialogue model’s good-response rate and lowers search-result error in human evaluations over DIRECTOR.The paper motivates the variant by noting that edits to bad replies may retain overlapping tokens that need not be punished.Source: §A.5.2 · pp. 14–15

B Model Training Setting

Training and evaluation use ParlAI, with default evaluation settings and fine-tuning runs on up to eight 32GB NVIDIA V100 GPUs. The reported tables cover final dialogue model results and comparisons with oracle and DIRECTOR baselines.

B Model Training Setting

  • All runs use ParlAI for training and evaluation, with evaluation metrics measured using default settings.Fine-tuning uses Adam and runs for up to 4,000 updates with batch sizes up to 128.Source: B Model Training Setting · pp. 15–16
  • Table 10 reports that DIRECTOR OVERLAP raises reply-correction F1, but using its corrections slightly raises final-model F1 while worsening perplexity.The caption says further investigation is needed to understand the discrepancy.Source: Table 10 · p. 16
  • Table 11 presents 100% oracle-method results and marks statistically significant human-evaluation improvements over the BB2 3B baseline.Table 12 compares JUICER with DIRECTOR OVERLAP and says the variant improves human-evaluation results over vanilla DIRECTOR.Source: Table 11 · p. 16, Table 12 · p. 16
  • Fine-tuned models use a maximum of eight 32GB NVIDIA V100 GPUs and are trained up to 4,000 updates; typical times are 8 hours for standard transformer encoder-decoders and 16 hours for retrieval-based models.Source: B Model Training Setting · p. 16
Loading 2210.15893v1…