PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Classification by Asking Informative Questions

Lili Yu, Howard Chen, Sida Wang, Tao Lei, Yoav Artzi

arXiv:1911.03598v2cs.CLcs.HCcs.IRcs.LG

TL;DR

Single-step natural-language classification does not interact to reduce ambiguity in underspecified or misinterpreted queries. This paper adds binary or multiple-choice clarification questions, chooses whether to ask or predict, and reports accuracy gains across two tasks.

  • Problem

    Single-step classification typically relies on one user input and does not interact to reduce ambiguity from underspecified requests or misinterpreted queries.

    Source: §1 · p. 1

  • Method

    The system uses natural-language representations to estimate label and response distributions, selects questions by information gain, and learns a controller to choose between asking and predicting.

    Source: §1 · p. 1, §4.3 · p. 4, §4.5 · p. 5

  • Results

    Across FAQ suggestion and bird identification, accuracy improves over the no-interaction baseline by over 100% in simulation and over 90% with human users, given at most five turns.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The system can be bootstrapped without interaction data and is effective on two tasks using limited question-based interaction.

    Source: §8 · p. 9

    The model assumes user responses depend only on the current question and target label, not past interactions, and that question selection is deterministic given the interaction history.

    Source: §4.1 · p. 3

Abstract

from arXiv · show

We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query using natural language, and the system asks for additional information using binary or multi-choice questions. At each turn, our system decides between asking the most informative question or making the final classification prediction.The simplicity of the model allows for bootstrapping of the system without interaction data, instead relying on simple crowdsourcing tasks. We evaluate our approach on two domains, showing the benefit of interaction and the advantage of learning to balance between asking additional questions and making the final prediction.

1 Introduction

The paper adds limited question-based interaction to natural-language intent classification to address ambiguity that single-step prediction cannot resolve. It bootstraps the system without interaction data and reports substantial accuracy gains on two tasks.

1 Introduction

  • Single-step classification does not interact to reduce ambiguity, although further interaction could mitigate low-quality decisions from underspecified requests or misinterpreted queries.The paper frames this as a low-overhead way to study interaction without the cost and complexity of interactive data collection.Source: §1 · p. 1
  • The system asks binary or multiple-choice questions after an initial natural-language query, limiting interaction while allowing clarification.The illustrated interactions cover FAQ suggestions and bird identification; the images are for illustration only and are not used by the approach.Source: §1 · p. 1, Figure 1 · p. 1
  • It selects questions by expected information gain and uses a learned policy controller to choose between asking again and returning a final prediction.The distributions are estimated independently from crowdsourced initial queries and keyword annotations, avoiding interactive data collection.Source: §1 · pp. 1–2
  • With at most five turns, accuracy improves over the no-interaction baseline by over 100% on both tasks in simulation and over 90% with human users.A single clarification question improves simulated accuracy by 40% for FAQ suggestion and 65% for bird identification.Source: §1 · p. 2

2 Technical Overview

The system tracks beliefs over labels through an interaction history, asks an information-gain-maximizing question when its controller chooses ASK, and otherwise returns the highest-probability label. Crowdsourced non-interactive tasks support learning and evaluation with simulated and human users.

2 Technical Overview

  • The model’s interaction history includes the initial query and question-response pairs; available questions can be binary or multiple-choice with predefined answers.Possible outcomes include returning a classification label to end the interaction.Source: §2 · p. 2
  • At each turn, a policy controller chooses whether to ask a question or return the label with the highest current probability; questions maximize information gain.The interaction begins with a user query, and each answer updates the belief over classification labels.Source: §2 · p. 2
  • Crowdsourcing supplies initial queries for each label and text tags that are converted heuristically into question-answer pairs for model training, simulation, and policy learning.These are non-interactive data-collection tasks.Source: §2 · p. 2
  • Evaluation measures classification accuracy and the trade-off between accuracy and turn count using both a user simulator and real human users.Human evaluation additionally collects qualitative ratings.Source: §2 · p. 3

3 Related Work

Related work spans feedback-based learning, interactive language tasks, and question-selection methods. This approach limits interaction to binary and multiple-choice questions while using natural-language descriptions and crowdsourced annotations.

3 Related Work

  • Unlike methods that use post-prediction human feedback as offline training data, this model uses interaction to increase prediction performance.Prior work also used language feedback to learn reward functions reflecting human preferences (Christiano et al., 2017).Source: §3 · p. 3
  • Many interactive language systems require recorded dialogues or Wizard-of-Oz annotations; limiting questions to binary or multiple-choice reduces annotation complexity while retaining effective interaction.The paper relates its task to the 20-question game and uses natural-language descriptions rather than categorical or structural representations.Source: §3 · p. 3
  • Like prior question-selection methods, the approach uses entropy reduction, but models the full interaction history and trains from crowdsourced annotations rather than real user-user interaction data.The cited learning-to-ask method models expected utility, with entropy as a special-case utility (Rao and Daumé III, 2018).Source: §3 · p. 3

4 Method

The system maintains beliefs over labels, asks questions selected for expected information gain, and uses a learned controller to decide when to stop. Its probability estimates and simulator are learned from separate crowdsourced or held-out annotations, without requiring interaction recordings.

4 Method

  • At each turn, the system updates label beliefs, asks a maximum-information-gain question when needed, or returns its current best label.A policy controller chooses between asking and stopping.Source: §4 · p. 3, §4.5 · p. 5

4.1 Belief Probability Decomposition

  • The belief over labels factors into an initial-query label probability and a response probability conditioned on the question and label, avoiding costly interaction recordings.The decomposition assumes responses depend only on the current question and target label, and that question selection is deterministic given interaction history.Source: §4.1 · pp. 3–4

4.2 Information Gain Question Selection

  • Questions are selected to minimize expected label entropy after possible answers, using the response and label distributions to update information gain during interaction.The entropy before asking is constant across candidate questions.Source: §4.2 · p. 4

4.3 Modeling the Distributions

  • A shared text encoder scores queries, question-answer pairs, and labels; response probabilities blend observed counts with a learned estimate that smooths incomplete annotations.The model is pretrained on query-label and question-answer-label text pairs, with negative sampling approximating the full-label loss.Source: §4.3 · pp. 4–5

4.4 User Simulator

  • A separate held-out dataset supplies initial queries and question-answer responses for training the simulator, whose sampled answers provide noisy interaction episodes.The simulator's response distribution is estimated from smoothed empirical counts and is not used to estimate the model's main distributions.Source: §4.4 · p. 5

4.5 Policy Controller

  • The policy controller uses top-k label probabilities and the current turn to choose ASK or STOP; ASK selects the maximum-information-gain question, while STOP returns the most probable label.Source: §4.5 · p. 5

4.6 Training Procedure

  • Training uses REINFORCE with rewards for correct predictions, penalties for incorrect predictions and each question, and keeps the text encoder fixed during policy-gradient updates.The response-distribution parameters and controller are updated using the policy gradient.Source: §4.6 · pp. 5–6

5 Data Collection

The data collection process creates initial queries and question-answer annotations for FAQ targets without conducting multiround user dialogues. It combines expert-defined tags and templates with non-expert response annotation.

Initial Query Collection

  • Workers wrote FAQ-search queries while asked to provide incomplete information rather than simply paraphrasing the target document.The process was intended to elicit realistic, diverse utterances from users with limited domain and system knowledge.Source: §5 · p. 6

Tag Collection

  • Experts defined free-form tags, removed duplicates, and combined some binary tags into categorical tags before questions were generated.Tags could be a phrase or a single word and were not restricted to a predefined ontology.Source: §5 · p. 6
  • Templates converted tags into binary or multiple-choice questions, and non-experts annotated target-tag associations or selected applicable answers.For binary questions, workers saw ten tags and a none-of-the-above option; multi-choice questions also allowed not applicable.Source: §5 · p. 6
  • Workers annotated only the current top 50 tags ranked for each target, and did not participate in multiround interaction, keeping collection cheap and scalable.The tag-ranking model was initially trained on the collected queries and later retrained on the complete data.Source: §5 · p. 6

6 Experimental Setup

The evaluation covers FAQ suggestion and bird identification, with human interaction tests and simulator analysis. It compares no-interaction systems, alternative question-selection strategies, and variants of the proposed approach.

Task I: FAQ Suggestion

  • FAQ Suggestion uses 517 troubleshooting documents, with 3,831 initial queries and 118,640 tag annotations split across 310 training, 103 development, and 104 test documents.Only training-split annotations are used for pretraining and policy learning.Source: §6 · p. 6

Task II: Bird Identification

  • Bird Identification uses CUB-200 captions as noisy initial queries and species as labels, with 8 categorical and 279 binary questions derived from attributes.Images are excluded from model training and used only to ground human evaluation.Source: §6 · p. 6

Baselines

  • Comparisons include four no-interaction models, random questions, query-independent static question selection, and variants with alternative stopping rules or no parameterized response estimator.Source: §6 · pp. 6–7

Evaluation

  • Human evaluation measures prediction accuracy and interaction rationality and naturalness; FAQ tests use either development/test tags or training annotations only.The latter setup evaluates unseen documents without adding tags, while simulator analysis reports Accuracy@k.Source: §6 · p. 7

7 Results

The simulated evaluation reports strong accuracy gains from interaction, including with one clarification question. Human evaluation reports stronger performance across accuracy and user-centric measures, while further analyses examine generalization, encoder choice, and policy training.

Encoder and policy training

  • The lightweight SRU RNN encoder performs better or similarly to other encoders, while policy-controller accuracy stabilizes after the first 1,000 exploratory training episodes.Different turn penalties lead to different expected turn counts.Source: §7 · pp. 7–8

Human evaluation

  • The full model balances classification accuracy with naturalness and rationality, achieving stronger performance across the human-evaluation measures.Figure 2 presents user ratings; Table 1 covers classification accuracy.Source: §7 · p. 7, Figure 2 · p. 8, Table 1 · p. 8

Simulated interactions

  • Accuracy@1 reaches 79% for FAQ Suggestion and 49% for Bird Identification with less than five turns, beating the No Interaction RNN baseline by 41% and 26%.These are results using the user simulator.Source: §7 · p. 8

Generalization and policy analysis

  • With question-and-answer access withheld during development and test evaluation, full-model performance drops only slightly to 75%, indicating generalization to unseen tags.The passage describes this as the full model’s performance under that evaluation condition.Source: §7 · p. 8
  • Alternative termination strategies underperform the full model, while the λ = 1 variant’s lower performance and No Initial Query Interaction confirm the importance of the learned natural-language embedding encoder.The λ = 1 variant has fewer probability components leveraging natural language than the full model.Source: §7 · p. 8

Accuracy-turn trade-off

  • A single clarification question yields relative accuracy boosts of 40% for FAQ and 65% for Birds over no-interaction baselines.The reported comparison varies interaction length and strategy to show the accuracy-turn trade-off.Source: §7 · p. 8

8 Conclusion

The paper proposes interactive classification for underspecified natural-language queries using simple binary or multiple-choice questions. It selects questions by information gain, controls whether to ask or predict, and reports effectiveness on two tasks without interaction data.

8 Conclusion

  • The system asks binary or multiple-choice questions to gather missing information from underspecified natural-language queries.The approach is designed to balance question asking with making the final classification prediction.Source: §8 · p. 8
  • Information gain selects the next question, while a lightweight policy controls whether to continue interacting.The design is intended to support extension to new classes and zero-shot generalization to unseen targets and questions.Source: §8 · p. 8
  • The paper reports effectiveness on two tasks and identifies handling users’ free-form responses as a future research direction.The authors describe their system as bootstrappable without interaction data.Source: §8 · p. 9

A.1 Data collection

For the FAQ task, the authors collected deliberately incomplete initial queries and annotated target-tag associations. They qualified workers and limited tag annotation to highly ranked candidates, with the reported curve plateauing at rank 50.

A.1 Data collection

  • Workers provided potentially insufficient initial queries after a two-stage qualification process that selected workers familiar with the FAQs.The authors collected 3,831 initial queries from the selected pool of 25 workers.Source: §A.1 · p. 12
  • For tag annotation, workers selected target-associated tags from lists that included negatives and a “none of the above” option.Workers had to annotate three targets without selecting negative tags to pass qualification.Source: §A.1 · p. 12
  • The authors showed workers the top 50 ranked tags per target, and the accumulated assignments plateaued at rank 50.The tags were divided into five non-overlapping lists of ten, with each list assigned to four workers.Source: §A.1 · p. 12, Figure A.1 · p. 12

A.2 Implementation Details

The implementation uses bidirectional SRU encoders for the two tasks, with fixed fastText embeddings. A feed-forward policy controller uses the current turn and top belief probabilities, and interactions are capped at ten turns.

A.2 Implementation Details

  • The FAQ task uses a single-layer bidirectional SRU, while bird identification uses a two-layer bidirectional SRU.Both use fixed 300-dimensional fastText embeddings, hidden size 150, batch size 200, and dropout 0.1.Source: §A.2 · p. 12
  • The implementation uses a Noam learning-rate scheduler with an initial rate of 1e-3, 4,000 warm-up steps, and a scaling factor of 2.0.For the no-interaction RoBERTa baseline, the paper fine-tunes RoBERTaBASE.Source: §A.2 · p. 12
  • The policy controller is a two-layer feed-forward network with a 32-unit hidden layer and ReLU activations.Its inputs are the current turn and the top 20 belief probabilities; the system allows at most 10 interaction turns.Source: §A.2 · p. 13

A.3 Additional Analysis

The additional analyses examine text-encoder inputs and encoders, while the human evaluation describes task-specific interaction procedures and reports that the full model performs best on accuracy and user-rated interaction quality.

A.3 Additional Analysis

  • Using pseudo-queries that combine existing queries with sampled target tags improves classification performance.Using tags instead of initial queries also improves performance, while concatenating queries and tags yields 76% Accuracy@1; the full model reaches 79% with about five tag queries.Source: §A.3 · p. 13, Table A.2 · p. 14
  • The analysis compares system performance using three different text encoders, with the maximum number of turns set to five.The supplied passage gives the comparison’s scope but not its outcomes.Source: §A.3 · p. 13, Table A.3 · p. 14

A.4 Human Evaluation

  • The full model performs best on Accuracy@1, naturalness, and rationality in the human evaluation.The survey asks users to rate interaction naturalness and whether they felt understood by the system.Source: §A.4 · p. 13
  • In bird identification, users may answer “not visible” for occluded attributes, after which the system skips remaining binary questions in that label group.The task has 200 fine-grained classes; most reported errors are between similar sparrow and warbler types.Source: §A.4 · p. 13
  • The bird-identification interface presents users with a bird image and asks them to describe it before answering attribute questions.The paper reports that interaction improves classification for both distinct and similar bird types.Source: §A.4 · p. 13, Figure A.2 · p. 15, Figure A.4 · p. 17
Loading 1911.03598v2…