Putting Humans in the Natural Language Processing Loop: A Survey

• HITL NLP frameworks continuously integrate human feedback into different stages of model deployment to improve accuracy, robustness, interpretability, usability, and trust.Source: §1 · p. 1, Figure 1 · p. 1
• Human feedback can reveal overlooked design details and provide data instances absent from training datasets, addressing limits of the traditional linear NLP pipeline.Source: §1 · p. 1
• HITL NLP spans a high-dimensional design space involving different feedback providers, intervention stages, interfaces, and feedback-learning methods across NLP, ML, and HCI.Source: §1 · p. 1
• The survey summarizes recent HITL NLP studies by task, goal, human interaction, and feedback learning method.Source: §1 · pp. 1–2
• The survey also identifies knowledge gaps and distills future research directions for integrating human feedback into NLP development.Source: §1 · p. 2

• Prior surveys cover interactive machine learning generally or focus specifically on reinforcement learning and offline dialogue systems, whereas this survey reviews HITL in NLP comprehensively.Source: §2 · p. 2

3.1 Text Classification

• HITL frameworks for text classification commonly train an initial classifier, collect human annotations based on model behavior, and retrain continuously on expanded data.Source: §3.1 · p. 2
• Active learning strategically selects documents or features for annotation, enabling comparable model improvement with fewer labels than random sampling.Source: §3.1 · p. 2
• Text-classification systems also support user-edited features, semantic dictionaries, web-based interaction, and deployment-integrated retraining.Source: §3.1 · p. 2

3.2 Parsing and Entity Linking

• Parsing and entity-linking systems solicit feedback on uncertain or poorly handled instances, then use soft constraints or strategic sampling to improve models with limited feedback.Source: §3.2 · p. 2

3.3 Topic Modeling

• Topic-modeling systems let users modify topic words, weights, documents, or rankings, increasingly emphasizing intuitive interfaces and non-expert end users.Source: §3.3 · p. 3

3.4 Summarization and Machine Translation

• For summarization and translation, human preferences or explicit and implicit feedback are learned through reward modeling or reinforcement learning, with reported gains in accuracy and generalization.Source: §3.4 · p. 4

3.5 Dialogue and Question Answering

• Dialogue and QA systems use online loops that continuously retrain from deployment feedback or offline loops that aggregate feedback before batch updating.Source: §3.5 · p. 4
• Offline updating can be more robust for dialogue because misleading user feedback makes direct model updates risky.Source: §3.5 · p. 4

• HITL significantly improves model performance, including classification accuracy, ranking metric hits, robustness, and generalization, with relatively small amounts of human feedback.These effects are reported across text classification, dialogue, question answering, summarization, and related tasks.Source: §4 · p. 4
• HITL improves interpretability and usability by enabling semantically meaningful features, probing model behavior, and supporting intuitive model interaction.User studies also report increased trust and confidence through HITL systems.Source: §4 · p. 4

5 Human-machine Interaction

• HITL systems use graphical and natural language interfaces to collect human feedback.The section connects interaction mediums with subsequent methods for using feedback to update models.Source: §5.1 · p. 4, §5 · p. 4, §5.1.2 · p. 5

5.1.1 Graphical User Interface

• Graphical interfaces support labeling, feature editing, rating, ranking, and visual inspection of model behavior.Their WIMP interactions provide users with more accurate control for refining models, including for non-expert users.Source: §5.1.1 · p. 5

5.1.2 Natural Language Interface

• Natural language interfaces support explicit responses and implicit sentence-level feedback while integrating naturally with dialogue systems.They are described as intuitive and requiring no additional tutorial, while providing users more freedom to refine models.Source: §5.1.2 · p. 5

5.2 User Feedback Types

• The surveyed systems organize feedback into binary, scaled, natural language, and counterfactual example types.Table 2 relates these feedback types to model learning methods.Source: §5.2 · p. 5, Table 2 · p. 6

5.2.1 Binary User Feedback

• Binary feedback is easy to collect through labels, feature edits, or short responses, but can oversimplify intentions by omitting intermediate situations.It can update training data or directly manipulate models.Source: §5.2.1 · p. 5

5.2.2 Scaled User Feedback

• Scaled feedback uses numerical categories such as five-point ratings, offering more intermediate options than binary feedback.It is commonly collected through graphical interfaces and can directly update models or provide implicit guidance.Source: §5.2.2 · p. 5

5.2.3 Natural Language User Feedback

• Natural language feedback is collected through natural language interfaces and is associated with multiple update modes, including offline, online, and direct model manipulation.The supplied survey material lists related works across binary, scaled, and natural language feedback categories.Source: §5.2.3 · pp. 5–6

5.3 Counterfactual Example Feedback

• Counterfactual feedback describes an alternative causal situation in the form “If X had not occurred, Y would not have occurred.”It is usually expressed as natural-language text and collected through a natural language interface.Source: §5.3 · p. 6
• HITL systems collect and analyze user-modified counterfactual examples before retraining the model.The survey connects this process to its later discussion of feedback use.Source: §5.3 · p. 6

5.4 Intelligent Interaction

• Active learning and reinforcement learning are commonly used intelligent interaction techniques in HITL NLP systems.Reinforcement learning treats user feedback as actions, helping models understand human intent and update accordingly.Source: §5.4 · p. 6

6 How to Use User Feedback

• HITL NLP systems use feedback through two major update methods: data augmentation and direct model manipulation.These methods determine how collected feedback changes the model or its training data.Source: §6 · p. 6

6.1 Data Augmentation

• Data augmentation treats human feedback as new ground-truth samples for retraining NLP models.A user's answer to a model question can become a training example.Source: §6.1 · p. 6
• Offline updates retrain models after enough feedback is collected, making them suitable for noisy feedback and computationally demanding models.Examples use human feedback as class labels or span-level annotations before retraining.Source: §6.1 · p. 7
• Online updates apply feedback immediately, supporting dialogue and conversational QA systems where recent input matters for reasoning.Incremental learning updates the current system in real time, including topic hierarchies and LDA sampling parameters.Source: §6.1 · p. 7

6.2 Model Direct Manipulation

• Numerical feedback can directly adjust model objectives, but it is easier to incorporate and less informative than natural-language feedback.Prior work uses binary or 5-point ratings as reinforcement-learning or bandit rewards; broader feedback types require quantitative and qualitative evaluation because they may be noisy.Source: §6.2 · p. 7

7.1 Broader Roles of HITL NLP System

• HITL NLP can improve interpretability and usability alongside model performance, and may help mitigate harms from NLP model bias.The survey encourages using feedback to understand models and improve end-user experience.Source: §7.1 · p. 7
• Future HITL systems could let end users inspect feature effects or modify model behavior through interactive controls and natural-language instructions.The survey gives feature-weight “what-if” operations and pronoun corrections as examples of greater user control.Source: §7.1 · p. 7

7.2 Human-centered System Design

• Most surveyed HITL NLP systems were designed by NLP researchers, motivating deeper HCI involvement in interface design and user studies.Poor interfaces can produce inconsistent, incorrect, or misleading feedback and thereby reduce downstream task performance.Source: §7.2 · p. 7
• Human-centered HITL research should study how interface cues, question framing, and question selection affect feedback quality.The survey also highlights choices about who provides feedback, how model changes are shown, and how users are guided toward useful feedback.Source: §7.2 · p. 8
• Because human feedback can be subjective, noisy, or misleading, researchers should evaluate feedback quality and conduct rigorous user studies beyond model-performance measurement.The survey also recommends open-source tools, shared protocols, and human feedback datasets.Source: §7.2 · p. 8

8 Conclusion

• The survey categorizes HITL NLP work by task, goal, human interaction, and feedback-learning method across NLP and HCI communities.It presents the field as nascent but diverse and identifies potential across many NLP tasks.Source: §8 · p. 8
• The authors highlight open challenges and future directions centered on collaboration between NLP and HCI researchers and practitioners.They frame this collaboration as important for creating NLP technologies aligned with people’s needs.Source: §8 · p. 8