Source-linked AI summary
A Survey of Human-in-the-loop for Machine Learning
Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, Liang He
TL;DR
Human-in-the-loop addresses the need for accurate models despite limited labeled data and costly annotation by integrating human knowledge and experience. This survey classifies existing work from a data perspective, summarizes approaches across application areas, and identifies open challenges and opportunities.
Problem
Large model parameter counts require massive labeled datasets, while data growth lags behind model growth and annotating new tasks is labor-intensive.
Method
The survey reviews papers since 1990 and classifies human-in-the-loop machine-learning work into data processing, interventional model training, and independent system design.
Results
The survey finds that relatively small amounts of human feedback can dramatically boost model performance across tasks, including NLP and computer vision.
Takeaways & Limitations
The survey provides a high-level synthesis of human-in-the-loop techniques and identifies open challenges and future directions for integrating human knowledge into robust systems.
Takeaways & Limitations
Human-in-the-loop preprocessing research includes very little work on computer-vision tasks, reflecting difficulty integrating human experience into image processing.
Abstract
from arXiv · showhide
Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field; along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.
1. INTRODUCTION
Human-in-the-loop integrates human knowledge and interaction into machine-learning pipelines to address data, modeling, and application challenges. This survey organizes prior work by data flow, from data processing and interventional training to independent human-in-the-loop systems.
- Motivation: Large models require massive labeled datasets, while data growth lags parameter growth and new tasks make exhaustive annotation labor-intensive.These constraints motivate generated datasets, pre-training, transfer learning, weak supervision, and few-shot learning.
- Significance: Human-in-the-loop incorporates human knowledge into machine-learning processes and spans computer science, cognitive science, and psychology.Human prior knowledge is especially relevant when data are sparse, including in clinical diagnosis.
- Organization: The survey classifies implementations into data processing, interventional model training, and independent human-in-the-loop system construction.The organization follows how human interaction relates to data processing, model training and inference, and system design.
- Research scope: The authors review human-in-the-loop machine-learning research through questions about challenges, data processing, model training, and applications.The paper frames these questions across NLP, computer vision, and other application areas.
- Contributions: The authors compare methods, summarize challenges and possible solutions, identify milestones, and analyze systems by components and applications.They present the review as the first survey focused specifically on human-in-the-loop for machine learning.
- Method: The survey searches Google Scholar literature, screens titles and abstracts, classifies papers, revises the classification through detailed reading, and synthesizes the final content.Its structure connects data preprocessing, annotation, training, inference, system construction, and applications.
2. DATA PROCESSING
Human-in-the-loop data-processing research addresses data scarcity and annotation costs through preprocessing, annotation, and iterative labeling. The survey organizes representative approaches across NLP, CV, and related applications, while noting limited CV preprocessing work.
- Motivation: Large models require massive labeled datasets, but annotation lags behind model growth and emerging tasks make labeling labor-intensive.The survey motivates human-in-the-loop methods as a way to use data information at lower cost.
- Organization: Human-in-the-loop data processing is organized into data preprocessing, data annotation, and iterative labeling.The survey presents this organization as a data-processing pipeline and reviews representative works in each category.
- Data Preprocessing: Data preprocessing methods use expert experience, interactive tools, crowdsourcing, and adaptive approaches to analyze or prepare complex real-world information.Examples include automated hyperparameter and preprocessing optimization, interactive dictionary expansion, and human-recycling components.
- Limitations: Most preprocessing studies concern extracting and analyzing complex real-world information, while human-in-the-loop preprocessing for CV remains limited.The survey attributes this pattern conjecturally to the lack of a perfect mode for integrating human experience into image processing.
- Data Annotation: NLP annotation work covers task-specific datasets such as entity extraction and linking, alongside abstract tasks including question answering and reading comprehension.Entity-processing methods combine automatic extraction and human labeling to address efficiency, accuracy, and cross-domain disambiguation challenges.
- Data Annotation: CV work includes weak labeling, unified intervention experiences, reinforcement-learning-based human-in-cycle updating, and multi-stage labeling for intricate image tasks.These approaches address newly accumulating data and complex settings such as 3D point-cloud labeling, while seeking to reduce annotation work.
3. MODEL TRAINING AND INFERENCE
Human intelligence is leveraged in NLP and CV to train models and support inference, including through deep-learning and human-machine hybrid methods. These heuristic approaches account for diverse human creativity in pursuing high-quality results.
- Cross-domain approaches: NLP and CV research uses human intelligence in both model training and inference through deep-learning and human-machine hybrid methods.The survey describes these approaches as spanning multiple artificial-intelligence fields.
- Cross-domain approaches: Heuristic human-in-the-loop methods incorporate diverse human creativity to achieve high-quality results.This statement applies to the related NLP and CV research discussed by the survey.
- Scope: The reviewed training and inference approaches cover both natural language processing and computer vision.The survey frames these domains as major areas containing varied methods that leverage human intelligence.
3.1. Natural Language Process
Human-in-the-loop NLP systems use feedback, rationales, preferences, and adversarial examples across tasks including classification, parsing, summarization, dialogue, question answering, and sentiment analysis. Surveyed results indicate that relatively small amounts of human feedback can improve performance while also supporting robustness, generalization, interpretability, and usability.
- Text Classification: Text-classification systems use journalist feedback to retrain models and incorporate human rationales while modeling worker reliability.MARTA jointly updates model parameters and human reliability in an iterative Bayesian framework.
- Syntactic and Semantic Parsing: Parsing systems can treat non-experts’ answers to parser-generated questions as soft constraints for retraining.The surveyed parsing approaches also face ambiguity, insufficient parser performance, and limited explainability.
- Text Summarization: Summarization methods can fine-tune pretrained language models with reinforcement learning from human-preference reward models.
- Dialogue and Question Answering: Dialogue and question-answering systems use online feedback for continual updates or offline feedback datasets, including adversarial examples for negative training.Online systems can generate new examples from successful conversations and retrain continuously.
- Summary: Across surveyed NLP tasks, small amounts of human feedback can improve accuracy, ranking hits, robustness, generalization, interpretability, and usability.Human rationales and worker-reliability weighting are reported as supporting more human-understandable models.
3.2. Computer Vision
Human-in-the-loop computer-vision systems integrate human feedback into tasks such as detection, restoration, segmentation, enhancement, and video segmentation. The surveyed studies primarily aim to improve model performance, with reported benefits including higher recall and reduced annotation requirements.
- Overview: Computer-vision frameworks integrate human feedback into deep-learning architectures to handle complex cases that models cannot handle independently.A general framework for model training and inferencing is outlined for computer vision.
- Object Detection: Interactive object-detection systems ask people to correct selected annotations, while occluded, tiny, and blurred objects remain difficult.One approach prioritizes unannotated images with the maximum predicted annotation cost.
- Image Restoration and Enhancement: Human-in-the-loop vision methods combine expert knowledge with algorithms for image restoration and use human filtering or preferences for enhancement.Applications include electron microscopy restoration and ranking-based learning of user preferences.
- Image Segmentation: Image-segmentation troubleshooting selects informative unlabeled images, has people filter them, and retrains the model on resulting counter-examples.The retraining process retains previously seen examples.
- Summary: 11% higher recall and 67% fewer manually annotated unlabeled samples were reported for pedestrian detection versus the cited baselines.The recall increase is reported over deep SORT, while annotation reduction is reported versus bi-directional deep SORT without AFSID.
4. SYSTEM CONSTRUCTION AND APPLICATION
Human-in-the-loop systems extend beyond data and model intervention into security, software development, simulation, and search applications. These systems allocate work between humans and algorithms to address safety, integration, usability, and feedback-driven improvement needs.
- Security Systems: Security systems use human-in-the-loop reasoning to identify causes of human failures and help designers address defects before deployment.Security applications also combine algorithms, experts, and crowdsourcing workers for fact-checking and misinformation control.
- Security Systems: Security workflows combine artificial-intelligence algorithms with experts or crowdsourcing workers in collaborative operational pipelines.
- Code Production Tools: Human-machine collaboration in software development targets repetitive programming and model-training work by helping developers integrate existing components.The surveyed tools support collaboration on existing projects rather than writing projects from scratch.
- Code Production Tools: Dynodroid automates fuzz testing of unmodified Android applications by generating and executing event sequences and collecting interactive feedback.
- Simulation Systems: Simulation systems require human interaction for decision-making, process forecasting, safety control, and improving the usability of soldier-system optimization.
- Search Engines: Search-engine research includes recommendation and image search, with recommender systems continuously learning from user feedback through interactive interfaces.The cited recommender combines information aggregation, enrichment, recommendation, review summarization, and rating prediction.
- Other Applications: Human-in-the-loop systems are also applied in bioinformatics, supervisory healthcare, and crowdsourcing, with further applications expected as interaction expands.
5. DISCUSSION AND FUTURE DIRECTIONS
The survey identifies open challenges for human-in-the-loop systems across NLP, computer vision, and real-world applications. Future work must address feedback sparsity, user reliability, richer interaction, safety, and robustness.
- Computer Vision: Human-in-the-loop research needs broader approaches for incorporating human experience and knowledge into computer vision beyond direct image labeling.Multimodal representations, graph neural networks, and inverse reinforcement learning are identified as possible directions.
- Data and Feedback: Active learning seeks accurate prediction with minimal labeling cost by selecting the most informative examples, but confidence is less informative for segmentation, regression, and detection.Selection criteria may provide useful guidance for these tasks.
- NLP: Generative NLP systems receive sparse feedback because users typically reward only the single output presented from a large output space.Future work should investigate richer feedback and scaling methods to larger unlabelled corpora or additional languages.
- Safety and Evaluation: Human-in-the-loop systems raise safety risks because malicious users may exploit human feedback to train persuasive or manipulative models.Potential harms include belief manipulation, radicalization, and fraud.
- Human Feedback: Future work should account for variation in human expertise, annotation credibility, feedback selection, user studies, and visualization of how feedback changes models.The survey also calls for more shared human-feedback datasets across NLP and computer vision.
- Real-world Applications: Real-world systems must choose intervention timing carefully and support user experience, complex feedback, robustness, and generalization under domain changes and out-of-range samples.These requirements extend beyond superficial acceptance, rejection, or directional judgments.
6. CONCLUSION
The survey organizes human-in-the-loop machine-learning research into data processing, interventional model training, and system construction. It finds that practical systems involve complex variable coordination and require robust integration of high-level human knowledge.
- Survey Organization: The survey classifies human-in-the-loop work into data processing, interventional model training, and system construction and application.Data processing covers preprocessing, annotation, and iterative labeling; model training is discussed for NLP and computer vision.
- Data Processing: Data-processing methods emphasize semi-supervised data collection and annotation more than selecting and identifying key dataset samples.The core data perspective asks how to achieve greater performance with fewer samples.
- Practical Applications: Practical human-in-the-loop applications require coordinating considerably more variables than the single-variable settings common in academic research.This complicates rapidly integrating high-level human knowledge while maintaining system robustness.
- Future Directions: The survey identifies large-scale pre-training and few-shot human-machine hybrid methods as promising directions for integrating high-dimensional human knowledge.It presents these directions as having development potential and broad prospects.