Source-linked AI summary
SURF: Improving classifiers in production by learning from busy and noisy end users
Joshua Lockhart, Samuel Assefa, Ayham Alajdad, Andrew Alexander, Tucker Balch, Manuela Veloso
TL;DR
Production classifiers need feedback to correct mislabeled data, but user silence is ambiguous because it may indicate agreement or non-response. The paper presents SURF, an extension of Dawid-Skene that estimates user busyness, and reports better ground-truth estimation than conventional crowdsourcing methods when users are busy, with a stated underperformance regime.
Problem
Production feedback systems must interpret user non-response, but silence can mean agreement with a classifier label or that a user is too busy to respond.
Method
SURF extends Dawid-Skene’s EM loop by estimating each user’s busyness or response rate when combining classifier labels and user relabelings.
Results
SURF outperforms conventional crowdsourcing algorithms when users are busy and converges to a better ground-truth estimate when users are noisy.
Takeaways & Limitations
Accounting for non-response ambiguity is vital for production feedback systems because assuming silence means agreement can inject noise when users are busy.
Takeaways & Limitations
SURF underperforms when user error probability is high and classifier error probability is low, possibly because it estimates user busyness poorly.
Abstract
from arXiv · showhide
Supervised learning classifiers inevitably make mistakes in production, perhaps mis-labeling an email, or flagging an otherwise routine transaction as fraudulent. It is vital that the end users of such a system are provided with a means of relabeling data points that they deem to have been mislabeled. The classifier can then be retrained on the relabeled data points in the hope of performance improvement. To reduce noise in this feedback data, well known algorithms from the crowdsourcing literature can be employed. However, the feedback setting provides a new challenge: how do we know what to do in the case of user non-response? If a user provides us with no feedback on a label then it can be dangerous to assume they implicitly agree: a user can be busy, lazy, or no longer a user of the system! We show that conventional crowdsourcing algorithms struggle in this user feedback setting, and present a new algorithm, SURF, that can cope with this non-response ambiguity.
1 INTRODUCTION
SURF addresses the ambiguity of user non-response in production feedback loops, where silence may reflect agreement or disengagement rather than correctness. It extends crowdsourcing-based label aggregation to estimate ground truth while accounting for user busyness.
- Production classifiers need end-user feedback to relabel mistakes and retrain models for improved future performance.
- Feedback labels are noisier than ordinary crowdsourcing because users may misunderstand the taxonomy, resist automation, or provide intentionally incorrect relabelings.
- Conventional crowdsourcing methods do not explicitly model feedback prompted by a classifier’s prior label, making their independence assumptions unsuitable for busy-user feedback.
- Non-response is ambiguous: users may silently agree with a classifier label, or fail to respond because they are busy, lazy, or disengaged.
- SURF extends Dawid-Skene’s expectation-maximisation framework by estimating each user’s response rate and incorporating the classifier alongside end-user relabelings.
- SURF copes with user non-response more effectively than crowdsourcing algorithms, suggesting that modeling non-response is vital in production systems.
2 METHODS
The methods model production feedback as classifier-triggered user relabeling with ambiguous non-response, then extend Dawid-Skene to estimate user busyness and infer ground-truth labels.
- Feedback setting: Non-response is ambiguous because silence may mean agreement with the classifier or that a busy user failed to provide feedback.Feedback is prompted by the classifier’s label, making user responses correlated with that label.
- Feedback setting: The feedback task combines classifier labels with user feedback to produce labels approximating the ground truth for retraining.The classifier is treated as an agent alongside users, and the inferred labels can support training a new classifier.
- Baseline: Dawid-Skene estimates user confusion probabilities with EM under an independence assumption about users’ reports.Its estimators are updated through an expectation-maximisation loop using indicator variables.
- SURF: SURF extends Dawid-Skene by estimating each user’s busyness and incorporating that estimate into the EM posterior-label calculation.The classifier is included as agent u0, while a_k_i records whether user k was assigned data point i.
- SURF: When a user’s feedback matches the classifier label, SURF allows for busyness; a differing label is treated as ordinary feedback.A busy user responds with the classifier label, whereas an engaged user responds through the user’s error function.
3 EXPERIMENTS
The experiments simulate classifier feedback on MNIST using noisy, potentially busy user agents. They vary classifier and user conditions and compare SURF with established crowdsourcing methods across repeated runs.
- Simulation setup: The simulation draws MNIST data with fixed ground-truth labels, generates classifier labels, and asks user agents for feedback on those labels.The goal is to recover ground-truth labels from the classifier and user feedback.
- Simulation setup: Users are modeled with pairwise-flipper confusion matrices, producing correct labels with probability 1 − p and incorrect labels with probability p.Each user receives one confusion matrix at the beginning of an experiment.
- Simulation setup: The classifier is modeled as an additional user agent so experiments can directly control classifier noise.The classifier agent has busyness probability 𝜖=0 and a randomly drawn confusion matrix with fixed error probability p_c.
- Experimental procedure: For each data point, a subset of M users is sampled from the pool to provide feedback, and the algorithms estimate labels from the resulting feedback.The experiments use K users and vary M, classifier error, user error, and user busyness while keeping homogeneous user parameters.
- Evaluation: SURF is compared with DS, iBCC, EBCC, LFC, ZenCrowd, CATD, GLAD, and Majority Vote across parameter settings.Each setting is run 10 times using newly sampled N = 1000 MNIST data points.
4 RESULTS
Experiments show that SURF remains robust as user busyness increases, unlike conventional crowdsourcing models, and continues to work with sparse feedback. Its main weakness appears when users are highly error-prone but the classifier is relatively accurate.
- Performance degradation from busyness is greater when classifier error probability is high.Busy users rarely correct labels from noisy classifiers, reducing feedback quality when non-response is treated as agreement.
- SURF remains effective when user error is low and classifier error is high, without injecting classifier bias into ground-truth estimation.The method treats the classifier as another user, allowing accurate users to correct an inaccurate classifier even as ε increases.
- SURF is resilient when only a random subset of users provides feedback for each data point.The experiments vary the number of feedbacking users M, including sparse-feedback settings.
- SURF underperforms when user error probability is high and classifier error probability is low.The authors suggest that poor estimation of user busyness may produce poor ground-truth estimates, especially at low busyness.
- SURF maintains high performance as user busyness increases, while conventional crowdsourcing models degrade.All models perform well when ε=0; increasing busyness harms every model except SURF.
5 CONCLUSION
The paper addresses ambiguity in feedback non-response, which can undermine noisy-label methods such as Dawid-Skene. SURF extends Dawid-Skene by learning user busyness from label uncertainty to improve ground-truth estimation.
- The feedback setting makes non-response ambiguous because users may agree with a label or simply be unavailable.
- SURF refines Dawid-Skene by learning each user’s busyness probability to estimate ground truth more accurately.
- SURF uses uncertainty about each data point’s label to distinguish agreement-driven non-response from disengagement.