PaperlayerView exact revision on arXiv

Source-linked AI summary

QUORUM: QUality-Optimized Routing Using Multiple annotators

Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Amin Mantrach, Fabrizio Silvestri

arXiv:2608.27974v1cs.CL

TL;DR

Human annotation is costly to scale, while LLM reliability depends on instance difficulty and contextual demands. QUORUM routes instances among human and LLM annotators under a fixed budget using feature-based difficulty signals and multi-annotator aggregation. Across evaluated tasks, it reports higher annotation quality and lower cost than competing methods.

  • Problem

    Human annotation is costly to scale, while LLM annotation quality varies substantially across instances requiring different levels of reasoning and contextual understanding.

  • Method

    QUORUM dynamically routes instances between human and LLM annotators under a budget, using contextual and budget signals, difficulty features, and additional annotations when warranted.

  • Results

    QUORUM reports up to 34.4% relative improvement in annotation quality and up to 8.8% lower cost across English and multilingual open- and closed-ended tasks.

  • Takeaways & Limitations

    QUORUM achieves strong performance–cost trade-offs across classification and summarization benchmarks through cost-efficient, adaptive annotation routing.

  • Takeaways & Limitations

    The evaluation is conducted primarily on English-language datasets, and more specialized domains remain to be evaluated.

Abstract

from arXiv · show

Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.

1 Introduction

QUORUM addresses budget-constrained annotation by routing instances between humans and LLMs based on difficulty and available resources. It supports multiple annotators and improves the quality–cost trade-off across evaluated tasks.

  • Motivation: Human annotation is difficult to scale, while LLM reliability varies with instance complexity and contextual demands.Some instances can be handled reliably by LLMs, whereas others require nuanced reasoning, domain expertise, or cultural understanding.
  • Motivation: Existing routing methods commonly use uncertainty estimates, but calibrated confidence, extra inference, and model-specific signals limit their applicability.These limitations increase computational overhead and make routing less general across annotators.
  • QUORUM: QUORUM dynamically assigns instances to human or LLM annotators while jointly optimizing annotation quality and cost under a budget.Its policies use contextual and budget signals rather than uncertainty estimation.
  • QUORUM: QUORUM supports multiple annotations and requests additional supervision when agreement is insufficient or its cost is justified.Human and LLM annotations can be combined through multi-annotator strategies.
  • Contributions: QUORUM provides theoretical guarantees on posterior concentration and asymptotic convergence toward oracle routing.The formulation is explicitly grounded in budget-aware routing.
  • Results: 34.4% relative improvement in annotation quality and 8.8% cost reduction are reported across English and multilingual open- and closed-ended tasks.On AG’s News, QUORUM is described as achieving the highest annotation quality while matching the cost of the cheapest competing methods.

2 Related Work

Related work balances annotation quality and cost through adaptive routing, predominantly using uncertainty or confidence signals. QUORUM instead uses feature-based difficulty signals to avoid model-derived uncertainty and additional inference steps.

  • Adaptive Routing: Adaptive routing methods seek to balance annotation quality and cost in human–LLM annotation.This focus has become increasingly common in recent work.
  • Uncertainty-Based Routing: Uncertainty-based approaches route low-confidence samples to humans using model-provided confidence scores.Examples include logits from LLM APIs and confidence estimates derived from model outputs.
  • Uncertainty-Based Routing: These methods depend on calibrated confidence scores that are not consistently available across models and require an additional inference step.Both limitations can increase the overhead of routing decisions.
  • Alternative Routing Strategies: Other strategies use retrieval, learned error signals, active learning, or error-aware triage to prioritize human annotation under limited budgets.ARAIDA combines model predictions with nearest labeled neighbors, while SANT integrates active learning with error-aware triage.
  • QUORUM: QUORUM relies on feature-based difficulty signals rather than model-derived uncertainty or additional inference steps.This is presented as enabling a more general and cost-efficient routing strategy under budget constraints.

3 Method

QUORUM formulates annotation routing as maximizing quality under a budget, using feature-based difficulty estimates and contextual, adaptive selection among human and LLM annotators. It supports iterative multi-annotation refinement, task-specific aggregation, and theoretical convergence toward oracle routing.

  • Routing formulation: QUORUM models annotation as selecting annotator actions with rewards and costs to maximize quality under budget constraint B.The framework includes human and multiple LLM annotators as available arms.
  • Feature definition: Feature-based signals estimate instance difficulty from linguistic complexity and semantic isolation rather than model-derived uncertainty.Features include lexical and syntactic statistics plus embedding distances to nearest neighbors.
  • Context construction: Each routing decision uses the sample representation together with remaining budget, dataset progress, and previously queried annotators.This contextual representation incorporates both instance properties and earlier resource use or annotations.
  • Routing mechanism: The router calibrates annotator reliability, explores early with decaying probability, exploits quality estimates later, and escalates low-quality predictions to humans.Human calibration labels initialize annotator-specific posteriors, whose means and covariance guide subsequent routing.
  • Iterative refinement: Samples can receive additional annotations when posterior difficulty, disagreement, or missing human supervision creates high priority.A reserved human budget helps prevent expert annotations from being exhausted before difficult late-stage samples appear.
  • Task-specific aggregation: For classification, QUORUM uses weighted majority voting; for summarization, it uses semantic agreement and selects the summary nearest the embedding centroid.Classification weights human annotations at 10 and LLM annotations at 1, while summarization uses cosine similarity between summary embeddings.
  • Theoretical properties: Annotator quality estimates become increasingly accurate as observations accumulate, and average loss relative to optimal routing vanishes over time.Under the stated assumptions, QUORUM asymptotically approaches oracle routing.

4 Experiments

QUORUM is evaluated across classification, multiple-choice question answering, and summarization benchmarks under fixed human-annotation and monetary budgets. Experiments compare it with representative routing baselines using task performance, human-allocation effectiveness, and annotation cost.

  • Baselines: The experiments compare QUORUM with retrieval-, confidence-, PAC-, active-learning-, and random-routing baselines.The baselines include ARAIDA, CoAnnotating, CDI, HyPAC, SANT, and Random.
  • Datasets: Experiments cover classification, multiple-choice question answering, and summarization benchmarks, including multilingual Global-MMLU and XLSum.The datasets include AG’s News, SST2, IMDB, PubMed, Global-MMLU, MMLU-Redux, CNN/DailyMail, and XLSum.
  • Experimental Settings: The AUDITOR STYLE regime routes between human and LLM supervision under a fixed budget of human annotations.The study varies the percentage of human-labeled data.
  • Experimental Settings: The DOLLARS regime routes among multiple annotators under monetary constraints with heterogeneous annotation costs.Available budgets are expressed relative to the cost of fully annotating the dataset with humans.
  • Metrics: Evaluation reports Accuracy or ROUGE-1, Machine Cumulative Accuracy, and total annotation cost.Machine Cumulative Accuracy measures whether instances sent to humans correspond to cases where the reference LLM would fail.

5 Results

Across routing settings, QUORUM generally improves the quality–cost trade-off, performs competitively with uncertainty-based methods, benefits from multiple annotations, and transfers across noisy human supervision and languages. Its strongest reported aggregate improvement is up to 34.4% in annotation quality with cost reductions of up to 8.8%.

  • RQ1. Budget-Aware Routing Evaluation: In AUDITOR STYLE, QUORUM delivers the strongest quality–cost trade-off, with especially pronounced improvements at 10%–30% human-annotation budgets.Gains are particularly evident on PubMed and MMLU-Redux; for summarization, it achieves the highest ROUGE-1 while maintaining the lowest cost.
  • RQ1. Budget-Aware Routing Evaluation: In DOLLARS, QUORUM continues to benefit from increased budgets while several baselines plateau early.On challenging PubMed, the larger low-to-high-budget gains indicate more effective selective use of expensive annotators.
  • RQ2. Feature-Based Difficulty Estimation: Feature-based difficulty estimation matches or outperforms uncertainty-based routing across datasets and budget regimes without additional model inference.A random-feature baseline performs worse, while QUORUM’s features use lightweight deterministic computations over input text.
  • RQ3. Impact of Multiple Annotations: Increasing annotations per instance from one to three improves downstream quality across datasets.On CNN, performance rises from 0.387 to 0.410, with larger gains for ambiguous instances.
  • Robustness to Human Supervision: With noisy heterogeneous human supervision on SentiMP-En, QUORUM outperforms competing strategies across all three annotators and improves over the standalone LLM baseline.The standalone LLM baseline has 60.4% accuracy.
  • RQ4. Multilingual Generalization: On Spanish and Japanese Global-MMLU and XLSum, QUORUM matches or outperforms competing methods across languages and annotation budgets while maintaining lower costs.Improvements are visible at higher budgets, where additional annotations are allocated more effectively.

6 Conclusion

The conclusion presents QUORUM as a feature-based, budget-aware routing framework that adaptively allocates samples between humans and LLMs while supporting cost-efficient performance–cost trade-offs. It identifies broader multilingual and specialized-domain evaluation as future work.

  • 6 Conclusion: QUORUM dynamically allocates samples between human annotators and LLMs using linguistic and semantic features rather than uncertainty estimates.The framework is designed for cost-efficient and adaptive annotation strategies.
  • 6 Conclusion: QUORUM consistently achieves strong performance–cost trade-offs across classification and summarization benchmarks.The conclusion summarizes results across both task families.
  • Limitations: The evaluation is conducted primarily on English-language datasets, motivating extension to additional multilingual or cross-lingual settings.Such extensions are presented as a direction for further validation of generality.
  • Limitations: Evaluating QUORUM on more specialized domains could provide additional insight into behavior under different annotation regimes.This is identified as a future direction despite the diverse task set already evaluated.
  • Method Details: The routing representation uses linguistic features and embedding-based nearest-neighbor distances as signals of annotation difficulty.The listed linguistic features capture complexity, ambiguity, and text characteristics, while embedding distances proxy atypicality or semantic isolation.

A.2 Proof of Proposition 3.1

The proof establishes that, under the stated contextual reward and sampling assumptions, posterior uncertainty shrinks asymptotically and quality estimates converge toward the true annotator parameters.

  • Concentration Argument: If an annotator is sampled infinitely often on sufficiently diverse contexts, the concentration bounds apply to its posterior estimates.The argument also uses bounded context norms.
  • Result: Posterior uncertainty shrinks asymptotically and quality estimates converge toward the true annotator parameters.This is the proof’s stated conclusion.
  • Assumptions: The analysis assumes a linear contextual reward model with bounded parameters, conditionally sub-Gaussian noise, and a uniformly separated oracle annotator.These assumptions support the concentration argument for routing quality.
  • Routing Rule: The routing rule combines exploration with exploitation based on estimated annotator quality.The proof considers an exploration probability and an exploitation choice among annotators.

Assume that the exploration mechanism samples

The analysis shows that QUORUM’s exploitation rounds eventually select the oracle annotator with zero regret, while exploration rounds become asymptotically negligible. Consequently, average regret vanishes under the stated concentration and margin conditions.

  • After a finite time T0, every exploitation round has zero regret because the greedy rule selects the oracle annotator.This follows on the high-probability concentration event and under the margin assumption.
  • Exploration rounds are represented by a binary indicator whose centered process forms a bounded martingale difference sequence.The strong law for martingale differences is then used to establish their asymptotic frequency.
  • Exploration rounds form a vanishing fraction of all rounds.
  • Average regret vanishes because the finite initial period and the exploration-round contribution both become negligible after division by T.
  • The QUORUM DOLLARS procedure combines calibration, adaptive routing, optional budget-aware re-annotation, and task-specific aggregation.Its decisions account for monetary budgets and heterogeneous annotator costs.

B Experiments - Additional Details

Experiments span classification, multiple-choice question answering, and abstractive summarization across diverse English benchmarks. The learned policy initially explores broadly, then favors cheaper annotators for routine instances and escalates difficult cases using budget and annotation-history signals.

  • Experimental Setup: LLM prompts are specified separately for classification, summarization, question answering, and confidence estimation.
  • Experimental Setup: Experiments cover classification, multiple-choice question answering, and abstractive summarization across benchmarks with varying difficulty, class distributions, and task formats.The datasets include AG’s News, SST2, IMDB, PubMed, Global-MMLU, MMLU-Redux, CNN/DailyMail, and XLSum.
  • Routing Analysis: The router initially explores annotators uniformly, then converges toward lower-cost annotators for routine instances and higher-cost annotators for difficult ones.
  • Routing Analysis: Remaining budget fraction is the dominant routing signal, followed by annotation-mask context and linguistic features such as vocabulary richness.The policy therefore relies primarily on budget state and annotation history rather than embedding-based features alone.

C.2 Sensitivity analysis of QUORUM

QUORUM remains relatively stable across routing and aggregation hyperparameter settings on AG’s News under a $450 DOLLARS budget. The human escalation threshold has the largest effect, while excessive calibration can reduce accuracy.

  • Sensitivity Analysis: Performance remains relatively stable across the tested routing and aggregation hyperparameter configurations.The sensitivity analysis uses AG’s News under the DOLLARS setting with a total budget of $450.
  • Sensitivity Analysis: The human escalation threshold τH has the largest impact among the evaluated hyperparameters.
  • Sensitivity Analysis: Moderate human escalation thresholds, including τH = 0.5, achieve the strongest reported sensitivity-analysis performance.
  • Routing Analysis: Figure 4 visualizes arm-selection proportions, posterior feature-by-arm weights, and mean-absolute-weight feature importance over a 2000-sample AG’s News window.
  • Sensitivity Analysis: Larger calibration fractions gradually reduce accuracy because they leave fewer resources for adaptive routing and iterative annotation refinement.

C.3 Time Analysis of QUORUM

QUORUM reduces routing overhead by replacing confidence estimation with embedding extraction and lightweight feature computation. In multilingual evaluations, it achieves the best or second-best performance across tasks and budgets, with a larger advantage at higher budgets.

  • Time Analysis: QUORUM consistently has lower computational overhead than confidence- or training-based alternatives while maintaining competitive annotation performance.
  • Time Analysis: QUORUM performs only embedding extraction and lightweight feature computation before routing decisions, requiring approximately four minutes on similarly sized datasets.
  • Multilingual Results: Multilingual evaluations use Japanese and Spanish Global-MMLU question answering and XLSum abstractive summarization tasks.
  • Multilingual Results: QUORUM achieves the best or second-best performance across multilingual tasks and budget levels.The results indicate transfer beyond English, including typologically distant languages where LLM annotators are less reliable.
  • Multilingual Results: 0.946 accuracy on Global-MMLU (ja) at 70% human annotation exceeds the second-best method by nearly 4 percentage points.
  • Evaluation Measures: The DOLLARS results report Accuracy or ROUGE-1, Machine Cumulative Accuracy, and performance under varying monetary-budget constraints.
Loading 2608.27974v1…