Source-linked AI summary
AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?
Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Irene Ying, Tianyi Zhou, Jordan Boyd-Graber
TL;DR
The paper asks how people decide when to delegate tasks to AI and how to adopt AI suggestions, addressing the rarity of studying both decisions together in realistic settings. It examines these choices in competitive human–AI question answering and finds that collaboration outperforms either humans or AI alone, although reliance decisions remain systematically imperfect. The authors therefore recommend calibrated confidence, evidence-grounded explanations, and mechanisms for refining trust.
Problem
Prior work rarely studies autonomous delegation and advisory adoption together with the same users in realistic, uncertain settings.
Method
The study examines skilled human–AI teams in a competitive trivia tournament that separately measures autonomous delegation and deliberative adoption.
Results
Human–AI collaboration performs better than either humans or AI alone, but teams under-rely on correct AI suggestions and over-rely on incorrect ones.
Takeaways & Limitations
The findings support calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust in human–AI collaboration.
Takeaways & Limitations
Generalizing beyond competitive, knowledge-intensive trivia to domains such as medical diagnosis or legal review requires caution.
Abstract
from arXiv · showhide
AI systems are fallible, and humans can make mistakes in deciding whether to trust AI over their own judgment. Thus, improving human-AI collaboration requires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: the delegation choice -- deciding when to let AI act autonomously without knowing its output, and the adoption choice -- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in a question-answering game in which humans can choose when and how to work with AI agents to win. Our 24 matches pair 23 expert humans with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, while confirmation bias drives higher under-reliance (64.5%) when an AI suggestion agrees with humans' initial incorrect answer. To close this gap, we recommend calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust.
1 Introduction: How much do you trust LLM output?
The paper studies how people decide when to delegate to AI and how to adopt AI suggestions, addressing the limited study of both reliance decisions in realistic settings. It finds that collaboration is beneficial but trust remains imperfect because humans and AI both contribute errors.
- Prior work rarely captures proactive delegation and deliberative adoption together in realistic, uncertain interactions.These decisions occur under time pressure, imperfect knowledge of AI capabilities, social dynamics, and limited opportunities for repeated learning.
- The study measures when and how skilled users rely on AI through autonomous delegation and evaluated adoption decisions.Delegation concerns letting AI act without reviewing its output, whereas adoption concerns accepting or rejecting AI output after evaluation.
- Human–AI collaboration is mostly synergistic, performing better than either humans or AI alone, but teams still miscalibrate trust.The main errors are under-reliance on correct AI suggestions and over-reliance on incorrect ones.
- 3.9% of opportunities show under-reliance on initially incorrect AI answers, while over-reliance occurs in 1.7% of cases when teams replace correct answers with incorrect AI suggestions.When humans and AI disagree, model confidence performs near chance for selecting the correct answer.
- 64.5% under-reliance occurs when an incorrect human answer is confirmed by an AI teammate, as agreement reinforces the team’s wrong judgment.High-skill teams are particularly susceptible because expertise can increase overconfidence in initial judgments.
- Evidence-grounded explanations help humans abandon wrong answers 12% more often, while practice reduces inaccuracies from 28% to 18%.Teams also reach correct answers in 5.5% of cases where neither humans nor AI were initially right.
2 Game Design for Human–AI Collaborative Question Answering
The game design separates autonomous AI answering from human-mediated adoption so the study can observe distinct reliance decisions. Bonus questions explicitly record human judgments before and after AI suggestions.
- Game structure: Teams of up to three humans and two AIs compete in games alternating between autonomous tossups and collaborative bonuses.Tossups permit immediate individual answers, while bonuses are three-part questions answered collaboratively by the successful team.
- Tossup Delegation: In tossups, teams decide whether to let an AI buzz autonomously or mute it, revealing beliefs about reliability without oversight.Muting prevents an AI from buzzing for the rest of the game but does not remove its bonus-phase suggestions.
- Bonus Adoption: In bonuses, humans first provide a consensus guess, then review two AI guesses with confidence scores and explanations before deciding the final answer.The pre-suggestion guess is recorded without affecting the score and is not labeled correct or incorrect to the team.
- Measurement: The two-stage bonus design compares the same team’s answers before and after AI assistance, isolating AI influence on human decisions.Recorded traces include initial and final answers, AI answers, confidence scores, explanations, and correctness at each stage.
- Interface: The interface presents AI suggestions with confidence and self-generated textual explanations, creating natural variation across agent systems.The explanations are produced by the agents rather than designed by experimenters.
3 Human–AI Cooperative Trivia Tournament
The tournament combines experienced trivia players with diverse AI agents in adversarial questions designed to require complementary human and AI capabilities. Its structure records collaboration behavior while preserving competitive stakes.
- Participants: Two tournaments included 23 experienced human players and 16 AI agents, with nine teams competing in person and online.Participants had 1 to 7+ years of competitive experience, averaging 3.2 years.
- Tournament Structure: The tournament ranks teams on trivia knowledge and their ability to form and work with AI teams.Opaque AI nicknames conceal model origins during drafting, while observed performance informs later team choices.
- Tournament Structure: Weaker teams receive first access to perceived-best AI teammates, partially offsetting human skill gaps and limiting runaway advantages.Online and in-person teams were pooled after no significant differences in bonus accuracy, switching rate, or muting behavior.
- Complementarity: Collaboration generally improves accuracy, although questions that challenge both humans and AI remain difficult.Figure 3 plots average human accuracy against AI accuracy, with bubble size representing team accuracy after deliberation.
- Question Design: Questions target known human or AI weaknesses, including temporal reasoning, cultural references, wordplay, and other adversarial themes.Bonus questions are designed so neither humans nor AI can answer trivially alone and each side’s strengths can compensate for the other’s.
- AI Agents: The 16 AI agents use diverse architectures and base models, producing accuracies ranging from 30 to 80% on the question set.Systems range from single-model prompted calls to multistep pipelines involving answer generation, verification, and confidence calibration.
4 How Humans Trust AI Assistance—and Where They Misjudge
Teams use AI strategically through both delegation and adoption, but their trust remains imperfect: collaboration improves accuracy while producing asymmetric reliance errors and calibration failures.
- Proactive delegation: Eight of nine teams gained points from muting, capturing 79% of the oracle’s maximum possible gain.Teams adapted muting to topics where AI systems performed poorly.
- Proactive delegation: Only 9% of muting decisions occurred at the optimal time, with teams generally muting later than the oracle policy.When teams muted early, they did so 49% earlier than optimal, or 9.8 questions earlier on average.
- Adoption and calibration: Confidence scores achieved 52.3% accuracy, barely above chance, whereas AI agreement reached 100% accuracy and domain knowledge reached 92.4%.Teams most often followed AI agreement, while explanations and confidence scores guided fewer decisions.
- Collaboration outcomes: 81.7% overall team accuracy exceeds humans alone, a random AI, and an oracle choosing the best AI per question.Collaboration is mostly synergistic but not perfect.
- Reliance errors: 3.9% under-reliance exceeds 1.7% over-reliance, showing missed correct AI help is more common than adopting incorrect AI advice.Teams are appropriately cautious overall but sometimes overly conservative.
- Adoption and calibration: When models disagreed, switching fell to 45% even when one model was correct, while confirmation by one AI raised under-reliance to 64.5%.When both AIs agreed on a correct answer, teams switched 82% of the time, compared with a 68% average.
- Explanations and trust: Humans rely on surface signals such as quotes, whereas question understanding and reasoning coherence better predict AI correctness.Evidence grounding is the only feature appearing among both correctness and trust predictors.
5 Related Work
Prior work has documented trust and reliance in human–AI decisions, but often lacks behavioral detail, genuine expertise, or simultaneous measurement of delegation and adoption.
- Related work: Field studies examine consequential decisions but often lack behavioral information about explanation use, confidence comparisons, and deliberation sequences.Lab studies may also rely on crowdworkers unfamiliar with the relevant domains.
- Contribution: This study addresses those gaps by studying experts in practice while capturing both proactive delegation and deliberative adoption.The dual approach examines how humans weigh evidence, calibrate trust, and integrate AI assistance.
- Contribution: The study finds sophisticated but imperfect calibration, including 3.9% missed opportunities and 1.7% over-reliance, with improvement over time.These reliance patterns connect expertise, confidence, explainability, and interpretability to trust decisions.
6 Conclusion and Discussion
The paper argues that effective human–AI collaboration requires better calibration, clear evidence, and informed control over when AI participates. It also points toward multimodal collaboration and team formation as important future directions.
- Implications: Better-calibrated systems, clear evidence, and decision-relevant information are needed to improve human–AI collaboration.The study focuses on text-only settings.
- Implications: Although AI agents surpassed human teams in this text-only setting, synergistic collaboration exceeded the sum of its parts.The authors suggest this may matter more in multimodal domains where AI accuracy still lags.
- Future directions: Future collaboration systems should support team formation by deciding which humans and AI contributors to use for a given problem.The authors release the dataset, tournament platform code, and analysis scripts to support future work.
7 Limitations
The study identifies limitations involving generalizability, statistical power, causal interpretation, interface effects, question design, and temporal scope. These boundaries motivate validation in other domains, larger samples, randomized interventions, alternative interfaces, and longer-term studies.
- Domain specificity: The trivia setting may not generalize to medical diagnosis or legal review, where decision pressures differ.The authors recommend testing whether under-reliance, confidence sensitivity, and explanation effects generalize to other expert domains.
- Sample size and statistical power: With 23 human players and 16 AI agents across 24 games, the study has limited power for nuanced individual differences and rare interactions.Larger studies could reveal additional player archetypes, more precise learning trajectories, and context-specific reliance strategies.
- Causality and interventions: The observational design identifies correlations between confidence, explanations, and reliance but cannot establish causality.Confounding remains possible because high-confidence AI may also produce better explanations or answer easier questions; randomized interventions are proposed.
- Muting interface effects: Explicit muting before each tossup may introduce friction that inflates muting rates.Default-on interfaces with quick toggles or voice commands could produce different strategic patterns.
- Question design and ecological validity: Adversarial questions create systematic human-AI skill gaps, while real-world contexts may contain less predictable AI failures.The authors argue that calibrated confidence, grounded explanations, and user control may generalize beyond adversarial settings.
- Temporal scope: Four tournament rounds over approximately two hours per session cannot assess long-term trust calibration.Longitudinal studies could examine whether positive experiences produce overconfidence or more sophisticated mental models.
8 Ethics Statement
The paper reports participant protections, anonymization, transparency about AI interactions, compensation practices, and data-release efforts. It also situates the work within prior reliance research and describes methodological contributions concerning delegation, adoption, confidence, and explanations.
- Ethics and participant protection: Human evaluation experiments were exempt under institutional IRB review, and human data collection procedures were reviewed to protect participants’ privacy and rights.Participants gave informed consent, participation was voluntary, and anonymized data were reported.
- Compensation: Players participated voluntarily for competitive interest, while question writers received $25/hour and tournament prizes were distributed through online gift cards.The supplied passages report compensation for players, model submitters, writers, and editors.
- Privacy: The study collected gameplay data and basic demographic information, while personally identifiable information beyond pseudonymous player IDs was excluded from the dataset and analysis.Participants were informed about AI characteristics but not implementation details that might affect strategy.
- Transparency: Participants knew they were collaborating with AI agents and saw AI-generated confidence scores and explanations without deception.AI interactions were transparently disclosed as AI-generated rather than human expert opinions.
- Data transparency: The research engaged the quizbowl community and released the full dataset, behavioral traces, application source code, and analysis scripts for reproducibility.Platform details are provided in Appendix H.
- Related work: Prior work distinguishes misuse from disuse and identifies trust calibration as matching reliance to actual automation capability.The paper builds on research spanning human factors, field studies, mental models, explanations, and algorithmic reliance.
- Contribution: The study bridges methodological gaps by capturing proactive muting and deliberative switching in experts’ naturalistic decisions.This dual-signal approach examines how humans weigh evidence, calibrate trust, and integrate AI assistance.
- Trust calibration: Grounded explanations increased appropriate switching by 12%, providing design guidance beyond generic explanation requirements.The paper jointly analyzes confidence and explanations and reports that confidence was not a strong predictor of switching.
A.3 Adversarial Evaluation and Question Answering
The paper uses adversarial question design and a competitive quizbowl setting to study complementary human–AI strengths and strategic reliance. It formalizes under-reliance and over-reliance around whether teams adopt AI assistance after forming an initial answer.
- A.3 Adversarial Evaluation and Question Answering: Quizbowl provides incremental questions and competitive risk, making it a setting for studying when systems should answer under uncertainty.Questions progress from obscure to obvious, and answering involves balancing accuracy against competitive costs.
- A.3 Adversarial Evaluation and Question Answering: Adversarial design creates collaboration opportunities by pairing AI-favoring questions with human-favoring questions that test appropriate reliance.This reframes adversarial evaluation from exposing AI weaknesses toward engineering productive partnerships.
- A.3 Adversarial Evaluation and Question Answering: The analysis records human answers before AI suggestions, AI correctness and recall, whether teams switch, and the final answer’s correctness.These variables support comparison of human decisions before and after AI assistance.
- A.3 Adversarial Evaluation and Question Answering: Under-reliance measures missed help opportunities, while over-reliance measures being misled by incorrect AI suggestions.Both optimal rates are defined as 0%.
B.2 Selection Method Breakdown
Teams selected AI suggestions most effectively through domain knowledge, while AI agreement provided a frequent and accurate shared signal. Conflicting suggestions increased evaluation difficulty and conservative behavior, and agreement could also amplify shared errors.
- B.2 Selection Method Breakdown: 55.0% of eligible revised cases involved AI agreement, and teams adopting the shared answer were perfectly accurate in this filtered set.The agreement share uses all eligible cases as its denominator.
- B.2 Selection Method Breakdown: 75.2% of non-agreement choices used domain knowledge, reaching 90.9% final accuracy.Teams were most effective when using their expertise to evaluate competing AI suggestions.
- B.2 Selection Method Breakdown: Model explanations guided 9.8% of non-agreement choices with 83.3% final accuracy, while confidence was used in 3.6% and reached 54.5% accuracy.Model reputation accounted for 4.2% of choices, and 7.2% appeared random with 50.0% final accuracy.
- B.2 Selection Method Breakdown: When AIs disagreed, teams faced competing suggestions, higher cognitive load, and more conservative behavior.When AIs agreed, switching was easier and under-reliance dropped substantially.
- B.2 Selection Method Breakdown: 64.5% under-reliance occurred when an incorrect human answer was confirmed by one AI while the other AI was correct.AI agreement can also amplify errors when both systems are wrong.
D Our Adversarial Questions
The adversarial question set creates varied collaboration conditions by distributing difficulty differently across humans and AI. Most questions were harder for humans, while packet-level difficulty varied more for humans than for AI.
- D Our Adversarial Questions: Questions are labeled Hard below 40% accuracy, Medium at 40–70%, and Easy above 70% for humans and AI separately.Relative difficulty then compares the two parties’ accuracies.
- D Our Adversarial Questions: 45% of questions were harder for humans, 26% were balanced, 18% were hard for both, and 11% were harder for AI.Relative difficulty compares human and AI accuracy, using a 20-percentage-point performance gap for the harder-for-one category.
- D Our Adversarial Questions: Human difficulty varies more across packets than AI difficulty, whose distribution is more uniform.This creates diverse collaboration scenarios within each tournament.
- D Our Adversarial Questions: The tournament’s draft procedures varied across settings to prevent opposing teams from selecting the same AI agent in a game.Serpentine drafting gave weaker in-person teams earlier access to perceived stronger agents.
E.4 Implications for Analysis
The analysis separates features predicting AI correctness from features predicting which explanation humans select. Draft variations are outside the main within-game reliance analyses, while draft strategy is left for future work.
- E.4 Implications for Analysis: Draft variations do not affect the main analyses, which focus on within-game muting and switching rather than cross-game selection.The paper leaves detailed draft-strategy analysis to future work.
- E.4 Implications for Analysis: Explanation features are evaluated separately for predicting AI correctness and predicting whether humans select an explanation.The analysis extracts 57 features and fits single-feature logistic regressions to isolate individual contributions.
- E.4 Implications for Analysis: AI correctness prediction uses bonus responses where both AI systems answered, with correctness defined against the gold answer.The outcome is binary.
- E.4 Implications for Analysis: Human selection prediction analyzes cases where humans choose between two AI explanations.The outcome is which explanation the team selected.
F.2 Feature Categories
The feature analysis organizes explanation signals into seven categories and shows a calibration gap: features predicting AI correctness differ from those predicting human selection. Evidence grounding is the only signal prominent in both lists.
- Epistemic Features: Epistemic features capture uncertainty expression, and confidence_alignment measures whether linguistic and numeric confidence match.Confidence alignment is identified as a strong predictor of correctness.
- Content Grounding Features: Content-grounding features describe how explanations relate to question content, but humans rely heavily on semantic_similarity (66%) and word_overlap_ratio (63%), which are weak correctness signals.This mismatch contributes to the reported calibration gap.
- Structural Features: Structural features capture formatting and discourse patterns; has_quotes predicts human selection at 70% despite providing a weak correctness signal.Surface signals such as quotes therefore predict trust better than answer quality.
- Cross-category comparison: Across outcomes, LLM-assessed features reach 70–76% correctness accuracy but 44–58% human-prediction accuracy, whereas surface and structural features show the opposite pattern.Only evidential_grounding appears among the strongest predictors of both correctness and human selection.
- Implications: The analysis recommends evidence-citing explanations and evaluating reasoning quality rather than surface familiarity.These recommendations follow from the divergence between correctness-predictive and human-trusted features.
- Feature extraction: The feature pipeline uses seven extractors with consistent category prefixes, including tools for linguistic, structural, content, reasoning, epistemic, pragmatic, and LLM-based signals.The LLM-based extractor uses GPT-4o for structured semantic-quality assessment.