Source-linked AI summary
Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence
Patrick Hemmer, Max Schemmer, Niklas Kühl, Michael Vössing, Gerhard Satzger
TL;DR
The paper asks how to model synergetic potential in human-AI decision-making and what contributes to complementary team performance. It formalizes complementarity and examines information and capability asymmetry in two studies, finding that both sources increase complementarity potential and realized effect.
Problem
The paper asks how human-AI decision-making can be modeled to understand team synergetic potential and what factors contribute to complementary team performance.
Source: §1 · p. 2Method
The authors formalize complementarity potential and complementarity effect, distinguish their inherent and collaborative components, and examine information and capability asymmetry in two experiments.
Source: §1 · p. 2Results
Both information and capability asymmetry increased inherent complementarity potential and realized effect in the two studies, resulting in complementary team performance.
Source: §1 · p. 2Takeaways & Limitations
The conceptualization provides a foundation for analyzing and designing human-AI collaboration in decision-making toward more effective outcomes.
Source: §1 · p. 2The laboratory-based experiments and vignette methodologies could limit generalizability and practical implications.
Source: §5.4 · p. 21
Abstract
from arXiv · showhide
Artificial intelligence (AI) has the potential to significantly enhance human performance across various domains. Ideally, collaboration between humans and AI should result in complementary team performance (CTP) -- a level of performance that neither of them can attain individually. So far, however, CTP has rarely been observed, suggesting an insufficient understanding of the principle and the application of complementarity. Therefore, we develop a general concept of complementarity and formalize its theoretical potential as well as the actual realized effect in decision-making situations. Moreover, we identify information and capability asymmetry as the two key sources of complementarity. Finally, we illustrate the impact of each source on complementarity potential and effect in two empirical studies. Our work provides researchers with a comprehensive theoretical foundation of human-AI complementarity in decision-making and demonstrates that leveraging these sources constitutes a viable pathway towards designing effective human-AI collaboration, i.e., the realization of CTP.
1 Introduction
The introduction frames complementary team performance (CTP) as the goal of human-AI decision-making, while noting that existing research does not yet explain team success systematically. It presents a model of complementarity and two studies of information and capability asymmetry.
1 Introduction
- Many studies show human-AI teams outperforming humans, but often do not test whether teams also surpass AI alone; other studies show teams can outperform AI.The introduction identifies a need to assess team performance against both members individually.Source: §1 · p. 2
- The paper identifies missing explanations for human-AI team success and asks how to model synergetic potential and which factors contribute to CTP.It focuses these questions on decision-making and team performance measured against both members individually.Source: §1 · p. 2
- The proposed conceptualization distinguishes complementarity potential from complementarity effect, each with inherent and collaborative components, and identifies information and capability asymmetry as sources.The inherent component reflects the individually more accurate member; the collaborative component reflects gains emerging through joint decisions.Source: §1 · p. 2
- Two experiments examine information and capability asymmetry, and report that both sources increase inherent complementarity potential and realized effect, resulting in CTP.The studies use real-estate valuation and image classification decision tasks.Source: §1 · p. 2
- The authors present the conceptualization and experiments as contributions to IS research, intended to support the design of more effective human-AI decision-making collaboration.They connect this aim to hybrid intelligence and an “AI with human” perspective.Source: §1 · p. 2
2 Theoretical Foundations and Related Work
The section reviews sources and forms of human-AI complementarity, prior collaboration research, and related concepts. It distinguishes information and capability asymmetry and identifies a theoretical and empirical gap in existing complementarity work.
2.1 Sources of Complementarity: Information and Capability Asymmetry
- Information asymmetry concerns differences in available decision inputs, while capability asymmetry concerns differences in processing those inputs; both may support synergies but can also accompany performance degradation.Information may be unavailable to one team member, while humans and AI may process identical information differently.Source: §2.1 · pp. 3–4
- The paper uses “AI” broadly for machine-learning models or systems, following terminology prevalent in computer science and HCI.The authors acknowledge the technical distinction between AI and machine learning.Source: §2.1 · p. 3
2.2 Means for Harnessing Complementarity: Human-AI Collaboration
- Human-AI collaboration describes team members working synergistically toward shared goals, while human-AI decision-making specifically applies that collaboration to decision tasks.The AI may offer recommendations and humans may combine them with expertise and judgment to reach a team decision.Source: §2.2 · p. 4
- Although research examines reliance on AI advice and related decision supports, team performance often remains below AI-alone performance and the mechanisms behind synergies remain poorly understood.The review notes that appropriate reliance may depend on the situation and that human biases or misleading AI explanations can contribute to suboptimal decisions.Source: §2.2 · p. 5
2.3 Concepts Related to Human-AI Complementarity
- Related paradigms include intelligence augmentation, human-machine symbiosis, and hybrid intelligence, but existing studies do not provide the paper’s combined theoretical and empirical treatment of complementarity.The reviewed complementarity theories do not distinguish CP and CE with their components or empirically analyze decision-making in behavioral experiments.Source: §2.3 · p. 5
3 Conceptualization of Human-AI Complementarity
The section develops a decision-making framework for human-AI complementarity, distinguishing theoretical improvement potential from the portion realized by an integrated team. It identifies information and capability asymmetries as sources that can shape both.
3 Conceptualization of Human-AI Complementarity
- The framework models human-AI collaboration in decision tasks and distinguishes the potential for improvement from the effect actually realized by the team.Its concepts are intended to support systematic analysis and design of collaboration.Source: §3 · p. 5, §3.5 · p. 12
3.1 The Principle of Human-AI Complementarity
- Different human and AI errors create opportunities to improve on either individual, while combining their information or capabilities may produce decisions neither could reach alone.In the motivating example, choosing the correct individual decision reduces errors from the AI’s 13 to 5; collaboration could potentially address five instances neither solves individually.Source: §3.1 · p. 6
3.2 Human-AI Decision-Making Setting
- The decision setting comprises task instances with ground-truth labels, independent human and AI decisions, and an integrated team decision that may differ from both.Decision quality is measured as loss relative to ground truth, with overall loss averaged across instances.Source: §3.2 · p. 7
3.3 Complementarity Potential
- Complementarity potential measures the better individual’s loss relative to perfect decisions and separates instance-level gains from the inferior member from gains requiring new knowledge through collaboration.In the introductory example, the potential of 13 comprises 8 instances solvable using the better individual decision and 5 requiring collaboration.Source: §3.3 · pp. 7–9
3.4 Complementarity Effect
- Complementarity effect measures how much of the potential the integrated decision realizes; it can be positive or negative depending on whether collaboration improves on the better individual.In the hypothetical example, the team makes 9 errors versus 13 for AI and 15 for the human, realizing an effect of 4, or 31% of the potential of 13.Source: §3.4 · pp. 9–11, Figure 4 · p. 9
3.5 Sources of Complementarity: The Impact on Complementarity Potential and Effect
- Information and capability asymmetries can shape both inherent and collaborative potential and effect: the team may use the better-informed or more capable member’s decision, or combine information or capabilities into a different decision.Examples of complementary capabilities include human experience and AI computational power.Source: §3.5 · pp. 11–12
4 Experimental Studies
The studies examine human-AI decisions in a setting where a human receives AI advice and makes the final team decision. They investigate information asymmetry and capability asymmetry in separate experiments.
4 Experimental Studies
- The studies use a human-with-AI-advice setting in which the human makes the final team decision, which may follow either member’s decision or integrate both.An integrated decision can differ from both individual decisions.Source: §4 · p. 12
- The first experiment tests information asymmetry by giving the human additional contextual information unavailable to the AI.Source: §4 · p. 12
- The second experiment tests capability asymmetry by varying the diversity between human and AI capabilities.Source: §4 · p. 12
4.1 Experiment 1: The Effect of Information Asymmetry
Experiment 1 tested whether giving humans house images unavailable to the AI changes team decisions. The images improved human and team performance, and the study reports realized complementary team performance.
4.1.1 Task and AI Model
- The Kaggle appraisal task covered 15,474 houses; a random forest trained on tabular features withheld images, then predicted 15 detached-house cases (AI MAE $163,080).The hold-out cases were used in the behavioral experiment.Source: §4.1.1 · p. 12
4.1.2 Study Design
- Participants were randomly assigned to one of two conditions, completed a mandatory tutorial and two training tasks, then completed 15 house-price tasks; after exclusions, data from 101 participants were analyzed.Source: §4.1.2 · pp. 13–14
4.1.3 Evaluation Measures
- The study measured human, AI, and team errors as absolute errors averaged into MAE, then tested condition differences with corrected significance tests.The authors used Student’s t-tests or Mann–Whitney U-tests depending on prerequisites and applied a Bonferroni correction.Source: §4.1.3 · p. 14, §4.1.4 · p. 14
4.1.4 Results
- Team MAE improved from $160,095 without UHCI to $148,009 with UHCI; the team outperformed the AI (MAE $163,080) significantly only with UHCI.The UHCI team improvement was $12,086 (7.55%) and significant; without UHCI, the difference from AI was not a significant improvement.Source: §4.1.4 · p. 14, Figure 8 · p. 14
- UHCI shifted complementarity potential from collaborative to inherent: CP_inh rose from $42,995 to $61,970, while CP_coll fell from $120,085 to $101,110.Total CP remained $163,080 in both conditions.Source: §4.1.4 · p. 15
- Participants used 45% of inherent complementarity potential with UHCI versus 34% without it.The difference was significant (d = 0.82, p < 0.001).Source: §4.1.4 · p. 15
- Collaborative complementarity effect did not differ significantly between conditions, although positive effects appeared for task instances 1, 4, and 13.The authors describe a negative collaborative effect as intuitive in this setup, where the AI outperformed its human teammate and human decisions tended to fall between the two individual predictions.Source: §4.1.4 · p. 15
- With UHCI, total complementarity effect (CE) was $15,071 versus $2,985 without UHCI, and the study reports realized CTP.The authors identify humans’ realization of a disproportionate amount of inherent complementarity potential through unique contextual information as the key finding.Source: §4.1.4 · p. 15
4.2 Experiment 2: The Effect of Capability Asymmetry
The second experiment increased human–AI capability asymmetry while holding overall AI performance constant, then examined team decisions in image classification. The complementary-AI team improved over baseline and over the AI alone.
4.2.1 Task and AI Model
- The complementary AI was trained with human predictions to target images difficult for humans; the two AIs had equal 26.66% error on the selected task images.The intervention increased capability asymmetry by creating non-overlapping errors while holding overall performance constant.Source: §4.2 · p. 15, §4.2.1 · p. 17
4.2.2 Study Design
- In the online between-subject experiment, 144 participants classified 15 images each, first independently and then after receiving an AI recommendation they could revise.Participants were assigned to Baseline AI (76) or Complementary AI (68); they were not shown the true image classes during the task.Source: §4.2.2 · pp. 17–18
4.2.3 Evaluation Measures
- Human, AI, and team performance were measured as average classification error across task instances; the study also calculated CP, CE, and their components.Measures were averaged across participants.Source: §4.2.3 · p. 18
4.2.4 Results
- Inherent CP rose from 0.0640 (Baseline AI) to 0.2480 (Complementary AI), while collaborative CP fell from 0.2026 to 0.0186.Inherent CP made up 93% of overall CP with Complementary AI, compared with 24% with Baseline AI.Source: §4.2.4 · p. 18
- 0.1461 classification error with Complementary AI versus 0.2473 with Baseline AI (a 41% improvement); both teams beat the AI’s 0.2666 error.Figure 11 presents classification error across conditions with 95% confidence intervals and marks AI performance with a red horizontal line.Source: §4.2.4 · p. 18, Figure 11 · p. 19
- Total CE was 0.0193 (Baseline AI) versus 0.1206 (Complementary AI); CEinh realization rose from 58% to 89%, while CEcoll fell from -0.0175 to -0.0990.CEinh was 0.0368 with Baseline AI and 0.2196 with Complementary AI; the authors describe a trade-off between exploiting capability asymmetry and not relying on AI after seeing errors on easier instances.Source: §4.2.4 · pp. 18–19
5 Discussion
The discussion frames complementary team performance (CTP) as a central aim as AI moves from decision support toward independent performance. It connects the paper’s conceptualization and experiments to theoretical and managerial implications, limitations, and future research.
5 Discussion
- As AI increasingly performs tasks independently, the discussion calls for collaboration that treats humans and AI as equitable team members and aims for CTP.The paper contrasts this with earlier AI systems used for partial support in a narrower range of lower-stakes tasks.Source: §5 · p. 19
5.1 Contributions
- The conceptualization measures complementarity potential and effect, and both information and capability asymmetry can contribute to CTP in the two experiments.Unique human contextual information affected inherent potential and increased the realized inherent effect; greater capability asymmetry contributed to realizing a larger share of inherent potential in the specific experiment.Source: §5.1 · pp. 20–21
5.2 Theoretical Implications
- The conceptualization offers measures for examining human-AI team dynamics beyond aggregate performance comparisons and for testing hypotheses about socio-technical factors.The authors present these measures as a foundation for future human-AI collaboration research.Source: §5.2 · p. 21
5.3 Managerial Implications
- Managers should consider both the complementarity source and team-decision mechanism, and use the conceptualization to identify suitable use cases and improve collaboration.The discussion also suggests collecting data for teamwork-compatible AI and training employees to enhance inherent complementarity potential.Source: §5.3 · p. 21, §5.2 · p. 21
5.4 Limitations
- Laboratory and vignette methods may limit generalizability, while controlled experiments leave complex human factors and sequential decision effects unaddressed.The authors identify motivation, engagement, self-efficacy, team behavior, and situational factors such as time pressure as relevant areas for further study.Source: §5.4 · pp. 21–22
5.5 Future Work
- Future work could refine complementarity sources, study larger teams and alternative collaboration mechanisms, and examine creative or generative tasks.Proposed avenues include unique AI contextual information, AI-led integration or delegation, and criteria for contextual-information utility and capability asymmetry.Source: §5.5 · p. 22
6 Conclusion
The conclusion argues that as more decision tasks become automatable, human-AI collaboration should be designed around complementary team performance, in which the team outperforms each member.
6 Conclusion
- As decision tasks become increasingly automatable, the conclusion identifies CTP—team performance exceeding that of either member—as the objective for human-AI collaboration.It presents the paper’s conceptual foundation and empirical studies as support for future research on this collaboration paradigm.Source: §6 · p. 22