PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Team Label, Different Evidence: A Full-Text Audit of Claim Denominators in Human-AI Teaming Research

Hanjing Shi, Kimberly Wang, Sabrina Doherty, Dominic DiFranzo

arXiv:2609.27849v1cs.HC

TL;DR

HAT labels can group evidence about different human arrangements, even though trust, coordination, performance, and accountability claims concern particular human relations and outcome levels. Full-text reading moved 36 of 74 apparent core candidates out of the core empirical denominator and separated evidence into advice, peer coordination, task orchestration, collective facilitation, and oversight. Team/collaboration outcome vocabulary appeared in 14/27 dyadic versus 20/23 peer-team records; unanimous surface triage diverged from full-text labels in 32% of screening and 34% of arrangement cases.

  • Problem

    HAT labels can group evidence about different human arrangements, even though trust, coordination, performance, and accountability claims concern particular human relations and outcome levels.

    Source: §1 · p. 1

  • Method

    Full-text reading moved 36 of 74 apparent core candidates out of the core empirical denominator and separated evidence into advice, peer coordination, task orchestration, collective facilitation, and oversight.

    Source: §8 · p. 18

  • Results

    Team/collaboration outcome vocabulary appeared in 14/27 dyadic versus 20/23 peer-team records; unanimous surface triage diverged from full-text labels in 32% of screening and 34% of arrangement cases.

    Source: §8 · p. 18

  • Takeaways & Limitations

    A claim-pooling checkpoint can assess comparability by tracing human arrangement, AI role, relational coupling, team process, and outcome level.

    Source: §6 · p. 16, §6.3 · p. 17

    Because the sample is purposive, its counts describe this 86-paper set rather than HAT research as a whole.

    Source: §7 · p. 17

Abstract

from arXiv · show

Human-AI Teaming (HAT) reviews often group studies by labels such as advisor, teammate, or coordinator. Yet the same label can describe one person taking AI advice, several people coordinating around AI, or a workflow that distributes authority and responsibility. Pooling these studies can therefore change the human unit behind a claim. We examine how full-text evidence changes the set of studies behind a claim. We audited 86 full texts purposively selected from a 419-record title/abstract map. We find that full-text reading changed core membership for 40 records: 36 of 74 apparent core candidates moved out, while 4 of 12 boundary candidates moved in. Team vocabulary did not reliably identify the social unit: 14 of 27 human-AI dyads and 20 of 23 multi-human peer teams used team or collaboration terms. Only 20 of 86 papers specified who could see AI output. Four blinded language-model runs unanimously labeled 53 screening cases and 59 arrangements, yet 32% and 34% of those consensus decisions differed from the full-text labels. These results identify claim-denominator drift as a synthesis problem in HAT research. We contribute a full-text audit centered on human arrangements and a claim-pooling checkpoint for deciding when evidence about trust, coordination, performance, efficiency, and accountability can be compared.

1 Introduction

HAT labels can group studies with different human arrangements, changing the unit behind a pooled claim. This audit traces how arrangement, AI role, team process, and outcome level affect whether evidence is comparable.

1 Introduction

  • HAT labels can pool evidence about a person, a team, a workflow, or an institution, although these settings do not produce the same kind of evidence.An endoscopist taking AI advice and an ICU team coordinating with an AI agent illustrate how a shared label can conceal different human units.Source: §1 · pp. 1–2
  • Trust, performance, collective intelligence, and accountability depend on human relations and outcome levels that cannot be inferred from the AI role alone.Trust may concern reliance on AI, confidence among teammates, or confidence in a deploying institution; performance may concern an individual, group, or organization.Source: §1 · p. 1
  • The audit asks which human arrangements and evidence roles become visible, how human and AI roles organize team processes, and whether outcome claims align with those relations.It examines visibility, control, responsibility, and the level at which outcomes are reported.Source: §1 · p. 2
  • The study combines a 419-record title/abstract map with an 86-paper full-text audit to distinguish search vocabulary from recoverable evidence about human relations.The map establishes vocabulary; the audit determines what evidence selected papers provide.Source: §1 · p. 2
  • The Human-AI Teaming Mechanism Chain links human arrangement and AI role to relational coupling, team process, and outcome claim, informing a checkpoint for deciding when evidence can be compared.The checkpoint addresses claims about trust, coordination, performance, and accountability.Source: §1 · p. 2

2 Related Work: From AI Roles to Human Arrangements

Prior HAT reviews classify definitions, interaction patterns, settings, configurations, and team types. This paper instead asks whether reports contain the relational evidence needed to support comparable claims about teams and outcomes.

Review as a diagnostic

  • Reviews can expose mismatches between a field's unit of study and the phenomenon it seeks to explain, as well as conceptual and methodological gaps.Examples include individual-focused misinformation interventions despite the relevance of prior beliefs and social context [Aghajari et al. 2023].Source: §2 · p. 2

Human-AI review frameworks

  • Human-AI co-creation frameworks organize agency, control, context, and application, whereas this audit evaluates control alongside authority, visibility, responsibility, team process, and outcome level.The cited frameworks support classification and design of co-creative systems [Shi et al. 2023; Zhang et al. 2025].Source: §2 · pp. 2–3

Related HAT reviews

  • Reviews classify HAT definitions, interaction patterns, testbeds, complementarity, and team types, while this audit asks whether studies report comparable human relations and outcome levels.The audit treats claim-denominator comparability as a question distinct from classifying systems or configurations.Source: §2 · p. 3

From team type to mechanism

  • Team taxonomies distinguish configurations, while the mechanism-chain analysis traces how arrangements make processes possible and which outcome levels reports can support.Recent syntheses foreground constellations, trust, shared cognition, and team types [Schmutz et al. 2024; Hughes and Habli 2026].Source: §2 · p. 3

Theoretical foundations

  • Cooperative-work and organizational theories connect coordination to shared artifacts, appropriation, common ground, expertise, awareness, articulation work, and accountability.These traditions motivate recording relations behind reliance, coordination, collective work, and accountability rather than treating “team” or an AI role as a stable explanatory unit.Source: §2 · p. 3

3 Method: Corpus Construction and Full-Text Audit

The review separates a broad title/abstract search map from a purposive full-text audit. Full-text evidence-location coding, rather than surface similarity, determines which selected records can support claims.

3.1 Review design

  • The review uses a title/abstract map to locate HAT vocabulary and a purposive full-text audit to assess evidence and human arrangements.The 86 audited papers were selected to expose comparison changes across varied arrangements.Source: §3.1 · pp. 3–4

3.2 Corpus construction

  • The search map combined targeted queries, a seed bibliography, OpenAlex expansion, and backward snowballing, leaving 419 primary records after duplicate checks.The search covered HAT and collaboration terms across several research areas.Source: §3.2 · p. 4
  • The 86-paper audit purposively sampled arrangement neighborhoods where comparison errors were most likely to matter, including dyads, peer teams, workflows, and oversight settings.Eligibility also required a relevant mechanism or outcome and sufficient full-text access for evidence-location coding.Source: §3.2 · p. 4
  • The other 333 primary records remain search context; only the 86 audited records received evidence-location coding for determining claim scope.The selection path from the title/abstract map to the full-text audit is summarized in Table 1.Source: §3.2 · p. 4, Table 1 · p. 5

3.3 Evidence roles and codebook development

The audit distinguishes evidence roles and arrangement types through explicit coding gates, then records evidence locations and mechanism memos so interpretive judgments can be reviewed.

3.3 Evidence roles and codebook development

  • Core empirical records report evidence tied to a recoverable human-AI arrangement, while boundary records inform mechanisms or roles without supporting the focal claim through a concrete arrangement.Reviews, conceptual frameworks, and theory sources are categorized as background.Source: §3.3 · p. 5
  • The codebook uses gates for screening, evidence role, arrangement, authority and stakes, and reporting quality, with a textual location cue for each included or boundary record.Coded dimensions include expertise, temporal structure, visibility, control, responsibility, outcomes, and accountability.Source: §3.3 · pp. 5–6, §3.4 · p. 6

3.4 Coding workflow

  • The workflow decomposed synthesis into retrieval, coding, evidence capture, mechanism memoing, normalization, agreement diagnostics, and adjudication.Twelve calibration records were coded independently by three coders; 24 of 86 records had overlapping coder entries.Source: §3.4 · p. 6

3.5 Mechanism-chain synthesis

The synthesis links reported outcomes to the human arrangement, AI role, relational coupling, and team process. It groups recurring coded sequences into five nonexclusive analytic routes.

3.5 Mechanism-chain synthesis

  • The final analytic pass connected human arrangement and roles, AI role, relational coupling, team process, and the level of the reported outcome.Visibility, autonomy, control, and responsibility informed the relational-coupling analysis.Source: §3.5 · p. 6
  • Outcome terms were assigned to individual, team, workflow, and institutional levels, with multi-label coding when papers used terms at several levels.The coded terms include individual reliance, team coordination, workflow efficiency, and institutional accountability.Source: §3.5 · pp. 6–8
  • The keyword rule classified coder-extracted reporting vocabulary, not whether every matched construct was independently operationalized or measured; 78 of 86 records were included.Records with outcomes marked unrecoverable from available full text were excluded.Source: §3.5 · p. 8
  • Five recurring analytic routes—advisory calibration, shared coordination, task orchestration, collective facilitation, and supervisory accountability—were synthesized qualitatively, not as mutually exclusive paper types.A system may advise one person while also exposing a trace to a supervisor, supporting different claims.Source: §3.5 · p. 8

3.6 Normalization, agreement diagnostics, and adjudication

  • The normalization script standardized spelling and codebook aliases while preserving raw entries and routing interpretation-changing values for lead review.It did not assign final codes or make substantive recoding decisions.Source: §3.6 · p. 8
  • Agreement diagnostics identified fields needing interpretive judgment, and lead adjudication assigned final codes and reasons to 171 queued issues and 141 flagged record-level cells.Finalized screening codes were used for denominator analysis.Source: §3.6 · p. 8

3.7 Surface-triage diagnostics

The surface-triage diagnostics compared title-and-abstract signals with adjudicated full-text labels. They used both a TF–IDF procedure and blinded language-model runs, while the results section proceeds from evidence role to arrangement, mechanism path, and outcome level.

3.7 Surface-triage diagnostics

  • Both surface-triage diagnostics were scored against adjudicated full-text labels kept in a separate file.The full-text codes remained the analytic labels rather than outputs of the diagnostics.Source: §3.7 · pp. 8–9
  • The TF–IDF diagnostic vectorized title, abstract, screening rationale, and matched query terms, then used leave-one-out nearest-centroid triage against the final screening decision.Records were clustered with cosine k-means for k=5 to 12, with k selected by silhouette.Source: §3.7 · p. 8
  • Claude Opus 5 and GPT-5.6 Sol each ran once under two blinded conditions, without full texts, coding sheets, adjudication trails, or other runs’ outputs.Condition B supplied title, venue, year, and abstract; abstracts were retrievable for 40 of 86 records.Source: §3.7 · pp. 8–9
  • The diagnostic runs were treated as surface-representation error analyses, not evaluations of general model capability.Each run assigned screening decisions, arrangement buckets, and self-reported confidence for all 86 records.Source: §3.7 · p. 9

3.8 Analytic materials

  • The results move from evidence role to arrangement, mechanism path, and outcome level, with Table 2 defining the coding gates.Analytic materials include published records, bibliographic metadata, and the research team’s coding work product; copyrighted full texts are not redistributed.Source: §3.8 · p. 9

4 Findings

Full-text coding changed which studies supported HAT claims, while team vocabulary often crossed distinct human arrangements. The audit linked evidence use to reported relations and showed why surface agreement cannot substitute for adjudicated full-text evidence.

4.1 Full-text audit changed the claim denominators

  • The audit identified 27 dyads and 23 peer teams; all 42 core records reported arrangements explicitly, while all eight context records were unrecoverable.Other arrangements included organizational workflows, clinical workflows, governance settings, and instructional groups.Source: §4.1 · p. 9
  • Full-text coding moved 36 of 74 apparent core candidates out and four of 12 boundary candidates in, yielding 42 core empirical records.The final base also included 23 boundary-mechanism, 13 background/theory, and eight context records.Source: §4.1 · p. 9, Figure 1 · p. 10

4.2 Team vocabulary crossed different human arrangements

  • Team-level terms appeared in 14 of 27 dyads and 20 of 23 peer teams, so outcome vocabulary alone could pool studies with different human relations.The adjudication rule followed the enacted arrangement rather than the title label.Source: §4.2 · p. 10

4.3 Relational coupling specified different mechanisms

  • Advisory AI most often paired with individual-level vocabulary (19 of 25), while facilitation paired with team-level terms (seven of eight); audience-specific visibility was reported in only 20 of 86 records.Responsibility patterns also overlapped descriptively with vocabulary: 18 of 27 records with individual final decision used individual-level terms, compared with 23 of 29 where responsibility was shared.Source: §4.3 · p. 11
  • Five analytic routes connect coupling to processes such as reliance, calibration, delegation, collective work, and accountability; the audited clinical studies support distinct advice, coordination, and oversight claims.The routes are not exclusive paper types or causal estimates; comparisons can proceed within a route or across routes when the arrangement change is itself part of the explanation.Source: §4.3 · pp. 11–12, Table 4 · p. 12

4.4 Surface triage reproduced the same collapse

  • TF–IDF triage matched 36 of 86 screening decisions, below the 42-of-86 majority baseline; adding rationale and matched terms did not improve either language model.Language-model arrangement accuracy ranged from 48 to 50 of 86 across four runs.Source: §4.4 · p. 12, Table 5 · p. 13
  • Four-run consensus labeled 53 screening cases and 59 arrangements alike, yet 32% and 34% respectively diverged from full-text labels; repeated agreement was not a correctness signal.Surface triage can prioritize reading but cannot replace the full-text step that fixes the claim denominator.Source: §4.4 · p. 13

4.5 Reporting and adjudication preserved claim scope

  • Reporting quality constrained evidence use, and final rules kept topic, mechanism, and observed relation separate; adjudication used textual evidence locations to resolve coding decisions.Across 171 field-level issues, 73 were normalized disagreements, 40 incomplete screening fields, 38 invalid or substantive values, and 20 semantic aliases.Source: §4.5 · pp. 13–14, Figure 2 · p. 14

5 The HAT Mechanism Chain and Claim-Pooling Checkpoint

The Human-AI Teaming Mechanism Chain turns full-text coding into a checkpoint for deciding which findings can share a claim denominator. Pooling requires comparable human relations, recoverable claim-relevant coupling and process, and matched outcome levels; the chain scopes evidence rather than modeling causation.

5 The HAT Mechanism Chain and Claim-Pooling Checkpoint

  • Pool findings only when the human relation is comparable, claim-relevant coupling and process are recoverable, and the outcome level matches.Otherwise, keep records in separate evidence bodies or use them only as mechanism or context evidence; the sequence is not a causal model.Source: §5 · pp. 14–15, Figure 3 · p. 15
  • The checkpoint applies to trust, performance, accountability, and complementarity claims by recording the social conditions under which each result was produced.Two studies may be pooled when the relation and process required by the outcome remain comparable; sharing a process alone may support mechanism evidence, not the same team-level outcome.Source: §5 · pp. 14–15
  • Authority changes what visibility means: a shared trace may support peer coordination but serve as evaluative evidence in an instructional or managerial hierarchy.The reviewer asks what the visible trace lets participants know and do, and who has authority over it.Source: §5 · p. 16

6 Discussion: From Team Configuration to Team Mechanism

The discussion argues that HAT evidence should be interpreted through the human arrangement and mechanism behind an outcome, not team configuration alone. It distinguishes outcome levels, proposes reporting and synthesis practices, and treats review judgments as accountable to texts and explicit rules.

6 Discussion: From Team Configuration to Team Mechanism

  • The mechanism chain asks what an arrangement lets participants know, do, contest, and be held responsible for, then locates the outcome at the individual, team, workflow, or institutional level.The same AI system may provide private advice, a shared coordination object, or a trace available to an evaluator; configuration alone does not establish what claim the evidence supports.Source: §6 · p. 16

6.1 AI roles acquire meaning through human relations

  • Private advice, peer-shared recommendations, and manager-visible records support different mechanisms because authority and visibility participate in producing outcomes.The literature often reports AI presence without specifying the audience, leaving visibility—the condition theory treats as central to common ground—least available for comparison. The chain makes the reporting path inspectable across domains without assuming the mediator stays unchanged.Source: §6.1 · p. 16

6.2 Trust, collective performance, and accountability occupy different levels

  • Trust claims must name their referent—AI, human teammates, the mixed team, or the deploying organization—because these constructs may interact but do not substitute for one another.Collective performance requires evidence that humans integrated knowledge through contribution integration, shared cognition, or coordinated action; efficiency claims must also specify the unit and cost.Source: §6.2 · p. 17
  • Accountability requires connecting an AI trace to authority and consequences: someone must be able to question, sanction, correct, or justify the decision, and control must be specified.A log alone does not create accountability; visibility may support peer awareness yet increase surveillance or shift blame in a hierarchy.Source: §6.2 · p. 17

6.3 How future HAT research can use the mechanism chain

  • Researchers can use the chain before data collection to hold the social unit and outcome level constant across AI conditions, while observing contribution integration or separating task speed from coordination costs.For accountability studies, report visibility alongside authority, control, and consequences.Source: §6.3 · p. 17
  • Only 20 of 86 audited records specified an audience for AI output; naming whether one participant, a group, or an evaluator sees it makes coordination claims checkable.Reviews can group findings by mechanism path before comparing effect directions, and benchmarks can annotate AI role alongside visibility, override rights, responsibility, and outcome level.Source: §6.3 · p. 17

6.4 Synthesis as accountable evidence work

  • The mechanism chain serves as a shared object for review teams, requiring coders to connect a role to a relation and process before assigning the scope of an outcome claim.Computational checks can find surface-near records and schema problems, but consensus alone cannot authorize a denominator; judgments remain accountable to text locations and explicit rules.Source: §6.4 · p. 17

7 Limitations and Future Work

The authors bound the audit's counts and coding conclusions to the sampled and adjudicated records. They also identify limits in the surface-triage diagnostics, keyword-based outcome assignment, and the scope of the mechanism paths.

7 Limitations and Future Work

  • The purposive audit's counts describe its 86-paper set, not the prevalence of arrangements across HAT research.Estimating prevalence would require a representative or exhaustive corpus and controlled exclusion reasons for records outside the audit.Source: §7 · pp. 17–18
  • Coder agreement statistics characterize the 24-paper overlap subset, while final counts are an application of the codebook, not an estimate of universal coding reliability.The remaining records contribute primary coding and evidence-location notes.Source: §7 · p. 18
  • Surface-triage comparisons describe performance on this audit, not what abstract text alone carries; abstract availability is confounded with record provenance and selection.Each condition was run once per model, and two model families cannot establish a universal pattern.Source: §7 · p. 18
  • Keyword-assigned outcome levels inherit imprecision from coder-extracted terms and describe reporting vocabulary, not measurement prevalence or analytic focus.The rule cannot distinguish a construct a paper measured from one it merely mentioned or interpreted; records with multiple terms receive multiple levels.Source: §7 · p. 18
  • The five mechanism paths are nonexclusive analytic routes, not an exhaustive HAT taxonomy or estimates of causal mediation.Future work can test prospectively whether mechanism-matched synthesis explains variation in reported effects better than AI-role or domain labels alone.Source: §7 · p. 18

8 Conclusion

Full-text reading changed which studies belonged in the core empirical denominator and revealed that team vocabulary and unanimous model decisions did not reliably recover the human arrangement. The paper proposes a claim-pooling checkpoint that keeps arrangements and processes visible when comparing outcomes.

8 Conclusion

  • 36 of 74 apparent core candidates moved out of the core empirical denominator after full-text reading, separating papers by evidence role.Source: §8 · p. 18
  • 14 of 27 dyads and 20 of 23 multi-human peer teams used team or collaboration vocabulary, while only 20 of 86 papers specified who could see AI output.Unanimous model decisions differed from full-text labels in 32% of screening cases and 34% of arrangement cases.Source: §8 · p. 18
  • The Human-AI Teaming Mechanism Chain links human and AI roles to relational coupling, team process, and outcome claims, with a checkpoint for deciding when findings can be pooled.Source: §8 · p. 18

A Record-Level Evidence Availability

Figure 4 maps reported or recoverable evidence fields and outcome vocabulary across the audited records, retaining evidence-role and arrangement groupings. The map shows where relational detail is available for empirical claim pooling and where audience visibility remains sparse.

A Record-Level Evidence Availability

  • Figure 4 sorts each audited record by final evidence role and human-arrangement family, then maps recoverable fields and outcome-level vocabulary.Counts are audit-set denominators, not prevalence estimates.Source: A Record-Level Evidence Availability · p. 22, Figure 4 · p. 22
  • The map tracks arrangement, authority, stakes, output audience, AI control, responsibility, and outcome vocabulary; outcome coding is multi-label.Source: A Record-Level Evidence Availability · p. 22
  • Core and boundary records usually provide enough detail to recover arrangement, control, responsibility, and outcome vocabulary, but audience-specific visibility remains sparse; background and context records may lack detail needed for empirical claim pooling.Source: A Record-Level Evidence Availability · p. 22
Loading 2609.27849v1…