Source-linked AI summary
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
David Piorkowski, Soya Park, April Yi Wang, Dakuo Wang, Michael Muller, Felix Portnoy
TL;DR
AI development requires collaboration across roles with unequal data-science expertise, creating communication gaps that AI developers must help bridge. Through interviews and communication artifacts analyzed with shared mental models, the paper identifies gaps centered on knowledge, trust, and expectations and describes how developers cross those boundaries. The findings offer a preliminary account of these practices, limited by the study’s small qualitative scope.
Problem
Multidisciplinary AI collaboration creates communication gaps because AI developers and external stakeholders have mismatched expertise, making it important to understand how those gaps are addressed.
Method
The study analyzes semi-structured interviews and communication artifacts from AI developers using shared mental models as its analytic lens.
Results
AI developers’ communication gaps center on educating others, establishing trust, and setting expectations, while developers use cross-boundary communication practices to address them.
Takeaways & Limitations
The findings identify communication practices and shared-mental-model principles that can inform understanding of collaboration between AI developers and other stakeholders.
Takeaways & Limitations
The findings are preliminary because the study has a small scope and qualitative research design, limiting generalization beyond the studied setting.
Abstract
from arXiv · showhide
The development of AI applications is a multidisciplinary effort, involving multiple roles collaborating with the AI developers, an umbrella term we use to include data scientists and other AI-adjacent roles on the same team. During these collaborations, there is a knowledge mismatch between AI developers, who are skilled in data science, and external stakeholders who are typically not. This difference leads to communication gaps, and the onus falls on AI developers to explain data science concepts to their collaborators. In this paper, we report on a study including analyses of both interviews with AI developers and artifacts they produced for communication. Using the analytic lens of shared mental models, we report on the types of communication gaps that AI developers face, how AI developers communicate across disciplinary and organizational boundaries, and how they simultaneously manage issues regarding trust and expectations.
1 INTRODUCTION
This study examines how AI developers address communication gaps with stakeholders whose expertise differs from theirs. Using shared mental models as a lens, it analyzes communication challenges, responses, trust, and expectations in an IBM data science team.
- Analytic lens: Shared mental models frame communication as a way for team members to build common understandings of responsibilities and information needs.The framework is used to understand how collaborators anticipate one another’s needs and work together.
- Method: The authors conducted semi-structured interviews with four AI experts across 10 sessions and analyzed communication artifacts shared by some participants.The study provides a formative, in-depth account of communication gaps across the AI lifecycle.
- Research questions: The study asks which communication gaps AI developers face and how they overcome those gaps while communicating across roles.These questions target both the nature of the gaps and the practices used to address them.
- Contribution: The paper documents tools and techniques AI developers use to communicate across roles and stages, interpreted through shared mental models.The findings are intended to inform future work on communication challenges and collaboration tools.
- Research focus: The study focuses on inter-role communication gaps between AI developers and external stakeholders with mismatched expertise.AI developers include data scientists and AI-adjacent team members who require AI knowledge for their work.
- Findings: The reported gaps stem from mismatched expertise, while teams respond through informal education sessions and continuous sync-ups that establish trust.The paper also discusses how these practices manage expectations and support communication across boundaries.
2 RELATED WORK
Related work positions this study at the intersection of AI software engineering, data science collaboration, communication challenges, and shared mental model theory. It extends prior research by focusing specifically on inter-role communication between AI developers and domain experts.
- Related-work scope: Prior research covers AI software engineering, emerging AI-team roles, collaboration practices, communication challenges, and shared mental model theory.The paper organizes this literature into four related areas.
- AI software engineering: AI development spans data preparation, model building, deployment, and iterative feedback loops across data-centric, model-centric, and product-centric work.Prior studies examine both data-focused workflows and the integration of models into AI products.
- AI software engineering: AI teams must coordinate different roles and expertise while addressing problem formulation, training-data analysis, evaluation, interactions, performance, and social-good concerns.These demands make collaboration central to building AI products.
- Collaboration practices: Earlier collaboration research examined professional, civic, domain-expert, and software-oriented data science teams, including technical and non-technical members.This study builds on that work with a deeper focus on communication gaps and collaboration practices in AI development teams.
- Communication challenges: Prior studies describe communication difficulties involving different languages, motivations, business-to-data-science translation, and the need for broker roles.The literature also reports miscommunication, language barriers, intimidation concerns, and poor attitudes in collaborative software settings.
- Communication challenges: Existing collaboration systems mainly support communication among technical AI developers rather than communication between AI developers and domain experts.The present study therefore investigates practices between AI developers and domain-expert collaborators.
- Shared mental models: Shared mental models provide a lens for examining common ground around task requirements, procedures, role responsibilities, and collaboration activities.The framework is used to discuss how AI developers build shared understandings with other stakeholders.
3 METHODOLOGY
The study used interviews and communication artifacts from four AI developers working on production-targeted IBM projects. Researchers analyzed these data qualitatively through a shared mental models lens, documenting team roles, collaboration practices, and communication challenges.
- Participants: Four participants from different IBM teams contributed 10 hours of semi-structured interviews about recently completed or nearly completed ML or AI projects.The sample was recruited within one company under privacy and confidentiality constraints.
- Interview Protocol: Researchers conducted four project-overview sessions and six artifact-in-depth sessions, with participants sharing one to three communication artifacts per artifact interview.Artifacts included slide presentations, documentation, software repositories, and README files.
- Interview Protocol: The project-overview interviews mapped roles to AI lifecycle phases using jointly constructed Mural visualizations that participants modified to match their projects.Participants placed roles from their own and collaborating teams into the represented development phases.
- Interview Protocol: Artifact interviews examined how communication artifacts were created, used, and assessed for success and limitations, focusing on model evaluation and deployment or integration stages.These stages involved many roles across teams and required explanatory artifacts.
- Team Roles: The AI team gathered stakeholder needs, translated problems into data science terms, built models, and maintained communication through data scientists and strategic consultants.Stakeholder teams supplied domain knowledge, guidance, verification, and eventual ownership or maintenance of the final product.
- Team Roles: The lifecycle view shows roles entering and leaving different phases, leaving substantial work outside others’ view and making shared knowledge difficult to maintain.The process was presented linearly for analysis but was described as highly iterative.
4 RESULTS
The results identify varied communication challenges for AI developers and organize them around communication gaps, education, and concrete strategies for addressing those gaps. Examples from interviews and slide decks illustrate how developers communicated across roles.
- Results: Participants faced a varied assortment of communication challenges, which the qualitative analysis organized into two complementary parts.The first part identifies communication-gap themes and education; the second presents examples of challenges and responses.
- Communication Strategies: Vignettes use shared mental model principles to connect communication gaps with the techniques and tools participants used to address them.Real-life slide decks provide examples of how AI developers explained concepts and made their points across roles.
4.1 RQ1: Kinds of Communication Gaps
AI developers faced communication gaps rooted in mismatched expertise, difficulties establishing trust, and differing expectations about AI work and outcomes. They responded with informal, often repeated and tailored education, transparency, demonstrations, and expectation management.
- Overview: Participants identified three central communication challenges: knowledge gaps across roles, establishing trust, and setting expectations.These challenges arose in collaborations among technically and domain-specialized team members.
- Knowledge Gaps: Stakeholders’ limited AI knowledge could lead them to misinterpret model performance and misunderstand the contribution and hurdles of data scientists.AI teams also struggled to explain algorithms, evaluation metrics, and advanced statistical concepts without excessive detail.
- Knowledge Gaps: AI teams used informal education sessions and repeated examples to translate machine-learning terminology into business terminology, but no effective teaching process existed.This led to substantial trial and error in finding explanations that worked for stakeholders.
- Knowledge Gaps: Education was bidirectional: domain experts explained domain knowledge and desired model behavior, while AI teams repeatedly sought clarification as data patterns and findings changed.Domain knowledge was difficult to capture in one session and required continued confirmation throughout collaboration.
- Establishing Trust: AI developers built trust through repeated contact, regular meetings, explanations of model concepts and metrics, and additional time for difficult topics.They also addressed differences between AI development and traditional software development.
- Setting Expectations: Teams managed expectations by explaining trial-and-error development, model uncertainty, and why incremental work might not improve performance.They demonstrated sample predictions, explained model architectures, and increased transparency for stakeholders uncomfortable with a closed box.
- Setting Expectations: Different stakeholders sought different performance metrics, so AI experts tailored explanations and persuasion to each case rather than using one general communication approach.Early demonstrations were also used to compare AI predictions with current service techniques.
- Shared Mental Models: Shared mental model principles frame education and expectation setting as ways to align teams, anticipate misunderstandings, maintain trust, and reconcile problems when they arise.This framework was used to interpret why particular communication strategies helped address the identified gaps.
4.2 RQ2: How AI Developers Cross Communication Gaps
AI developers crossed communication gaps by building shared mental models through education, contextualized examples, documentation, and reactive explanations. These practices also helped manage stakeholder trust, expectations, and uncertainty across the AI development workflow.
- Communication gaps: Limited contextual evidence motivated the study’s focus on how AI developers communicate with other roles and why they do so.The authors frame the discussion around shared mental model principles, while noting that coordination evidence was absent.
- Communication gaps: Mismatched expertise produced different perspectives and languages about project goals, making shared understanding difficult.AI developers often understood the data science while stakeholders brought business or domain perspectives.
- Trust and expectations: Reactive communication addressed underperforming models, distrustful users, and shifting expectations by explaining evaluations, alternatives, and prediction validity.Participants diagnosed possible causes, prepared next-iteration approaches, and worked to persuade or educate users that predictions made sense.
- Documentation and shared resources: Repeated questions led teams to consolidate answers in shared repositories and documents, reducing interruptions and making information publicly available.Examples included Box, Dropbox, GitHub README files, and shared resources that supported consistency across asynchronous work.
- Education and contextualization: AI developers used education and stakeholder-relevant examples to explain model concepts and connect them to business problems.Presentations explained concepts such as precision, recall, and near-match measures, then mapped them to stakeholder roles and salient use cases.
- Documentation and shared resources: Documentation served both proactive and bidirectional roles by communicating model information while capturing domain knowledge and stakeholder feedback.High-level business and technical documentation supported varied audiences, while project documentation was revisited with stakeholders to confirm observations.
4.3 Artifact Analysis: Real-World Examples
Artifact analysis shows AI developers use carefully tailored visual, explanatory, and reusable materials to bridge communication gaps, adapting information to particular audiences. Creating these materials is labor-intensive, while reuse depends largely on developers’ memory.
- Analysis approach: The study triangulated interview accounts by analyzing communication artifacts and identifying how documented practices appeared in presentation materials.The artifact collection was not comprehensive and focused on selected AI development stages where participants shared materials.
- Communication techniques: AI developers used visualizations, examples, comparisons, and analogies to explain data science concepts and model behavior.Examples included confusion matrices, feature-change tables, cross-dataset comparisons, and a scale analogy for normalization.
- Communication techniques: Images served both explanatory and recall functions, with repeated visual elements helping participants remember prior discussions.A descriptive confusion matrix was intentionally reused to trigger memories of earlier conversations.
- Audience adaptation: Slides were highly targeted, typically conveying one contextualized message and tailoring information to the intended audience.The artifacts were described as precisely crafted to suit particular audiences rather than following a single communication format.
- Reuse and effort: There was no one-size-fits-all approach: designing information was time-consuming but considered necessary for building an effective shared mental model and supporting project success.Some explanatory content, including confusion matrices, was reused and adapted to new contexts when developers remembered it.
- Reuse and effort: Reuse depended on memory because the available tools lacked affordances for searching prior explanations, and the study could not generalize beyond its participants.Participants iteratively customized information until a communication gap was crossed, then reused it when possible.
5 DISCUSSION
The discussion identifies recurring practices through a shared mental model lens: planning, documentation, iterative feedback, and staged expansion help AI teams coordinate with stakeholders. These practices support mutual understanding while keeping stakeholders engaged with model development.
- Shared mental models: The study reports communication challenges and specific responses, framing observed practices through shared mental model theory.The analysis maps emergent practices to shared mental model components to explain why they may be successful.
- Planning and self-correction: Planning and brainstorming give AI teams time to learn domain knowledge and adjust implementation through self-correction.The paper identifies this as a commonality between machine-learning and software-engineering development.
- Documentation: Documentation provides contextualized communication, answers repeated stakeholder questions, bridges teams, and supports knowledge management.The study reports greater documentation use in its participating teams than prior literature had suggested.
- Documentation: The participating teams’ ad-hoc structure may help explain their documentation practices: independent teams formed to integrate machine learning into a business model.This organizational arrangement differed from patterns reported in earlier literature about data-science documentation.
- Stakeholder involvement: AI teams minimize time to deliver executable models so stakeholders stay involved, provide feedback, and gain confidence using the model.The study also describes gradual expansion from one business sector to others after the model becomes stable.
- Stakeholder involvement: After successful launch in one sector, teams expand models to other sectors and streamline the process using prior modeling experience.The paper characterizes this iterative expansion as reflexivity.
5.2 Design Implications for Collaboration Tools
The design implications call for collaboration and documentation tools that adapt information to different roles and remain connected to rapidly changing AI work. The paper also identifies automation and better integration as possible directions, while noting unresolved trust and filtering questions.
- Role-specific tools: Collaboration tools should tailor both information and presentation to the roles receiving them because one size does not fit all.Tools should account for what different roles need to process, care about, and share.
- Role-specific tools: Documentation tools should dynamically vary lifecycle content by role and provide context rather than isolated statistics or visualizations.Static documentation does not address how different roles require different information during AI development.
- Current tool limitations: Participants relied on PowerPoint for communication and education, but it was poorly suited to maintaining complete, timely documentation.One participant reported spending 2–3 days creating slides, which captured snapshots of rapidly changing work.
- Current tool limitations: PowerPoint’s flexibility and familiarity may support knowledge transfer, including when stakeholders reuse presentations for further communication.The paper presents this as a possible reason participants tolerated the cost of creating slides.
- Tool integration: The high cost of translating work into slides suggests closer integration between data-science and communication tools, although more research is needed.The paper mentions combining chat and computational notebooks as one example of such integration.
- Automation and coordination: Automatic summarization could monitor code and repository changes to keep team members updated, but trust and role-specific information filtering remain open questions.The paper links improved visibility with potentially better collaboration while retaining these unresolved design concerns.
- Automation and coordination: Coordination brokers and other intermediaries can create a ‘Game of Telephone’; suitable tools and education are proposed as ways to make cross-role conversation more efficient.The paper connects this issue with emerging work in AutoML and machine teaching.
5.3 Threats to Validity, Limitations, and Future Work
The authors characterize the findings as preliminary and bounded by a small, single-company, one-sided qualitative study. They call for broader samples, additional perspectives, other methods, and multiple theoretical lenses in future work.
- Scope and sample: The qualitative findings are preliminary because the study included only four interviews and 10 interview sessions.The authors recommend larger and more varied informant samples, surveys, and ethnographic studies.
- Scope and sample: Because participants came from a single company, other organizations may have different team structures, role compositions, and communication gaps.The authors caution against generalizing the findings beyond the studied organizational context.
- Future work: Future work could combine evidence from successful and failed communication practices to learn which practices lead to successful communication.The authors also position the study as one contribution toward a broader understanding built from qualitative accounts across contexts.
- Perspective: The study presents only the AI developers’ perspective even though communication is bidirectional between AI developers and stakeholders.Future work should examine complementary stakeholder-team perspectives.
- Theoretical scope: The shared mental model lens clarified selected issues but may marginalize others, motivating future studies using multiple theoretical lenses.The interviews also did not explicitly capture the coordination principle, possibly because of the interview design or focus on later development phases.
- Study phases: The study’s focus on model evaluation and deployment may have missed coordination activities concentrated in earlier requirements and data-gathering phases.The authors expect earlier phases to contain more coordination-principle-related details.
6 CONCLUSION
The case study identifies communication gaps across the AI development lifecycle and examines how AI developers address them across roles. Its findings center on educating others, establishing trust, setting expectations, and informing best practices.
- The shared mental model lens identified communication gaps involving knowledge, trust, and expectations across the AI development lifecycle.
- Interviews revealed varied information requests from colleagues and how AI developers approached and fulfilled them.
- The findings informed best practices for AI development workflows and offered insights for researchers and designers addressing communication challenges.
- The paper concludes that participants still lack a dedicated tool to help overcome communication gaps.
- Participants described the need for templates or tools that could guide communication artifacts and reduce brainstorming effort.
A TEAM COMPOSITION OF EACH TEAM
The interviewed teams included different roles across stages of the machine-learning workflow, and their composition varied with project circumstances. The figures identify team-specific configurations for Data1, Data2, Data3, and Strat1.
- Team configurations show how different roles participate in each stage of the machine-learning workflow.
- Team composition varies with data availability, whether data is already labeled, and role responsibilities across teams.
- The figures label team configurations for Data1, Data2, Data3, and Strat1.