PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding Explainability: Towards Social Transparency in AI systems

Upol Ehsan, Q. Vera Liao, Michael Muller, Mark O. Riedl, Justin D. Weisz

arXiv:2101.04719v1cs.HCcs.AI

TL;DR

AI-mediated consequential decisions require explanations that address more than algorithmic behavior, yet XAI has largely centered on technical transparency. The paper introduces and explores Social Transparency through scenario-based design and a formative study, developing a framework that makes technological, decision, and organizational contexts visible.

  • Problem

    XAI has predominantly centered on algorithmic transparency despite AI systems and explanations being embedded in socio-organizational contexts.

  • Method

    The authors use scenario-based speculative design and formative studies to explore Social Transparency and develop its design elements and conceptual framework.

  • Results

    The framework identifies technological, decision, and organizational contexts made visible by Social Transparency and their potential effects.

  • Takeaways & Limitations

    Social Transparency expands the conceptual and practical design space of XAI by incorporating socio-organizational context into AI-mediated decision-making.

  • Takeaways & Limitations

    The study's insights are formative rather than evaluative and are constrained by the dependency between its scenario and data.

Abstract

from arXiv · show

As AI-powered systems increasingly mediate consequential decision-making, their explainability is critical for end-users to take informed and accountable actions. Explanations in human-human interactions are socially-situated. AI systems are often socio-organizationally embedded. However, Explainable AI (XAI) approaches have been predominantly algorithm-centered. We take a developmental step towards socially-situated XAI by introducing and exploring Social Transparency (ST), a sociotechnically informed perspective that incorporates the socio-organizational context into explaining AI-mediated decision-making. To explore ST conceptually, we conducted interviews with 29 AI users and practitioners grounded in a speculative design scenario. We suggested constitutive design elements of ST and developed a conceptual framework to unpack ST's effect and implications at the technical, decision-making, and organizational level. The framework showcases how ST can potentially calibrate trust in AI, improve decision-making, facilitate organizational collective actions, and cultivate holistic explainability. Our work contributes to the discourse of Human-Centered XAI by expanding the design space of XAI.

1 INTRODUCTION

The paper argues that algorithm-centered XAI misses the social and organizational contexts in which explanations and AI-mediated decisions are situated. It introduces Social Transparency (ST) as a first step toward socially situated XAI.

  • Explanations support reasoning, justification, sense-making, decision-making, and coordination in personal and social contexts.
  • Human-centered research finds popular XAI techniques ineffective, potentially risky, and underused in real-world contexts.
  • Dominant algorithm-centered XAI privileges technical and model transparency, leaving significant gaps from how people seek and produce explanations.
  • Because consequential AI systems are embedded in socio-organizational settings, technical explanations alone cannot provide a holistic understanding of AI-mediated decisions.
  • The paper introduces Social Transparency (ST), incorporating socio-organizational context into explainability and expanding XAI beyond algorithmic transparency.
  • Using scenario-based speculative design and a formative study, the paper develops ST design elements and a framework spanning technological, decision, and organizational contexts.

2 RELATED WORK

Related work identifies limits in algorithm-centered XAI, including weak user evidence, mixed effects, and insufficient attention to explanations as social and organizational processes.

  • Limits of technical XAI: There is limited understanding of how people perceive and consume AI explanations, while studies report mixed effects on trust and task performance.
  • Limits of technical XAI: Explanations can induce over-trust, overestimate model capabilities, increase cognitive workload, and provide little evidence of improving perceived accountability or control.
  • Limits of technical XAI: Current XAI often represents model internals, although explanations also transfer knowledge and should address users’ beliefs and knowledge gaps.
  • Sociotechnical gap: AI systems are socially situated, yet sociotechnical perspectives remain mostly absent from XAI, despite organizational interpretation involving cooperation and mental-model comparison.
  • Sociotechnical gap: The paper expands XAI by incorporating socio-organizational factors and proposing Social Transparency as an operational step toward sociotechnical XAI.

3 SOCIAL TRANSPARENCY IN AI SYSTEMS: A SCENARIO BASED DESIGN EXPLORATION

The authors use scenario-based design to explore Social Transparency in AI-mediated decisions and develop 4W features representing users’ socio-organizational context.

  • Concept and approach: Social Transparency adds socio-organizational context to AI systems to support explainability of AI-mediated decision-making.
  • Concept and approach: Scenario-based design uses narrative descriptions and visual mock-ups to envision future use possibilities while retaining interpretive flexibility.
  • Concept and approach: Four workshops with 21 people from eight technology companies explored AI-mediated decision-making scenarios in cybersecurity, hiring, healthcare, and sales.
  • 4W design features: Participants converged on the 4W—who did what with the AI system, when, and why—to provide socio-organizational context around decisions.
  • Design choices: Pilot studies found that presenting the entire scenario created cognitive and visual clutter, so interview blocks were revealed sequentially.

4 STUDY METHODS

The study used semi-structured remote interviews with 29 AI users and practitioners, a speculative sales scenario, and qualitative thematic and grounded-theory analysis.

  • Interview procedure: The visual scenario presented an AI price recommendation, technical explanation, ST summary, and three sequential 4W blocks for a sales decision.
  • Recruitment: The researchers recruited 29 participants with experience using, developing, or designing AI systems interacted with by multiple users.
  • Recruitment: Participants included sellers and stakeholders from multiple domains and six companies, with eight recruited sellers and non-sellers clustered mainly in healthcare and cybersecurity.
  • Recruitment: Interviews were conducted remotely during Covid-19, which added recruitment burden and limited the setting to video conferencing.
  • Interview procedure: Participants gave initial and final prices and confidence levels, ranked 4W elements, discussed transfer scenarios, and considered unwanted consequences.
  • Analysis: The study was not designed to quantitatively evaluate Social Transparency; summary statistics instead guided the qualitative analysis.
  • Analysis: The interviews averaged 58 minutes, yielding roughly 29 hours of data analyzed through open coding, thematic analysis, grounded theory, and consensus-building.

5 FINDINGS

Participants found that technical transparency alone did not meet explainability needs, motivating Social Transparency’s inclusion of broader decision and organizational context. The study reports effects across technological, decision-making, and organizational levels.

  • Technical transparency alone did not meet participants’ explainability needs because important decision context extended beyond the algorithm.Participants described technical explanations as insufficient or unintelligible without contextual guidance from colleagues.
  • Social Transparency made additional context visible around AI-mediated decisions, including technological, decision-making, and organizational levels.The findings organize ST’s effects across three levels and summarize them in Table 2.
  • Participants repeatedly emphasized that algorithms cannot capture all contextual factors relevant to decisions, including social relationships and organizational activity.Examples included client allegiances, internal projects, colleagues’ actions, and relationship-dependent considerations.
  • The findings suggest that ST information helped participants set prices more cautiously and feel more confident by supplying context beyond the machine.Participants described context as filling gaps needed to understand and evaluate AI-mediated decisions.
  • 24 out of 29 participants increased decision-confidence ratings, reaching 8.3 out of 10 (SD=0.9) versus an initial mean of 6.4 (SD=1.7).Participants associated the change with understanding the situation more holistically.

5.2 Technological (AI) context made visible

Technological context made visible by ST included the AI’s past outputs and interactions with people, helping participants gauge performance and calibrate trust. Participants also connected these human elements to social forms of trust and adoption.

  • ST made visible the AI’s past decision outputs and others’ interactions, providing evidence of performance that model internals and metrics could not communicate.Participants used this context to understand the AI’s limitations and actual performance.
  • Participants reported recalibrating trust in the AI, helping address both over-reliance and AI aversion.The recalibration included becoming more confident in their own decisions while trusting the AI less.
  • ST infused human elements into AI-mediated decision-making, reflecting participants’ view that human aspects of practice cannot be replaced by machines.Participants from sales, cybersecurity, and healthcare highlighted the human aspect of their work.
  • Participants described transitive trust from peers to AI when colleagues used the system or accepted its recommendations.Organizational hierarchy, authority, identity, and social presence were discussed as social cues for evaluation.
  • Knowing who else used an AI recommendation and why could help users evaluate it through organizational authority and social endorsement.Examples included senior analysts, directors, and radiologists’ peers.

5.3 Decision-making context made visible

Decision-making context in ST exposed local histories, crew knowledge, and comparable outcomes that algorithms may not formalize. Participants used this context for analogical reasoning, greater self-confidence, follow-up actions, and contestability.

  • ST made visible local context from past decisions, including client-specific quirks and other factors absent from the AI’s feature space.Participants described these factors as tacit, idiosyncratic, or constantly changing.
  • Crew knowledge is informal, experience-based knowledge learned through colleague interactions and essential to doing the job.The paper notes that participants originally used the term “tribal knowledge.”
  • ST provided in-situ access to elements of crew knowledge and supported knowledge sharing through a consolidated platform.Participants viewed this as a way to surface knowledge that is difficult to formalize algorithmically.
  • Participants used similar past decisions and actual outcomes to compare current and prior contexts through analogical reasoning.They considered both similarities and differences between the cases.
  • ST-supported context increased confidence in participants’ own decisions while sometimes reducing confidence in the AI.Participants also reported learning how to evaluate AI insights and justify decisions to clients or supervisors.
  • Social validation from others’ decisions could reduce individual vulnerability and support decision-making resilience and contestability of AI.Knowing that others had acted similarly helped distribute perceived risk when challenging or departing from the machine.

5.4 Organizational context made visible

Organizational context in ST exposed norms, values, responsibilities, expertise, and institutional knowledge around AI-mediated work. Participants associated this visibility with collective action, accountability, training, and support, while also identifying surveillance risks.

  • ST made organizational norms and values visible through others’ actions, helping participants understand expectations and why decisions were made.Participants linked examples such as discounts to what the company accepted and valued.
  • Knowing who did what and why could promote accountable actions, audits, and postmortems by making past decisions traceable and socially situated.Participants described this as providing peripheral vision for evaluating and attributing responsibility.
  • Participants warned that traceability and accountability could become surveillance when organizational culture treats monitoring as a goal.The paper therefore presents organizational culture as an important boundary for ST’s effects.
  • Organizational context could support collective action by revealing who knows what, who knows whom, and whom to contact.Participants connected this visibility to expertise location in larger, distributed organizations.
  • ST could help create institutional memory, preserve legacy knowledge, and support training and peer-to-peer assistance.Participants imagined shared repositories containing practical knowledge for newcomers and virtual teams.
  • Repeated visibility of colleagues’ decision processes could support a Transactive Memory System and, over time, a shared collective mind.The paper connects these possibilities to collective knowledge distribution and shared decision schemas.

5.5 Design for ST: the 4W

Participants ranked the 4W elements as complementary forms of socially situated context, with “what” and “why” generally prioritized before “who” and “when.” These features supported overview, rationale, social validation, and contextualized judgment, while raising concerns about quality, privacy, and bias.

  • 4W elements: The 4W—Who did What, When, and Why—were developed as constitutive design elements of Social Transparency and ranked across the sales scenario and participants’ domains.Participants ranked the elements twice and found overall preference patterns despite domain-dependent variation.
  • What: “What” provided a snapshot of AI performance and others’ actions, helping participants decide whether to investigate further and avoid over-relying on recommendations.Participants described the outcome summary as a concise entry point, with further information available when needed.
  • Why: “Why” supplied the context behind human and organizational decisions, informing AI assessment, actionable decisions, social validation, and understanding of organizational norms.Prior rationales could also support similar decisions or justify rejecting the machine’s recommendation.
  • Design tensions: ST information requires quality control, standardization, and legal safeguards because unstructured comments may mislead or reveal private or proprietary details.Participants emphasized that not all rationales are equally useful and that compliance requirements constrain disclosure.
  • Who: Participants valued “who” for locating expertise and assessing judgment, but identity information could shape social validation, transitive trust, and bias.Organizational role, experience, hierarchy, collective consistency, names, and profile pictures all affected how participants interpreted the information.
  • When: “When” made prior decisions more relevant and actionable by placing them in temporal perspective and strengthening their rationale.Timing helped participants determine which prior decision deserved greater weight.

5.6 Transferability of ST to other domains

Participants saw Social Transparency as transferable across cybersecurity, healthcare, and other decision-support settings because it integrates technical recommendations with peer and organizational context. They associated this integration with localized judgment, knowledge sharing, peer review, and holistic explanations.

  • Cybersecurity: Cybersecurity participants saw ST as addressing limited awareness of peers’ AI-assisted decisions and supporting client-specific, socially validated threat assessment.Historical false-positive timelines and peer knowledge could help calibrate decisions and improve resilience.
  • Cybersecurity: ST could augment standardized cybersecurity AI with local legal and organizational contexts, helping teams interpret threats and justify decisions using human and AI data.Participants highlighted differing international laws, regional contexts, and the limits of North American training data.
  • Cross-domain implications: Participants described ST as holistic explainability because social signals reveal organizational context, multiple rationales, and human elements beyond the AI model.Some data scientists also envisioned feeding social signals back into AI to improve performance and explanations.
  • Healthcare: In healthcare, ST could personalize treatment decisions by combining standard-dataset recommendations with comparable doctors’ cases and surrounding clinical contexts.Participants connected this use to radiology, oncology, and other multi-stakeholder treatment decisions.
  • Healthcare: Integrating technical and socio-organizational decision support in one place could reduce context switching and support peer review when clinicians’ mental models diverge from AI recommendations.Participants compared comment-based peer feedback with tumor-board discussions of difficult cases.

5.7 Challenges around ST

Participants identified four interconnected challenges around ST: transparency can conflict with privacy, social information can introduce bias, increasing content can overload workflows, and contributors may lack incentives to participate.

  • Implications: The authors frame these challenges as requiring mitigation rather than treating ST as universally applicable or fully implementable.Future work is directed toward addressing the identified negative consequences and contribution barriers.
  • Privacy: Transparency creates privacy and organizational risks when sensitive activities, performance information, or personal identities become visible to others.Participants questioned who could access shared information and suggested anonymized or aggregated presentations.
  • Bias: ST may encourage groupthink, conformity, or deference to senior people and friends, with these biases potentially varying in consequence across domains such as HR.The same social heuristics that support judgment can also distort decision-making.
  • Information consumption: Information overload and workflow integration become concerns as the number of ST entries grows, especially in time-sensitive clinical decision-support settings.Participants considered filtering, summarization, statistics, and on-demand access as ways to manage consumption.
  • Incentives: ST depends on people contributing knowledge, but time-pressed professionals may view contribution as unpaid extra work and decline to participate.Participants noted that systems may fail if contributors do not sustain the feedback loop.

6 DISCUSSION & IMPLICATIONS

The discussion positions ST as a sociotechnical expansion of XAI that makes organizational context and human contributors visible alongside AI outputs. The authors argue that this can support holistic explanations and collective human-AI practices, while requiring context-sensitive safeguards and implementation choices.

  • Holistic explainability: ST expands explainability beyond algorithmic outputs by incorporating organizational context and human elements into explanations of AI-mediated decisions.Participants described this broader view as holistic and useful for answering multiple forms of “why.”
  • Holistic explainability: The 4W can help justify decisions to clients and regulators because it humanizes the process in ways technical transparency alone may not.This extends explanation access to non-primary stakeholders.
  • Broader stakeholders: ST could support model developers and auditors by exposing situated technological, decision, and organizational contexts relevant to performance, failure, bias, and safety.The authors identify these groups as additional potential consumers of ST information.
  • Human-AI assemblage: Making human activity explicit strengthens the human-AI assemblage and could support shared decision schemas and collective minds through repeated visibility of decision processes.The proposed connection draws on organizational meta-knowledge and transactive memory processes.
  • Technical and sociotechnical considerations: ST implementation should be localized to stakeholder values because privacy, bias, information overload, motivation, and domain differences constrain what information can be gathered and shown.The authors note that some organizations cannot collect all 4W information and may need alternatives.
  • Technical and sociotechnical considerations: Scenario-based design enabled conceptual exploration without fixing system operations or technical details, leaving implementation questions for future work.The authors also identify information quantity, validation, model change, and incomplete 4W collection as practical concerns.

7 LIMITATIONS & FUTURE WORK

The paper presents Social Transparency as a work-in-progress that expands XAI by incorporating socio-organizational context, while treating its findings as formative. It calls for further work on design elements, transferability, and longitudinal trust.

  • Limitations: The authors caution that scenario-based design creates dependencies between the scenario and data, so findings should be interpreted as formative rather than evaluative.This limits direct interpretation of the study as an evaluation of ST-infused systems.
  • Future work: Future work should expand ST’s design space, examine additional design elements, and assess where transferring the insights may be inappropriate.The authors explicitly identify transferability as an open issue requiring further investigation.
  • Future work: Longitudinal research is needed to investigate how ST affects user trust over extended use of ST-infused XAI systems.The paper leaves trust over time as a future research question.
  • Scope and framing: The authors frame Social Transparency as an ongoing, self-reflective effort that continuously examines its own blind spots.They position the work as a beginning rather than a finished account of explainability.
  • Contributions: The study’s formative insights come from identifying socio-organizational context as a neglected constitutive design element in XAI.The authors describe this as their initial design-oriented contribution to expanding the field’s boundaries.

8 CONCLUSION

The paper challenges algorithm-centered XAI by introducing Social Transparency, which incorporates socio-organizational context into explanations of AI-mediated decision-making. Through a scenario-based design and 29 stakeholder interviews, it develops a framework spanning technological, decision, and organizational contexts and expands the conceptual and design space of human-centered XAI.

  • 8 CONCLUSION: Algorithm-centered XAI overlooks the socio-organizational context in which AI systems and explainability are situated.The paper identifies this omission as an epistemic blind spot in consequential decision-making contexts.
  • 8 CONCLUSION: Social Transparency incorporates socio-organizational context to enable holistic explainability of AI-mediated decision-making.The concept is presented as a sociotechnical extension of explainability rather than a machine-only account.
  • 8 CONCLUSION: The authors explored ST through a scenario-based design embodying four constitutive elements—who, what, when, and why—and 29 interviews with AI stakeholders.The 4W design features operationalize the proposed context for formative empirical exploration.
  • 8 CONCLUSION: The study distinguishes technological, decision, and organizational levels of context made visible by ST and examines their effects.This three-level framework refines the conceptual development of Social Transparency.
  • 8 CONCLUSION: The work contributes design insights and identifies potential challenges of incorporating socio-organizational context into AI systems.These insights support further exploration of the design space by researchers and practitioners.
  • 8 CONCLUSION: The paper advances a socially-situated, human-centered XAI discourse by expanding XAI’s conceptual and design space.Its contribution is positioned as a formative step toward sociotechnical XAI systems.
Loading 2101.04719v1…