Source-linked AI summary
Responsible Institutional Analytics: Interpreting Bias with AI Support
Francielle Marques, Ariel Ortiz-Beltrán, Ishari Amarasinghe, Davinia Hernández-Leo
TL;DR
IA dashboards support higher-education decisions, but interpreting their indicators responsibly requires attention to analytical and contextual factors that may bias interpretation. This paper introduces FACTRIA and uses it to guide a chatbot that prompts reflection; across four authentic IA cases, stakeholders shifted toward more reflective, context-aware, conditional interpretations.
Problem
Method
The study introduces FACTRIA and a FACTRIA-aware chatbot, then evaluates stakeholder interactions across authentic IA cases using transition network analysis.
Source: §4.3 · p. 13, §4.4 · p. 13, §8 · p. 22Results
Across participants and four IA cases, the chatbot supported a clear shift toward more reflective, context-aware, conditional interpretations.
Source: §6 · p. 20Takeaways & Limitations
FACTRIA provides an adaptable scaffold for making biasing factors explicit, and chatbot prompts can support reflective sense-making about IA visualizations.
Source: §3 · p. 5, §6 · p. 20, §8 · p. 22The study is a proof of concept, and expressed think-aloud reasoning is evidence of sense-making rather than a direct measure of comprehension.
Source: §7 · p. 22
Abstract
from arXiv · showhide
Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes potential biasing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while analyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.
1 Introduction
Institutional Analytics (IA) supports higher-education decision-making, but responsible interpretation requires attention to biasing factors in the data context and analytical process. This paper introduces FACTRIA and a chatbot that prompts stakeholders to reflect on those factors while interpreting IA.
1 Introduction
- IA extends course-level learning analytics across programs, departments, and cohorts to inform institutional systems, planning, and stakeholder action.It draws on indicators such as learning-management-system data, student performance, and satisfaction surveys.Source: §1 · pp. 1–2
- Responsible interpretation depends on users’ awareness of potential biases in analytics, data, context, and communication—not only on dashboard indicators.Interpretations should make assumptions and limitations transparent and remain bounded by the data.Source: §1 · p. 2
- Existing frameworks address responsible implementation or data sources, but do not specify data-context and analytical sources of bias affecting interpretation.The cited examples include DELICATE and Analytics Layers for Learning Design.Source: §1 · p. 2
- FACTRIA organizes biasing factors, and a FACTRIA-aware chatbot uses retrieval-augmented generation to prompt reflection rather than provide interpretations or recommendations.The paper illustrates the framework with four real IA cases and assesses chatbot use with stakeholders using transition network analysis.Source: §1 · p. 2
- The paper uses “responsible interpretation of IA” for drawing conclusions from visualizations while accounting for factors that condition what they can support.FACTRIA operationalizes this process through dimensions that specify factors to consider when interpreting a visualization.Source: §1 · pp. 2–3
2 Background
Prior work documents the contextual sensitivity of IA indicators and the interpretive demands of analytics dashboards. The background motivates tools that make analytical and contextual conditions explicit while distinguishing FACTRIA from related approaches.
2 Background
- IA indicators—including satisfaction, teaching quality, and retention—can vary with demographic, institutional, course, and learning-design conditions.The cited examples include gender, COVID-19-related policy conditions, workload, and discipline-specific dropout risk.Source: §2 · p. 3
- Dashboards can exceed users’ interpretive capabilities when analytical-process factors remain implicit, risking overlooked limitations or poorly grounded conclusions.The background also calls for clear visualizations paired with support for attending to analytical and contextual conditions.Source: §2 · p. 3
- Prior visualization chatbots generate explanations, whereas FACTRIA draws on knowledge of factors that can hinder responsible interpretation of IA visualizations.The background discusses VizChat and a multi-agent system for learning-analytics dashboard explanations.Source: §2 · pp. 3–4
- FACTRIA specifies factors to consider when interpreting IA visualizations, unlike governance frameworks focused on institutional principles or visualization-literacy approaches focused on graphical features.The paper positions FACTRIA at the moment of interpretation as a basis for its chatbot.Source: §2 · p. 4
3 FACTRIA framework
FACTRIA organizes potential influences on IA interpretation into four categories and offers an evidence-informed scaffold for reflection rather than prescribing decisions. It is intended to remain adaptable as practices and evidence evolve.
3 FACTRIA framework
- FACTRIA groups potential biasing factors into analytics pipeline, institutional, course, and demographic categories, while leaving decisions to stakeholders.The framework is described as a structured scaffold for surfacing assumptions and questioning comparability, not providing definitive conclusions or recommendations.Source: §3 · pp. 4–5
- FACTRIA is an adaptable theoretical construct: additional biasing factors may be incorporated as IA practices and evidence evolve.The framework is presented as a foundation for responsible IA.Source: §3 · p. 5
3.1 Analytics Pipeline Factors
- Pipeline interpretation should account for data quality, aggregation, analytical techniques, visualizations, and their assumptions.Educational data may be incomplete or inconsistent, and high-level aggregation can obscure granular nuances; technique choices also shape analysis.Source: §3.1 · p. 5
3.2 Institutional Factors
- Institutional interpretation should consider academic-unit differences, teaching practices, and university modality, which shape context and data coverage.Department composition and modality-specific data sources can affect how comparisons should be interpreted.Source: §3.2 · p. 6
3.3 Course Factors
- Course factors include class size and level, learning design, workload, and student performance, each of which can relate to satisfaction or outcomes.The cited work notes both expected influences and cases that call for balanced interpretation.Source: §3.3 · pp. 6–7
3.4 Demographic Factors
- Demographic considerations include gender, age, and language, whose potential influences can interact and are not deterministic.The cited examples include differing findings on gender and evaluations, and lower evaluations for some non-native accents despite no measured learning-outcome differences.Source: §3.4 · p. 7
4 Methods
The study examined whether FACTRIA and FACTRIA-aware chatbot guidance support stakeholders’ responsible interpretation of institutional analytics. Participants reviewed the framework, interpreted dashboard visualizations with and without chatbot support, and researchers analyzed their interactions.
4.1 Study Design
- Participants interpreted the same dashboard visualization first without and then with chatbot guidance; condition order was not counterbalanced.They first reviewed FACTRIA, then completed two sessions using cases selected for relevance to their stakeholder contexts.Source: §4.1 · p. 8
4.1.1 Institutional Analytics Cases
- Four context-relevant visualizations covered teaching satisfaction, grades by methodology and workload, high-interaction courses, and LMS configurations.The visualizations came from previous institutional studies the participants had not encountered; the cases differed in data structure, analytical focus, and interpretive requirements.Source: §4.1.1 · pp. 8–9, Figure 3 · p. 9, Figure 4 · p. 10
4.2 Participants
- The 11 participants were professors, academic managers, and technicians from a Spanish university; four were male and seven female.Participant profiles were anonymized.Source: §4.2 · p. 12
4.3 FACTRIA-Aware Chatbot Assistant
- The web-based LLM chatbot uses FACTRIA-aware retrieval-augmented generation and cycles through relevant factors to prompt reflection on visible data.Its dashboard interface combines visualizations with the chatbot, while a deterministic state machine connects users’ initial interpretations to the framework.Source: §4.3 · p. 13
4.4 Data Collection and Analysis
- Researchers annotated recorded screen activity and audio, independently coded transcripts with substantial agreement (Cohen’s k = 0.76), then resolved disagreements through discussion.Transition network analysis examined interaction patterns for each case and across cases; four narratives represented variation across cases and participant profiles.Source: §4.4 · pp. 13–14
5 Results
Across three scenarios, participants considered more sub-factors with the FACTRIA-aware chatbot, though time constraints kept them from addressing all relevant factors. In four case examples, prompts helped participants revise initial readings by considering data, analytical, course, and demographic context.
5 Results
- Across Cases 1–4, participants identified 3, 5, 5, and 4 relevant factors, respectively, and produced final summaries after chatbot interactions.The transition network showed prominent chatbot reflection and explanation actions alongside user insight-sharing; the figure includes transitions above 0.25 relative probability.Source: §5 · pp. 14–15, Figure 9 · p. 15
- Ten of 11 participants rated the chatbot 4 or 5 out of 5 for contextualizing data; usability scored 70.84 ± 16.39, while confidence and experiences varied.On-demand explanations of domain-specific terms helped users interpret visualizations, while one participant found the prompts too generic and another described use as challenging.Source: §5 · pp. 14–15
- All three scenarios showed significantly more addressed sub-factors with the chatbot: Case X + Case Y (p < .001; d = 2.1530), first cases (p < .001; d = 4.4947), and second cases (p < .001; d = 1.7866).Time constraints prevented participants from addressing all sub-factors related to the visualizations with the chatbot.Source: Table II · p. 16, §5 · p. 16
5.1.1 P7 – Illustrative example
- P7 moved from reading gender differences as direct evidence of bias toward considering data quality, small samples, and student–professor gender interactions.The participant also developed a more conditional interpretation involving aggregated statistics and institutional differences, while calling for further inquiry.Source: §5.1.1 · pp. 16–17
5.2.1 P10 – Illustrative example
- P10 shifted from treating lower box plots as weaker courses to considering course-level aggregation, class size, workload, and learning design.The chatbot clarified that the visualization showed course aggregates rather than individual student data.Source: §5.2.1 · pp. 17–18
5.3.1 P6 – Illustrative example
- P6 reconsidered LMS interaction as a possible reflection of workload and learning design, while recognizing that unclear indicator definitions limited conclusions.Institutional and demographic factors did not enter this participant’s reasoning.Source: §5.3.1 · pp. 18–19
5.4.1 P2 – Illustrative example
- P2 shifted from a satisfaction-based reading toward LMS task diversity and learning design after prompts clarified the Shannon Index and data assumptions.The revised interpretation considered task variety and workload, but institutional factors did not emerge in the exchange.Source: §5.4.1 · pp. 19–20
6 Discussion
Participants often began with narrow readings that overlooked analytical and contextual factors, and framework familiarity alone did not prompt broader consideration. Across four cases, chatbot scaffolding supported more reflective, context-aware interpretations and suggests a role for factor-aware guidance in IA tools.
6 Discussion
- Before chatbot interaction, participants referenced an average of 1.50 sub-factors (SD = 0.80), and framework familiarity alone did not prompt broad initial consideration.Across three scenarios, the chatbot surfaced factors participants had not considered independently.Source: §6 · p. 20
- Across all participants and four IA cases, chatbot prompts supported shifts from visually driven readings toward more reflective, context-aware, conditional interpretations.Participants reconsidered assumptions about sample size, aggregation, indicator definitions, learning design, and demographic interactions.Source: §6 · p. 20
- The chatbot guided reflection by asking questions rather than supplying answers, extending AI-mediated dashboard assistance toward critical reflection in institutional analytics.The analysis recorded chatbot reflection, interpretation, and explanation actions as part of this question-led interaction.Source: §6 · p. 21
- The findings point to embedding factor-aware guidance in analytics tools or training to connect dashboard indicators with contextual considerations in institutional decision-making.The paper suggests that point-of-interpretation cues can help prevent premature conclusions and promote transparent, evidence-aligned decisions.Source: §6 · p. 21
7 Limitations
The study is exploratory, with limited observations per case and several constraints on attributing participants’ reflections to chatbot support alone. Its findings are expressed sense-making rather than a direct measure of comprehension, and FACTRIA may require adaptation across contexts.
7 Limitations
- The 11-participant study had limited observations per case, so findings are treated as exploratory and case-level rather than comparative.Each participant engaged with two of four cases.Source: §7 · p. 21
- Because participants read FACTRIA before chatbot use, chatbot effects cannot be cleanly separated from increasing familiarity with the framework.Future work should use alternative experimental conditions to distinguish these effects.Source: §7 · p. 21
- Think-aloud responses may reflect reactivity and articulation ability or willingness, so they indicate expressed sense-making rather than directly measuring comprehension.The paper also notes possible automation bias and over-reliance on chatbot guidance.Source: §7 · pp. 21–22
- The work is presented as a proof of concept, and FACTRIA may need adaptation across cases and institutional contexts.The study includes diverse stakeholder perspectives, but the authors call for broader testing.Source: §7 · p. 22
- Future research should test FACTRIA across more IA cases and universities, incorporate additional bias sources, and examine GenAI risks and role- or experience-sensitive prompting.The authors identify oversimplification, occasional inaccuracies, and inconsistent alignment with institutional contexts as GenAI risks.Source: §7 · p. 22
8 Conclusion
The paper addresses the lack of structured support for stakeholders interpreting institutional analytics responsibly. It synthesizes biasing factors in FACTRIA, operationalizes the framework in a GenAI chatbot, and offers a conceptual and empirical foundation for more contextually sound interpretation.
8 Conclusion
- The paper targets the lack of structured support to help stakeholders consider factors shaping responsible IA interpretation.It presents FACTRIA as a synthesis of biasing factors and uses a GenAI chatbot to scaffold reflective sense-making.Source: §8 · p. 22
- A qualitative design with institutional decision-making stakeholders, authentic interpretive cases, and transition network analysis assessed FACTRIA’s applicability.The conclusion describes the approach as an initial response to the identified gap.Source: §8 · p. 22
- The work provides a conceptual and empirical foundation for developing more transparent, fair, and contextually sound IA interpretive practices.The authors call for further research to address study limitations and investigate new directions.Source: §8 · p. 22
Funding
The work acknowledges support from two funded projects and an additional academic program.
Funding
- The work was supported by PID2023-146692OB-C33; the passage also names 2023-26 CEX2021-001195-M as granted by MICIU/AEI/10.13039/501100011033 and acknowledges D. Hernández-Leo’s AGAUR ICREA Academia support.Source: Funding · p. 23
- The work was also supported by 2023-26 CEX2021-001195-M, granted by MICIU/AEI/10.13039/501100011033.Source: Funding · p. 23
- D. Hernández-Leo acknowledged support from the AGAUR ICREA Academia programme.Source: Funding · p. 23
Factor Description of the factor
The factor descriptions span analytics-pipeline, institutional, course, and demographic considerations. The section also gives examples of chatbot interaction categories and identifies tables for usability, interaction coding, and factor counts with and without chatbot use.
Factor Description of the factor
- Analytics-pipeline factors include data collection, feature extraction, sample, data reliability, aggregation level, analytical techniques, visualizations, and analysis assumptions.Source: Factor Description of the factor · p. 30
- Institutional and course context includes institution type, academic centers, department gender composition, teaching methodologies, class size, course level, content, assessments, schedule, and performance.Source: Factor Description of the factor · p. 30
- Demographic factors include student and professor gender, age, and language, including the language of instruction.Source: Factor Description of the factor · p. 30
- The section identifies tables for the System Usability Survey, the interaction coding scheme, and counts of sub-factors addressed with and without the chatbot.Source: Table V · p. 31, Table VI · p. 32, Table VII · p. 35
- The interaction examples distinguish sharing insight, statements, summaries, factor- or framework-related questions, reflections, interpretation, directing attention, visualization questions, and clarification requests.Source: Factor Description of the factor · p. 33
- Examples include prompts about gender and satisfaction, learning design and workload, data quality, visualization choices, and users’ interpretations of teaching-methodology outcomes.Source: Factor Description of the factor · pp. 33–34
A FACTRIA-Aware Chatbot: Technical Details
The chatbot uses FACTRIA entries and case-specific chart data as a fixed evidence set, with prompts designed to elicit reflection rather than provide direct interpretations. A deterministic dialogue flow and structured practitioner summary support the interaction, although adherence to safeguards was not measured quantitatively.
A FACTRIA-Aware Chatbot: Technical Details
- Each request combines chart context, a current FACTRIA factor, relevant framework entries, and verbatim case descriptions and chart data in an auditable fixed evidence set.The injected blocks are designated as authoritative, bounding generation without query-time search.Source: A FACTRIA-Aware Chatbot: Technical… · p. 36
- Three safeguards restrict unsupported statements, but the authors did not quantitatively measure adherence and note residual risk.The safeguards limit responses to injected content and numerical values, while near-deterministic decoding reduces variability.Source: A FACTRIA-Aware Chatbot: Technical… · p. 36
- A deterministic state machine routes messages among five prompted components, including classifiers, clarification, primary-response, and summarization agents.The primary agent uses short, jargon-free responses, avoids direct chart interpretations, and redirects users who overlook factors.Source: A FACTRIA-Aware Chatbot: Technical… · p. 36
- After dialogue, a schema-constrained agent produces a 150-word practitioner synthesis with factor coverage labels, user insights, and factors not reached.The system uses Gemini 2.0 Flash with conservative decoding settings, token caps, and English, Spanish, and Catalan language support.Source: A FACTRIA-Aware Chatbot: Technical… · p. 36
B Transition Network Analysis
Across four case-specific transition networks, assistant prompts coded around reflection, interpretation, and factors accompany user insights, caution, doubts, and clarification. The networks include transitions observed at least twice with probability greater than 0.2.
B Transition Network Analysis
- The case-specific networks retain transitions observed at least twice and with probability greater than 0.2.Source: B Transition Network Analysis · p. 38
B.1 Case 1
- Case 1 shows assistant reflection, interpretation, and factor prompts preceding user insights or agreement; some corresponding transitions have P = 1.Source: §B.1 · p. 38, Figure 10 · p. 38
B.2 Case 2
- In Case 2, higher-probability transitions show user explanation following assistant reflection and user reflection following insight sharing.The network also includes user caution and assistant prompts about reflection, visualization, and factors.Source: §B.2 · p. 39, Figure 11 · p. 39
B.3 Case 3
- Case 3 shows varied user contributions alongside reflective assistant questions, explanations, and validation, consistent with guidance rather than answer provision.User actions include sharing insights, seeking clarification, expressing doubts, and caution; assistant prompts include factors, interpretation, and reflection.Source: §B.3 · p. 40, Figure 12 · p. 40
B.4 Case 4
- In Case 4, user insight follows assistant factor, interpretation, and reflection prompts; doubt or clarification requests are followed by clarification and renewed reflection.Source: §B.4 · p. 41, Figure 13 · p. 41