PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Gender-Inclusive Practices Actionable: Evaluating a Research-Informed Computing Education Toolkit

Alina Berry, Susan McKeever, Brenda Murphy, Sarah Jane Delany

arXiv:2609.03936v1cs.CYcs.HC

TL;DR

Research-backed interventions for gender inclusion in computing are difficult for educators to translate into actionable practice. This paper presents and evaluates TechMate, a web toolkit offering practical guidance, through educator task engagement and feedback. Educators selected initiatives across the toolkit, with influence-and-support mechanisms receiving the most selections in this sample.

  • Problem

    Research has identified practices supporting under-represented groups in computing, but translating that evidence into actionable educator guidance remains a challenge.

    Source: §1 · p. 1

  • Method

    The paper presents TechMate, a practical research-informed web toolkit, and evaluates it through educators’ perceptions of usefulness and usability and their interactions with it.

    Source: §1 · p. 2

  • Results

    Participants selected 18 of 26 initiatives, and influence-and-support mechanisms received the most selections, with 12.

    Source: §5 · p. 6

  • Takeaways & Limitations

    TechMate may lower barriers to evidence-informed practice, but institutional culture, workload, and recognition structures shape whether its potential is realized.

    Source: §6 · p. 8

    Recruitment through a gender-equality network likely favored educators already motivated and knowledgeable about inclusion, limiting generalisability most directly to early adopters and champions.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

The persistent gender imbalance in computing remains a global concern, and universities offer a key part of the pipeline to address it. Although research has identified practices that support under-represented student groups, translating this evidence into actionable guidance remains challenging. This paper first presents a novel web- based toolkit (TechMate) designed to address this gap by helping computing educators implement gender- inclusive initiatives through practical research-informed guidance. The toolkit defines over 25 actions, ranging from operational to strategic, and provides case studies and implementation resources. Second, this work reports on the evaluation of TechMate, capturing educators first impressions of its usefulness and usability through authentic tasks, and eliciting unanticipated insights about structural barriers to gender-inclusive practice in computing higher education. Eighteen computing educators of varying seniority from eight universities in Ireland assessed the toolkit using nine research-driven attributes: four usefulness measures (novelty, relevance, trustworthiness, actionability) and five usability measures (effectiveness, aesthetics, navigation, terminology, user satisfaction). Data was analysed using mixed methods: deductive coding, quantitative sentiment scoring, and inductive thematic analysis. Educators rated TechMate highly for user satisfaction and praised its novelty and trustworthiness, with over 60% of participants reporting positively on all attributes, suggesting that research -informed tools can be designed for real-world use. Thematic analysis highlighted other challenges, including women students isolation, institutional resistance to change, and the scarcity of women role models, while also surfacing proposed solutions such as leveraging student ambassadors and fostering supportive learning environments.

1 Introduction

Women remain under-represented in computing, and the causes span individual, social, and structural factors. TechMate addresses the gap between research evidence and actionable guidance, and this study evaluates educators’ first impressions rather than long-term outcomes.

1 Introduction

  • Women comprise less than 20% of EU ICT specialists, nearly half leave technical careers before age 35, and women were 21% of introductory computing students in Ireland and Denmark.The paper describes this pattern as a leaky pipeline.Source: §1 · p. 1
  • Research-informed practice is difficult to translate into accessible, actionable guidance for computing educators.Educators nevertheless have responsibility and opportunity to support recruitment, retention, and inclusive experiences for women and other under-represented groups.Source: §1 · p. 1
  • TechMate is a freely available web toolkit offering practical, research-informed resources educators can access on demand, unlike high-level frameworks or policy documents.Its focus is design and first-use value; studies of long-term effects on gender ratios or student outcomes are future work.Source: §1 · p. 2
  • The evaluation asks how educators perceive TechMate’s content usefulness and usability, and what unanticipated needs or insights emerge during use.The paper links these questions to the toolkit’s initial evaluation rather than long-term effectiveness.Source: §1 · p. 2

2 Background/Related Work

Research documents interventions that can support under-represented groups in computing, but the evidence is dispersed and difficult to translate into practice. Existing resources offer guidance, while their implementation detail, transferability, evidence links, and usability evaluation remain limited.

2 Background/Related Work

  • Research identifies interventions including women-only programmes, introductory-course rebranding, curriculum redesign, and mentoring, but relevant evidence is dispersed across academic and grey literature.Reviews synthesize intervention strategies, yet actionable translation for educators remains challenging.Source: §2 · p. 2
  • Prior toolkits and frameworks vary in scope and often lack detailed implementation guidance, direct links to supporting research, or transferability across institutional contexts.Some require educators to interpret separate resources, while others target specific contexts or offer limited institutional implementation guidance.Source: §2 · p. 2
  • Usability and educator experience of gender-focused educational toolkits have received limited systematic evaluation, despite established digital-system assessment models.Prior studies have focused more on intervention effectiveness than on the digital resources that curate and present interventions.Source: §2 · p. 2

3 The Toolkit - TechMate

TechMate is an on-demand web toolkit intended to bridge research evidence and educators’ practice. It organizes interventions into categories and presents them with implementation guidance and resources, with navigation designed for both broad exploration and targeted discovery.

3 The Toolkit - TechMate

  • TechMate offers persistent, on-demand access to actionable resources, in contrast with time-limited workshops and typically high-level policy documents.Users can consult the toolkit at their own pace and return as institutional contexts evolve.Source: §3 · p. 2
  • TechMate organizes more than 25 actions into policy, pedagogical, influence-and-support, and promotion-and-engagement categories.Examples include balanced staff recruitment, hybrid programmes, buddy systems, and school outreach.Source: §3 · p. 3, Figure 1 · p. 3
  • Each action page combines research evidence, implementation steps, evaluation approaches, ready-to-use resources, and references in a consistent six-section format.Buddy Systems guidance includes mentor recruitment and matching, training, participation incentives, and suggested evaluation materials.Source: §3 · p. 3, Figure 2 · p. 3
  • Class/Lab Dynamics recommends practical steps to ensure women and other minoritised students are included during classroom work.Examples include pairing a woman student with at least one other woman, using students’ names, and checking that quieter students can contribute.Source: §3 · p. 3
  • Users can browse categories and subcategories or filter initiatives by goal, impact level, student cohort, and user role.Category navigation supports first-time visitors seeking a broad understanding; filters support targeted discovery.Source: §3 · p. 3

4 Evaluation Methodology

The study evaluates educators’ first impressions of TechMate’s usefulness and usability through theory-driven attributes and seeks unanticipated insights. It uses authentic toolkit tasks, interviews, and separate deductive scoring and inductive thematic analysis.

4 Evaluation Methodology

  • The evaluation assesses perceived usefulness and usability through theory-driven attributes while eliciting unexpected insights during educator sessions.Usefulness comprises novelty, relevance, trustworthiness, and actionability; usability comprises effectiveness, aesthetics, navigation, terminology, and satisfaction.Source: §4 · pp. 3–4

4.1 Data Collection

  • Eighteen educators from eight Irish universities completed two representative toolkit tasks while thinking aloud, followed by semi-structured interviews.Sessions lasted 30–90 minutes and assessed how educators located and interpreted content, not whether they implemented an initiative; actionability therefore reflects perceived clarity and concreteness, not verified implementation.Source: §4.1 · pp. 4–5

4.2 Data Analysis

  • Researchers deductively assigned meaningful units to predefined usefulness and usability attributes and inductively coded separate data for emergent themes.Three independent coders scored attribute units on a five-point scale; post-discussion inter-rater reliability was ICC 0.814 (95% CI [0.752, 0.856]).Source: §4.2 · pp. 5–6

5 Results & Discussion

Educators selected a broad range of initiatives and rated TechMate positively across usefulness and usability measures. Their task-based feedback also surfaced structural constraints and practical needs for gender-inclusive practice.

5 Results & Discussion

  • Participants selected 18 of 26 initiatives (69%); influence and support mechanisms received the most selections (12), while 70% of academic management chose policy-change initiatives.Teaching-focused faculty favored pedagogical practices for an implementable task, while their unlimited-resources choices shifted toward influence and support mechanisms.Source: §5 · p. 6

5.1 Quantitative Scoring

  • Actionability received 83% positive ratings, Trustworthiness 72%, and Novelty and Relevance at least 78%.Participants valued implementation guidance, evaluation approaches, research links, and unfamiliar strategies consolidated in one place.Source: §5.1 · p. 6
  • All attributes averaged 3.8–4.5, and every participant rated them positively or neutrally; User Satisfaction was strongest, with 95% positive ratings and a 4.5 average.Figure 4 presents results across multiple evaluation dimensions.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
  • Effectiveness and Navigation each received 67% positive ratings, while Aesthetics received 61% and Terminology 66%.Participants requested clearer labels and visual hierarchy, larger fonts, tooltips, and better-signalled filters; accessibility was not formally evaluated.Source: §5.1 · p. 7

5.2 Thematic Analysis

  • The findings may generalize most directly to early adopters and champions, and whether barriers delay or prevent implementation remains unknown.Participants were recruited through a gender-equality network and referrals; the authors propose longitudinal research to track implementation and institutional change.Source: §5.1 · p. 7, §5.2 · p. 8
  • Participants described time and funding constraints, institutional resistance, scarce women role models, and women students’ isolation and reduced confidence in male-dominated classrooms.Some said the toolkit’s consolidated actions and evaluation resources could help them address these issues and implement initiatives in their departments.Source: §5.2 · pp. 7–8
  • Participants proposed ambassador programs, mentoring, role-model visibility, and curriculum redesign, and requested case studies, practitioner stories, videos, and community features.They also suggested clearer authorship, update indicators, and visual refinements to improve trust and design.Source: §5.2 · p. 8
  • Participants favored student-centred teaching practices, including small-group work, early support for struggling students, and framing programming as learnable rather than innate.They noted that these practices are time-intensive and can conflict with existing workloads.Source: §5.2 · p. 8

6 Conclusion

TechMate appears adoptable and empirically supported, but uptake depends on institutional conditions and the evaluation has limited generalisability. Future work will track adoption and downstream recruitment and retention.

6 Conclusion

  • The evaluation’s small Irish sample and likely motivated early adopters may limit generalisability and inflate positive usefulness and relevance evaluations.It captured initial impressions, not longitudinal outcomes, and focused on third-level education.Source: §6 · p. 8
  • Future work will track toolkit adoption across national and international contexts, enable community-contributed case studies, and assess recruitment and retention impact.Source: §6 · p. 8
  • TechMate appears adoptable and empirically supported, but institutional culture, workload, and recognition shape whether its potential is realised.The authors frame these conditions as a governance and policy challenge as well as a design one.Source: §6 · p. 8
Loading 2609.03936v1…