PaperlayerView exact revision on arXiv

Source-linked AI summary

Observing the Conduct of Systematic Reviews with Generative AI Support: An Experience Report from a Graduate Software Engineering Course

Danilo Monteiro Ribeiro, Gilberto Sussumu Hida

arXiv:2609.26057v1cs.CYcs.AIcs.SE

TL;DR

Teaching systematic reviews requires practical exposure to authentic methodological decisions, while empirical documentation of how graduate Software Engineering students engage with these methods is limited. This paper reports an observed classroom experience with AI-supported review activities, finding both reduced barriers and risks such as delegation and superficial validation. It concludes that learning benefits require human supervision, decision records, and critical reflection.

  • Problem

    Empirical documentation of how graduate Software Engineering students engage with review methods in practice is limited, although teaching requires exposure to authentic methodological decisions.

    Source: §1 · p. 1

  • Method

    The paper reports a classroom experience observing groups’ varied use of AI during practical review activities with shared infrastructure and planned manual-versus-AI contrasts.

    Source: §2 · p. 2, §3.6 · p. 5

  • Results

    LLMs enriched practical SLR activities while also exposing risks of insufficiently critical acceptance, delegation of structural decisions, dependence, and superficial validation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    AI should support, not direct, methodological decisions, with records and reflection balancing exploratory use against pedagogical control.

    Source: §5 · p. 10, §4.5 · p. 10

    The single-course, small-group, single-day experience with uneven phase separation and heterogeneous stages provides situated reflections, not causal evidence about AI’s effect on SLR quality.

    Source: §3.6 · p. 5, §5 · p. 10

Abstract

from arXiv · show

Context: Secondary studies are fundamental practices in Evidence- Based Software Engineering, but teaching them requires activities that expose students to authentic methodological decisions. Objective: This paper reports an experience in a graduate course in which ten doctoral students in Software Engineering, organized into three groups, piloted secondary studies with and without support from generative AI. Method: A single-day classroom session was organized and observed, in which the groups conducted pilot systematic reviews with and without generative AI support. Classroom observations, produced artifacts, and interaction threads with assistants configured in ChatGPT were analyzed to reconstruct how each group appropriated the technology throughout the activity. Results: LLMs reduced initial barriers, accelerated the generation of alternatives, and made methodological problems more explicit, but they also favored excessive delegation, superficial validation, operational difficulties, and a shift in focus from conducting the SLR to using the tool. Conclusion: The experience offers a situated, observational account of how doctoral students engaged with generative AI during a systematic review activity, and the resulting insights also inform the design of a subsequent controlled study. The findings indicate that generative AI can support practical learning about SLRs, provided that its use is accompanied by human supervision, decision records, and critical reflection on its limitations.

1 Introduction

The introduction frames practical review teaching as a response to limited empirical understanding of how graduate students make methodological decisions. It reports an observed pilot activity that revealed both learning-support potential and risks in students’ use of generative AI.

1 Introduction

  • Practical review activities let students experience chained methodological decisions, revise mistakes, and see how early choices can affect later selection, extraction, and synthesis.Such decisions include question formulation, search strategy, selection criteria, and recording justifications for transparent, auditable work.Source: §1 · p. 1
  • Empirical documentation of how graduate Software Engineering students engage with review methods in practice remains limited, despite the importance of teaching literature reviewing.The gap includes which decisions students prioritize, what shortcuts they take, and where they struggle.Source: §1 · p. 1
  • In education, LLM use raises questions about how students interpret, validate, and incorporate outputs, alongside potential learning support and risks to critical thinking..Source: §1 · p. 1
  • The paper observes three groups of doctoral students piloting systematic reviews with and without generative AI support.The observed tasks included planning, question formulation, search-string construction, search, screening, and initial extraction.Source: §1 · pp. 1–2
  • LLMs reduced initial barriers and supported review decisions, but also exposed technical difficulties, inconsistent responses, insufficiently critical acceptance, and a need for teacher mediation.The activity focused on how groups used AI during pilot reviews, not on causally measuring its impact or advocating review automation.Source: §1 · p. 2
  • The report documents how doctoral students used AI and the risks that emerged, offering lessons for instructors designing similar activities rather than prescribing AI use.It addresses a need for empirical grounding in educational decisions about incorporating AI into Software Engineering courses.Source: §1 · p. 2

2 Theoretical Background and Context of the Experience

The experience is grounded in practical engagement with authentic review decisions and treats LLMs as support rather than substitutes for human judgment. Its shared classroom infrastructure enabled observation of varied ways groups incorporated AI into their methodological decisions.

2 Theoretical Background and Context of the Experience

  • Learning systematic reviews requires practical contact with authentic methodological decisions, including scope, questions, search strategies, selection criteria, extraction, and evidence analysis.The framing emphasizes planning, traceability, rigor, and auditability.Source: §2 · p. 2
  • LLMs may support review stages, but their use requires critical observation, human supervision, pedagogical guidance, response validation, and human judgment.This caution is especially relevant while students are still developing methodological expertise.Source: §2 · p. 2
  • The activity was designed to observe graduate students appropriating technology during authentic pilot-review tasks, prompted by students’ earlier interest in contrasting manual and AI-assisted execution.It was not conceived as training in an AI-use protocol.Source: §2 · p. 2
  • A shared infrastructure aimed to reduce unnecessary differences across groups, while variation in their AI interactions revealed different modes of appropriation in the classroom.The tool was treated as support, with students responsible for assessing suggestions and making final decisions through guided reflection.Source: §2 · p. 2

3 Design and Conduct of the Activity

The activity combined pilot SLR practice with observation of how students, teachers, and LLMs shaped methodological work. In execution, the planned separation between manual and AI-assisted stages became less clear, making the activity more observational than controlled.

3.1 Educational and Observational Objective of the Activity

  • The activity paired practical pilot-review experience with observation of how LLM availability shaped group work, artifacts, and methodological decisions.The observation focused on when students invoked AI, delegated tasks, accepted responses, encountered conflicts, and sought teacher mediation.Source: §3.1 · pp. 2–3

3.2 Originally Planned Design

  • The planned workflow paired manual and AI-supported work across review stages, including planning, search-string construction, screening, and extraction.Screening decisions were to be recorded as included, excluded, or uncertain; Figure 1 depicts the planned workflow.Source: §3.2 · p. 3, Figure 1 · p. 4

3.3 Ethical Considerations

  • Participation was voluntary and ungraded, with informed consent, withdrawal permitted, and anonymization of reported findings.Students were informed that assistant interactions and artifacts would be recorded and analyzed.Source: §3.3 · p. 3

3.4 Practical Organization of the Classroom Activity

  • The one-day activity used autonomous group work with organizational and supervisory support from two instructors.Groups produced review artifacts and preserved assistant threads, enabling observation of artifact construction and interactions such as delegation, validation, and teacher mediation.Source: §3.4 · pp. 3–4

3.5 Observed Execution

  • Planning, search, and screening produced the clearest evidence, while extraction was less standardized and synthesis remained embryonic.Only one group produced a more complete outline integrating planning, search, screening, extraction, and synthesis.Source: §3.5 · p. 4

3.6 Contrast Between the Planned and the Realized

  • The planned linear stages became overlapping negotiations among methodological understanding, practical limitations, tool suggestions, and teacher mediation.Questions, strings, criteria, and screening decisions surfaced iteratively in group interactions with AI.Source: §3.6 · p. 5
  • Groups alternated or combined manual and AI-assisted work rather than following the planned sequence, reducing the strength of strict comparisons between modes.This made the report more useful for observing real technology appropriation in an educational context.Source: §3.6 · p. 5

4 What Worked, What Failed, and What We Learned

The section interprets classroom observations, group artifacts, and ChatGPT interaction threads to examine how AI shaped pilot-review work, rather than evaluate final SLR quality.

4.1 Observed Types of AI Use

  • AI shifted among refinement, alternative generation, and decision execution; delegating central protocol choices created the greatest risk when students did not justify them.The authors identify refinement as most aligned with the pedagogical objectives and decision execution as carrying the greatest risk of cognitive dependence.Source: §4.1 · p. 8

4.2 Perceived Benefits and Key Learning Moments

  • AI lowered the barrier to producing review artifacts, surfaced weaknesses in broad research questions, and enabled rapid trial-and-revision for classroom discussion.The tool helped students generate more structured initial formulations and made methodological problems, such as an overly broad or hard-to-operationalize question, more explicit.Source: §4.2 · pp. 8–9

4.3 Difficulties, Errors, and Human–AI Conflicts

  • Operational and interaction difficulties—including file limits, prompt adjustments, and output formatting—sometimes shifted students’ effort from conducting the review to operating the tool.Long, rigid prompts did not ensure the expected output format, creating rework.Source: §4.3 · p. 9
  • Convincing AI responses and compact screening labels could encourage insufficient validation and obscure the reasoning behind inclusion, exclusion, or uncertain decisions.The authors also describe the risk that formally organized artifacts may not reflect students’ understanding of the methodological reasons behind them.Source: §4.3 · p. 9

4.4 Teacher Mediation and Human Supervision

  • The teacher mainly preserved the activity’s organization and framed AI as support, while avoiding intervention in groups’ internal decisions so observed interactions could unfold naturally.Student–AI interactions made the internal execution of activity blocks more iterative and less linear.Source: §4.4 · p. 9

4.5 What We Would Do Differently and Recommendations for Replication

  • For replication, the authors recommend splitting the intensive single-day activity and preparing students for LLM limits, file handling, formats, and validation before the main session.The one-day scope created pressure for shortcuts, while limited LLM literacy sometimes redirected attention from the review to tool operation.Source: §4.5 · p. 9
  • They also recommend preserving initial, AI-supported, and student-justified final artifacts, varying prompt scaffolds, planning infrastructure, and pausing for reflection at each stage.These measures are presented as conditions for supporting methodological learning rather than merely accelerating artifact production; instructors may need to adjust scope and pacing for less experienced students.Source: §4.5 · p. 10

5 Conclusion

The report describes an observed classroom experience in which generative AI supported pilot reviews, revealing both increased fluency and pedagogical risks. It recommends balancing exploratory use with pedagogical control and treats the findings as situated input for a more controlled study.

5 Conclusion

  • LLMs enriched practical SLR activities while also exposing students to dependence, superficial validation, and delegation of structural decisions.The report identifies gains in fluency alongside these risks as its central empirical contribution.Source: §5 · p. 10
  • AI integration requires exploratory openness alongside records and reflection, so the activity does not become task automation without methodological appropriation.AI should support the review process rather than conduct methodological decisions.Source: §5 · p. 10
  • The findings are situated reflections, not causal evidence about AI's impact on SLR quality, given the course, participant, timing, and workflow limitations.The single-day activity had uneven separation between AI and non-AI moments, partial and heterogeneous review stages, and occasional infrastructure deviations.Source: §5 · p. 10
  • The pilot informs a planned controlled comparison of manual and LLM-assisted SLR execution, including checkpoints, decision records, AI literacy, and structured screening and extraction formats.The authors also judge a single-day format insufficient to observe the full SLR workflow with adequate control.Source: §5 · p. 10

Use of AI Tools

ChatGPT 5.2 was used for language revision and visual refinement, not for the study's substantive research activities.

Use of AI Tools

  • ChatGPT 5.2 supported language revision, translation, and textual clarity improvement.It did not contribute to study design, data collection, analysis, interpretation, or scientific conclusions.Source: Use of AI Tools · p. 10
  • ChatGPT 5.2 refined workflow figures based on structures already defined by the authors.The authors retained responsibility for defining those structures.Source: Use of AI Tools · p. 10
  • ChatGPT 5.2 did not contribute to the study's design, data collection, analysis, interpretation, or scientific conclusions.Its stated use was limited to language and visual refinement.Source: Use of AI Tools · p. 10
Loading 2609.26057v1…