PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI Guide Testing Without Generating Tests? A Comparative Study of AI-Assisted Unit Testing

Zihan Fang, Ron Asoulin-Handelman, Helen Wu, Yu Huang, Gina R. Bai

arXiv:2610.36185v1cs.SEcs.AI

TL;DR

Novices need to judge test quality as well as code, but evidence is limited on AI support that does not generate tests. This study compares TestCompass, which gives targeted feedback without executable tests, with general-purpose AI; most objective suite-quality measures were comparable, while TestCompass users achieved similar requirement coverage with fewer test methods.

  • Problem

    Novices struggle to choose test cases, assess suite sufficiency, and verify test correctness, while evidence is limited on feedback-only AI for testing compared with general-purpose AI that generates tests.

    Source: §I · p. 1

  • Method

    TestCompass combines coverage analysis and generative AI to give targeted feedback on test-suite weaknesses without generating executable tests, and was compared with general-purpose AI in two Java unit-testing tasks.

    Source: §I · p. 2, A. TestCompass Design · p. 3

  • Results

    Most objective test-suite quality differences were not statistically significant, and TestCompass participants achieved similar requirement coverage with fewer test methods.

    Source: §I · p. 2, §VII · p. 10

  • Takeaways & Limitations

    Feedback-oriented AI can support testing while keeping students involved in developing and verifying tests, though future tutors need more guidance on implementation and verification.

    Source: §I · p. 2

Abstract

from arXiv · show

Recent advances in generative AI have enabled assistants to generate increasingly sophisticated executable unit tests. However, novices may accept these outputs without independently identifying what to test, evaluating test-suite completeness, or verifying correctness, potentially limiting opportunities to develop essential testing skills. Although prior work has explored AI-assisted test generation and feedback-oriented programming tutors, little is known about whether AI can support novices' testing practices without generating tests for them. To investigate this, we introduce TestCompass, a generative AI tutor that provides targeted feedback on weaknesses in students' test suites without generating executable tests. In a classroom study with 18 students, we compared TestCompass with a general AI assistant across two Java unit-testing tasks, examining students' testing behaviors, test-suite quality, and subjective experiences. We found that participants using the general-purpose AI assistant produced more tests, whereas TestCompass participants spent more time developing and verifying tests themselves. However, most test-suite quality measures were comparable between conditions, and TestCompass participants achieved similar requirement coverage with significantly fewer test methods ($p=0.012$) in one task. Neither group reported short-term improvements in perceived testing knowledge or confidence. These findings suggest that feedback-oriented AI can support meaningful testing outcomes while preserving opportunities for hands-on testing practice, offering a complementary approach to AI-assisted testing education.

I. INTRODUCTION

Novices must evaluate tests as well as AI-generated code, yet evidence is limited on whether feedback-only AI can support testing without supplying executable tests. This study introduces TestCompass and compares it with general-purpose AI in a classroom study of unit testing.

I. INTRODUCTION

  • Novices struggle to choose what to test, judge suite sufficiency, and evaluate test correctness, while software testing often receives limited curricular emphasis.Generative AI also lets students obtain tests for verifying code, adding the need to evaluate AI-generated tests.Source: §I · p. 1
  • Prior evidence is limited on how testing-specific, feedback-only AI compares with general-purpose assistants that generate executable tests, especially for novice behavior and suite quality.The comparison can inform AI tools intended to support testing while students remain engaged in test design and verification.Source: §I · p. 1
  • In the classroom comparison, general-purpose AI participants spent more time interacting with assistants, while TestCompass participants spent more time developing and verifying tests; most suite-quality measures were comparable.TestCompass participants achieved similar requirement coverage with significantly fewer test methods.Source: §I · pp. 1–2
  • TestCompass provides targeted feedback on weaknesses in students’ test suites without generating executable tests.The study compares it with general-purpose AI across students’ testing behaviors, test-suite quality, and perceptions.Source: §I · p. 2
  • The authors present design insights for future testing tutors, including a need for greater guidance on test implementation and verification.They frame TestCompass as supporting effective testing without taking test development away from students.Source: §I · p. 2

A. LLM-Based Support for Software Testing

  • Prior LLM testing work targets executable-test quality, using program context, iterative refinement, or validation, while this study examines feedback that helps students build their own suites.Prior work also reports correctness issues in ChatGPT-generated tests, including compilation errors and incorrect assertions.Source: A. LLM-Based Support for Software… · p. 2

B. Learning and Teaching Software Testing

Prior testing-education work explores conceptual feedback and structured scaffolding, while programming tutors offer guidance without complete solutions. TestCompass combines coverage analysis and generative feedback to identify testing gaps without writing tests for students.

B. Learning and Teaching Software Testing

  • Testing-education studies have examined conceptual feedback and hints, but evidence remains limited on dynamically generated, testing-specific feedback versus general AI that can generate executable tests.Prior findings include conceptual feedback associated with higher coverage and fewer redundant tests, and instructor-written hints associated with a small but significant improvement in mutant detection.Source: B. Learning and Teaching Software… · p. 2

C. Pedagogical AI Support in Programming Education

  • Programming tutors such as CodeHelp, CodeAid, and Iris provide guidance without complete solutions; prior comparisons found no significant differences in learning gains or task performance between a tutor, ChatGPT, and no AI.The study builds on this work by comparing feedback-only AI with general-purpose AI for unit testing.Source: C. Pedagogical AI Support in… · p. 2

A. TestCompass Design

  • TestCompass combines coverage analysis with generative AI to identify untested code and provide targeted feedback, while leaving students to write and verify their own tests.It ranks production methods by uncovered lines, supplies code, coverage, and existing tests as context, and prompts for recommendations describing behavior, action, and expected outcome; students then implement tests and rerun coverage.Source: A. TestCompass Design · p. 3
  • Before students see recommendations, a second LLM pass reviews them and deterministic checks assess required structure and task-specific properties; failures trigger a correction pass.The process aims to reduce incorrect or internally inconsistent feedback.Source: A. TestCompass Design · p. 3

B. TestCompass Feedback Evaluation

The study used two Java unit-testing tasks and evaluated TestCompass feedback with a five-dimension rubric. Expert review and pilot sessions rated the feedback above four on every dimension.

B. TestCompass Feedback Evaluation

  • The rubric assessed recommendation correctness, relevance, actionability, clarity, and pedagogical usefulness on a five-point scale.Pedagogical usefulness concerned whether feedback encouraged independent testing decisions rather than supplying ready-made solutions.Source: B. TestCompass Feedback Evaluation · p. 3
  • TestCompass feedback was rated above 4 on every rubric dimension across expert review and three pilot sessions.Two researchers independently checked recommendations against specifications, code, existing tests, and coverage, resolving disagreements through discussion.Source: B. TestCompass Feedback Evaluation · p. 3

1) Unit-Testing Tasks:

  • The two Java tasks used a fixed order and common language, framework, and environment, with participants given implementations, specifications, and minimal starter tests.MarsRover covered 12 requirements including state transitions and boundaries; BowlingScoreKeeper covered 13 requirements including scoring dependencies across frames.Source: 1) Unit-Testing Tasks · pp. 3–4

2) Surveys:

The study recruited students with Java and introductory JUnit experience, assigned them to TestCompass or general-purpose AI, and had them complete two testing tasks. Screen recordings, submitted suites, and surveys supplied the study data.

2) Surveys:

  • Pre- and post-study surveys covered testing experience and confidence, challenges, tool use and helpfulness, perceived knowledge, and open-ended feedback.The pre-study survey assessed self-reported Java unit-testing experience and confidence in writing comprehensive tests.Source: 2) Surveys · p. 4

D. Participant Recruitment and Group Assignment

  • Eighteen students were randomly assigned to conditions; after one TestCompass-assigned participant used general AI, analysis included eight TestCompass and ten AI participants.All reported less than one year of unit-testing experience; most identified as novices, and the analysis groups showed no significant differences in education, gender, programming experience, or Java experience.Source: D. Participant Recruitment and Group… · p. 4

E. Study Protocol

  • Participants completed both tasks in IntelliJ IDEA, could run tests and inspect failures and coverage, and were screen-recorded; each session lasted about 1.5 hours.They used condition-specific tools, submitted project files and recordings after each task, and completed a post-study survey.Source: E. Study Protocol · p. 4

F. Data Analysis

  • The dataset comprised 36 test suites with 798 identified test methods, 36 screen recordings, and matched pre- and post-study survey responses from all participants.The researchers analyzed these materials to examine tool interaction, testing performance, and perceptions.Source: F. Data Analysis · p. 4

1) Interaction Analysis:

The study analyzed recorded testing activities and assessed test-suite quality across completeness, effectiveness, maintainability, and implementation barriers. It also used multiple coverage measures and participant-level comparisons to evaluate those dimensions.

1) Interaction Analysis:

  • 22.7 hours of recordings were segmented by observable activity, manually annotated with AI assistance, and independently reviewed for half the participants.Activity duration and frequency were calculated per participant and task; proportions of classified task time were averaged within conditions.Source: 1) Interaction Analysis · p. 4

2) Test-Suite Quality Analysis:

  • Four dimensions captured suite quality: completeness, effectiveness, maintainability, and implementation barriers.They covered intended-behavior exercise, fault detection, code quality, and difficulties implementing correct tests.Source: 2) Test-Suite Quality Analysis · pp. 4–5
  • Completeness was measured through requirement, line, and branch coverage, with JaCoCo checks testing the robustness of coverage comparisons.Requirement coverage was manually assessed against 12 MarsRover and 13 BowlingScoreKeeper requirements, regardless of implementation errors or incorrect expected results.Source: 2) Test-Suite Quality Analysis · p. 5
  • Effectiveness used PIT mutation coverage, while maintainability measures included static-analysis issues, test-name ratings, and redundant methods.Researchers also recorded issue counts per test method to account for suite size.Source: 2) Test-Suite Quality Analysis · p. 5
  • Implementation barriers were assessed through submitted test-method counts and the proportion of tests with compilation, intention, assertion, logic, or exception-handling errors.The study reported the number of inventoried methods and the percentage classified as erroneous.Source: 2) Test-Suite Quality Analysis · p. 5

3) Survey Analysis:

The results compare students’ activity patterns across the two AI conditions, including their use of AI and hands-on testing. TestCompass participants devoted more time to test work and were more likely to execute and inspect tests, while general-AI participants spent more time consulting their assistants.

4) Statistical Analysis:

  • The study used inductive thematic analysis for open-ended responses and participant-level, task-specific comparisons with exact permutation or Fisher’s exact tests and Holm correction.Continuous outcomes used Mann–Whitney U statistics with Cliff’s delta; the significance threshold was α = 0.05.Source: 3) Survey Analysis · p. 5, 4) Statistical Analysis · p. 5

A. RQ1: How Do Students Interact with Different AI Tools for Unit Testing?

  • 27.1 vs. 8.9 minutes: TestCompass participants spent significantly more time working on tests in BowlingScoreKeeper.Test-related work accounted for 72.6% vs. 31.8% of time in MarsRover and 59.1% vs. 22.9% in BowlingScoreKeeper, for TestCompass vs. general AI.Source: A. RQ1: How Do Students Interact with… · p. 6
  • 13.5 vs. 2.3 minutes and 12.4 vs. 2.5 minutes: general-AI participants spent significantly more time consulting assistants in MarsRover and BowlingScoreKeeper, respectively.The between-condition differences had p = 0.015 and p = 0.011, respectively.Source: A. RQ1: How Do Students Interact with… · p. 6
  • 60% and 80% of general-AI participants used their assistant for multiple purposes in MarsRover and BowlingScoreKeeper, while one participant used it only for test planning.Purposes included understanding program behavior, planning or generating tests, troubleshooting, debugging, and reviewing coverage; TestCompass users mainly reviewed coverage and recommendations.Source: A. RQ1: How Do Students Interact with… · p. 6
  • 87.5% of TestCompass participants executed or inspected JUnit results in each task, compared with 50% of general-AI participants in MarsRover and 60% in BowlingScoreKeeper.Time spent on these activities was similar: 9.2% vs. 9.5% in MarsRover and 10.0% vs. 7.7% in BowlingScoreKeeper, for TestCompass vs. general AI.Source: A. RQ1: How Do Students Interact with… · p. 6

B. RQ2: How does objective test-suite quality differ between the conditions?

Across two tasks, objective test-suite quality was mostly comparable between conditions, despite larger suites and numerically higher coverage and mutation scores in the general-AI condition. TestCompass participants achieved similar complete requirement coverage with fewer methods, significantly so in BowlingScoreKeeper.

a) Completeness:

  • 90.8% vs. 85.4% requirement coverage in MarsRover and 96.2% vs. 89.4% in BowlingScoreKeeper favored general AI numerically, but differences were not significant after correction.Four participants per condition covered all MarsRover requirements; six TestCompass and seven general-AI participants covered all BowlingScoreKeeper requirements.Source: B. RQ2: How does objective test-suite… · p. 7

b) Effectiveness:

  • 84.9% vs. 79.2% mutation coverage in MarsRover and 91.5% vs. 80.8% in BowlingScoreKeeper favored general AI numerically, but neither difference was statistically significant.The conclusions were unchanged when one TestCompass participant with two compilation-error tests was assigned a mutation score of zero.Source: B. RQ2: How does objective test-suite… · p. 7

c) Maintainability:

  • General-AI suites had fewer maintainability issues per method in both tasks, but only BowlingScoreKeeper was significant (0.19 vs. 0.40; p = 0.044).Test-name ratings favored general AI without significant differences, while redundancy comparisons were also not significant.Source: B. RQ2: How does objective test-suite… · p. 7

d) Implementation Barriers:

  • 17.9 vs. 27.6 test methods: TestCompass participants submitted significantly fewer methods in BowlingScoreKeeper (p = 0.012), while complete requirement coverage was similar between conditions.In MarsRover, participants submitted 12.0 vs. 28.3 methods for TestCompass vs. general AI.Source: B. RQ2: How does objective test-suite… · p. 7
  • Confirmed-error rates did not differ significantly: 0.0% vs. 0.3% in MarsRover and 11.0% vs. 2.6% in BowlingScoreKeeper, for TestCompass vs. general AI.The authors characterize most objective suite-quality differences as not statistically significant.Source: B. RQ2: How does objective test-suite… · p. 7

C. RQ3: What are students’ subjective experiences with the tools?

Participants described different roles for the tools: TestCompass was used to guide testing decisions and assess completeness, while general AI was consulted more continuously and rated more helpful. Neither group reported short-term gains in perceived testing knowledge or confidence, and TestCompass users wanted clearer explanations for applying recommendations.

a) Self-Reported Tool Use:

  • 75% of TestCompass and 70% of general-AI participants sought help identifying what to test next, while 62.5% vs. 20% sought guidance on whether testing was sufficient.Six TestCompass participants (75%) reported consulting the tool before submission; general-AI participants reported more frequent tool consultation during testing (80% vs. 25%).Source: C. RQ3: What are students’ subjective… · pp. 7–8

b) Perceived Helpfulness and Recommendation:

  • 100% of general-AI respondents rated helpfulness 4 or 5, compared with 37.5% of TestCompass respondents (p = 0.007), but recommendation ratings were more similar.At least 4 recommendation ratings were given by 60% of AI respondents and 50% of TestCompass respondents.Source: C. RQ3: What are students’ subjective… · p. 8

c) Self-Assessed Knowledge and Confidence:

  • Mean self-assessed knowledge remained 3.0 before and after the session in both conditions, and confidence ratings were similarly stable.TestCompass participants’ confidence increased slightly.Source: C. RQ3: What are students’ subjective… · p. 8

d) Reported Challenges:

  • Participants reported difficulty identifying test cases, understanding specifications and expected outputs, and deciding when testing was sufficient; TestCompass users also wanted explanations of recommendation reasoning.Their feedback suggests that identifying gaps alone may not provide enough guidance to translate recommendations into effective tests.Source: C. RQ3: What are students’ subjective… · p. 8
  • General-AI participants valued test-generation convenience, while some reported failed generated tests or project-setup difficulties and one wanted to learn programming without AI.The responses point to a tension between immediate assistance and developing independent testing judgment.Source: C. RQ3: What are students’ subjective… · p. 8

A. Internal Validity

Random assignment and comparable self-reported experience helped align the conditions, but unmeasured participant differences and variation in general-purpose AI tools remain potential sources of variability. Fixed task order and screen recording also constrain interpretation of behavior.

A. Internal Validity

  • Unmeasured differences in programming proficiency, testing skills, or AI familiarity may have influenced participants’ performance despite random assignment and comparable self-reported experience.Participants were recruited primarily from the same course, with one exception from a senior-level course.Source: A. Internal Validity · p. 8
  • Participants could use different general-purpose AI assistants, so the comparison concerns two forms of AI support rather than a standardized assistant.The study frames this heterogeneity as a potential source of variability.Source: A. Internal Validity · pp. 8–9
  • Fixed task order makes task-specific effects difficult to separate from practice, fatigue, or carryover; screen recording may also have affected behavior.Screen recording applied to both conditions, and participants were told performance would not affect course grades.Source: A. Internal Validity · p. 9

B. External Validity

The study’s small, single-institution sample, two Java tasks, and short duration limit generalization. Its measures capture important but partial aspects of test quality, while the findings motivate AI support that adapts guidance to students’ implementation needs.

B. External Validity

  • Most participants came from one course at one institution, and the short study of two Java tasks may not generalize to other learners, developers, or testing contexts.The study lasted about 1.5 hours and measured short-term behavior, performance, and perceptions rather than long-term learning, retention, or transfer.Source: B. External Validity · p. 9

C. Construct Validity

  • Coverage does not necessarily measure test correctness or semantic quality, and mutation coverage depends on generated mutants and mutation operators.Static-analysis issues, test-name quality, and redundancy may not fully capture long-term maintainability; submitted-test counts and confirmed errors only partly represent implementation challenges.Source: C. Construct Validity · p. 9
  • Survey responses may be biased, and screen recordings show observable behavior but cannot fully reveal participants’ reasoning or decisions.Manual coding and qualitative analysis involved researcher judgment, although consistent criteria and review by a second researcher were used to reduce subjectivity.Source: C. Construct Validity · p. 9

A. Teaching Software Testing in the AI Era

  • General-purpose AI participants produced larger suites and generally higher mean coverage, but comparable proportions of TestCompass participants covered all requirements; coverage differences were not significant after correction.TestCompass participants spent more time writing, revising, and inspecting tests, while general AI participants spent more time consulting assistants; these patterns do not establish better learning.Source: A. Teaching Software Testing in the AI… · p. 9

B. Designing AI Tutors That Bridge Guidance and Implementation

  • Tool choice may depend on instructional goals: general AI can support unfamiliar concepts, while recommendation-based tutors suit exercises requiring students to implement and evaluate tests.TestCompass users sometimes struggled to interpret lengthy recommendations, translate them into tests, and resolve execution problems; future tutors could provide progressively more concrete implementation guidance.Source: A. Teaching Software Testing in the AI… · p. 9, B. Designing AI Tutors That Bridge… · p. 10

C. Feedback Must Support Oracle and Domain Reasoning

Participants struggled to derive and judge expected outcomes, so testing guidance may need to explain the reasoning behind assertions. Perceived helpfulness and task involvement do not establish learning gains, which the single-session study could not determine.

C. Feedback Must Support Oracle and Domain Reasoning

  • Participants in both conditions struggled to interpret specifications, determine expected outputs, and judge test sufficiency, especially for BowlingScoreKeeper’s scoring rules.Recognizing a relevant case did not guarantee a correct assertion, and a test could run without checking the correct behavior.Source: C. Feedback Must Support Oracle and… · p. 10
  • Future tutors could explain which requirement a test addresses and how its expected result is derived, revealing concise guidance progressively as needed.This is proposed to help students understand their test reasoning and evaluate AI-generated expected outputs rather than accept them unverified.Source: C. Feedback Must Support Oracle and… · p. 10

D. Perceived Helpfulness Does Not Necessarily Reflect Learning Support

  • General-purpose AI was rated more helpful, but this did not yield a significant requirement-coverage advantage, and neither condition showed consistent self-assessed knowledge or confidence gains.The single-session study cannot establish whether students’ additional involvement with recommendation-based support leads to better learning outcomes.Source: D. Perceived Helpfulness Does Not… · p. 10

VII. CONCLUSION

  • TestCompass offered recommendations without generating tests, but participants’ difficulty interpreting guidance and determining expected outcomes points to a need for added implementation support.The conclusion calls for adaptive tutors combining recommendations with implementation and verification support, and for longitudinal study of independent testing skills.Source: §VII · p. 10
Loading 2610.36185v1…