AI-powered software testing tools: A systematic review and empirical assessment of their features and limitations

1 INTRODUCTION

• Conventional test automation improves testing effectiveness and efficiency but remains vulnerable to fragile scripts, high maintenance costs, and limited adaptability to dynamic environments.Software testing is also described as resource-intensive and essential to software quality, reliability, and security.Source: §1 · p. 3
• AI-powered tools use machine learning, natural language processing, and computer vision for test generation, self-healing, visual regression testing, and failure analysis.They are presented as promising less manual intervention, greater test accuracy, and optimized execution workflows.Source: §1 · p. 3
• The study addresses limited real-world evaluation by reviewing 56 AI-powered testing tools and empirically assessing two tools on two open-source projects, while identifying limitations and future research directions.The review categorizes tools by capability; the empirical study assesses effectiveness and efficiency.Source: §1 · p. 3

2.1 Background: Emergence of AI-powered testing tools

• Terms for AI-enabled testing tools vary and are not standardized, and their wording may imply different levels of AI involvement.Table 1 lists the different terms identified by the authors.Source: §2.1 · pp. 3–4, Table 1 · p. 4
• A six-level autonomy model ranges from manual testing to fully autonomous testing, with self-healing at Level 3 still requiring human validation.At Level 4, testers supervise AI-generated and executed test suites; Level 5 assigns AI the full testing lifecycle without human involvement.Source: §2.1 · p. 4, Figure 1 · p. 5
• AI testing support includes general-purpose assistants and dedicated testing products; Parasoft Selenic exemplifies the latter by self-healing broken UI locators.General-purpose tools can assist with test activities, but output accuracy depends on information and prompt precision.Source: §2.1 · p. 5

2 BACKGROUND AND RELATED WORK

• The background reviews AI-powered testing tools alongside prior secondary studies, positioning this study’s systematic screening and analysis as a deeper examination of the field.The paper reports that no prior work conducted an analysis at this level of depth.Source: §2 · p. 3, §2.2 · p. 5

2.2 Related work

• The review systematically screened 56 tools available by the end of 2024 and examines their features, benefits, and limitations through three research questions.Earlier review papers provide useful insights, but the authors report that none conducted an analysis at comparable depth.Source: §2.2 · p. 5, Table 2 · p. 5

3 RESEARCH METHOD AND APPROACH

• The study pairs a systematic tool review of AI-powered testing tools with an empirical evaluation of two selected tools on two software systems.Source: §3 · p. 6

4 DESIGN AND EXECUTION OF THE SYSTEMATIC TOOL REVIEW

• The Systematic Tool Review follows established software-engineering review guidelines.Source: §4 · p. 6

4.1 Approach for searching for tools

• Google searches for “AI-X software testing tools,” varying X across seven industry terms, yielded 74 tools for screening.Source: §4.1 · p. 6

4.2 Inclusion and exclusion criteria

• Tools had to explicitly claim AI use and be publicly available; excluding general-purpose non-AI tools and tools without public information left 56 for analysis.Source: §4.2 · p. 7

4.3 Approach for extraction, management and synthesis of data

• The review method outlines data extraction, data management, and synthesis of findings.Source: §4.3 · p. 7

4.3.1 Data extraction approach for each RQ

• For RQ1, researchers reviewed tool homepages and demos for AI features; for RQ2 and RQ3, they assessed documentation, demos, and user reviews without installing each tool.When available sources lacked information, they queried ChatGPT for tool details.Source: §4.3.1 · p. 7

4.3.2 Data management: Google spreadsheet

• Extracted data were recorded in an online spreadsheet containing 56 tools, feature categories, mappings to research questions, and G2 user feedback on benefits and limitations.Source: §4.3.2 · p. 7

4.3.3 Approach for synthesis of qualitative data and evidence

• Qualitative descriptions of tool benefits and limitations were synthesized using qualitative coding, an established software-engineering and grounded-theory approach.Source: §4.3.3 · pp. 7–8

5.1.1 AI-powered test generation

• 43 tools (77% of those reviewed) provide AI-powered test generation, which can create large test suites in seconds or minutes rather than hours or days.The review describes tools drawing on application behavior, defects, code, requirements, user behavior, coverage gaps, and configuration data.Source: §5.1.1 · p. 8

Test generation using AI-powered NLP; Test generation using AI-powered test recording

• Test generation approaches included AI-powered NLP in 20 tools, record/playback in 9, coverage analysis in 13, and other approaches in 3.NLP converts natural-language scenarios into executable scripts, lowering the technical barrier and supporting collaboration; AI-enhanced recording adapts to UI and timing changes and can reduce flaky tests.Source: §5.1.1 · p. 8, Test generation using AI-powered NLP · p. 8, Test generation using AI-powered test… · p. 9

Test generation using AI-powered coverage analysis; Test generation using other AI-powered approaches

• Coverage analysis targets untested or high-risk code, while other approaches use models, agentic workflows, or behavioral analytics to generate context-aware tests.Models can represent system states and workflows; agentic tools interact with the system and refine testing strategies, while usage analytics can prioritize scenarios based on real-world patterns.Source: Test generation using AI-powered… · p. 10, Test generation using other AI-powered… · p. 10

5.1.2 AI-powered self-healing of UI test-code

• 32 tools (57% of those reviewed) provide self-healing that repairs broken UI locators, reducing test maintenance costs and improving stability.The tools can identify alternative locators by analyzing element attributes, DOM hierarchy, or visual similarity.Source: §5.1.2 · p. 11

5.1.3 AI-powered visual testing

• 16 of 56 tools (29%) provide AI-powered visual testing, using computer vision and machine learning to detect visual regressions while reducing false positives.Human oversight is still needed to refine results and address dynamic content and false positives.Source: §5.1.3 · pp. 11–12

5.1.4 AI-powered test-result and failure analysis

• 11 tools (20% of those reviewed) provide AI-powered test-result and failure analysis to identify probable causes, categorize failures, and recommend fixes.Some also cluster similar failures, offer CI/CD diagnostics, and analyze logs for performance bottlenecks or API failures.Source: §5.1.4 · p. 12

5.1.5 Other AI-powered features

• 23 tools offer additional AI features, including test prioritization, predictive analytics, exploratory testing, and test-data generation.These features target high-risk areas, anticipate failures, navigate applications without predefined scripts, or generate realistic and diverse test inputs.Source: §5.1.5 · p. 12

5.2 RQ2: Benefits of AI features in increasing test effectiveness and efficiency

• The review synthesized benefits from reviews, vendor documentation, reports, demonstrations, and white papers, rather than directly measuring every tool’s effectiveness.The authors note that their findings reflect vendor claims and user experiences; they empirically assessed two tools separately in Section 6.Source: §5.2 · p. 15

5.2.1 Potential increase in test effectiveness using AI features

• AI-powered testing may improve effectiveness through better defect detection, broader test coverage, and less human bias in test design and selection.Effectiveness concerns detecting defects, supporting software reliability, and increasing confidence in the SUT; efficiency concerns reducing testing effort and time.Source: §5.2.1 · p. 15

1. More comprehensive test coverage (reducing testing gaps)

• AI-generated test artifacts identify missing scenarios and test cases that manual testers might overlook.They analyze code structure, system behavior, and historical defect data.Source: §1 · p. 15
• Machine-learning models can predict defect-prone areas and direct coverage toward under-tested code regions.Source: §1 · p. 15
• This approach is particularly useful for large-scale and complex applications where traditional test design may miss execution paths.Examples include Diffblue Cover and TestRigor.Source: §1 · p. 15

2. Higher defect detection rate (catching more bugs)

• AI-powered tools can potentially catch hidden defects and regressions that traditional predefined test suites may miss.The passage frames this as a potential benefit, not a measured result.Source: §2 · p. 15
• AI tools analyze runtime behavior, execution logs, and edge cases, while detecting flaky tests and anomalies that may indicate failures.Source: §2 · p. 15
• This capability may be particularly valuable for dynamic web and mobile applications, where UI elements and backend responses frequently change.Applitools and Mabl are examples of tools mentioned.Source: §2 · p. 15

3. Making test coverage more objective (reducing human bias)

• Manual test selection can favor familiar or frequently used cases, while AI can assess historical defect trends, execution logs, and real-world usage data to reduce that bias.The resulting coverage is intended to align test cases with actual risk levels rather than human intuition.Source: §3 · p. 15
• AI-based risk analysis supports a more structured, data-driven approach to test coverage.Examples named include Functionize and Tricentis Tosca.Source: §3 · p. 15

4. Prioritizing critical and risky areas of the system

• The Pareto Principle suggests that testing a small portion—20% of test cases or code—can reveal a significant majority—80%—of defects.Source: §4 · p. 16
• AI can dynamically prioritize business-critical and frequently modified components for more testing than stable, low-risk sections.Qodo and Testim are given as examples.Source: §4 · p. 16
• Historical defect data, code complexity metrics, and version-control changes can inform which system areas receive more intensive test execution.Source: §4 · p. 16
• Risk-based prioritization is described as improving test-execution effectiveness and reducing the likelihood of defects escaping into production.Source: §4 · p. 16

5. Detecting defects earlier in the development cycle

• AI can analyze code changes and pull requests to generate targeted tests, while automated regression testing after each commit helps catch defects before they reach later SDLC stages.The stated aims include reducing defect leakage and extensive post-release testing; LoadMill and ReportPortal are given as regression-testing examples.Source: §5 · p. 16
• AI-generated test artifacts require human peer review and validation because outputs may be incorrect or lack contextual relevance and practical applicability.The paper also frames human oversight as necessary to ensure accuracy and quality.Source: §5 · p. 16
• AI can expand coverage and identify high-risk areas, but its limited domain understanding may produce irrelevant tests or miss business-relevant edge cases.Models may rely on historical defects, execution logs, and code complexity while failing to grasp business logic, workflows, or regulatory constraints; engineers should refine artifacts and align optimizations with actual risks.Source: §5 · p. 16
• AI defect detection can generate false positives or misclassifications, including mistaking minor visual shifts for regressions or missing functional issues when the interface appears unchanged.Human testers must interpret flagged defects and distinguish critical failures from non-issues.Source: §5 · p. 16
• The paper presents AI as an augmentation of human testers, combining automation with human judgment to refine coverage, validate failures, and maintain contextual flexibility.This hybrid approach is offered as a way to use AI efficiency while ensuring high-quality testing outcomes.Source: §5 · p. 16

5.2.2 Potential increase in test efficiency using AI features

• The efficiency analysis draws on tool documentation and practitioner feedback, defining efficiency gains in terms of reduced time, effort, and cost for a given testing quality.This differs from test effectiveness, which concerns defect detection and test quality.Source: §5.2.2 · p. 16

1. Faster test execution with AI optimization

• AI can dynamically analyze test results, historical data, and system behavior to prioritize fast, relevant tests, reducing execution time when full regression suites are impractical.Machine-learning models may predict tests likely to fail based on recent code changes, limiting runs and CI delays; LoadMill, ReportPortal, and Testify Pulse are examples.Source: §1 · p. 17
• AI orchestration can parallelize tests across environments and dynamically distribute them by priority and resource availability without increasing hardware costs.LambdaTest and SauceLabs are examples of cloud-based platforms described.Source: §1 · p. 17

• Self-healing automation detects and updates broken locators, keeping tests functional despite UI changes and reducing manual maintenance effort.Source: §2 · p. 17
• Tools such as Testsigma, Tricentis Tosca, and ACCELQ can replace obsolete element locators without human intervention, reducing maintenance costs.Source: §2 · p. 17
• Tools such as Testim and Functionize can stabilize flaky tests by adjusting waits, locators, or execution strategies, reducing debugging of false failures.Source: §2 · p. 17

• AI analyzes past execution data to remove redundant or duplicate cases that do not improve defect detection, focusing runs on high-value tests.Filtering ineffective cases optimizes execution time and computing resources.Source: §3 · p. 17
• AI clusters overlapping tests for merging and skips tests already validated against unchanged code, as in Functionize, Parasoft SOAtest, and TestResults.io.Source: §3 · p. 17
• Cloud-based AI monitors execution performance and recommends removing tests that never fail or adding tests where needed to balance speed and quality.Examples include Testim and Keysight Eggplant; this management reduces wasteful execution cycles.Source: §3 · p. 17

• Automating repetitive test artifact creation reduces test engineers’ burden and lets them focus on exploratory testing and analysis.AI can generate scripts for different configurations and scenarios.Source: §4 · p. 17
• AI tools can turn natural-language requirements into scripts and dynamically generate relevant cases by analyzing application behavior, defect history, and business logic.Examples include Qodo, Testim, and Virtuoso.Source: §4 · p. 17
• AI-generated test data provides realistic, diverse inputs across environments and targets edge cases and high-risk areas to improve test coverage.The passage identifies predictive analytics as the mechanism and Avo Automation and Katalon as examples.Source: §4 · p. 17

5. Seamless integration with CI/CD for continuous testing

• AI impact analysis selects tests relevant to code changes, while smart scheduling allocates runs across on-premises and cloud environments to reduce cycle time without overloading CI/CD pipelines.Examples include Applitools, Mabl, and Tricentis Testim for impact analysis, and LambdaTest and SmartBear TestComplete for smart scheduling.Source: §5 · pp. 17–18
• Human oversight is necessary because AI test selection or execution optimization can miss edge cases, low-frequency defects, and security risks, while incorrect self-healing or predictive analysis can cause false successes or failures.Test engineers should validate optimizations and ensure high-risk areas receive adequate coverage; a hybrid AI-human workflow is proposed to preserve reliability while streamlining repetitive work.Source: §5 · p. 18

5.3 RQ3: Limitations of AI-powered testing features

• Reported limitations include opaque AI decisions, ongoing model maintenance and retraining, difficult integration, overfitting and false positives, bias, weak handling of edge cases, and limited user control over training.These themes were identified through qualitative analysis of user reviews and existing literature, and may change as AI technology and tools advance.Source: §5.3 · pp. 18–19
• Feature-specific limits include redundant tests from coverage analysis, dependence on accurate models or historical data, and visual testing’s over-detection of minor differences and difficulty with dynamic content.Visual-testing mitigations such as tunable sensitivity, DOM-based heuristics, and context-aware comparisons are not yet fully reliable.Source: §5.3 · p. 19

5.4 The STR raised the need for an empirical study

• Because review findings based on documentation, vendor claims, and practitioner reviews showed context-dependent effectiveness, the study selected Parasoft Selenic and SmartBear VisualTest for empirical evaluation of self-healing and visual testing.The tools were chosen for capability diversity, maturity and adoption, and their evaluability against traditional automation approaches.Source: §5.4 · p. 20

6.1.2 Tools under assessment

• Selenic repairs failed Selenium UI-object locators, while VisualTest uses AI computer vision and DOM information to compare captured UI states and identify visual changes.VisualTest can distinguish intentional changes from regressions and classify differences by severity and relevance.Source: §6.1.2 · p. 21, Tool 1-Parasoft Selenic: AI feature… · p. 21, Tool 2-SmartBear VisualTest: AI… · p. 21

6.1.4 Development of a Selenium automated test-suite code-base for each of the two SUTs

• The researchers built Selenium suites covering main user flows: Spring-PetClinic had 15 test cases and 992 LOC, and Java-Blog-Aggregator had 18 cases and 1,804 LOC.They used the Page-Object-Model pattern to support maintainability, reduce duplication, and enable modular, reusable scripts.Source: §6.1.4 · pp. 23–24

Mutation-testing approach for assessing the AI self-healing feature of Parasoft Selenic:

• Mutation testing manually altered UI locators for Selenic and visual elements for VisualTest, using five and three Selenic mutants and five and four VisualTest mutants across the two SUTs.Manual mutation analysis was chosen because no suitable tool was available for the setup, adding one would complicate execution, and the planned mutant set was manageable.Source: §6.1.5 · p. 24, Mutation-testing approach for… · pp. 24–25

6.2 Findings of the empirical study

• Selenic healed all five Spring-PetClinic locator mutants but repaired only one of three Java-Blog-Aggregator mutants.It selects replacement locators using a Confidence Factor derived from Weight and Stability metrics; the tool’s rationale was considered systematic and explainable.Source: §6.2 · p. 25

6.2.2 RQ1: Assessing the benefits of the tool 2 (SmartBear VisualTest) – AI feature: Visual testing

• VisualTest detected all 9 injected UI mutants, including layout shifts, missing components, and styling inconsistencies, using comparisons against baseline screenshots.The assessment describes its image-based comparison as context-aware, including resolution and browser type.Source: §6.2.2 · p. 26

Limitations of the AI self-healing features of tool 1 (Parasoft Selenic)

• Selenic did not heal every affected locator, including tag-name locators when an element changed from an input to a button, and it struggled with large test suites.The study also observed that some locators healed in one test were not healed in other affected tests.Source: Limitations of the AI self-healing… · p. 27

Limitations of the AI visual-testing features of tool 2 (SmartBear VisualTest)

• VisualTest could highlight an entire HTML DIV rather than pinpointing multiple changes inside it, so minor visual inconsistencies, such as a changed footer character, could be missed.The authors suggest a second set of highlighters to identify changes within the DIV.Source: Limitations of the AI visual-testing… · p. 27

6.3 Summary of the empirical study

• Both tools increased test-automation effectiveness and efficiency: Selenic reduced test-maintenance time, while VisualTest enabled more efficient detection of visual defects.The empirical summary reports these overall findings in Table 6.Source: Table 6 · p. 28, §6.3 · p. 28

6.4 Generalizability of the findings from our empirical study

• The two tools reduced test maintenance effort and improved defect detection, but locator self-healing failed partially and visual defect identification had inconsistent granularity.The study assessed Parasoft Selenic and SmartBear VisualTest on two open-source systems.Source: §6.4 · p. 29
• Enterprise applicability is plausible because the open-source projects model real-world architectures and the tested features address widespread UI-change, maintenance, and regression-testing challenges.Enterprise systems with large UI components and frequent release cycles may be particularly suited to AI-powered test automation.Source: §6.4 · p. 29
• Enterprise transfer remains uncertain because legacy systems, customized workflows, security constraints, and large CI/CD pipelines add complexity; larger-scale evaluations are needed.Future work could assess scalability in high-traffic environments and adaptability to domain-specific needs such as healthcare, finance, and embedded systems testing.Source: §6.4 · p. 29
• AI testing still requires human oversight for complex scenarios and false positives, with explainability, failure handling, and adaptability identified as improvement priorities.The findings reinforce the Systematic Tool Review's insights into efficiency gains and remaining limitations.Source: §6.4 · p. 29

7.1 Fully autonomous software testing remains a distant goal

• Levels 1–3 of testing autonomy are present in current tools, while Levels 4–5 remain largely theoretical because of limits in context, accuracy, explainability, adaptation, and human-like judgment.The authors conclude that AI should assist rather than replace test engineers, who must review, refine, and validate AI-generated artifacts.Source: §7.1 · pp. 29–30

7.2 Implications of findings for practitioners (test engineers and QA teams)

• Practitioners should use AI testing tools to reduce maintenance, improve defect detection, and accelerate execution, while retaining human oversight to interpret results and refine artifacts.Human review also helps mitigate risks in AI-generated test artifacts.Source: §7.2 · p. 30

7.3 Implications of findings for tool developers

• Tool developers should improve explainability, reliability, and adaptability, reduce adoption barriers, and prioritize human–AI collaboration over full automation.These changes are intended to make tools more effective and trustworthy in real-world testing environments.Source: §7.3 · pp. 30–31

7.4 Potential threats to validity

• The study assessed internal, external, construct, and conclusion validity threats across its systematic review and empirical study, with mitigation strategies reported.The threats and mitigation strategies are presented in Table 10.Source: §7.4 · p. 31, Table 10 · p. 31

7.5 Conclusions and future work directions

• The 56-tool review and two-tool assessment found efficiency, maintenance, and defect-detection benefits, alongside false positives, contextual limits, and shortcomings in self-healing and visual testing.The empirical study found that self-healing reduced maintenance effort and visual testing improved UI validation accuracy, but locator self-healing had partial failure handling and visual testing lacked granular highlighting.Source: §7.5 · p. 32
• Future work should advance explainable AI, model generalization, context-aware risk prioritization, defect localization, CI/CD integration, and AI for exploratory and security testing.Proposed approaches include diverse training data, hybrid models combining computer vision and contextual and historical defect information, and AI that dynamically schedules tests or explores application behavior.Source: §7.5 · p. 32
• Future research should develop effective human–AI collaboration, with AI acting as an intelligent assistant rather than a replacement for test engineers.The section also calls for continuous model improvement, stronger lifecycle integration, and collaboration with human testers.Source: §7.5 · pp. 32–33

APPENDIX A-DETAILS OF THE EMPIRICAL RESULTS

• Tables 11 and 12 present empirical assessment results for Parasoft Selenic and SmartBear VisualTest, respectively.The supplied table passages identify the tables’ scope but do not provide their result values.Source: Table 11 · p. 35, Table 12 · p. 36
• A link-text locator was altered to use partial link text.The passage records the locator change but does not provide further performance results.Source: Appendix A · p. 35
• The empirical details include a J3 edit link whose target is constructed from the blog ID.The snippet shows an edit link with an href built from `/blog-form?blogId=` and `blog.id`.Source: Appendix A · p. 35
• Appendix B describes asking ChatGPT about a tool when its website does not clearly state its AI features.The passages provide the appendix heading and the condition for using this interaction.Source: Appendix A · p. 37

DUPLICATE OF TABLE 3

• The duplicate classification table includes Parasoft tools and SmartBear VisualTest, whose listed capability is AI-powered object recognition.The fragments name Parasoft JTest, Selenic, and SOAtest, and associate object recognition with Smartbear VisualTest.Source: Table 3 · p. 39, DUPLICATE OF TABLE 3 · p. 40
• One listed AI-assisted testing capability is describing test methods, generating mocks and assertions, and deciding whether to use mocks.The duplicate table fragments also identify Machinet AI Unit Tests as an IntelliJ plugin.Source: DUPLICATE OF TABLE 3 · p. 40
• The duplicate table describes AI features for API behavior clustering and dependency analysis, plus virtual-user testing and adaptive visual hints.The clustering passage says it groups documented and undocumented API patterns to improve coverage and detect edge cases; the visual-testing passage describes finding interaction points without predefined hints.Source: DUPLICATE OF TABLE 3 · p. 40