Source-linked AI summary
Testing Scientific Software: A Systematic Literature Review
Upulee Kanewala, James M. Bieman
TL;DR
Scientific software is important for consequential scientific work, but testing is difficult because of software characteristics and scientist-led development contexts. This systematic literature review analyzes 62 primary studies to identify testing challenges, techniques, limitations, and unresolved problems. It finds two main challenge categories and concludes that oracle problems remain prevalent while adapted software-testing techniques can help.
Problem
Testing scientific software is difficult because its characteristics and scientist-led development contexts create challenges such as oracle problems and limited software engineering familiarity.
Method
The paper conducts a systematic literature review using structured searches, study selection, quality assessment, and data extraction.
Results
The review identifies two categories of testing challenges and surveys techniques, limitations, and open problems across 62 primary studies.
Takeaways & Limitations
Existing software-testing techniques can address some challenges, but scientific software requires techniques that account for characteristics such as prevalent oracle problems.
Takeaways & Limitations
The review relies on keyword-based searches in three databases and cannot ensure that all relevant studies were retrieved.
Abstract
from arXiv · showhide
Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code. Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software. Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software. Results: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them. Conclusions: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
1. Introduction
Scientific software supports consequential scientific and engineering activities, yet testing receives limited attention despite faults that can alter results without crashes. This review examines testing challenges, proposed responses, and unresolved problems, including challenges arising from software characteristics and scientist-led development.
- Motivation: Scientific software supports critical decisions, climate predictions, and evidence reported in research publications.Examples include nuclear simulations, climate models, and computational results used in scientific publications.
- Motivation: Scientists often develop scientific software because it requires specialized domain knowledge, but may lack familiarity with software engineering practices.The paper defines scientific software broadly as software used for scientific purposes, with developers ranging from scientists without software engineering knowledge to professional developers.
- Motivation: Inadequate systematic testing allows subtle faults to change program outputs without crashes, contributing to losses in precision, compromised measurements, and retracted publications.Reported examples include one-off errors in seismic data processing, compromised coordinate measuring machine performance, and retracted published work.
- Motivation: Faults can arise while discretized models are translated into algorithms and code, while developers commonly emphasize scientific-model validation and computational verification.The review notes that systematic testing has received less attention than other development activities.
- Review focus: The review identifies challenges caused by software characteristics, such as oracle shortages, and by cultural differences between scientists and software engineers.It also surveys techniques addressing oracle and test-case challenges, their limitations, and open problems.
2. Research Method
The authors conducted a guideline-based systematic literature review using predefined research questions, database searches, selection criteria, quality assessment, and data extraction. The process retained 62 primary studies for synthesis, although keyword-based database searching may have missed relevant work.
- Protocol: The review protocol defines four research questions covering scientific software definitions, testing difficulty, adaptable testing methods, and unresolved challenges.The protocol was specified before the review to reduce researcher bias and included source selection, search, quality assessment, and extraction procedures.
- Search and selection: The search used Google Scholar, IEEE Xplore, and ACM Digital Library without a fixed publication time frame, with searches conducted in January 2013.The search string combined synonyms for challenges, techniques, testing, faults, and scientific or numerical software.
- Data extraction and assessment: Data were extracted with a structured form and assessed using binary quality questions, with all selected primary studies reported as high quality.The extraction and assessment followed Kitchenham’s guidelines, using different checklists for quantitative and qualitative studies.
3. Results
Testing scientific software is difficult because of both software-specific characteristics and cultural differences between scientists and software engineering communities. The review identifies oracle problems, complex numerical behavior, limited systematic testing, and adapted testing techniques as central themes.
- Challenges fall into two categories: characteristics of scientific software and cultural differences between scientists and software engineering communities.
- Scientific-software challenges: Complexity, round-off effects, large input spaces, long execution times, and difficult-to-replicate physical contexts complicate test-case selection and execution.
- Scientific-software challenges: Reliable test oracles are difficult to create because scientific programs may have complex outputs, uncertain models, numerical tolerances, and limited requirements.These difficulties make subtle faults harder to detect.
- Cultural challenges: Cultural challenges include ad-hoc testing, late testing, limited unit testing, weak awareness of verification testing, and misconceptions about testing benefits.Unit testing is reported as almost non-existent in the climate modeling community, partly because of legacy code and hidden internal structure.
- Testing practice: Only 12 of 62 primary studies applied unit, integration, system, acceptance, or regression testing, and 8 of those 12 used only one testing method.
- Proposed solutions: Existing approaches include pseudo oracles, analytical solutions, simplified and reference data, statistical oracles, metamorphic testing, boundary-focused tests, random testing, symbolic execution, and automated test generation.Each approach has limitations, including oracle disagreement, unavailable analytical solutions, inadequate simple tests, and uncertain statistical decisions.
- Proposed solutions: Reducing oracle tolerances can improve fault detection, while decomposing algorithms into independently tested steps can reduce compounding truncation and round-off errors.
4. Discussion
The review groups scientific software testing challenges into software-characteristic problems and cultural differences, then examines techniques, limitations, and unresolved research needs.
- Challenges due to software characteristics: Testing challenges arise from scientific software characteristics, including scarce real-world data, difficult physical-context replication, oracle problems, long execution times, and errors masking faults.Oracle problems occur when outputs are unknown or calculations are too complex to verify; 30% of primary studies reported them.
- Cultural challenges: Cultural challenges include viewing code and its model as inseparable, limited understanding of testing processes, ad-hoc testing, and failure to apply methods such as unit testing.These challenges reflect differences between scientist developers and the software engineering community.
- Potential solutions: Pseudo oracles, analytical solutions, experiments, measurements, simplified data, and professional judgment address oracle problems, but their fault-detection effectiveness lacks empirical evaluation.Metamorphic testing has been evaluated in research, yet the review found no evidence of its practical use.
- Potential solutions: Automated test-generation techniques were applied only to narrow program sets, so their applicability in practice remains unresolved.Random generation was modified for equivalence classes, while other techniques targeted floating-point programs.
- Testing practice: Few studies combined testing types, none used more than three of unit, system, integration, acceptance, and regression testing, and only two evaluated test coverage.Regression testing was reported, but automation and prioritization were unclear; only two studies used JUnit frameworks for unit testing.
- Unsolved problems and research opportunities: Software engineering research can improve testing by addressing oracle problems, while clone detection may expose duplicated faults in legacy scientific code.Many test-selection techniques assume a perfect oracle, limiting their suitability for scientific programs.
5. Conclusion and future work
This review synthesizes how scientific software testing is defined, the challenges it faces, existing testing approaches, and unresolved problems. It concludes that both software characteristics and development practices require tailored attention.
- Scientific software is usually developed by multidisciplinary teams comprising scientists and software developers, with a large computational component.
- Testing challenges arise from scientific software characteristics, such as oracle problems, and cultural differences involving how scientists view code and models.
- Studies report unit, system, and integration testing, while pseudo oracles, experimental results, and metamorphic testing help address missing oracles.
- Test-case creation techniques include combining random and designed cases, modeling input dependencies, and using symbolic execution for floating-point programs.
- Oracle problems remain prevalent and require further attention because existing techniques do not fully address them.
- Scientists should incorporate established software-testing techniques, while software engineers should account for scientific software’s special challenges when developing testing methods.