Source-linked AI summary
Usability testing: a review of some methodological and technical aspects of the method
J. M. Christian Bastien
TL;DR
The paper reviews user-testing procedures and tools relevant to evaluating medical and health care informatics applications, including participant numbers, test procedures, remote evaluation, usability tools, and mobile applications. It reports that the first five participants uncovered 35% of usability problems, while finding 85% on the studied websites required considerably more than five users; the review is not exhaustive and further research is needed on participant numbers.
Problem
User-testing work addresses applications whose use can involve error-prone interactions, with implications for improving health care delivery and increasing patient outcomes.
Method
The paper reviews work that specifies or clarifies user-test procedures and develops tools to help conduct tests, focusing on topics relevant to medical and health care informatics applications.
Results
35% of usability problems were uncovered with the first 5 participants, while the studied websites required considerably more than five users to find 85% of the problems.
Takeaways & Limitations
The selected topics provide a basis for evaluating medical and health care informatics applications, while participant-number guidance remains unresolved and requires further research.
Takeaways & Limitations
The review is not exhaustive, and the appropriate number of users to test remains far from solved.
Abstract
from arXiv · showhide
The aim of this paper is to review some work conducted in the field of user testing that aims at specifying or clarifying the test procedures and at defining and developing tools to help conduct user tests. The topics that have been selected were considered relevant for evaluating applications in the field of medical and health care informatics. These topics are: the number of participants that should take part in a user test, the test procedure, remote usability evaluation, usability testing tools, and evaluating mobile applications.
Usability testing:
The supplied opening passages identify the authors’ institutional affiliation and intended publication venue.
- The work is associated with Université Paul Verlaine–Metz’s Laboratoire Lorrain de Psychologie.
- The laboratory includes the Équipe Transdisciplinaire sur l’Interaction et la Cognition.
- The paper was intended for publication in the International Journal of Medical Informatics.
Abstract
The abstract keywords center on user-centered usability testing, remote evaluation, automated interaction capture, and usability-testing tools.
- The paper concerns user-centered design and user testing.
- It addresses remote usability evaluation and diary-based or automatic capture of user events.
- The topics include log files, user traces, and usability testing tools.
Introduction
Health-care technologies span patient and provider applications but can be error prone, making usability evaluation important. The paper reviews selected user-testing procedures and tools relevant to medical and health-care informatics, while noting that the review is not exhaustive.
- Introduction: Medical and health-care informatics includes patient-facing services, mobile applications, and provider technologies such as infusion pumps, CPOE, and electronic medical records.
- Introduction: Usability evaluation assesses whether interactive systems fit users and tasks while supporting effectiveness, efficiency, and positive responses.
- Introduction: Inspection-, User-, and Model-Based Evaluations are the three standard approaches, but model-based methods are described as limited, immature, expensive, and largely restricted to research teams.
- Introduction: The standard usability test raises questions when applied to domains involving Internet use or mobile devices.
- Introduction: The review is not exhaustive and does not present the three evaluation approaches in detail or provide a framework for evaluating health-care applications.
- Introduction: The paper reviews user-testing work on procedures and tools, selecting topics relevant to medical and health-care informatics.
User-based evaluation
User-based usability evaluation observes users performing tasks to identify design flaws and inform ergonomic improvements. The reviewed evidence highlights unresolved questions about participant numbers, test procedures, remote evaluation, and mobile applications.
- User-based evaluations involve users directly performing typical tasks while their behavior is observed and recorded to identify design flaws.Observed measures include task-completion time, completion rates, and error counts.
- The test procedure: A user test typically defines objectives, recruits and qualifies participants, selects tasks and scenarios, chooses measures, prepares materials and environments, and analyzes and communicates results.The procedure also includes selecting the tester, designing the protocol, and preparing satisfaction questionnaires.
- The review argues that new and validated approaches and methods should be available, including tools for usability testing and evaluation of mobile applications and services.
- How many users do we have to test?: The appropriate number of participants remains unresolved because recruitment involves practical, economic, and scientific considerations.The review notes that research since the 1990s has not produced a final answer.
- The test procedure: Paired-user testing produced better success rates without statistically different task-completion times, whereas individual testing found more problems and paired sessions better revealed user difficulties.Paired sessions also generated more spontaneous utterances, discussions, and task justifications.
- Remote usability evaluation: Remote synchronous testing produced results comparable to laboratory testing, with high correlations for task completion and task time and similar usability information overall.Both methods identified the most critical issues, although each also uncovered some unique issues.
- Remote usability evaluation: Remote asynchronous methods were equivalent in some evaluations but took considerably more time for subjects and identified fewer usability problems.
Conclusion
The paper reviews work to clarify user-testing procedures and develop tools for conducting tests, while illustrating the complexity of testing in specific situations. Although non-exhaustive, it indicates both existing work and remaining needs for valid usability-evaluation methods.
- The review examines efforts to specify or clarify user-testing procedures and develop tools to help conduct user tests.
- User testing can be complicated in specific situations, motivating attention to methods and supporting tools.
- The review is far from exhaustive but aims to indicate work conducted and work still needed for valid usability-evaluation methods such as user testing.