Source-linked AI summary
The Systems Paper is Dead. Long Live the Systems Paper
Bjoern Hartmann
TL;DR
UIST systems research has relied on single prototypes and confirmatory evaluations because building working systems was difficult, but rapid implementation challenges that premise. The paper proposes comparing prototype families, pursuing discovery-oriented evaluation, and studying evolving software, while noting that evaluation may become the new bottleneck. It concludes that systems papers should seek deeper contributions beyond implementation.
Problem
UIST systems research assumes that building a working interactive system is hard and slow, an assumption challenged by rapid prototype generation.
Method
The paper proposes three speculative methodological directions: compare prototype families, ask more of evaluations, and develop methods for software that evolves during use.
Results
The paper argues that faster implementation should enable different questions and conclusions, including tradeoff analysis, broader knowledge from evaluation, and studies of evolving systems.
Takeaways & Limitations
UIST papers should rethink contributions and pursue deeper knowledge rather than merely producing more of the same prototypes faster.
Takeaways & Limitations
When evaluation cannot be automated or accelerated, evaluation effort becomes the bottleneck constraining how many ideas can be tested.
Abstract
from arXiv · showhide
The way we structure, conduct, and write up interactive systems research in UIST papers rests on assumptions about constraints that may no longer hold today. What should an impactful UIST paper look like when building working systems is no longer hard? I argue that it should look different, and that we should ask more of our papers once implementation stops being a bottleneck.
1 The UIST Paper Formula, and Why It May Be Time for Methodological Change
UIST systems research has typically treated a polished prototype and confirmatory evaluation as an advance because working interactive systems were hard and slow to build. As implementation becomes faster, the paper argues for adapting research methods and raising expectations about the knowledge systems papers should generate.
- The canonical UIST process invents a concept, builds a working prototype, and evaluates whether it works, often with users.
- This process assumes that building a working interactive system is hard and slow, making one well-realized prototype with confirmatory evaluation a meaningful advance.
- Rapid code generation challenges that assumption because prototypes can now be built in days rather than consuming a graduate student’s semester.
- Producing more of the same faster increases the intensive margin, whereas researchers should expand the extensive margin by engaging in new activities.
- Research methods should adapt to abundant prototypes and consider both the promises and perils of faster system implementation.
- The proposed methodological change extends earlier critiques of formulaic usability testing and prior method shifts prompted by new technologies.
2 Three Speculative Directions
The paper proposes three speculative directions for research after implementation becomes easier: compare families of prototypes, seek knowledge beyond confirmation, and study software that evolves during use. Together, these directions shift attention from accelerating existing practices to asking different questions and drawing different conclusions.
- 2 Three Speculative Directions: Three speculative directions aim to ask different questions and draw different conclusions rather than merely accelerate existing research practices.
- 2.1 Think About Multiples: Families of prototypes can instantiate competing theories and empirically map tradeoffs among alternative designs.
- 2.1 Think About Multiples: Automated evaluation enables sensitivity analyses across whole design families, but evaluation effort becomes the bottleneck when evaluation cannot be automated or accelerated.
- 2.2 Ask More of the Evaluation: Discovery-oriented evaluations extend beyond confirming that a specific artifact works toward broader implications and unexpected knowledge.
- 2.3 Develop New Methods for Software That Keeps Evolving: Real-time evolution of working interfaces could let participant confusion trigger live regeneration, while raising questions about consent, analysis, and replicability.
3 What Will Your Future UIST Contribution Look Like?
The paper asks researchers to rethink what counts as a contribution once implementation is no longer the bottleneck. Future systems papers could gain insight from competing prototypes, tradeoff analyses, or continuously evolved systems studied in the wild.
- Once implementation is no longer the bottleneck, researchers should rethink what counts as a contribution.
- Papers comparing related prototypes that enact competing theories and analyze their tradeoffs could yield more insights than a single polished interface and lab study.
- A paper could instead study a system that participants collaboratively and continuously evolved during a multimonth deployment in the wild.
- The community may need to reconsider its research goals and reallocate research time toward deeper, further, and higher contributions.