Criteria for Rigor in Visualization Design Study

1 INTRODUCTION

• Process models and software validation leave limited guidance for producing and judging the diverse knowledge design studies can yield.The authors describe a resulting emphasis on working software over situated, complex learning.Source: §1 · p. 1, §2.4 · p. 3

2 THEORETICAL BACKDROP

• The paper draws on four years of engagement with debates about knowledge generation and rigor across social science, information systems, and design.The authors describe these debates as contradictory, dynamic, and nuanced.Source: §1 · p. 1, §2 · p. 1

2.1 Philosophical Positions

• Positivist work prioritizes objectivity, repeatability, and reproducibility, while interpretivist inquiry treats knowledge as subjective and socially constructed; the field lacks shared rigor standards.Source: §2.1 · pp. 1–2

2.2 Interpretivist Criteria

• Interpretivist rigor criteria remain contested, and the authors propose visualization-specific criteria rather than wholesale assimilation of general standards.Lincoln and Guba’s trustworthiness criteria are one influential contribution to this ongoing debate.Source: §2.2 · p. 2

2.3 Design Research

• Action research develops knowledge through iterative intervention and reflection, while research through design treats design activities as knowledge production.Action design research also links researcher intent with organizational context.Source: §2.3 · p. 2

2.4 Visualization Design Study

• Visualization design study combines real-world domain problems, expert collaboration, system design, validation, and reflection, but existing models offer little guidance for generating or judging varied knowledge claims.Source: §2.4 · pp. 2–3

• The perspective extends and deepens the existing definition of visualization design study.Source: §3 · p. 3
• It treats design as a subjective method for constructing and communicating knowledge under assumptions of multiple, mind-dependent realities.Source: §3 · p. 3
• The authors synthesize approaches from qualitative social science and research through design into four statements, each linked to rigor criteria discussed later.Source: §3 · p. 3

Design study uses design for inquiry and expression.

• Designers learn by exploring possibilities through making, prototyping, and reflecting on evidence gathered with domain experts.Design instances and artifacts can express learning, sometimes implicitly through their form and interaction.Source: Design study uses design for inquiry… · p. 3
• Written reports and other media make design-study knowledge available through explicit, appropriately scoped claims.Reports may include prose, figures, imagery, software, digital artifacts, and annotated videos.Source: Design study uses design for inquiry… · p. 3
• The proposed criteria call for designs informed by existing designs, abundant in observations and descriptions, plausible in interpretation, resonant, and transparent in claims and evidence.Source: Design study uses design for inquiry… · p. 3

Design study tackles wicked problems.

• Wicked problems shift as problem definitions and solutions shape each other, so design study calls for multiple perspectives and transparent reporting of this dynamic process.The authors identify abundance and transparency as rigor criteria for addressing this changing problem-solution relationship.Source: Design study tackles wicked problems · pp. 3–4

Design study is inherently subjective.

• Design-study knowledge is personal and context-dependent; researchers’ values and choices shape the artifacts, problems, observations, interpretations, and details they report.The relevant context includes domain experts, datasets, and organizational and social constraints.Source: Design study is inherently subjective · p. 4
• A relativist perspective treats observed realities as multiple, relative, changing, and mind-dependent, rather than as one reality viewed objectively.This position contrasts with positivist approaches prevalent in visualization research.Source: Design study is inherently subjective · p. 4
• Four criteria address subjectivity through reflexive practice, transparent communication of researcher effects, plausible claims, and appropriately informed analysis.Researchers can draw on methods from other fields to diversify perspectives and recognize their own role in constructing knowledge.Source: Design study is inherently subjective · p. 4

Design study produces diverse knowledge claims.

• Design studies produce knowledge about visualization idioms, design guidelines and methods, and problem domains, through design practice and purposeful examination.These topics concern representations in context, ways to develop solutions, and relationships among people, data, and technology.Source: Design study produces diverse… · p. 4
• Recording observations, sketching ideas, and manipulating code or data each produce interpreted knowledge expressions through researchers’ acts of abstraction.Researchers decide which details are meaningful and how to record them.Source: Design study produces diverse… · p. 4
• Design-study knowledge ranges from specific, situated accounts to more general claims; the balance trades explanatory potential against the accuracy of the specific.Where knowledge becomes theory is contested, and narrowly scoped theories can have practical impact.Source: Design study produces diverse… · p. 4
• One design study’s claims ranged from creativity methods to the data-sculpting interaction concept and a software artifact for comparing household appliance energy use and scheduling.Six criteria support this diversity: informed and reflexive practice, abundant evidence, and transparent reporting that supports plausible and resonant claims.Source: Design study produces diverse… · pp. 4–5

4 SIX CRITERIA FOR RIGOR

• The six criteria guide context-sensitive rigor without requiring every study to meet them all; researchers choose and justify methods, and readers assess how well reports support the criteria.They were developed through four years of engagement with social-science and design literature, including action design research, trustworthiness, and Tracy’s big-tent criteria.Source: §4 · p. 5

4.1 INFORMED

• Existing knowledge informs design choices and interpretation while leaving room for context-specific solutions and new interpretations.Rather than dictating each design step, prior knowledge provides a backdrop against which researchers shape their processes and artifacts to the situation’s richness and complexity.Source: §4.1 · p. 5

4.2 REFLEXIVE

• Reflexivity makes researchers’ roles, biases, and effects on the study explicit, while attending to participants’ responses and missing voices.Researchers can sustain this practice through observation, reflection, notes, discussions, and candid reporting.Source: §4.2 · pp. 5–6

4.3 ABUNDANT

• Abundance uses detailed evidence, diverse perspectives, and tested design alternatives to support nuanced interpretations, with sufficiency judged in context and saturation explained.Long-term collaboration, participatory methods, and rapid prototyping can broaden input, but collecting and documenting more evidence can consume resources and disrupt design work.Source: §4.3 · p. 6

4.4 PLAUSIBLE

• Plausible claims link sufficient, complementary evidence to coherent justifications, using reflective analysis to move from situated details to broader concepts.The appropriate methods depend on the claim: particular claims rely on suitable processes and evidence, while broader claims require analytic generalization.Source: §4.4 · p. 7

4.5 RESONANT

• Resonant research affects audiences by supporting transfer between contexts and inspiring understanding, empathy, or action through evocative reports and artifacts.Rich, situated details can help readers judge whether findings transfer and make reports compelling.Source: §4.5 · pp. 7–8

4.6 TRANSPARENT

• Transparency supports scrutiny rather than reproducibility by documenting what researchers did, how, and why, including the problem knowledge implicit in design artifacts.A data and task abstraction can make explicit the problem a visualization artifact is intended to address.Source: §4.6 · p. 8
• Transparent analysis and evidence help readers judge whether findings ring true and enable new interpretations, while frank reporting includes errors, failures, and dead ends.Source: §4.6 · p. 8
• Academic papers often cannot accommodate thick description and reflexive detail, while growing supplementary materials make navigation, search, and interpretation harder.Supplementary materials can provide richer details, including images, software, videos, observations, analysis, field notes, reflexive notes, and audit trails; the passage asks whether other reporting forms may serve design study.Source: §4.6 · p. 8
• Transparency means justifying design choices and research stance, disclosing reflexive notes, sharing interpretable evidence, describing analysis and failures, and communicating artifact knowledge for transfer.Source: §4.6 · p. 8
• Transparency should be embraced throughout the design study, not treated as a task reserved for the end.Source: §4.6 · p. 8

5 QUESTIONS FOR DEBATE

• The authors’ perspective emerged from their experiences, wide reading, and discussions with each other and the broader community.They note that few ideas are new individually, while their combination points to new approaches and opportunities for inquiry.Source: §5 · p. 8
• The paper acknowledges that its perspective has implications and that many open questions remain.Source: §5 · p. 8
• The authors use open-ended questions to challenge perspectives, stimulate debate, and expose implicit assumptions and practices associated with design study.Source: §5 · p. 8

• Visualization courses and textbooks seldom cover ontologies, epistemologies, or the range of methods needed for design and relativist inquiry.Source: Is our field prepared for rigorous… · p. 8
• Design study already requires broad technical and collaborative skills, while capturing and interpreting rich qualitative data may stretch researchers further.Source: Is our field prepared for rigorous… · p. 9
• Current paper and supplementary formats, together with condensed review times, work against the effective exposition and consideration of the advocated design study.Source: Is our field prepared for rigorous… · p. 9
• Interpretivist design study can yield diverse insights into people’s relationships with data and technology, relevant to changing visualization practices and the human side of data science.Source: Is our field prepared for rigorous… · p. 9
• Rigor standards could clarify methodological trade-offs and help improve inquiry and guide review, although calls for more work and skills risk undermining design study’s acceptance.Source: Is our field prepared for rigorous… · p. 9
• The authors encourage continued community discussion to better understand, achieve, support, and assess rigor in design study.Source: Is our field prepared for rigorous… · p. 9

• Rigor does not necessarily make a design study high quality, according to the authors.Source: Are these criteria enough? · p. 9
• Tracy’s broader criteria include topic worthiness, relevance, timeliness, significance, morality, practicality, and the study’s ethical stance.Source: Are these criteria enough? · p. 9
• Underconsidered ethical issues include procedures involving participants, funding and project focus, and whether post-study exit leaves collaborators with sustainable benefits.Source: Are these criteria enough? · p. 9

• A four-year engagement with related fields left the authors asking where visualization’s own philosophical and theoretical discourse takes place.Source: Does our field have adequate… · p. 9
• Visualization discussions of disciplinary foundations occur mainly in informal, ephemeral venues, making community views difficult to understand and collate.Source: Does our field have adequate… · p. 9
• The authors call for public, critical, frequent discussion of disciplinary boundaries, theory, knowledge assessment, and socio-political concerns.Source: Does our field have adequate… · p. 9

• Positivist approaches suit many perceptual, cognitive, and computational phenomena, while relativist approaches often benefit studies of complex, contextualized social reactions.Source: §6 · p. 9
• The authors synthesize work from related disciplines through reading, conference attendance, and engagement with those communities.Source: §6 · p. 9
• Their aims include clarifying what design study does and can know, valuing both particular and general knowledge, and informing judgments of legitimacy.Source: §6 · p. 9
• The authors seek methodological foundations that can change how situated design-study knowledge is approached, created, assessed, and used.Source: §6 · p. 10
• They reposition design study as a subjective interpretive inquiry that can rigorously address real-world problems and complement positivist research, guided by six criteria: INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT.Source: §6 · p. 10
• The criteria are dynamic, contradictory, imperfect, narrowly scoped to design-study rigor, and do not cover all aspects of high-quality research; transfer beyond visualization remains speculative.Source: §6 · p. 10

INFORMED

• Existing knowledge grounds research and supports interesting, meaningful insights and interpretations.Source: INFORMED · p. 13
• Researchers can consult textbooks, academic papers, and technical documentation to build knowledge of visualization and its domain.Source: INFORMED · p. 13
• Broadening exposure to varied design ideas is another suggested way to develop informed research.Source: INFORMED · p. 13

REFLEXIVE

• Reflexivity recognizes and leverages researchers’ subjectivity, including the reciprocal influence between researchers and their research.Source: REFLEXIVE · p. 13
• Researchers can support reflexivity through note-taking, recording learning, structured reflection, critique, and critical feedback.Source: REFLEXIVE · p. 13

ABUNDANT

• Abundance creates opportunities to uncover, relate, understand, and justify meaningful insights.Source: ABUNDANT · p. 13
• Suggested practices include diverse perspectives, explaining saturation, thick descriptions, sustained collaboration, co-design, and parallel prototyping.Source: ABUNDANT · p. 13

PLAUSIBLE

• Plausible knowledge claims are evidenced, appropriate, and persuasive.Source: PLAUSIBLE · p. 13
• Researchers can strengthen claims with memos, diverse analysis, explicit justifications, and crystallization across evidence.Source: PLAUSIBLE · p. 13

RESONANT

• Resonant research inspires understanding and invites action.Source: RESONANT · p. 13
• Researchers can foster resonance through situated context, engaging artifacts, evocative communication, narrated media, and discussion of transfer opportunities.Source: RESONANT · p. 13

TRANSPARENT

• Transparent reporting invites scrutiny.Source: TRANSPARENT · p. 13
• Suggested practices include comprehensive, self-critical reports, candid accounts of failures, supplementary materials, and an annotated evidence base with an audit trail.Source: TRANSPARENT · p. 13