Source-linked AI summary
Guidelines Are Not Rules: Characterizing Terminologies around Visualization Design Guidelines
Anna L. Chinni, Md Dilshadur Rahman, Bon Adriel Aseniero, Petra Isenberg, Kushin Mukherjee, Ghulam Jilani Quadri, Paul Rosen, Karen B. Schloss, Daniel Weiskopf
TL;DR
Visualization research often expresses actionable findings as guidelines, but the term is ambiguous and guidance terminology lacks shared meaning. The paper combines expert exploration, crowdsourced mapping and rating, and analysis of IEEE VIS publications to characterize this landscape. It finds persistent variation and concentrated use of a narrow vocabulary, motivating more nuanced and precise communication of research outcomes.
Problem
Guidance terminology in visualization is inconsistently understood, making it unclear how terms convey applicability, strictness, scope, and evidentiary basis.
Method
The paper combines expert discussions, crowdsourced mapping and rating studies, and corpus analysis of guidance terminology in published IEEE VIS research.
Results
Guidance terms vary across strictness, breadth, evidentiary basis, and theoretical versus practical orientation, while six terms account for 98.2% of verified guidance-adjacent term–paper pairs.
Takeaways & Limitations
The authors advocate more flexible, nuanced, and precise terminology to communicate guidance-related findings and their relative utility.
Takeaways & Limitations
The initial expert study used a small, non-representative workshop sample and researcher-defined rather than data-driven dimensions.
Abstract
from arXiv · showhide
A common expectation in visualization research is that outcomes recommend how researchers and practitioners take action or make design decisions. We often express these as "guidelines." Yet, the term "guideline" is both ambiguous and loosely defined, and what one researcher considers a guideline may be too broad, too loose, or too strict for another. We take a closer look at a broader set of terms that can express desirable results around visualization research, and untangle how these words are understood in the community in relation to other similar terms. We base our work on an exploratory study with experts, followed by a crowdsourcing study with a separate mapping phase (n=30) and rating phase (n=42) targeting input from the broader visualization community, and an analysis of the use of terminology in 3,877 IEEE VIS papers published from 1990 to 2024. Based on our findings, we call for more nuanced, precise discussions of research outcomes and their communication to the broader community, including practitioners and students.
1 INTRODUCTION
Visualization research increasingly expects findings to become generalizable, actionable guidance, yet guidance terminology is inconsistently understood and used. The paper examines this mismatch and argues for more precise communication of research outcomes.
- Researchers and reviewers increasingly expect visualization studies to produce quickly applicable, generalizable design guidance, even when guidelines are inappropriate.
- GUIDELINE ranges from a broad rule of thumb to a strict rule, reflecting disagreement about what the term means.
- Inconsistent terminology makes contributions difficult to compare when guidance rests on different evidentiary bases, such as author opinion versus validated empirical research.
- The authors combine expert discussions, crowdsourced studies, and literature analysis to investigate how guidance-related terms are understood and used.
- Guidance terms differ across strictness, breadth, evidentiary basis, and theoretical versus practical orientation, while GUIDELINE remains near the center of these dimensions.
- The paper calls for nuanced, precise terminology that supports communication with practitioners and students and helps articulate guidance for research, systems, and teaching.
2 RELATED WORK
Prior visualization and HCI work identifies tensions over how empirical knowledge becomes practical guidance, but it rarely defines or differentiates the terminology used for that guidance. This gap matters for organizing, searching, recommending, and theorizing about visualization contributions.
- Visualization spans differing views of rules and guidelines, including preferences for established codified knowledge and calls to account for tacit and situated knowledge.
- The field needs precise shared terminology because differing interpretations can create conflicts among readers, authors, reviewers, and practitioners.
- VisGuides and related initiatives established venues for discussing and organizing visualization guidelines, but uptake and discussion on the accompanying website remained limited.
- The Dagstuhl Seminar broadened this agenda through working groups on terminology, practitioners, competing value systems, and guidelines plus AI.
- Few papers define guidance terms; Chen et al. describe a guideline as actions that may lead to a desired outcome or prevent an undesired one, and a principle as a rule that must be followed.
- Guidance may draw on objective empirical observation or critical reflection on situated practice, depending on the research viewpoint.
- Prior HCI critiques warn that usability evaluations and empirical findings can be generalized into design recommendations beyond the supporting evidence.
- Existing work shows variation in guidance origin, evidence, and scope but does not distinguish terms suited to different types of visualization guidance.
3 STUDIES ON GUIDANCE TERMINOLOGY
The paper studies guidance terminology through an exploratory expert investigation, broader crowdsourced embedding studies, and a structured vocabulary assembled for community discussion. These stages progressively examine term meanings, dimensions, and possible sources.
- The authors first explored guidance terminology and potential sources at a Dagstuhl Seminar, then used formal embedding and dimensionality-reduction procedures with broader community data.
- The initial exploration sought a shared foundation for understanding how guidance terms vary in meaning and where different kinds of guidance may originate.
- Six senior authors and one contributor assembled alternative terms to capture differences in applicability, breadth, source, and strictness.
- The authors used dictionary definitions, AI assistance when dictionaries were inadequate, and collective author review to produce the term definitions.
- Researchers iteratively positioned terms by perceived similarity and difference to begin mapping their semantic space.
- Table 1 presents definitions for community consideration, distinguishing terms included in the initial exploration and indicating their likely sources from survey data.
DESIGN PATTERN
The paper situates DESIGN PATTERN within a multidimensional landscape of guidance terms, examining how terminology varies in strictness, coverage, evidentiary basis, and theoretical versus practical orientation. Its studies show that these distinctions can support more precise terminology choices for communicating visualization research.
- DESIGN PATTERN: DESIGN PATTERN is characterized as a structured, reusable solution to a recurring design problem grounded in assimilated knowledge and practice.
- Exploratory dimensions: The exploratory study organized guidance terms around strictness and coverage, ranging from loose to strict and broad to specific.Examples included GUARDRAIL near the strict endpoint, RULE OF THUMB near the loose endpoint, PRINCIPLE near the broad end, and STYLE GUIDE near the specific endpoint.
- Exploratory findings: GUIDELINES were judged near the center of strictness and coverage while also receiving ratings spread across the full range, unlike more consistently specific and strict terms such as STANDARD and NORM.
- Exploratory visualization: The visual map retained frequently selected strictness–coverage cells and used color to distinguish loose versus strict and specific versus broad terms.Darker cells indicate more respondents selecting a combination; connected cells preserve additional frequent ratings beyond each term’s highest-frequency location.
- Crowdsourced study: The crowdsourced study used triadic similarity judgments for mapping and bipolar-scale ratings for dimensional analysis, with 2,408 triplet trials collected for embedding estimation.
- Crowdsourced study: PCA indicated that four components accounted for 95% of embedding variance, aligning with strictness, evidentiary basis, coverage, and theoretical versus practical orientation.The rating phase confirmed PC1 as loose versus strict, PC2 as assimilated knowledge versus direct evidence, PC3 as specific versus broad, and PC4 as theoretical versus practical.
4 TERM USAGE IN VISUALIZATION LITERATURE
The corpus analysis shows that guidance-adjacent terminology is concentrated in a small vocabulary, but these labels serve different roles and do not consistently specify a contribution’s scope, evidence, or strictness.
- Corpus and method: The analysis classified guidance attributed to papers’ own findings, analyses, or design processes while excluding prior-work and unrelated uses.The corpus comprised 3,877 IEEE VIS papers published from 1990 to 2024, with occurrence-level classification and manual review of guidance-adjacent cases.
- Distribution of guidance-adjacent terms: 625 verified guidance-adjacent occurrences appeared in 319 papers, and 98.2% of 392 term–paper pairs involved six terms.The six terms were GUIDELINE, RECOMMENDATION, CONSIDERATION, IMPLICATION, PRINCIPLE, and DESIGN PATTERN.
- Distribution of guidance-adjacent terms: GUIDELINE, RECOMMENDATION, and DESIGN PATTERN appeared as guidance-adjacent uses in 16.7%, 14.6%, and 10.9% of papers, respectively, whereas common terms such as STANDARD and REQUIREMENT had no retained guidance-adjacent occurrences.Overall word frequency therefore did not indicate how often a term presented guidance from a paper’s own work.
- Temporal patterns: Each of the six frequent terms had a higher mean annual guidance-adjacent share in 2018–2024 than in 1990–2011.GUIDELINE rose from 1.0% to 6.9%, RECOMMENDATION from 0.5% to 6.3%, and CONSIDERATION from 0.3% to 6.4%; these averages do not establish a steady annual increase or explain the pattern.
- Term co-use: 252 of 319 papers, or 79.0%, used only one guidance-adjacent term, indicating that indiscriminate switching was not the main pattern.The most frequent pair was GUIDELINE and RECOMMENDATION, found in 23 papers; co-use does not imply synonymy.
- Term use and interpretation: GUIDELINE alone can denote evaluative criteria, tentative empirical advice, conditional prescriptions, or guidance based on multiple forms of evidence.The label does not by itself specify the outcome’s form, evidentiary basis, intended scope, or required strictness.
5 RECOMMENDATIONS
The paper recommends that authors, reviewers, researchers, and educators use guidance terminology deliberately and share a nuanced understanding of what different terms communicate.
- Authors: Authors should choose terminology that fits a contribution’s strictness, breadth, evidentiary basis, and theoretical or practical orientation.A design-space paper may support practice descriptions or recommendations without supporting strict requirements or even loose considerations when direct evidence is absent.
- Reviewers: Reviewers should avoid demanding stricter, broader, or more actionable terms than the paper’s evidence supports.Pushing conclusions too far can produce published messages that are potentially misleading or unreliable.
- Research community: The research community is asked to adopt shared, nuanced terminology to make outcomes more reliable and support cumulative knowledge building.Clearer terminology could help assemble and merge knowledge across surveys, textbooks, course notes, and future meta-studies.
- Educators: Educators should clearly articulate what kind of message terminology is intended to communicate to students and practitioners.The paper notes that communication with these audiences may involve less shared terminology and background knowledge than communication among researchers.
6 CONCLUSION
The paper uses multiple complementary studies and corpus analysis to clarify how visualization guidance terminology is interpreted and used, then advocates more precise communication while retaining flexibility.
- Conclusion: The study combines an expert exploratory study, crowdsourcing with separate mapping and rating phases, and analysis of visualization literature.Its main goal is to raise awareness of the guidance terminology landscape and illuminate community interpretation and use.
- Conclusion: The authors name their own Section 5 RECOMMENDATIONS, while acknowledging that other valid terms could have fit and that their term list is not comprehensive.They specifically identify RULES, CONSTRAINTS, and SUGGESTIONS as additional potentially useful terms.
- Conclusion: The paper advocates flexibility and a negotiated shared understanding of terminology to better communicate guidance-related findings.The authors frame this as accommodating the varied facets and nuances of visualization research outcomes.
AI USE STATEMENT
The paper reports AI assistance for illustrative artwork, occurrence labeling, editorial support, and code debugging, while stating that research content, findings, and conclusions were not AI-generated.
- AI use statement: AI tools assisted with Fig. 1 artwork, corpus-label assignment, manuscript editing, and experiment or figure-code debugging.The data plots were not AI-generated, and the stated assistance did not generate the research content, findings, or conclusions.
S.1 Terms and Definitions from the Initial Exploratory Study
The exploratory study assembled guidance-related terms, preliminary definitions, and likely sources to establish a shared terminology landscape.
- Table S.1 lists guidance-related terms with preliminary definitions and icons indicating their likely sources.The source indications were determined using survey data from the exploratory study.
- Principle is defined as a fundamental primary or general law or truth from which others are derived.
- Rule of Thumb is defined as a rough method of procedure based on experience and common sense.
- Style Guide is defined as a set of rules or guidelines that help maintain consistency in presentation.
- Guideline is defined as a suggestion or recommendation that steers actions or decisions without being a strict rule.
S.2 Example trials the Mapping and Rating Phases of the Crowdsourced Study
The crowdsourced study used example trials to illustrate its two distinct tasks: mapping term similarities and rating terms.
- Figure S.2 depicts an example triadic similarity judgment trial for the mapping phase and an example rating trial for the rating phase.Task instructions are described in Sections 3.2.2 and 3.2.3 of the main text.
S.3 Embeddings from the Mapping Phase of the Crowdsourced Study
The mapping-phase analysis embedded guidance terms in a multidimensional space and evaluated candidate dimensionalities before applying PCA to identify its structure.
- The four principal components represented loose versus strict, assimilated knowledge versus direct evidence, specific versus broad, and theory versus practical.Supplementary figures plot the guidance terms across each pairwise combination of these components.
- The analysis fit embeddings at one through five dimensions and evaluated how well each solution predicted held-out similarity judgments.The five-dimension solution was the highest dimensionality for which a stable embedding could be estimated.
- The model used majority responses from triplet combinations as human-response targets and compared embedding-space Euclidean distances with held-out judgments.Agreement across participants with the majority response provided a baseline for human-response variability.
- All embedding solutions performed similarly on held-out human judgments, while the four-dimensional model achieved the lowest final loss.Adding a fifth dimension did not improve performance.
- PCA on the five-dimension solution showed that the first four components explained 95% of the embedding variance.The fifth component contributed relatively little additional variance, supporting a likely four-dimensional terminology space.