PaperlayerView exact revision on arXiv

Source-linked AI summary

Mini-VLAT: A Short and Effective Measure of Visualization Literacy

Saugat Pandey, Alvitta Ottley

arXiv:2304.07905v2cs.HC

TL;DR

The 53-item VLAT takes about 22 minutes, limiting its use primarily to academic settings. The paper evaluates Mini-VLAT through test-development and validation procedures, reporting reliability, VLAT correlation, and association with unfamiliar-visualization ability.

  • Problem

    The 53-item VLAT takes about 22 minutes, and its time commitment limits its use primarily to academic settings.

    Source: §1 · p. 1

  • Method

    The authors developed and evaluated Mini-VLAT using item generation, reliability assessment, expert content validation, VLAT correlation, and predictive-capacity testing.

    Source: §1 · p. 2, §4.1.2 · p. 5

  • Results

    Coefficient omega = 0.72 and average content validity ratio = 0.6; Mini-VLAT strongly correlated with VLAT and was positively associated with ability to read and understand unfamiliar visualizations.

    Source: §6.1 · p. 8

  • Takeaways & Limitations

    Mini-VLAT is presented as a practical, time-effective substitute for VLAT for assessing reading and interpretation, one aspect of multidimensional visualization literacy.

    Source: §6.1 · p. 8, §6.2 · p. 8

    Mini-VLAT covers 12 visualization types drawn from VLAT and excludes other common representations, including icon arrays, radar charts, and heat maps.

    Source: §8 · p. 9

Abstract

from arXiv · show

The visualization community regards visualization literacy as a necessary skill. Yet, despite the recent increase in research into visualization literacy by the education and visualization communities, we lack practical and time-effective instruments for the widespread measurements of people's comprehension and interpretation of visual designs. We present Mini-VLAT, a brief but practical visualization literacy test. The Mini-VLAT is a 12-item short form of the 53-item Visualization Literacy Assessment Test (VLAT). The Mini-VLAT is reliable (coefficient omega = 0.72) and strongly correlates with the VLAT. Five visualization experts validated the Mini-VLAT items, yielding an average content validity ratio (CVR) of 0.6. We further validate Mini-VLAT by demonstrating a strong positive correlation between study participants' Mini-VLAT scores and their aptitude for learning an unfamiliar visualization using a Parallel Coordinate Plot test. Overall, the Mini-VLAT items showed a similar pattern of validity and reliability as the 53-item VLAT. The results show that Mini-VLAT is a psychometrically sound and practical short measure of visualization literacy.

1. INTRODUCTION

The paper introduces Mini-VLAT as a shorter measure for a field that lacks a validated, widely deployable test. It reports that the measure is reliable and strongly correlated with VLAT.

1. INTRODUCTION

  • Visualization is pervasive in daily life, while people’s capacity to interpret and use graphics varies; visualization literacy is crucial for understanding interactive visual designs [BHH∗21].The paper frames measuring visualization literacy as increasingly important as visualizations convey complex data.Source: §1 · p. 1
  • The 53-question VLAT takes about 22 minutes, and its time commitment limits its use primarily to academic settings.Each question has a 25-second limit; the authors note that the length may cause cognitive tiredness and erroneous answers for some respondents.Source: §1 · p. 1
  • Mini-VLAT comprises 12 multiple-choice questions across 12 visualization types and takes about five minutes to complete.The authors evaluated it through five phases, including item generation, refinement, reliability evaluation, VLAT correlation, and predictive-capacity testing.Source: §1 · p. 1
  • Mini-VLAT is described as psychometrically sound and reliable, with a strong correlation to VLAT and similar validity and reliability properties.The authors followed established test-development procedures and assessed reliability, VLAT correlation, and predictive capacity.Source: §1 · p. 2

2. RELATED WORK

Prior work has proposed differing definitions and assessment approaches for visualization literacy. This paper treats it as multidimensional and focuses on reading and interpreting visualized data, while positioning Mini-VLAT against earlier measures and the length of VLAT.

2.1. Definitions of Visualization Literacy

  • Definitions describe visualization literacy as translating between data and visual queries, interpreting visual patterns, or reading and extracting information from visualizations.The cited definitions arose from work on visualization literacy and the VLAT.Source: §2.1 · p. 2
  • Visualization literacy has no clear consensus definition; the authors treat it as multidimensional and focus on reading and interpreting visually represented data.They argue a single scale cannot measure its full scope and describe this focus as a necessary skill across proposed definitions.Source: §2.1 · p. 2

2.2. Measuring Visualization Literacy

  • Early measures assessed graph reading through three comprehension levels, while Boy et al. applied item response theory to six tasks and three chart types.The tasks were minimum, maximum, variation, intersection, average, and comparison.Source: §2.2 · p. 2
  • Börner et al. assessed familiarity with visual designs in science museums, and a later framework organized insight extraction into seven hierarchical typologies.The museum study selected 20 designs and showed five to visitors; the framework included definitions, exercises, and assessments.Source: §2.2 · p. 2
  • VLAT established a 53-item test across 12 visualizations, while later work developed literacy tests for parallel coordinates and treemaps.The VLAT was validated with five visualization experts; the specialized tests assess comprehension and interpretation of their respective designs.Source: §2.2 · p. 2
  • VLAT’s length constrains larger-scale research, motivating this study’s 12-item abbreviated test with reliability and validity comparable to VLAT.VLAT’s comprehensive range of designs and question types makes a shorter form less complete.Source: §2.2 · p. 2

3. WHY 12-ITEM Mini-VLAT?

The authors sought a shorter VLAT form suitable for non-expert settings without discarding visualization-type diversity. They chose one item per visualization type as a way to reduce length while aiming to preserve reliability and validity.

3. WHY 12-ITEM Mini-VLAT?

  • For research purposes, Lee et al. reduced the VLAT from 53 to 41 items and Mansoor et al. from 53 to 22, yet either test could still be too long [LKY∗19, MVH17].Lee et al. removed the 12 items with the lowest discriminating values; Mansoor et al. cited participants’ performance degradation over time due to fatigue in a pilot study [LKY∗19, MVH17].Source: §3 · pp. 2–3
  • To retain visualization-type diversity for non-expert settings, the authors chose to preserve the variety of designs rather than the variety of task types.They note that any short form would be less complete than VLAT.Source: §3 · p. 3
  • The authors hypothesized that carefully selecting one item from each visualization type could yield a reliable, valid test with substantially shorter completion time.They intended this approach to make the test suitable for widespread use.Source: §3 · p. 3

4. DEVELOPMENT OF THE Mini-VLAT

The paper develops Mini-VLAT as a short, practical measure intended for widespread use. Its five-phase process selects and refines items, then evaluates reliability and correlation with the full VLAT.

4. DEVELOPMENT OF THE Mini-VLAT

  • The study sought a short, valid measure for widespread use and developed Mini-VLAT through a five-phase process.The process included item selection, piloting and revision, reliability evaluation, correlation with VLAT, and an independent aptitude test.Source: §4 · p. 3

4.1.1. Replicating VLAT

  • Of 200 replication participants, one was excluded as a random clicker, and responses for most items overlapped the original study's responses.The researchers used the replicated data to select Mini-VLAT items.Source: §4.1.1 · p. 3

4.1.2. Expert Feedback

  • Five visualization experts rated item relevance on a three-point scale; CVR above zero indicated that more than half the panel agreed an item was relevant.The experts rated each VLAT item against its respective visualization type.Source: §4.1.2 · p. 5

4.1.3. Item Selection

  • For each visualization type, the team selected items with positive CVR and high item-total correlation, excluding some highest-correlation items with nonpositive CVR.The item-total correlation was chosen as the selection measure, while positive CVR was retained as a validity condition.Source: §4.1.3 · p. 5

4.2. Phase II: Piloting and Item Refinement

  • Eight pilot participants reported difficulty interpreting questions, prompting revisions and replacement of the politically sensitive party-approval data with Tokyo Olympics data.The pilot also surfaced questions that participants found too long or needed to reread.Source: §4.2 · pp. 5, 7

4.3. Phase III: Reliability Evaluation

  • Mini-VLAT had acceptable reliability, with coefficient omega = 0.72.The authors describe scores as adequately consistent and not unduly influenced by random error.Source: §4.3 · p. 7

4.4. Phase IV: Correlation Between Mini-VLAT & VLAT

  • r = 0.75 (n = 30, p < 0.001) was a strong positive correlation between Mini-VLAT and VLAT scores.Mini-VLAT completion averaged 3 minutes 45 seconds (SD = 45 seconds), versus 12 minutes 2 seconds (SD = 2 minutes 48 seconds) for VLAT.Source: §4.4 · p. 7

5. PHASE V: TESTING Mini-VLAT’S PREDICTIVE CAPACITY

The study tested whether Mini-VLAT scores are associated with participants’ ability to learn an unfamiliar visualization. Scores showed a strong positive correlation with performance on the Parallel Coordinate Test.

5. PHASE V: TESTING Mini-VLAT’S PREDICTIVE CAPACITY

  • The study used an independent assessment because Mini-VLAT and VLAT items overlap, and prior work linked VLAT scores to learning an unfamiliar visualization.The independent criterion was the Parallel Coordinate Test (P-Lite).Source: §5 · p. 7
  • Participants completed Mini-VLAT, viewed six static Parallel Coordinate Plot tutorials, then answered 14 P-Lite items; both sections were randomized.Thirty participants were recruited, three were excluded as random clickers, and the final sample was 27; none had prior familiarity with Parallel Coordinate Plots.Source: §5 · pp. 7–8
  • r = 0.62 (n = 27, p <0.001) showed a strong positive correlation between Mini-VLAT scores and Parallel Coordinate Test scores.The test assessed participants’ aptitude after tutorials on Parallel Coordinate Plots.Source: §5 · p. 8, Figure 2 · p. 8

6. DISCUSSION

Mini-VLAT is presented as a reliable, valid short-form measure focused on reading and interpreting visualizations. The discussion situates that scope within a field without consensus on visualization literacy and points toward multiple instruments for its different dimensions.

6. DISCUSSION

  • The discussion frames measuring visualization literacy as increasingly important while noting that visualization-literacy tools remain at an embryonic stage.Source: §6 · p. 8

6.1. Summary of Results

  • Mini-VLAT showed coefficient omega = 0.72, average content validity ratio = 0.6, and strong correlation with VLAT, alongside a strong positive association with ability to read unfamiliar visualizations.The authors describe it as a reliable and valid 12-item short form of VLAT.Source: §6.1 · p. 8

6.2. Toward Measuring Other Dimensions of Visualization Literacy

  • The authors argue that visualization literacy is multidimensional and that this study assessed users’ ability to read and interpret data visualizations.They anticipate multiple instruments or exams for different aspects of visualization literacy.Source: §6.2 · p. 8, §6.1 · p. 8

7. FUTURE WORK

The authors identify broader-scale and longitudinal study of visualization literacy as future work. They also propose using Mini-VLAT in underrepresented populations and adapting it for other languages, with translation validation by native speakers.

7. FUTURE WORK

  • Longitudinal studies could track the public’s ability to comprehend and interpret visualizations over time, and Mini-VLAT could assist broader-scale research.Source: §7 · p. 9

7.1. Visualization Literacy and Inclusivity

  • Mini-VLAT could help measure visualization literacy in populations neglected by prior studies and could be adapted to other languages.The authors caution that machine translations may be inaccurate, so native speakers should validate translated questionnaires.Source: §7.1 · p. 9

8. LIMITATION

The authors identify limits in both the range of visualization types covered and the scope of literacy that a single score can represent. They also note the trade-off between a short test and the breadth or depth of assessment.

8. LIMITATION

  • Mini-VLAT covers 12 VLAT-derived visualization types, excluding other common representations such as icon arrays, radar charts, and heat maps.The authors say the assessment’s range should be expanded.Source: §8 · p. 9
  • Visualization literacy may be too complex to quantify with one score, while taxonomy-based assessment may require expert intervention and be unsuitable for large-scale experiments.The authors suggest tools covering individual stages of the cognitive taxonomy as one possible solution.Source: §8 · p. 9
  • The authors acknowledge that Mini-VLAT’s length and item count are discretionary, and recommend weighing abbreviated-form advantages and disadvantages from participants’ perspective.They describe the 12-item form’s reliability and validity as almost as strong as those of the 53-item VLAT.Source: §8 · p. 9

9. CONCLUSION

The authors developed Mini-VLAT as an efficient, validated measure of visualization literacy and demonstrated its predictive capacity for learning unfamiliar visualizations. They argue it could support broader-scale assessment and show that shortening a test can maintain acceptable internal consistency and validity.

9. CONCLUSION

  • Mini-VLAT measures visualization literacy, and its predictive capacity was demonstrated for aptitude in learning unfamiliar visualizations.Source: §9 · p. 9
  • The authors demonstrated that a test can be redesigned or shortened while maintaining acceptable internal consistency and validity for measuring visualization literacy.Source: §9 · p. 9
  • The authors believe Mini-VLAT could help evaluate visualization literacy on a broader scale, and they intend to study diverse populations.Source: §9 · p. 9
Loading 2304.07905v2…