Source-linked AI summary
Kori: Interactive Synthesis of Text and Charts in Data Documents
Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, Nam Wook Kim
TL;DR
Readers must connect information across spatially separated text and charts, a task that requires shifting attention. Kori analyzes text–chart references and offers a mixed-initiative interface for authoring them. Its evaluation suggests effective authoring support, while automatic detection remains less reliable for interval and grouped references than for point references.
Problem
Spatially separated text and charts make it challenging for readers to synthesize information across the two media.
Source: §1 · p. 1Method
Kori combines an analysis of text–chart references with a mixed-initiative interface that suggests links automatically and supports manual construction.
Source: §1 · p. 2Results
Point-reference detection achieved F1 = 0.47, compared with F1 = 0.25 for interval detection and F1 = 0.26 for reference grouping; a user study also suggested the interface was intuitive and easy to use.
Source: §1 · p. 2, §7.1.2 · p. 7Takeaways & Limitations
Kori offers authors a flexible way to create references across text and charts, with manual construction complementing automatic suggestions.
Source: §1 · p. 2, §9 · p. 9The small dataset led the system to rely on separate heuristics rather than a joint model, motivating exploration of an end-to-end approach.
Source: §8.1 · p. 9
Abstract
from arXiv · showhide
Charts go hand in hand with text to communicate complex data and are widely adopted in news articles, online blogs, and academic papers. They provide graphical summaries of the data, while text explains the message and context. However, synthesizing information across text and charts is difficult; it requires readers to frequently shift their attention. We investigated ways to support the tight coupling of text and charts in data documents. To understand their interplay, we analyzed the design space of chart-text references through news articles and scientific papers. Informed by the analysis, we developed a mixed-initiative interface enabling users to construct interactive references between text and charts. It leverages natural language processing to automatically suggest references as well as allows users to manually construct other references effortlessly. A user study complemented with algorithmic evaluation of the system suggests that the interface provides an effective way to compose interactive data documents.
1 INTRODUCTION
Charts provide graphical summaries while text explains data stories, but readers must shift attention between them to connect information. Kori combines analysis of text–chart references with automatic suggestions and manual authoring tools.
1 INTRODUCTION
- Charts engage audiences and represent data perceptually, while text guides readers and explains additional context.The paper presents the two media as complementary parts of data stories.Source: §1 · p. 1
- Spatial separation between text and charts makes it challenging to synthesize their information and requires readers to shift attention back and forth.The paper describes this as a cognitive burden on working memory that can negatively affect learning.Source: §1 · p. 1
- Text–chart references resemble chart selections: phrases can form hierarchical structures whose marks combine through unions or intersections.The analysis extended prior news-media examples to additional chart types and scientific journals.Source: §1 · p. 2
- Kori suggests text–chart references automatically and lets authors accept, reject, or trigger suggestions, as well as construct references manually.Natural language processing derives point- and interval-level references, then groups semantically related references using parsed dependencies.Source: §1 · p. 2
- An evaluation combining algorithmic assessment and a study with eleven participants suggests Kori offers an intuitive, easy-to-use interface for authoring interactive data-driven documents.Participants reproduced and created references, then completed a usability survey and reflection interview.Source: §1 · p. 2
2 RELATED WORK
Prior work studies the value of combining text and visualizations, tools for authoring explanatory visualizations, and natural-language approaches to linking text with visual content.
2.1 Symbiotic Relationship of Text and Visualization
- Experiments on text–visualization linking found improved recall but not comprehension in one study, and improved comprehension for low-literacy participants with gaze-driven linking in another.A Bayesian-reasoning study found that people tend not to consolidate information well across text and visualizations.Source: §2.1 · p. 2
2.2 Systems for Authoring Explanatory Visualizations
- Existing storytelling tools support textual descriptions, annotations, customized marks, and narrative formats, but generally do not support interactive synthesis of text and visualizations.Authors otherwise rely on programming frameworks, while some publishing tools offer only simple figure-reference links; the paper targets accessible authoring for users without programming expertise.Source: §2.2 · p. 2
2.3 Natural Language Understanding in Visualization
- Prior NLP-based linking approaches use keyword or template matching for limited chart types and point references, whereas Kori also targets interval references and syntactic groupings.Kori focuses on references between existing text and charts in a communication context.Source: §2.3 · p. 2
3 UNDERSTANDING TEXT–CHART REFERENCES
The analysis of news and scientific text–chart pairs characterizes references as selections that can be grouped hierarchically. It identifies ambiguity, data transformations, and varied visual-analysis tasks as important features of those references.
3.1 Data Collection and Analysis
- The researchers expanded an existing news collection to 77 articles and 110 paragraph–chart pairs, then analyzed 227 sentence–chart pairs with manually constructed minimal references and open coding.The collection included additional chart types and sources, including scientific journals.Source: §3.1 · pp. 2–3
3.2 Analysis Results
- Of 676 minimal references, 366 were point, 294 multi-point, and 16 interval references; axes, visual marks, and legends were the most-referenced chart features.The referenced data values were mostly categorical (407), with numerical (244) and temporal (25) values also represented.Source: §3.2 · p. 3
- Minimal references can be grouped into higher-order references in hierarchies of up to four levels, with the sentence as root; grouping multiple references yields unions or intersections of data-point selections.The collection contained 175 first-level and 28 second-level ancestor references.Source: §3.2 · p. 3
- References correspond to visual-analysis tasks: sentence-level references included 149 identification, 49 comparison, and 29 summarization tasks.Higher-level references included comparison and summarization tasks in addition to identification.Source: §3.2 · p. 4
- Reference phrases can be ambiguous or require transformations: 311 matched chart labels exactly, while others involved partial matches, inference, synonyms, approximate ranges, or derived measures.Derived measures such as means or percentages may require computation from underlying data, and charts often do not annotate them.Source: §3.2 · p. 4
3.3 Limitations
- The sample remains limited in size and representativeness and excludes advanced coordinated multiple visualizations; the qualitative analysis could benefit from computational linguistic methods.The authors leave computational linguistic analysis for future work.Source: §3.3 · p. 4
4 DESIGN CONSIDERATIONS
Kori’s design balances automated assistance with user control. Suggestions remain optional, users can create additional or combined references, and assistance should not distract from authoring.
4 DESIGN CONSIDERATIONS
- Kori suggests potential references rather than creating them automatically, because detection can produce false positives and users may not want every suggested link.Users can accept or reject suggestions.Source: §4 · p. 4
- Users can construct references beyond suggestions and combine suggested references into higher-level links through smooth, intuitive interactions across chart types.Construction relates selected visual marks to text.Source: §4 · p. 4
- Reference assistance should blend into authoring without distracting users, becoming useful during revision and potentially inspiring deeper data communication.Source: §4 · p. 4
5 KORI USAGE
Kori’s editor lets authors add charts and compose interactive text–chart references through automatic suggestions and direct manipulation. Authors can preview, accept, or create references while writing.
5 KORI USAGE
- Kori’s editor separates a chart gallery from a text-editing area, where authors insert charts and compose their article.Source: §5 · p. 4
- While authors type, Kori suggests chart references such as New York, which they can preview by hovering to highlight the corresponding line.Source: §5 · p. 4
- Authors can type “@” to request suggestions, or select text and construct a reference by choosing a chart and filtering an interval.Source: §5 · p. 4
- Authors can directly select multiple chart marks to create a reference that compares timelines for several states.Source: §5 · p. 4, Figure 4 · p. 5
- Authors can preview a suggestion for states with more than 10,000 deaths, accept it, and adjust the opacity of inactive map regions.Source: §5 · p. 4
6 THE KORI SYSTEM
Kori is a web-based system for authoring interactive text–chart references. It supports multiple chart types and combines automatic detection with manual construction, while its reference detection and chart support have defined limitations.
6 THE KORI SYSTEM
6.1 Charts
- Kori supports bar, line, scatter, distribution, heat-map, and choropleth charts, but not advanced visualizations, coordinated views, or interactive charts.Source: §6.1 · p. 5
6.2 Reference Detection
- Text–chart references explicitly link a text phrase to corresponding chart marks, and can target points, multiple points, or intervals.Source: §6.2 · p. 5
- Detection extracts chart encodings and matches chart features to text n-grams using FastText, with a similarity threshold above 0.5 and n-grams up to size five.Source: §6.2 · pp. 5–6
- Rules identify numerical intervals from words and part-of-speech patterns, including forms such as “between X and Y” and “at least X.”Source: §6.2 · p. 6
- The method maps intervals to axis references by minimum dependency-tree distance, but in a failure case paired price with 2006–2008 (distance 2) rather than over $400 (distance 3).Source: §6.2 · p. 6
6.3 Reference Construction
- Authors can request suggestions with “@” or text selection; chart thumbnails and autocomplete help them choose a feature.Source: §6.3 · p. 6
- Direct manipulation selects chart marks, while filtering sets categorical or numerical dimension values; combining filters creates higher-level references through intersection.Source: §6.3 · p. 6
7 EVALUATION
The evaluation tested Kori’s automatic reference detection and conducted a user study.
7.1 Algorithmic Evaluation: Reference Detection
The evaluation curated a 92-pair chart–text dataset and tested automatic reference detection. Performance varied by reference type, and the discussion identifies matching and grouping challenges.
7.1.1 Dataset Curation
- The curated dataset contained 92 chart–text pairs across 15 chart types, split into 55 validation pairs and 37 test pairs.Source: §7.1.1 · p. 7
7.1.2 Results
- On validation data, a 0.5 similarity threshold and maximum n-gram size of 3 yielded the highest F1 score, 0.34.Source: §7.1.2 · p. 7
- 57 of 137 true references were identified, with 90 false positives and 93 false negatives; point references reached F1 0.47, versus 0.25 for intervals and 0.26 for grouping.Source: §7.1.2 · p. 7
- On Kong et al.’s 42 pairs, Kori’s average distance (1−F1) was 0.57 versus 0.39 for their approach, which refined user-extracted base references.Source: §7.1.2 · p. 7
7.1.3 Discussion
- Failures include contextually incorrect FastText matches, number-form mismatches, undetected month and day intervals, and grouping errors that chart extents, scales, and units might help avoid.Source: §7.1.3 · p. 7
7.2 User Study: Authoring
The study examined how people author interactive text–chart references with Kori, using three tasks and a usability survey. Participants completed the tasks, created both suggested and manual references, and rated the system positively while identifying limits in suggestion quality and chart exploration.
7.2.1 Participants
- Eleven participants with varied backgrounds took part; all had experience with document-editing tools and chart creation, and eight had used charts in word processors.Participants included students, visualization experts, and a user-interface designer.Source: §7.2.1 · p. 7
7.2.2 Procedure and Tasks
- Participants replicated a tutorial example, converted nine references across a scatterplot and heatmap, then created a 5–6-sentence story from charts.Sessions ended with a five-point usability survey and semi-structured interviews.Source: §7.2.2 · p. 8
7.2.3 Results
- All participants completed all three tasks; Kori suggested six references in the first two tasks and 64 in the story task.Participants completed the first two tasks with minimal experimenter intervention.Source: §7.2.3 · p. 8
- 48 of 64 suggested references were correct, 16 were incorrect, and participants manually created 40 references.The 64 suggestions included 25 explicitly triggered suggestions; the 48 correct references included all 25 triggered suggestions, which were point references.Source: §7.2.3 · p. 8
- Participants rated overall experience at median and mode 4 (IQR 0), and usefulness at median and mode 5 (IQR 0).They rated reference construction higher than automatic suggestions, with both reported at median and mode 4; IQR was 1 for construction and 0 for suggestions.Source: §7.2.3 · p. 8
- Participants valued flexible manual linking, but reported that suggestions were not smart enough, could be delayed or wrong, and could not be grouped through the interface.One participant also could not brush the chart’s color legend; direct manipulation was limited to marks inside the chart area.Source: §7.2.3 · p. 8
- Some participants used references to explore data and verify assumptions, while others noted that pre-made, non-modifiable charts limited investigative reporting.Participants also identified missing convenience features, including editing or deleting references and grouping suggestions.Source: §7.2.3 · pp. 8–9
8 DISCUSSION
The discussion identifies constraints in Kori’s linking and evaluation scope, while describing opportunities to connect authoring with data exploration and reduce the burden of synthesizing text and charts. It also considers additional automation and spatial or audio approaches to external cognition.
8.1 Limitations and Opportunities
- Automatic suggestions do not resolve which chart a user intends to link, especially when charts share a dataset, and Kori supports only one-to-one text-to-chart links.The authors leave the desirability of one-to-many or many-to-many links open, noting possible interface complexity and limited or adverse reading benefits.Source: §8.1 · p. 9
- Kori’s user study evaluated authoring rather than reading, and its small dataset led the suggestion pipeline to rely on heuristics rather than a joint model.The authors propose collecting substantial annotated examples to explore end-to-end learning and more contextualized embeddings.Source: §8.1 · p. 9
- The authors suggest adapting text-generation methods to propose prompts about interesting data facts during chart-to-text authoring.They identify data description and table-to-text generation as possible sources for this future direction.Source: §8.1 · p. 9
8.2 Closing the Loop from Explanation to Exploration
- Participants used references to inspect data while writing, suggesting a need for Kori to support exploration as well as prepared-chart storytelling.Kori assumes users have already explored the data and prepared charts; the study instead supplied pre-made charts, and exploration was frequent with dense, unlabeled scatterplots.Source: §8.2 · p. 9
8.3 Improving External Cognition in Data-driven Articles
- Interactive highlighting addresses split attention through temporal contiguity, while spatially integrating word-sized graphics or adding audio are discussed as alternatives for reducing distance between information modes.The discussion frames multimodal cognitive burden as an avenue for future research.Source: §8.3 · p. 9
9 CONCLUSION
Kori is a mixed-initiative system for creating text–chart references, informed by analysis of references in existing articles. Its evaluation indicates benefits for authoring across datasets and visualizations, while the conclusion identifies reliability and broader linking as future work.
9 CONCLUSION
- Kori supports authors in creating references between text and charts through mixed-initiative interaction.The system was informed by analysis of text–chart references in existing articles.Source: §9 · p. 9
- Findings about point and interval references and their grouping informed the automatic suggestion pipeline, while flexible manual construction complements it.The authors report algorithmic evaluation and a user study as demonstrating benefits for creating references across different datasets and visualizations.Source: §9 · p. 9
- Future work includes improving suggestion reliability and accuracy, extending linking beyond unidirectional text-to-chart references, and evaluating configurations that combine manual and automated linking.The authors also identify adapting the approach to data comics as a future direction.Source: §9 · p. 9