Source-linked AI summary
Natural Language Generation enhances human decision-making with uncertain information
Dimitra Gkatzia, Oliver Lemon, Verena Rieser
TL;DR
People may struggle to understand numerical uncertainty, while prior text-quality evaluations did not measure its effects on decision-making. This paper compares textual, graphical, and combined weather forecasts in a task-based game, finding better scores with NLG and combined presentations than with graphics alone.
Problem
People may not understand or act on numerical uncertainty, and earlier text-quality evaluations did not measure effects on comprehension or decision-making.
Source: §1 · p. 1Method
Results
NLG representations yielded a 24.8% increase in average task score versus Graphics-only, while multimodal presentations produced a 44% average game-score increase versus Graphics-only.
Source: §6.1 · p. 4Takeaways & Limitations
The effects of representation vary across user groups, so user adaptation is necessary when generating multimodal presentations of uncertain information.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Decision-making is often dependent on uncertain data, e.g. data associated with confidence scores or probabilities. We present a comparison of different information presentations for uncertain data and, for the first time, measure their effects on human decision-making. We show that the use of Natural Language Generation (NLG) improves decision-making under uncertainty, compared to state-of-the-art graphical-based representation methods. In a task-based study with 442 adults, we found that presentations using NLG lead to 24% better decision-making on average than the graphical presentations, and to 44% better decision-making when NLG is combined with graphics. We also show that women achieve significantly better results when presented with NLG output (an 87% increase on average compared to graphical presentations).
1 Introduction
This paper investigates how NLG can communicate uncertain data to support human decisions, addressing data-to-text systems’ assumption of precise inputs and the limits of numerical risk communication. It compares NLG strategies with graphical presentations and introduces an evaluation focused on decision-making and user groups.
1 Introduction
- About 30% of participants in a German-American study could not identify which of 1 in 100, 1 in 1000, or 1 in 10 represented the biggest disease risk.The cited study illustrates the difficulty of acting on numerical uncertainty.Source: §1 · p. 1
- The paper investigates whether NLG can communicate uncertain inputs in ways that support human decision-making, unlike data-to-text systems that assume precise data.It uses weather forecasts as an exemplar task.Source: §1 · p. 1
- Prior weather-forecast evaluations assessed text quality but did not measure effects on people’s understanding of uncertainty or decision-making.Earlier approaches also had not focused on communicating uncertain information or referring to meteorological probabilities.Source: §1 · pp. 1–2
- The paper contributes a principled uncertainty-to-language study, a game-based evaluation of confidence and task success, and evidence that representation effects vary across user groups.The authors state that user adaptation is necessary for multimodal presentations of uncertain information.Source: §1 · p. 2
2 The Extended Weather Game
The Extended Weather Game asks participants to choose a destination for an ice-cream vendor using uncertain weather forecasts. It presents forecasts in graphical, textual, or combined form and scores decisions, confidence, and risk literacy.
2 The Extended Weather Game
- Players choose where to send an ice-cream vendor to maximize sales using four weeks of forecasts for two locations.The forecasts concern predicted rainfall and temperature levels.Source: §2 · p. 2
- Table 1 presents the WMO-based mapping of likelihoods.Source: Table 1 · p. 2
- Forecasts and their likelihoods appear as graphics, text, or combined graphical and textual presentations.The graphical presentation is the original game version; the text is generated by two rule-based NLG approaches.Source: §2 · p. 2
- The game scores decisions and stated confidence, presents monetary gains based on outcomes, and assesses risk literacy using the Berlin Numeracy Test.Rain occurrence is estimated by sampling the probability distribution.Source: §2 · p. 2
3 Natural Language Generation from Uncertain Information
The paper develops two NLG systems to describe uncertain rainfall and temperature forecasts. One follows WMO uncertainty-reporting guidelines; the other imitates expert weather forecasters.
3 Natural Language Generation from Uncertain Information
- Both WMO-based and NATURAL generate textual descriptions of rainfall and temperature data that address forecast uncertainty.Both systems use SimpleNLG.Source: §3 · p. 2
- WMO-based is a rule-based system that applies WMO guidelines for reporting uncertainty.For a forecast of sunny intervals with a 30% probability of rain, it says rain is possible and less likely than not.Source: §3 · p. 2
- NATURAL imitates forecasters’ natural reporting, with rules derived by observing experts such as BBC weather reporters.For sunny intervals with a 30% probability of rain, it generates “Mainly dry with sunny spells.”Source: §3 · pp. 2–3
4 Evaluation
The evaluation compares graphical, text-only, and combined presentations in the Extended Weather Game. Its conditions vary whether forecasts are shown as graphics, NATURAL text, or WMO-based text, alone or paired with graphics.
4 Evaluation
- The evaluation uses five conditions to investigate how presentation formats support understanding and action under uncertainty.Source: §4 · p. 3
- The graphics-only condition presents weather forecasts using the graphs that scored best for human comprehension in prior work.Source: §4 · p. 3
- Multimodal conditions pair those graphics with text produced by either NATURAL or the WMO-based system.Source: §4 · p. 3
- Text-only conditions present forecasts from either NATURAL or the WMO-based system without graphics.Source: §4 · p. 3
5 Data
The study recruited 442 unique players and collected 450 game instances; the anonymised data will be released with the submission.
5 Data
- 442 unique players were recruited through social media: 197 females, 241 males, and 4 non-disclosed.Women made up 44.5% of the subjects.Source: §5 · p. 3
- 450 unique game instances were collected, as a few people played twice.Source: §5 · p. 3
- The anonymised data will be released as part of the submission.Source: §5 · p. 3
6 Results
Decision scores and confidence were analyzed across representations, with results differing between women and men. Multimodal output outperformed graphics-only overall, while women showed a substantial NLG-associated score increase.
6 Results
- The analysis used monetary gain as the game score and also examined rainfall predictions against confidence ratings on a 10-point scale.Source: §6 · p. 3
6.1 Results for all adults
- 44% average game-score increase: multimodal representations significantly outperformed graphics-only (p = 0.03, effect = +36.36), while NLG-only averaged 24.8% above graphics-only.There was no significant game-score difference between NLG-only and multimodal representations, or between the WMO and NATURAL NLG conditions.Source: Table 2 · p. 4, §6.1 · p. 4
6.1 Results for all adults; 6.2 Results for Females
- Multimodal presentations increased confidence over NLG-only for adults (p < 0.01, effect = 17.7%) and educated, risk-literate females (p = 0.01, effect = 16.7%).Among females, multimodal output added little decision-making effectiveness over NLG-only but enhanced confidence by 15%.Source: §6.1 · p. 4, §6.2 · p. 4
6.1 Results for all adults
- Prior risk-decision experience, weather-model familiarity, and correct literacy-test answers predicted understanding of uncertainty, reflected in confidence and game scores.Education level, gender, and native English-speaker status did not contribute to confidence or game scores in the all-adult analysis.Source: §6.1 · p. 4
6.2 Results for Females
- 87% increase: females scored significantly higher with either NLG output than with graphics-only (p < 0.05, effect = +53.03).The multimodal presentation also significantly outperformed graphics-only for females (p = 0.05, effect = 60.74%).Source: Table 3 · p. 4, §6.2 · p. 4
6.3 Results for Males
- Males obtained similar game scores across all representation types.Source: §6.3 · p. 4
7 Conclusions and Future Work
In a game-based study of uncertain-data descriptions, multimodal presentations improved adult scores over graphics, while NLG yielded a substantial score increase for women. The paper identifies the interplay of uncertainty, risk-taking, and gender as a future research question.
7 Conclusions and Future Work
- Multimodal presentations led to a 44% increase for all adults over graphics, while women’s task success increased 87% with NLG over graphics.People were also more confident with multimodal representations.Source: §7 · p. 4
- The study reports gender differences across multimodal, NLG, and graphical versions, and notes that prior representation comparisons had not distinguished gender.Source: §7 · p. 4
- The interplay between uncertainty, risk-taking behaviour, and gender is identified as a future research question.Source: §7 · p. 5