Source-linked AI summary
Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO
Benjamin Icard, Vincent Claveau, Ghislain Atemezing, Paul Égré
TL;DR
Determining whether a statement is factual or opinionated without first-hand knowledge matters for assessing information quality, but the linguistic cues behind such judgments are not fully accounted for. The paper combines symbolic VAGO with a neural clone trained on VAGO scores. Satirical French press articles score higher in vagueness and subjectivity and lower in detail than regular-press articles.
Problem
Judging factuality versus opinion without first-hand knowledge matters for information quality, while prior public judgments did not explain the linguistic cues that convey objectivity or subjectivity.
Source: §I · p. 1Method
Results
p = 4.99 × 10^-11 for vagueness; satirical articles were also more subjective (p = 1.69 × 10^-9) and less detailed (p = 3.36 × 10^-22) than regular-press articles.
Source: D. French VAGO on regular versus… · p. 4Takeaways & Limitations
LIME’s influential terms largely correspond to VAGO lexicon items or likely additions, supporting VAGO-N’s use for lexicon enrichment and neural versions in other languages.
Source: §V · p. 6
Abstract
from arXiv · showhide
We present a hybrid approach to the automated measurement of vagueness and subjectivity in texts. We first introduce the expert system VAGO, we illustrate it on a small benchmark of fact vs. opinion sentences, and then test it on the larger French press corpus FreSaDa to confirm the higher prevalence of subjective markers in satirical vs. regular texts. We then build a neural clone of VAGO, based on a BERT-like architecture, trained on the symbolic VAGO scores obtained on FreSaDa. Using explainability tools (LIME), we show the interest of this neural version for the enrichment of the lexicons of the symbolic version, and for the production of versions in other languages.
I. INTRODUCTION
The paper frames factuality judgments as a question of linguistic cues and introduces VAGO to measure vagueness and subjectivity. It also distinguishes vagueness types and adds a detail measure to better capture informativeness.
I. INTRODUCTION
- In a 2018 survey, only 26% identified all factual statements and 35% all opinion statements, while the study did not explain which linguistic cues convey objectivity or subjectivity.Source: §I · p. 1
- VAGO is introduced as a symbolic tool for measuring vagueness versus precision and subjectivity versus objectivity in texts.Source: §I · p. 1
- Prior work reported that French opinion articles more often contain certain punctuation markers, while factual articles contain more named entities.Source: §I · p. 1
- The approach combines rule-based VAGO with neural VAGO-N, using multidimensional vague adjectives as reliable markers of subjectivity and opinion.Source: §I · p. 1
A. Typology of vagueness and measures of detail
- VAGO distinguishes approximation, generality, degree, and combinatorial vagueness; approximation and generality are treated as factual, while degree and combinatorial vagueness are subjective.Examples include “environ” for approximation and “beau” for multidimensional gradable adjectives.Source: A. Typology of vagueness and measures… · pp. 1–2
- The original VAGO marks a sentence vague or subjective if it contains any corresponding marker, but its precision measure is only the absence of vague markers.The paper adds a relative detail score based on named entities compared with vague expressions, distinguishing sentences with different informativeness.Source: A. Typology of vagueness and measures… · p. 2
B. Scores: Vagueness, Subjectivity, Detail
VAGO uses a French-and-English lexical database and named-entity detection to calculate sentence scores for vagueness, subjectivity, and detail. Text-level vagueness is based on sentences with nonzero scores, while text-level detail averages sentence ratios.
B. Scores: Vagueness, Subjectivity, Detail
- VAGO’s French-and-English lexicon contained 1,640 terms, and named entities were identified with spaCy to measure detail.The lexicon was divided into approximation, generality, degree, and combinatorial vagueness categories.Source: B. Scores: Vagueness, Subjectivity,… · p. 2
- A sentence’s vagueness score is its vague-word count divided by its total word count, with subjectivity based on subjective vague expressions and factual vagueness on approximation and generality expressions.Source: B. Scores: Vagueness, Subjectivity,… · p. 2
- The detail score counts named entities relative to sentence length, while detail/vagueness measures named entities relative to vague terms.The examples yield detail/vagueness scores of 9/10 and 1/3; text-level detail/vagueness averages sentence ratios.Source: B. Scores: Vagueness, Subjectivity,… · p. 2
- The online interface displays vagueness and subjectivity as separate text-level barometers.Source: B. Scores: Vagueness, Subjectivity,… · p. 2
C. English VAGO on factual versus opinion statements
On the 10-sentence PRC benchmark, VAGO labeled eight statements as opinions and two as facts, matching PRC on the two facts and all five PRC-labeled opinions. Its detail/vagueness ratio also distinguishes statements that receive the same subjectivity score, while unhandled superlatives explain some disagreements.
C. English VAGO on factual versus opinion statements
- The benchmark comprises 10 sentences—five labeled factual and five opinion—whose PRC classifications were supported by participant majorities.Mode agreement ranged from 54% to 77% for factual statements and 68% to 80% for opinion statements; PRC tested 5,035 participants.Source: C. English VAGO on factual versus… · p. 3
- VAGO classified eight of ten statements as opinions and two as facts, matching PRC on both facts and all five PRC-labeled opinions.This indicates a stricter criterion for factual statements and a laxer one for opinions in this sample.Source: C. English VAGO on factual versus… · p. 3, Figure 1 · p. 3
- For statements 4 and 9, equal subjectivity scores of 1 coexist with detail/vagueness ratios of 4/5 and 1/3, respectively.The ratio distinguishes their informativeness despite identical subjectivity scores.Source: C. English VAGO on factual versus… · p. 3
- Four of five PRC-labeled opinions were identified by VAGO through VC expressions such as “great” and “wasteful,” and the VD marker “very.”The authors hypothesize participants relied on these items to identify subjective values or beliefs.Source: C. English VAGO on factual versus… · p. 3
- VAGO counts “highest” and “largest” as degree-vagueness markers because its current rules do not cancel vagueness in superlatives.The authors regard these superlatives as objective and factual in the example sentences, unlike “greatest.”Source: C. English VAGO on factual versus… · pp. 3–4
D. French VAGO on regular versus satirical press articles
On FreSaDa, VAGO scores satirical articles as significantly more vague and subjective, and less detailed, than regular press articles. Scores for vagueness, subjectivity, and detail/vagueness also classify the two document types effectively, reaching a high-accuracy plateau with 500 training documents.
D. French VAGO on regular versus satirical press articles
- FreSaDa contains 11,570 French press articles, of which VAGO processed 10,969; 601 were excluded for unsuitable formats.The corpus comprises 5,648 regular-press and 5,922 satirical articles.Source: D. French VAGO on regular versus… · p. 4
- Satirical articles were significantly more vague (p = 4.99 × 10−11), more subjective (p = 1.69 × 10−9), and less detailed (p = 3.36 × 10−22) than regular press articles.The tests used per-text scores and two-tailed t-tests with Bonferroni correction at α = 0.05.Source: D. French VAGO on regular versus… · p. 4, Figure 3 · p. 4
- VAGO ratio scores served as effective XGBoost classification cues, with high accuracy reached at a training-set size of 500 documents.The classifier used minimum, median, average, and maximum sentence-level ratios for vagueness, subjectivity, and detail/vagueness.Source: D. French VAGO on regular versus… · p. 4
A. Training of the VAGO clone
VAGO-N is a BERT-based neural counterpart trained to predict VAGO’s sentence-level scores on FreSaDa. Its regression results show high accuracy in reproducing symbolic scores, with subjectivity somewhat harder to detect than factual vagueness.
A. Training of the VAGO clone
- VAGO-N combines a BERT architecture, regression layer, and MSE loss to predict VAGO-assigned sentence scores from FreSaDa.The training data comprised 106,000 randomly selected sentences, split into 85,000 training and 21,000 test sentences; RoBERTa Large was used.Source: A. Training of the VAGO clone · p. 4
- VAGO-N replicates symbolic VAGO scores with high accuracy across the evaluated tasks.Table I reports regression results for subjective vagueness, factual vagueness, and detail/vagueness.Source: A. Training of the VAGO clone · p. 4, Table I · p. 4
- Subjectivity detection was somewhat more difficult than factual-vagueness detection.Source: A. Training of the VAGO clone · p. 4
B. Comparison of the versions of VAGO
LIME analysis finds that VAGO-N often relies on lexical cues already in VAGO, while also identifying potential additions and differences in how cues are weighted.
B. Comparison of the versions of VAGO
- The analysis uses LIME to inspect terms contributing most or least to VAGO-N scores, including cases where its predictions diverge from VAGO.Figure 4 gives an example of LIME output for a French sentence; “neutre” denotes the inverse of vagueness and contributes negatively to the score.Source: B. Comparison of the versions of VAGO · p. 5, Figure 4 · p. 5
- For factual vagueness, VAGO-N highlights many terms already in VAGO, but weights them differently from the symbolic system.Words from morpho-syntactic categories not considered by VAGO may amplify or diminish the neural score.Source: B. Comparison of the versions of VAGO · p. 5
- For subjective vagueness, LIME identifies existing VAGO adjectives and terms of excess alongside new combinatorial-vagueness candidates, with some exceptions.Examples of proposed additions include “durable,” “particulièrement,” “ringard,” and “actuel”; “sabbatique” is an exception.Source: B. Comparison of the versions of VAGO · p. 5
A. Validation and enrichment of the symbolic VAGO
The authors use LIME-derived token contributions to test whether VAGO-N recovers VAGO’s French lexical cues, and find both existing cues and candidates for lexicon enrichment.
A. Validation and enrichment of the symbolic VAGO
- The authors aggregate LIME contribution scores across 141,137 FreSaDa sentences to obtain a normalized ctok score for each token.They order tokens by decreasing ctok and measure precision against VAGO lexicon instances, counting inflections.Source: A. Validation and enrichment of the… · p. 5
- The French ctok ROC results support the hypothesis that VAGO-N can reconstruct the lexicon at the core of symbolic VAGO.Table II compares precision at different thresholds for French lexicon tokens ordered by ctok.Source: A. Validation and enrichment of the… · p. 5, Table II · p. 5, Figure 5 · p. 5
- Of the 100 highest-ctok tokens for subjective vagueness, 81 were already in VAGO and seven absent words were validated as relevant additions.Some top-ranked verbal forms were treated as false positives because the current lexicons list only adjectives and adverbs.Source: A. Validation and enrichment of the… · p. 5
- VAGO-N retrieves VAGO’s lexical cues while identifying new or non-standard forms.The authors report robustness to noisy text, including typos and abbreviations.Source: A. Validation and enrichment of the… · p. 6
B. Developing multilingual versions of VAGO-N
Because VAGO lexicons cannot be translated automatically, the authors train VAGO-N on translated FreSaDa sentences and compare its highest-ranked English terms with the English lexicon.
B. Developing multilingual versions of VAGO-N
- Automatic translation of VAGO’s lexicons is not possible because their expressions are idiomatic and out of context.The English lexicon comparison uses ctok-ranked tokens and reports accuracy at different thresholds, with corresponding ROC curves in Figure 6.Source: B. Developing multilingual versions of… · p. 6, Figure 6 · p. 6
- The method assumes vagueness scores are preserved across languages, translates FreSaDa from French into English, and trains VAGO-N with the French-training hyperparameters.The English-corpus regression results are described as similar to those for French.Source: B. Developing multilingual versions of… · p. 6
- 90 of the 100 highest-ranked English vagueness-prone terms were already in VAGO’s English lexicon.Some absent top-ranked terms could fit the combinatorial-vagueness category, while four were not clearly vague; among the next 100, absent terms were adjectives that could fit that category.Source: B. Developing multilingual versions of… · p. 6
V. CONCLUSION
The conclusion presents VAGO as a rule-based scoring system and VAGO-N as a BERT-based clone trained on VAGO scores alone. It highlights lexicon enrichment and other-language versions as uses, while noting an unresolved genericity test.
V. CONCLUSION
- VAGO calculates text scores for vagueness, subjectivity, and detail/vagueness; VAGO-N is a BERT-based clone trained solely on VAGO scores.VAGO-N is trained without access to the lexicon underlying the symbolic system.Source: §V · p. 6
- LIME’s highest-contribution terms are existing VAGO entries or terms likely to belong there, suggesting VAGO-N decisions are largely explained by symbolic lexical cues.The authors propose using the neural model to complete existing lexicons and produce neural and symbolic VAGO versions in other languages.Source: §V · p. 6
- The authors plan to test genericity by masking named entities and checking whether VAGO-N scores remain stable.The planned analysis is intended to estimate how much the lexicon—which contains no named entities—influences VAGO-N decisions; they also describe extending VAGO with explicit subjectivity markers and contextual features.Source: §V · p. 6