Source-linked AI summary
Count Evidence, Not Sentences: Tempered Evidence Fusion of LLM Judgments for Long-Text Value Measurement
Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, Tianyi Zhang, Zihan Dong, Jiaxin Liu, Yujie Chen, Guang Zhang
TL;DR
Long social media posts often mix contextual material with only a few stance-bearing sentences, while existing direct prediction and voting approaches do not account well for uneven evidence. The paper proposes TEF, which tempers sentence log-odds by information gain, and reports average accuracy and macro-F1 gains over the strongest baseline across five LLMs and two languages.
Problem
Long posts mix background, quotations, and concessions with only a few stance-bearing sentences, while direct prediction or voting can mishandle this uneven evidence.
Source: §1 · p. 1Method
TEF is a training-free decision-fusion rule that weights sentence log-odds by normalized information gain, discounting uncertain judgments while preserving decisive evidence.
Source: §1 · p. 1Results
TEF exceeds the strongest Direct, Majority Vote, and Soft Vote baseline by 4.5 accuracy points and 4.6 macro-F1 points on average across five LLMs and two languages.
Source: §1 · p. 1Takeaways & Limitations
On Qwen2.5-7B, TEF achieves the lowest expected calibration error and overconfidence gap among the compared methods, making its confidence better suited to selective review.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix background, quotations, concessions, and only a few stance-bearing sentences. Existing approaches either ask the model to predict a document-level label directly, which can be overconfident, or aggregate sentence-level predictions by majority or soft voting, which treat uncertain and decisive sentences as equally informative. We formulate long-text value measurement as a decision-fusion problem and propose Tempered Evidence Fusion (TEF), a training-free rule that weights each sentence's log-odds by its normalized information gain, as derived from a generalized Bayesian posterior. This makes the fused score nearly vanish for uncertain sentences while preserving the Bayes-optimal weight of decisive evidence. We further introduce Multi-event Insight Network Dimensions (MIND), a benchmark of 8,358 Chinese and English posts spanning five years of public events and six value dimensions. On MIND, TEF outperforms the strongest baseline among Direct, Majority Vote, and Soft Vote by an average of 4.5 accuracy points and 4.6 macro-F1 points across five LLMs and two languages. MIND dataset and code are available at https://github.com/Kzczc/ICASSP2027-TEF.
1. INTRODUCTION
Long social media posts can mix background, quotations, and concessions with only a few stance-bearing sentences, making document-level prediction or ordinary voting ill-suited to uneven evidence. The paper formulates value measurement as decision fusion and introduces TEF, reporting gains over baseline methods across models and languages.
1. INTRODUCTION
- Long posts may mix background, quotations, and concessions with only a few stance-bearing sentences, so uncertain sentences can outvote decisive evidence under majority or soft voting.Direct prediction can instead collapse heterogeneous evidence into one overconfident label.Source: §1 · p. 1, Figure 1 · p. 1
- The paper frames each sentence as a soft local detector and notes that, under conditional independence and calibrated sentence posteriors, the Bayes-optimal document rule sums log-likelihood ratios.Majority voting retains only each judgment’s sign, soft voting underweights highly decisive sentences, and directly summing LLM logits is unreliable because next-token probabilities are only approximately calibrated.Source: §1 · p. 1
- TEF is a training-free rule that weights sentence log-odds by normalized information gain, nearly nulling uncertain judgments while preserving decisive evidence.It is derived from a generalized Bayesian posterior and has a soft-censoring effect similar in spirit to distributed-detection rules.Source: §1 · p. 1
- TEF exceeds the strongest Direct, Majority Vote, and Soft Vote baseline by 4.5 accuracy points and 4.6 macro-F1 points on average across five LLMs and two languages.The paper also reports better-calibrated estimates.Source: §1 · p. 1
2. RELATED WORK
Related work spans longer-text stance measurement, LLM confidence, and classical decision fusion, but long-text LLM measurement still largely relies on equal-weight voting.
2. RELATED WORK
- Signal-processing work includes probabilistic opinion pools, reliability-weighted local decisions, and censoring schemes, while long-text LLM measurement still largely relies on equal-weight voting.The paper positions TEF as an evidence-aware fusion rule that discounts uncertain sentences while preserving decisive ones.Source: §2 · p. 2
3. PROPOSED TEF
The paper frames long-text value measurement as decision fusion over sentence-level judgments with uneven evidential strength. TEF aggregates reliability-weighted log-odds so uncertain segments contribute less than decisive ones.
3. PROPOSED TEF
- Hard voting can discard reliability information because a segment's hard label is a deterministic function of its probability distribution.The data processing inequality gives I(Y; p_i) ≥ I(Y; ŷ_i).Source: §3 · p. 2
- Unlike majority or soft voting, TEF retains each segment's full probability distribution and assigns different contributions to segments with the same hard label.Majority voting keeps only hard labels, while soft voting averages probability vectors with equal weight.Source: §3 · p. 2
- Under calibrated, conditionally independent segment posteriors and equal priors for a binary contrast, Bayes-optimal document log-odds add sentence-level log-likelihood ratios.TEF uses a one-vs-rest log-odds transform as its additive evidence scale.Source: §3 · p. 2
- TEF weights one-vs-rest log-odds by normalized information gain, giving near-uniform segments little influence while preserving near-Bayesian contributions from decisive segments.This soft-censoring means a few stance-bearing sentences can dominate even when uncertain sentences are numerous.Source: §3 · p. 2
4. THE MIND BENCHMARK
MIND is a longitudinal, cross-lingual benchmark for measuring public value orientations, built from public posts through collection, sequential annotation, and agent-driven evolution.
4. THE MIND BENCHMARK
- MIND contains 8,358 posts—5,474 Chinese and 2,884 English—covering six value dimensions from January 2020 to January 2025.Candidate posts came from X, Weibo, Reddit, Zhihu, discussion forums, and news feeds, with collection anchored to major public events.Source: §4 · p. 3
- Three annotation rounds label value dimension, subcategory, and orientation in sequence, accepting labels with at least two of three annotators' agreement and arbitrating disputed cases.Samples without a stable majority are discarded; accepted samples pass distribution and quality checks.Source: §4 · p. 3
- Agent-driven evolution addresses class imbalance and semantic ambiguity through synthetic oversampling, label-alignment review, and text revision, retaining only verified samples.Balancing agents generate underrepresented categories; critic and semantic-edit agents review and clarify low-confidence or ambiguous samples.Source: §4 · p. 3
5. EXPERIMENTS
TEF achieves the highest accuracy and macro-F1 across the evaluated models, languages, and dimensions, while remaining robust to prompt changes. Calibration, ablation, and efficiency analyses further examine its behavior and cost.
Main Results
- 4.5 accuracy points and 4.6 macro-F1 points on average over the strongest baseline, with TEF best on both metrics for every model in both languages.It is best or tied in 116 of 120 dimension-level comparisons and never trails the strongest baseline by more than 2.6 points.Source: §5 · p. 3, Table 3 · p. 4
Ablation Study
- On Qwen2.5-7B, both ablations retain most of TEF’s average gain over Soft Vote, but on DeepSeek-V3.2 both fall below Soft Vote in both languages.The largest accuracy drops occur on DeepSeek-V3.2’s English environment dimension: 22.9 points without entropy weighting and 23.8 without the log-odds transform.Source: §5 · p. 3, Table 4 · p. 4
Calibration
- TEF has the lowest expected calibration error and overconfidence gap across Chinese and English on Qwen2.5-7B.Direct and both voting rules are overconfident; TEF’s confidence is described as better suited to selective review.Source: Figure 4 · p. 3, §5 · p. 4
Robustness
- TEF remains best under verbose and minimal prompts and degrades the least under either perturbation.The prompts respectively add role background and annotation guidelines, or retain only the classification directive.Source: §5 · p. 4, Table 1 · p. 4
Efficiency
- TEF uses the same single-token queries as voting, with time to first token differing by only about two milliseconds.Sentence-level methods cost roughly three times as much as Direct, but TEF is the only one that consistently turns this overhead into a performance gain.Source: §5 · p. 4, Table 2 · p. 4
6. CONCLUSION
The paper frames long-text value measurement as fusion of soft sentence decisions and derives TEF to temper sentence log-odds by information gain. On MIND, TEF outperforms direct prediction and voting across five LLMs and two languages, is better calibrated on Qwen2.5-7B, and retains gains across prompt changes.
6. CONCLUSION
- The conclusion presents TEF as a fusion rule that tempers each sentence’s log-odds by its information gain.On MIND, it is more accurate than direct prediction and voting across five LLMs and two languages, better calibrated on Qwen2.5-7B, and its gain survives prompt changes.Source: §6 · p. 4