PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy

Pierre-Antoine Lequeu, Salim Hafid, Paul Lerner, Nazanin Shafiabadi, Laurène Cave, David Mas, Jean-Philippe Cointet, Benjamin Piwowarski, François Yvon

arXiv:2609.02990v2cs.SIcs.AI

TL;DR

Participatory democracy consultations produce sparse voting data that require preference inference, yet inferred votes shape the collective preference landscape used in downstream decisions. This paper benchmarks preference inference with a collective-centric framework and finds that predictive accuracy alone does not establish whether models preserve that landscape.

  • Problem

    Sparse voting data require preference inference, whose predictions reconstruct the preference landscape used to identify groups, select propositions, and produce summaries.

    Source: §1 · p. 1

  • Method

    The paper evaluates whether inferred votes preserve proposition-level approval rates and participant-level preference structures, including voting communities and socio-demographic patterns.

    Source: §1 · p. 2, Characterizing the Landscape of… · p. 3, Measuring the Distortion of Preferences · p. 4

  • Results

    Across evaluated consultations, no existing method faithfully preserves the preference landscape, and high predictive accuracy can coincide with substantial distortion of collective structure.

    Source: Results · p. 6, §5 · p. 7

  • Takeaways & Limitations

    Preference inference in democratic settings should be evaluated for preservation of collective structure, not solely for individual vote-prediction accuracy.

    Source: §1 · p. 2, §5 · p. 7

    The framework operationalizes a specific, non-exhaustive set of democratic principles grounded in deliberative theory; alternative conceptions may motivate additional criteria.

    Source: Limitations and Ethical Considerations · p. 8

Abstract

from arXiv · show

To scale up collective decision-making, participatory democracy platforms such as Polis and Remesh enable online deliberation among thousands of participants. However, at this scale, participants cannot review every opinion submitted by others, producing highly sparse voting data that misrepresent patterns of consensus, conflict, and minority support. Platforms therefore increasingly rely on Preference Inference (PI) models to predict missing votes. Yet this automation is not neutral: inferred preferences can artificially amplify, suppress, or reorder existing patterns of support, ultimately reshaping how the outcomes of a deliberation are interpreted. More generally, we lack a systematic understanding of how existing PI methods affect the collective preference landscape. To address this gap, we benchmark several existing PI approaches in this context. Moving beyond conventional user-centric evaluations centered on the accuracy of individual predictions, we introduce a collective-centric evaluation framework that measures whether inferred votes preserve salient properties of the broader preference landscape. We further contribute the largest multilingual dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages. Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure. These results demonstrate that accuracy alone is insufficient for evaluating PI in democratic settings. By contributing a novel comprehensive and collective-centric evaluation benchmark for the task of PI, this work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.

1 Introduction

Sparse voting data make preference inference necessary in large consultations, but inferred votes can distort consensus and group representation. This work proposes evaluating those collective effects alongside individual prediction accuracy and contributes a multilingual benchmark.

  • Preference Inference predicts whether participants would support or oppose propositions they have not reviewed, addressing voting-matrix sparsity in large consultations.The inferred preferences are used in downstream tasks including identifying opinion groups and producing summaries for decision-makers.Source: §1 · p. 1
  • Uneven inference errors can create artificial consensus or conflict and under-represent particular socio-demographic or interest groups.The proposed goal is to preserve both proposition support patterns and participant group structure without erasing genuine disagreement.Source: §1 · p. 2
  • Standard metrics aggregated across participant–proposition pairs can miss changes to approval patterns, group profiles, and vote-based communities.A model can attain high accuracy while distorting preferences across socio-demographic groups.Source: §1 · p. 2
  • The paper’s benchmark complements predictive metrics by assessing proposition-level approval rates and participant-level preference structures, including communities and socio-demographic groups.The framework formalizes the preference landscape along proposition-positioning and participant-grouping dimensions.Source: §1 · p. 2
  • Four consultations make up a multilingual dataset with almost 90k participants, 1M votes, and 22 languages.The authors also report that models with high predictive accuracy can substantially distort consensus and socio-demographic group preferences.Source: §1 · p. 2

2 Related Work

Participatory democracy platforms use inference strategies to handle sparse voting matrices, and the inferred data support later representation and summarization. Prior preference-inference research spans latent-variable and language-based models, while this paper argues that evaluation must assess whether those models preserve collective structure.

Participatory Democracy Platforms

  • Polis and Remesh use different strategies to handle sparse participant–proposition voting matrices, whose inferred entries support opinion-group identification and automated summarization.Polis imputes missing entries using proposition-level mean votes before dimensionality reduction and clustering; Remesh combines language-model representations with latent factors from observed votes.Source: Participatory Democracy Platforms · p. 2
  • Preference Inference also acts as algorithmic representation because it attributes unexpressed preferences and shapes how groups become visible to decision-makers.The paper connects this perspective to evaluating accuracy across socio-demographic groups and vote-based communities.Source: Participatory Democracy Platforms · p. 2

Preference Models for Collective Representation

  • Preference-inference methods developed from latent-utility and adaptive-survey approaches to matrix factorization, collaborative filtering, matrix completion, and language-model representations.These methods estimate missing responses from shared structure across participants and propositions; multilingual content poses an additional challenge.Source: Preference Models for Collective… · pp. 2–3
  • Equal predictive performance across groups does not ensure faithful representation: group-fairness metrics can miss when inference erases groups’ opposing approval profiles.The paper’s metrics therefore assess whether group-specific profiles and vote-based community structure remain represented.Source: Preference Models for Collective… · p. 3

3 Theoretical Framework

The framework treats preference inference as completing a partially observed voting matrix while preserving the collective structures used to interpret preferences. It evaluates those structures across proposition-level support and participant-level organization.

3 Theoretical Framework

  • Preference Inference predicts missing entries in a partially observed voting matrix, leaves observed votes unchanged, and is assessed for both prediction accuracy and preservation of collective properties.The matrix records support and opposition as +1 and −1, with ⊥ marking unobserved votes.Source: §3 · p. 3

Characterizing the Landscape of Preferences

  • The preference landscape comprises proposition positioning by support and opposition, plus participant organization by voting behavior and socio-demographic attributes.Proposition approval rates distinguish broad support, broad rejection, and contested ideas; participants may form vote-based communities or socio-demographic groups.Source: Characterizing the Landscape of… · p. 3

Measuring the Distortion of Preferences

  • Approval Rate Hit Ratio checks whether inferred proposition approval rates remain within confidence intervals estimated from observed votes.The interval is a 95% Wilson confidence interval, and the metric assesses whether proposition-level support and opposition patterns remain statistically credible.Source: Measuring the Distortion of Preferences · p. 4
  • Consensus Distortion measures group-level approval changes, taking the largest group distortion for each proposition and averaging those maxima across eligible propositions.The maximum captures the concern that distortion affecting even one group can mislead judgments of cross-group consensus.Source: Measuring the Distortion of Preferences · p. 4
  • Community Stability assesses whether similar-voting participant groups persist after inference, while fairness metrics capture worst-group accuracy and cross-group error-rate disparities.Community Stability compares clusterings with Adjusted Rand Index; the framework also includes minimum group Balanced Accuracy and Equalized Odds Ratio.Source: Measuring the Distortion of Preferences · p. 4

4 Experiments and Results

The experiments compare preference-inference methods across diverse consultation datasets and find that predictive accuracy does not reliably track preservation of the preference landscape. No model performs best across metrics; STUMP models offer a favorable trade-off, while several distortions persist as more votes become available.

Datasets

  • The evaluation combines two existing Polis and Remesh datasets with four new Make datasets, including non-English consultations and Eurhope’s 892k votes from 54k participants.Across datasets, the team retains agree/disagree votes, excludes neutral and pairwise preferences, and requires at least 25 participants per protected-attribute group.Source: Datasets · p. 5

Models

  • Collaborative filtering infers votes from correlated user histories or proposition similarities, while nearest-neighbor models transfer a user’s vote from the closest text-embedded proposition.The nearest-neighbor search uses propositions on which that user has already voted.Source: Models · p. 5
  • Other approaches include FM and STUMP latent representations, SBGCL’s signed-graph contrastive learning, and LLM prompts containing users’ observed positive and negative votes.LLM results are reported only for the three datasets with protected attributes because inference is not scalable to large consultations.Source: Models · pp. 5–6, Experimental Protocol · p. 6

Results

  • STUMP models offer the strongest accuracy–fidelity trade-off, while SBGCL leads B-Acc but ranks fifth on ARHR and CS and sixth on CD; no model excels across metrics.No model exceeds ARHR 0.95 on any dataset; the lowest average CD is 21.7 for FM, and CF has the best average CS at 22.9.Source: Figure 2 · p. 6, Results · pp. 6–7

Experimental Protocol

  • Models use a random 70-15-15 train-validation-test split; learned-model checkpoints are selected by validation B-Acc, with a maximum of 400 epochs.B-Acc, EOR, Min. B-Acc, and CD are measured on the test subset, whereas ARHR and CS use the full voting matrix.Source: Experimental Protocol · p. 6

Results

  • As observed votes per participant increase, B-Acc, Min. B-Acc, and CD improve, but ARHR, EOR, and CS remain largely unchanged, suggesting structural limitations beyond sparsity.SBGCL is not evaluated with only one observed preference because it requires at least two.Source: Figure 3 · p. 7, Results · p. 7

5 Conclusion

The conclusion finds that preference inference is essential for reconstructing sparse consultation data, but existing methods can distort collective preferences despite high predictive accuracy. The authors propose collective-centric evaluation and caution that preference inference can complement, but not replace, meaningful democratic participation.

5 Conclusion

  • Sparse voting data make preference inference important for reconstructing the landscape used in downstream decision-making.The conclusion frames this as a role for inference, not a replacement for participation.Source: §5 · p. 7
  • Across evaluated consultations, no existing preference inference method faithfully preserves the underlying preference landscape.Models with high predictive accuracy can still substantially distort consensus patterns and the preferences of particular groups.Source: §5 · p. 7
  • The released multilingual benchmark spans four consultations, nearly 90,000 participants, one million votes, and 22 languages.Eurhope is described as the largest publicly available consultation dataset, with eight times more participants and twice as many propositions as the largest previous benchmark.Source: §5 · p. 7
  • The authors argue that future preference inference evaluation should include faithful collective representation alongside predictive accuracy.They present collective-centric evaluation as a standard objective for future systems.Source: §5 · p. 7

Limitations and Ethical Considerations

  • Preference inference should complement—not substitute for—citizen consultations designed for broad, representative, and meaningful participation.The framework reflects deliberative-theory principles, which the authors note are not exhaustive; alternative conceptions may motivate additional criteria.Source: Limitations and Ethical Considerations · pp. 7–8

Related papers

Loading 2609.02990v2…