PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Political Text Scaling Comparable: Infrastructure and Hyperparameter Sensitivity for 17 Algorithms

Patrick Parschan

arXiv:2609.17602v1cs.CLcs.LG

TL;DR

CT-IPE methods are often compared as named algorithms, although applying them involves choices that configure how text becomes position estimates. This paper compares those choices across 17 algorithms using shared infrastructure and sensitivity analyses. It finds that profile-level shared shifts are small for most algorithms, with sensitivity concentrated in a few consequential decisions.

  • Problem

    CT-IPE comparisons have focused primarily on named algorithms, while researcher choices configuring them are difficult to identify and compare across heterogeneous methods.

    Source: §1 · p. 1

  • Method

    The paper treats CT-IPE methods as configurable measurement pipelines and compares researcher choices across 17 algorithms in a shared experiment using variance partitioning and SHAP-based sensitivity analysis.

    Source: §1 · pp. 1–2

  • Results

    Within the evaluated specification space, profiles account for less than 10% of context-adjusted variance through a shared shift for 13 of 17 algorithms; sensitivity concentrates in a few choices.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The named algorithm remains a reasonable comparison unit where shared shifts are small, but construct-defining choices can steer estimates and deserve explicit reporting.

    Source: §5 · p. 8

    The analysis covers choices varied in this experiment, and sensitivity may change with additional algorithms, corpora, or methodological decisions.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Computational text-based ideal point estimation (CT-IPE) methods are usually compared as named algorithms, yet applying them involves numerous researcher choices that configure how political text is turned into position estimates. This paper argues that CT-IPE methods are better understood as configurable measurement pipelines than as fixed estimators. Building on a large-scale comparative experiment spanning 17 CT-IPE algorithms, 5,537 experimental runs, and approximately 4.25 million left-right position estimates, I describe the shared infrastructure that makes these heterogeneous methods jointly executable and quantify how sensitive their estimates are to alternative hyperparameter choices. Variance-partitioning and SHAP-based sensitivity analyses show that, for most algorithms, hyperparameter profiles explain little residual variance through a shared shift: 13 of the 17 algorithms exhibit ICC values below .10. Where this profile-level sensitivity is present, it is concentrated in a small number of consequential researcher choices, most notably the selection of the underlying language or embedding model, the seed keyword lists that anchor the construct, and the number of topics.

1 Introduction

CT-IPE research has largely compared named algorithms, while the choices configuring those methods remain difficult to identify and compare across approaches. The paper treats CT-IPE as configurable measurement pipelines and examines how researcher choices influence estimates.

1 Introduction

  • Methodological comparisons have primarily assessed agreement with humans and, more limitedly, agreement with other algorithms.The named algorithm consequently became the central unit of methodological investigation.Source: §1 · p. 1
  • Applying CT-IPE also requires choices about constructs, text representations, model configurations, external information, and position estimation, which are rarely compared across methods.Conceptually similar researcher choices therefore remain difficult to identify and compare across otherwise heterogeneous approaches.Source: §1 · p. 1
  • The paper asks how heterogeneous algorithms can become jointly executable and systematically comparable, and what patterns of researcher choice emerge from comparing them.It frames CT-IPE methods as configurable measurement pipelines rather than fixed estimators.Source: §1 · p. 1
  • For most CT-IPE algorithms, alternative hyperparameter profiles explain little remaining variance through a shared shift; sensitivity, when present, concentrates in a few choices.The study compares researcher choices across 17 algorithms using a shared experiment and variance-partitioning and SHAP-based sensitivity analyses.Source: §1 · p. 2

2 Researcher Choices in CT-IPE

The paper treats CT-IPE as a configurable measurement procedure: theoretically or methodologically justifiable choices can alter estimated positions, and their role is broader than software hyperparameters.

2 Researcher Choices in CT-IPE

  • CT-IPE choices configure how political text is transformed into position estimates, a feature obscured when methods are discussed only by algorithm name.The paper’s examples of algorithm names include Wordscores, Wordfish, Party Embeddings, TBIP, and LaMP.Source: §2 · p. 2
  • A researcher choice is a theoretically or methodologically justifiable decision that may alter estimated political positions.A single choice can be implemented through different numbers or combinations of parameters across algorithms.Source: §2 · p. 2
  • Researcher choices are broader than software hyperparameters; some parameters merely implement a choice, while others are purely technical.The analyses treat hyperparameters as part of the measurement procedure and examine decisions at the level of individual choices.Source: §2 · p. 2

3 Methods

The methods combine a large comparative experiment with a shared execution and analysis layer for heterogeneous CT-IPE algorithms. The design preserves native implementations while standardizing selected quantities for comparison.

3.1 Comparative Experiment

  • The experiment evaluates 17 unsupervised and semi-supervised CT-IPE algorithms spanning word-frequency, topic-modeling, embedding-based, and LLM-based methods.The included methods fall within the text-based scope; network-based approaches using non-textual signals are outside it.Source: §3.1 · p. 2
  • The algorithms estimate left–right positions in six German and U.S. corpora of party manifestos, parliamentary speeches, and political tweets.The paper identifies these genres as those for which CT-IPE methods have most frequently been developed and applied.Source: §3.1 · p. 2
  • The experiment attempted 5,537 runs; 4,565 completed successfully, producing approximately 4.25 million political position estimates.Run counts varied because grids differed in size and runtime costs limited configurations for more expensive methods.Source: §3.1 · p. 2
  • The evaluated specification space derives implementation parameters and candidate values systematically from a CT-IPE literature review, but other designs or choices may yield different sensitivity profiles.The experiment aimed to make this space as comprehensive as possible.Source: §3.1 · p. 2

3.2 Code Implementation

  • Algorithm-specific wrappers connect existing methods to shared corpora, metadata, and a hyperparameter grid while preserving their native implementations and estimation logic.The wrappers were designed to make heterogeneous methods jointly runnable, not to reimplement or force them into one common API.Source: §3.2 · pp. 3–4
  • Native outputs remain unchanged during inference; analysis aggregates document estimates where needed and robustly standardizes estimates to a common scale.A robustness check separately aligns polarity per run using fixed anchor parties.Source: §3.2 · p. 3
  • Thin wrappers, validation of configuration keys, and resource-aware grids were consequential engineering lessons for joint CT-IPE experiments.Unknown grid keys are now rejected at planning time; runtimes range from seconds to tens of minutes per configuration, making equal-sized grids often infeasible.Source: §3.2 · p. 3
  • The shared execution layer makes heterogeneous CT-IPE algorithms jointly executable and systematically comparable under one design.Analysis-time standardization is kept separate from native outputs.Source: §3.2 · pp. 3–4

4 Results

The analysis estimates how alternative hyperparameter profiles shift standardized party scores after holding experimental context constant, then attributes shared shifts to individual researcher choices. Most algorithms show little profile-level sensitivity; where sensitivity is present, it is concentrated in a few choices.

4.1 Estimand and Estimation

  • The ICC measures between-profile shared shifts in standardized party scores after holding context constant; party reordering and polarity reversals fall outside it.Scores are aggregated to parties and robustly standardized within each algorithm; the analysis separates profile-level ICCs from SHAP attribution of individual choices.Source: §4.1 · p. 4, §4 · p. 4

4.2 Profile-Level Sensitivity

  • Wordscores’ point estimate is zero but uncertain ([.00, .74]); polarity alignment leaves near-zero algorithms near zero while moving SemScale from .06 to .16.CGCoT falls from .81 to .36 and LSS from .19 to .10 after alignment, indicating that some measured sensitivity reflects scale reversals.Source: §4.2 · pp. 5–6
  • ICC values quantify shared shifts within the evaluated framework, not methodological quality; sensitivity to construct-defining seed keywords can be desirable.Such choices steer which textual variation is treated as the relevant signal for the construct.Source: §4.2 · p. 6

4.3 Which Researcher Choices Drive Sensitivity?

  • SHAP attribution concentrates importance in one or two choices—model, NLI definition, seed lists, topics, and batch size—but weak adjusted R2 makes many shares exploratory.Preprocessing thresholds show no measurable influence within the evaluated grids for Wordscores, Wordfish, Correspondence Analysis, and Wordshoal.Source: §4.3 · pp. 6–7, Figure 2 · p. 7
  • Under alternative LSS seed-keyword choices, the AfD shifts from far right with links/rechts to far left with Fightin’ Words, illustrating substantial landscape variation.These German-manifesto results are an illustration for one algorithm and corpus, not a comparative dictionary ranking; keyword families also determine coupled auto-seeding.Source: §4.3 · p. 7, Figure 3 · p. 8
  • Semantic Scaling has greater profile-level sensitivity (.28) than LaMP (.08), Asking & Averaging (.09), and Benoit et al. (2026) (.02), driven by its NLI model and dimension definition.For generative methods, prompt-related parameters partition sensitivity that is small in absolute terms.Source: §4.3 · pp. 7–8
  • CGCoT sensitivity is especially uncertain: its high computational cost limited configurations, and only 8 of 16 attempted specifications produced valid results.CGCoT was evaluated on speeches and tweets, not manifestos, and its ranking procedure requires more than two political actors.Source: §4.3 · p. 8

5 Conclusion

The study finds small shared shifts for most algorithms, while sensitivity in the remaining cases concentrates in a few researcher choices. It concludes that algorithm-level comparison remains reasonable when shared shifts are small, but consequential choices should be reported explicitly.

5 Conclusion

  • Within the evaluated specification space, hyperparameter profiles account for less than 10% of context-adjusted variance through a shared shift for 13 of the 17 algorithms; the broader pattern remains after anchor-based polarity alignment.Source: §5 · p. 8
  • Where profile-level sensitivity is present, it concentrates in the language or embedding model, seed keywords anchoring the construct, and number of topics.The paper describes these as a few consequential choices rather than effects spread across the pipeline.Source: §5 · p. 8
  • The named algorithm remains a reasonable comparison unit where shared shifts are small, but consequential construct-defining choices deserve explicit reporting.Party-specific reordering lies outside the ICC estimand.Source: §5 · p. 8

Limitations

The analysis covers only researcher choices varied in the experiment, and its estimates are bounded by the executed specification space and analysis design. The paper identifies uneven grids and the ICC’s treatment of party-specific reordering as important qualifications.

Limitations

  • The analysis covers choices varied in this experiment, not every conceivable implementation choice, and results may change with additional algorithms, corpora, or decisions.Future work should broaden the specification space, examine additional choices, and study links between sensitivity and substantive conclusions.Source: Limitations · pp. 8–9
  • A near-zero ICC does not imply immunity of the party landscape to researcher choices because party-specific reordering is absorbed into the residual and illustrated only for selected algorithms and corpora.The reported ICC is a profile-level shared-shift quantity.Source: Limitations · pp. 8–9
  • Experimental grids are uneven because algorithms expose different numbers of hyperparameters and resource constraints limited configurations for more expensive methods, so cross-algorithm sensitivity contrasts are descriptive, especially for thin samples.The paper recommends broadening the evaluated specification space.Source: Limitations · p. 9
  • A post-hoc audit found that validation had silently dropped unknown YAML keys, so planned grids and executed runs could diverge; unknown keys are now rejected at planning time.Sensitivity estimates describe the executed specification space, and pretrained-embedding training knobs were placeholders rather than additional trained variation.Source: Limitations · p. 9

A Included Algorithms and Relevant Papers

Table 2 overviews the reviewed CT-IPE algorithms and identifies the main paper relevant to each algorithm and its implemented comparison.

A Included Algorithms and Relevant Papers

  • Table 2 overviews the reviewed CT-IPE algorithms and lists the main relevant paper first for each algorithm and implemented comparison.Source: Table 2 · p. 12

B Overview of Algorithm Runs

Table 3 reports per-algorithm execution and sensitivity-analysis information, alongside aggregate run and position-estimate totals.

B Overview of Algorithm Runs

  • 5,537 attempted runs and 4,565 successful runs produced 4,252,357 position estimates across all algorithms.Source: Table 3 · p. 13
  • Table 3 reports per-algorithm attempted and successful runs, mean runtime, peak RAM, cleaned hyperparameter profiles, and retained substantive parameters entering sensitivity analysis.Source: Table 3 · p. 13
  • Runs fail or are skipped when an algorithm is inapplicable to a corpus or a configuration does not complete.Source: Table 3 · p. 13

C Polarity-Alignment Robustness Check

After per-run anchor-based polarity alignment, the substantive sensitivity picture remains, but alignment removes arbitrary sign indeterminacy while also removing some choice-induced reversals.

C Polarity-Alignment Robustness Check

  • The substantive sensitivity picture remains after per-run anchor-based polarity alignment, though the aligned analysis excludes runs without a decidable anchor order.Runs are sign-flipped when their fixed anchor parties appear in reversed order; the table reports cleaned-profile ICCs before and after alignment.Source: C Polarity-Alignment Robustness Check · p. 13, Table 4 · p. 13
  • Alignment removes a nuisance for unsupervised sign indeterminacy but also part of the measured researcher-choice effect when choices define scale direction.Flipped shares therefore mix arbitrary sign indeterminacy with substantive choice-induced reversal.Source: C Polarity-Alignment Robustness Check · p. 13
  • For SS and CPTM, the aligned-versus-headline ICC difference partly reflects restricting to alignable runs; their unaligned ICCs on that subset are .36 and .06, respectively.Figure 4’s SHAP shares should not be substantively interpreted for algorithms with near-zero ICC or near-zero model fit.Source: Table 4 · p. 13, Figure 4 · p. 14

E Party Landscapes under Alternative Researcher Choices

Alternative researcher choices yield distinct party-landscape patterns across Semantic Scaling, TBIP, and Wordfish on German manifestos, from party reordering to near-identical landscapes.

E Party Landscapes under Alternative Researcher Choices

  • Semantic Scaling’s NLI-model choice can reorder parties and reverse the AfD’s polarity across models.The NLI model is the dominant SHAP driver for Semantic Scaling in these landscapes.Source: E Party Landscapes under Alternative… · p. 14
  • TBIP’s topic-count choice leaves the German-manifesto party order largely intact in this cell despite a pooled ICC of .34.High profile-level sensitivity need not imply landscape reordering in every corpus.Source: E Party Landscapes under Alternative… · p. 14
  • Wordfish landscapes under alternative minimum term- and document-frequency thresholds are nearly identical, consistent with its ICC of zero.The figure varies min_termfreq and min_docfreq; CGCoT is omitted because manifesto corpora were excluded from its executed grid.Source: E Party Landscapes under Alternative… · p. 14, Figure 5 · p. 15

F Online Supplementary Material

Online supplementary material and two repository zip files provide the experimental pipeline, replication materials, and analysis data and code.

F Online Supplementary Material

Loading 2609.17602v1…