Source-linked AI summary
Enhancing LLMs in Predictive Political QA with Semi-Structured Data
Yinan Liu, Zihan Zhou, Zichun Jin, Xinyu Wang, Bin Wang, Xiaochun Yang
TL;DR
Predictive political QA cannot be solved by direct factual retrieval because external records rarely contain answers about future political behavior, motivating evidence representations beyond knowledge-based retrieval. PSL converts semi-structured records into inference-oriented evidence by combining semantic actor stances with vectorized high-order structure signals. Across three real-world datasets and multiple LLMs, it consistently outperforms existing augmentation methods, with ablations supporting complementary gains from both signal types.
Problem
Predictive political QA lacks direct answers in external records, while existing augmentation methods mainly represent those resources as knowledge-based evidence rather than prediction-relevant signals.
Method
PSL uses a semantic view to extract actor stances and a vector view to learn high-order structure representations from semi-structured political records and actor interaction graphs.
Results
PSL consistently outperforms existing augmentation methods across multiple real-world datasets and LLMs, with ablations confirming complementary benefits from stance and structure signals.
Takeaways & Limitations
Inference-oriented evidence that combines textual semantics with graph structure is effective for predictive political QA beyond knowledge-based evidence.
Takeaways & Limitations
The evaluation is limited to U.S. political settings and requires broader countries, political systems, language contexts, and richer records to assess generalizability.
Abstract
from arXiv · showhide
Predictive political question answering (QA), such as predicting how a political actor will vote, goes beyond factual lookup. External political resources offer rich historical evidence, but rarely contain the answer itself. Existing LLM augmentation methods, including actor-profile-based simulation and knowledge graph evidence injection, improve political reasoning but largely treat external resources as knowledge-based evidence, leaving prediction-relevant signals under-modeled. We identify two complementary signals for predictive political QA: actor stances that capture issue-specific preferences, and high-order structure signals that capture indirect dependencies among political actors. We propose PSL, a dual-view framework that converts semi-structured political records into inference-oriented evidence for LLMs. PSL extracts stance signals from question-relevant actor records in a semantic view, and learns structure-aware actor representations from an actor interaction graph in a vector view. Across three real-world datasets and multiple LLMs, PSL consistently outperforms baselines, with ablations confirming the complementary gains of stance and structure signals.
1 Introduction
Predictive political QA requires more than factual retrieval because political records usually provide indirect evidence about behavior that has not yet occurred. PSL addresses this gap by combining actor stances with high-order structural signals from semi-structured political data to strengthen LLM inference.
- Predictive political QA cannot rely on direct factual retrieval because relevant future behavior is rarely recorded in external political resources.External data supplies historical records, but not the answer itself when the behavior has not yet occurred.
- Existing methods improve political reasoning but treat external resources mainly as knowledge evidence, limiting implicit group influence or fine-grained context.Profile simulation struggles with indirect actor influence, while knowledge-graph extraction can lose contextual detail.
- Actor stance signals capture issue-specific preferences linked to values, ideology, or partisan interests.These subjective attitudes can provide greater reasoning potential than factual knowledge in complex contexts.
- High-order structure signals model indirect dependencies and group influence within political actor interaction networks.Semi-structured data preserves contextual detail while supporting representation of network-embedded relationships.
- PSL uses semantic and vector views to extract stance evidence and model high-order structure from semi-structured political records.It builds actor profiles, links actors through shared records, and integrates both signal types into LLM inputs.
- Across three real-world datasets and different LLMs, PSL significantly outperforms all baseline methods, while ablations verify complementary benefits from stance and structure signals.The experiments evaluate the framework across multiple datasets and language models.
2 The PSL Framework
PSL converts semi-structured political records into inference-oriented evidence through semantic stance extraction and vector-based modeling of high-order actor structure. These signals are jointly integrated into LLM reasoning for predictive political QA.
- Framework overview: PSL uses semi-structured political data to model actor stances in a semantic view and high-order structure signals in a vector view.The semantic view expresses stances as natural-language judgments, while the vector view represents structural information through actor embeddings.
- Actor behavior profiles: Political behavior profiles organize legislative and diplomatic records around individual actors without excessively compressing their underlying information.Records include attributes such as bill titles, descriptions, votes, diplomatic actions, and involved parties.
- Actor stance acquisition: Ladder retrieval decomposes each question into a primary actor and question focus, then iteratively selects records using attribute-level semantic similarity.After n iterations, retrieval produces the top k_n records most relevant to the question.
- Actor stance acquisition: PSL distills factual records into concise stance evidence that exposes issue-aligned cues such as behavioral consistency and ideological orientation.A large teacher model is distilled into a smaller language model for stance inference.
- High-order structure signals: The actor interaction graph connects political actors to shared records, assigning voting weights of 1 for “Yea” and −1 otherwise.Diplomatic interactions receive weight 1, while absent diplomatic edges represent no interaction.
- High-order structure signals: A weighted-sum aggregator propagates information across the interaction graph to construct neighbor-aware actor representations that capture structural influence.Record embeddings come from a frozen pretrained embedding model, and graph normalization uses first-hop neighborhood sizes.
- Synergistic enhancement: PSL co-embeds actor and question representations so actor embeddings become question-conditioned and structural noise can be filtered before LLM integration.The resulting collaborative relationship helps the LLM use high-order structure signals alongside explicit and implicit stance signals.
- Learning: The MLP is trained with Bayesian personalized ranking to score observed records above unobserved records, using positive and negative actor-record relations.Negative records include explicit negative votes and unobserved diplomatic records.
3 Experiments
Experiments evaluate PSL across three political datasets, multiple LLMs, and comparisons with diverse external-knowledge baselines. PSL consistently outperforms baselines, while ablations and parameter studies support complementary stance and structure signals.
- Experimental setting: The experiments cover RCVP, ICEWS, and StaId, using macro F1 for binary classification and accuracy for the ICEWS multiple-choice task.The study evaluates Llama-3.1-8B-Instruct and Mistral-7B-Instruct, among other settings.
- Effectiveness study: PSL outperforms all baselines across the three datasets and different LLMs.Compared methods include knowledge-generation, document-enhanced, knowledge-graph-enhanced, and actor-profile-based approaches.
- Effectiveness study: PSL consistently outperforms PEG and PAA in all experimental setups, while also demonstrating competitive time efficiency.Both PEG and PAA use external knowledge similar in content to PSL’s resources.
- Ablation study: Removing stance evidence causes significant drops across all datasets, showing that retrieved records capture critical issue-specific positions.The ablation is conducted with Llama-8B.
- Ablation study: Removing structure evidence degrades performance, notably on RCVP and ICEWS, indicating the value of group-level behavioral dependencies.The w/o Structure setting eliminates collaborative embeddings and retains stance and question text.
- Parameter study: Performance peaks at 5 retrieved records for RCVP and ICEWS, 15 for StaId, layer 2 for RCVP and ICEWS, and layer 3 for StaId.More records can introduce irrelevant-content noise, while deeper propagation layers can cause over-smoothing and reduce discriminative power.
4 Further Analysis
Further analysis shows that targeted stance retrieval and vectorized structure modeling outperform alternative evidence presentations, while retrieval volume has dataset-specific optima and noise trade-offs.
- Stance acquisition: All retrieval variants surpass the baselines and the “w/o Stance” ablation, while PSLLR achieves the best performance among retrieval variants.Lexical methods underperform because they rely on lexical overlap; PSLLR leverages hierarchical structure and handles missing data through score inheritance.
- Stance acquisition: The full PSL framework outperforms all variants, indicating that targeted stance information reduces noise and boosts overall performance.The comparison separates retrieval and stance inference effects from the complete dual-view framework.
- Neighbor and record context: RCVP benefits from distributing retrieval across more actors, whereas ICEWS declines with larger context and StaId improves with retrieval volume.The dataset-specific differences reflect distinct sensitivity to indirect-record noise and missing statement profiles.
- Neighbor and record context: All text-based settings underperform PSL, supporting collaborative vector modeling over expressing complex actor relations entirely in natural language.The analysis attributes the gap to information overload in textual representations.
5 Related Work
Prior political reasoning methods augment LLMs with textual profiles or knowledge-graph triples, but predictive QA requires evidence that captures both contextual detail and indirect group influence.
- Predictive political QA: Predictive political QA differs from factual retrieval because external resources record past behavior rather than the target behavior itself.The target behavior has not yet occurred, so evidence must support prediction rather than directly state the answer.
- Political QA augmentation: PEG injects retrieved triples from a political knowledge graph, whereas PAA simulates political actors from textual profiles.Both methods adapt external-resource augmentation to political reasoning.
- Limits of existing evidence: KG triples lose fine-grained context, while textual profiles weakly capture implicit group influence.These limitations motivate deriving stance evidence and structure signals from semi-structured political records.
6 Conclusion
PSL transforms semi-structured political records into inference-oriented evidence by combining textual semantics with graph-structural features. Across multiple datasets and LLMs, it consistently outperforms existing augmentation methods, supporting prediction-oriented evidence beyond knowledge-only augmentation.
- Conclusion: PSL transforms semi-structured political records into inference-oriented evidence for predictive political QA.Its dual-view design jointly uses textual semantics and graph-structural features.
- Conclusion: PSL better preserves contextual detail than KG triples and more fully models actor relations than text-profile simulation.These comparisons describe the evidence representation targeted by the framework.
- Conclusion: PSL consistently outperforms existing augmentation methods across multiple real-world datasets and LLMs.The paper presents this result as initial evidence that inference-oriented evidence beyond knowledge-based evidence is effective for predictive political QA.
Limitations
The validation is limited to U.S. political settings and public records, leaving PSL’s generalizability across countries, political systems, and languages untested.
- Data scope: PSL is validated in U.S. political settings using public records such as legislative votes, bill information, and diplomatic events.The study constructs semi-structured profiles and an interaction graph from these records.
- Data scope: Generalizability across other countries, political systems, and language contexts remains to be assessed.The paper identifies broader geographic, institutional, and linguistic evaluation as future work.
- Data scope: Future extensions could add co-sponsorship, committee activities, public statements, campaign donations, and policy texts as behavioral evidence.The paper states that these richer sources would not require changes to PSL’s basic design.
A EFFICIENCY STUDY
The efficiency study identifies intermediate generation as the main source of runtime degradation, while PSL reduces this cost relative to existing generative approaches.
- Intermediate generation is the primary factor contributing to efficiency degradation.
- PSL has shorter intermediate generation time than RECITE, MindMap, and PEG.
- Table 5 reports running efficiency using relative time against Vanilla GPT-3.5-Turbo and separates retrieval, intermediate generation, and final inference.
B EVALUATION
The evaluation follows a multiple-choice prompting setup and uses regular expressions to match answers when model outputs do not explicitly state the options.
- The evaluation prompt provides multiple options and instructs the model to output its choice.
- Regular expressions match answers when the model output does not explicitly state the options.
C MLP TRAINING
The MLP is trained on actor-profile records by pairing actor embeddings with positive and randomly selected negative record embeddings, using a large sampled dataset.
- MLP training data is constructed from actor profiles, pairing each actor embedding with positive and randomly selected negative record embeddings.
- Table 7 summarizes the MLP training parameters.
- Over 900,000 training samples are aggregated, with 500,000 randomly selected using seed 42.
- The sampled dataset is split into training, validation, and test subsets at 70%, 15%, and 15%.
D LoRA TUNING
The paper applies LoRA fine-tuning while keeping pretrained parameters frozen, with experiments using limited training resources and actor-record examples.
- LoRA introduces trainable low-rank decomposition matrices while keeping the original model parameters frozen.
- LoRA reduces trainable parameters and lowers computational and storage costs while adapting models to downstream tasks.
- Experiments use 4 × NVIDIA A6000 GPUs and limit LoRA training to 800 samples.
- The section includes a sample stance of Seth Moulton based on repeated yea votes and legislative support patterns.