Source-linked AI summary
Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment
Deepak Pandita, Christopher M. Homan
TL;DR
AI evaluation and alignment need to account for disagreement and diverse human perspectives rather than relying on aggregated consensus. This proposal develops a perspective-aware framework, finding that multiple responses per item can improve reliable evaluation under fixed budgets while supporting pluralistic alignment. Its analyses are limited by independence assumptions and a scope focused partly on specific reasoning tasks.
Problem
Conventional evaluation can overlook pervasive human disagreement through plurality-based aggregation, while trustworthy alignment requires diverse human values and perspectives.
Method
The proposal combines perspective-aware reproducible evaluation studies with human-centered approaches for pluralistic LLM alignment.
Results
Increasing responses per item is often more effective than increasing items for reliable evaluation, with the trade-off depending on the metric.
Takeaways & Limitations
Strategically collecting multiple responses and using diverse perspectives can support more effective, budget-conscious evaluations and greater confidence in model performance.
Takeaways & Limitations
The evaluation treats responses as independent across items, does not account for sampling bias, and ProRefine is evaluated mainly on mathematical and multi-step reasoning tasks.
Abstract
from arXiv · showhide
Humans play a vital role at every stage of AI development, from data collection and curation to model development and evaluation. However, humans often disagree with each other and sometimes with themselves over time. It is essential to take disagreement into account when building human-centered AI systems, especially in domains where it is prevalent, such as AI safety, content moderation, or sentiment analysis. Disagreement often arises from subjective human opinion and can vary with one's identity, beliefs, and social environment. Despite this, current LLM evaluation approaches frequently rely on aggregating labels (often via plurality voting) to represent consensus, thereby obscuring minority perspectives. By failing to account for human disagreement, these evaluation methods contribute to the reproducibility crisis in AI. Human feedback is also crucial for ensuring that AI systems align with human values. For these systems to be trustworthy, it is critical to ensure that they reflect diverse human values and perspectives. In this thesis proposal, we present a human-centered and perspective-aware framework for reproducible ML evaluation and AI alignment.
1 Introduction
AI evaluation faces a reproducibility crisis, partly because conventional methods overlook pervasive human disagreement. The proposal therefore advances perspective-aware evaluation and pluralistic LLM alignment.
- 1 Introduction: AI systems must be reliable and trustworthy, yet researchers often cannot reproduce previous study results.Incomplete documentation, varying methods and implementations, and unreliable gold-standard data contribute to this crisis.
- 1 Introduction: Conventional evaluations often treat human disagreement as noise and aggregate only 3–5 labels per item through plurality voting.This practice overlooks disagreement that is endemic in human responses.
- 1 Introduction: Perspectivist approaches use and publish disaggregated labels to account for human label variation in model evaluation.The proposal connects this perspective-aware approach with trustworthiness and reproducibility.
- 1 Introduction: The thesis statement identifies diverse perspectives as critical to reproducible machine learning evaluation and pluralistic LLM alignment.The proposal investigates disaggregated responses, annotation budgets, diverse raters, textual feedback, and pluralistic adaptation.
- 1 Introduction: Human feedback aligns LLMs with human values and preferences through reinforcement learning from human feedback, but alignment can favor specific political ideologies.The proposal argues that AI systems should represent diverse groups rather than a single group.
2 Background
Human responses vary with subjectivity, identity, and beliefs, while majority voting inadequately incorporates that variance. AI reproducibility is also challenged by nondeterministic methods and unreliable human evaluation and statistics.
- 2 Background: Subjectivity causes variance in human responses, and disagreement is linked to rater identity and beliefs.Examples of identity factors include race, gender, age, education, and first language.
- 2 Background: Majority voting is inadequate for incorporating response variance.The background contrasts aggregation-based consensus with approaches that preserve disagreement.
- 2 Background: Identical deep-learning training runs can produce different models and test results because machine-learning methods, algorithms, and implementations are nondeterministic.This remains true even when code is shared.
- 2 Background: Human evaluation studies show low reproducibility, while statistical significance is often misapplied and reported outcomes are frequently unreliable.These concerns add inadequate statistical analysis to the reproducibility challenges in AI and NLP.
- 2 Background: Automatic prompt-optimization methods use gradient-based search, reinforcement learning, or LLM-generated prompts, with recent work extending automatic prompt refinement.The cited approaches include AutoPrompt, RLPrompt, Promptomatix, and EvoAgentX.
3 Perspective-Aware Reproducible ML Evaluation
The framework models human-response variability and uses statistical simulation to determine annotation allocations for reproducible comparisons between ML models. Results indicate that increasing responses per item often matters more than increasing items, with metric-dependent tradeoffs.
- Method: The approach estimates the numbers of items (N) and responses per item (K) needed to compare two models under difference ϵ and metric Γ.It computes p-values for candidate N, K, ϵ, and Γ settings.
- Results: P-values decreased as K increased across a wide range of N × K values, and similar trends appeared across metrics, distortions, and datasets.Figure 1 uses ΓMAE with ϵ = 0.05 for Toxicity and ϵ = 0.1 for MultiDomain.
- Results: Current practices may be insufficient for confident comparison: with 1000 items, MAE required at least 25 raters for significance.The authors note that test sets with 25,000–50,000 annotations are rarely used.
- Method: The simulator models categorical responses by sampling gold and model A from one Dirichlet-categorical distribution and perturbing model B by ϵ.The categorical simulator extends earlier continuous-response work.
- Experiments: The experiments evaluate annotation allocation across multiple datasets, metrics, fixed budgets N × K, and perturbation levels.Budgets range from 100 to 50,000, with K varied from 1 to 500.
- Results: Increasing K was often more effective than increasing N, while the existence and location of tradeoffs depended more on the metric than the data source.Distribution-sensitive metrics benefited greatly from higher K.
4 Human-Centered AI Alignment
This section examines disagreement-aware human-centered AI alignment and introduces methods that use rater perspectives and textual feedback to improve LLM reasoning and pluralistic adaptation.
- Rater Disagreement: Rater cohesion differs across political and demographic groups in offense annotation, with Independents most cohesive and Republicans least internally cohesive.Democrats are least cohesive with other groups, while gender and political affiliation both influence cohesion levels.
- Learning Using Textual Feedback: ProRefine adaptively refines prompts for multi-step reasoning by combining task generation, textual feedback, and prompt optimization without additional training or ground-truth labels.The iterative process generates k additional tokens, evaluates outputs, and updates the prompt until a maximum step count or EOS token is reached.
- Preliminary Results: ProRefine significantly improves performance over zero-shot CoT in all but one experiment and outperforms TextGrad in 11 of 15 cases.Across five reasoning datasets and three task models, performance gains are strongest for Llama3.1-8B-instruct, which outperforms both baselines on all five datasets.
- Proposed Work: The proposed perspective-adaptation work will use feedback models representing specific perspectives to optimize prompts or directly adapt task-performing LLM outputs.Perspective-optimized outputs may be synthesized, and inference-time refinement supports black-box scenarios where model weights are inaccessible.
5 Conclusion
The conclusion presents perspective-aware evaluation and pluralistic alignment as complementary goals, emphasizing budget-conscious annotation and future use of perspective-specific textual feedback.
- Conclusion: The proposal argues that integrating diverse perspectives is necessary for reproducible machine learning evaluation and pluralistic LLM alignment.Its evaluation work uses simulations and real-world datasets to study annotation allocation, while future work targets perspective-aware alignment.
- Conclusion: Increasing responses per item is often more effective than increasing the number of items for reliable evaluation under a fixed annotation budget.The trade-off depends on the evaluation metric, so annotation allocation should be metric-sensitive.
- Conclusion: The research offers practitioners a data-driven methodology for more effective, budget-conscious evaluations and suggests more strategic rater recruitment through cohesion analysis.The perspective-aware approach is presented as a way to strengthen trust and confidence in model performance.
Limitations
The work is limited by assumptions about response distributions and independence, restricted empirical scope, and simplified demographic and ideological categories.
- The simulator’s effectiveness depends on how accurately its probabilistic models capture realistic response distributions.Distribution-family choices for regression tasks were based on visual inspection.
- Responses are treated as independent across items, although they may depend on the annotator or model instance producing them.
- The hypothesis tests estimate sampling error but do not account for sampling bias or representativeness.
- The rater-cohesion findings may not generalize to education, cultural background, or economic-status groups.
- Political ideology is simplified into Democrats, Republicans, and Independents, which may omit multidimensional beliefs and identities.The authors propose more granular analysis as future work.
- ProRefine increases inference-time latency and computational cost, lacks a formal convergence guarantee, and may degrade or plateau after repeated refinement.Its evaluation currently focuses on mathematical and multi-step reasoning tasks.
A.1 Reproducible ML Evaluation
Figure 5 examines how p-values change with the number of responses per item while keeping the total evaluation budget fixed at N × K = 2500.
- Figure 5 plots p-value against K across various metrics under a fixed budget of N × K = 2500.Each data point is estimated from 10,000 samples.
A.1.1 Power Analysis
The power analysis evaluates statistical power for toxicity data as sample size changes, finding faster gains for Multistage Bootstrap as the number of responses increases.
- Multistage Bootstrap reaches 90% power before baseline hypothesis tests as the number of items or responses increases.Power is calculated using α = 0.05.
- The power of all tests benefits from more responses, but Multistage Bootstrap improves markedly faster.
- Figure 6 reports toxicity-data power estimates using 1000 outer-level samples of 10000 inner-level samples at ϵ = 0.1.
A.2 Rater Cohesion
The section presents simulation procedures for modeling categorical and noise parameters, prompt optimization, and post-filtering cohesion and accuracy analyses. The supplied material identifies the relevant algorithms and table encodings but does not report their numerical outcomes.
- Simulation procedures: Algorithms 1 and 2 define separate simulations for Halt and Hnull.The passages name both simulation algorithms and indicate that Algorithm 2 uses the same steps as Algorithm 1.
- Simulation procedures: The simulation inputs include N, K, M, α, ρ, and ϵ, with Dirichlet sampling for categorical and noise parameters.The categorical parameters βi and noise parameters ϱi are sampled from Dirichlet distributions, then combined as γi = (1 −ϵ) ∗βi + ϵ ∗ϱi.
- Prompt optimization: The prompt-optimization procedure takes a query, initial prompt, step limits, and task, feedback, and optimizer LLMs, returning an optimized prompt p∗.The procedure checks for an EOS token during optimization before returning the final prompt.
- Cohesion and alignment analyses: Table 2 encodes in-group and cross-group cohesion results after CrowdTruth filtering, while Table 3 reports vicarious alignment results under the same filtering.The table descriptions define significance at p = 0.05 and distinguish results before, after, or both before and after filtering.
- Accuracy evaluation: Table 4 reports test accuracy with 95% confidence intervals across five benchmark datasets and models.The table marks statistically significant improvements over baseline methods and identifies the highest accuracy for each dataset-method combination.