PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Feature Discovery from Unstructured Data Using LLMs and Statistical Choice Models

Baptiste Arnaudo, Amine Bennouna, Sébastien Martin

arXiv:2610.38082v1stat.MLcs.AIcs.LG

TL;DR

Opaque embeddings and costly hand-engineering make interpretable feature extraction from unstructured data challenging, while direct LLM extraction can miss or conflate features. The paper discovers features by aggregating pairwise LLM rationales with a statistical choice model. It reports improved feature recovery over direct extraction, comparable predictive accuracy to established baselines on ChangeMyView, and an extension to auditing LLM decisions.

  • Problem

    Feature extraction from unstructured data lacks a practical route to interpretable, decision-relevant features and per-item scores, while direct LLM extraction can miss or conflate features.

    Source: §1 · pp. 1–2

  • Method

    LLMs compare item pairs and state distinguishing features; a statistical choice model aggregates these rationales to recover relevant features and per-item scores.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The method recovers planted features better than direct LLM extraction and reaches accuracy comparable to the state of the art on ChangeMyView while providing interpretable features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The recovered features capture most predictive signal with fewer, less redundant features while matching established-baseline accuracy, and the rationales can also support audits of LLM decisions.

    Source: §6 · p. 10

    The score model assumes score differences drive both feature-citation likelihood and which item is more likely to win when a feature is cited.

    Source: §3 · p. 3

Abstract

from arXiv · show

Feature extraction from unstructured data typically relies on opaque embeddings unsuited to interpretable decision-making. Interpretable features, on the other hand, typically require costly, and often infeasible, hand-engineering. Large language models (LLMs) offer a natural bridge, as they can read raw data and describe it in interpretable terms. One could simply ask an LLM to extract features from a large sample of the dataset, but context limitations and diluted attention can lead to unsatisfactory results. We introduce a method that automatically discovers interpretable, decision-relevant features from unstructured data. Our key idea is to perform pairwise comparisons between items, asking an LLM to articulate, in interpretable terms, the features distinguishing each pair. We treat these explanations, or rationales, as data: we develop a statistical choice model that combines these rationales across the comparisons to identify relevant features and estimate per-item, per-feature scores. We first test our method on a synthetic hiring dataset of cover letters with known ground-truth features and show that it significantly outperforms direct LLM feature extraction in feature recovery. We then apply it to the ChangeMyView (Reddit) debate dataset, where it discovers several interpretable features associated with persuasion while reaching predictive accuracy comparable to established baselines. We also show how the same rationales can be used to audit LLM decisions, yielding a holistic score whose decomposition over interpretable features reveals what the LLM rewards.

1 INTRODUCTION

Interpretable, decision-relevant features are difficult to extract from unstructured data without costly hand-engineering or opaque embeddings. The paper uses pairwise LLM rationales and a statistical choice model to discover features and estimate item-level scores, outperforming direct extraction in feature recovery.

Motivation

  • Opaque embedding-based features do not support transparent decision-making, while interpretable feature engineering can be costly or infeasible.The paper frames the goal as discovering interpretable, discriminative features and per-item scores that can support decisions and make LLM decisions auditable.Source: §1 · pp. 1–2
  • Directly asking an LLM to extract features from an entire dataset faces long-context demands, and the paper reports that this approach misses present features and conflates others.The authors characterize the challenge as managing context so LLMs can identify relevant features reliably.Source: §1 · p. 2

Contributions

  • Pairwise LLM comparisons articulate which features distinguish items and which item is stronger, and enough rationales let a statistical model recover feature relevance and per-item scores.The paper calls each articulated comparison a rationale.Source: §1 · p. 2
  • The choice model combines whether a feature is cited with which item wins, estimates per-item feature scores and confidence intervals, and filters features whose scores are unreliable.The likelihood is non-convex and grows in parameter count with the number of items, but the paper proves a consistent local maximizer under mild assumptions as the number of items grows.Source: §1 · p. 2
  • The method correctly recovers all planted features in synthetic cover letters, while direct LLM extraction misses features and blurs several together; on ChangeMyView it matches state-of-the-art accuracy while providing interpretable features.The paper also uses rationales to audit LLM decisions and reveal what mattered in comparisons.Source: §1 · p. 2
  • The procedure can be supplied as LLM instructions, with a lead agent delegating graph-based pairwise comparisons to sub-agents and a statistical model aggregating their rationales.The paper connects this orchestration to work on scaling AI through multiple agents (Li et al., 2024a; Kim et al., 2026).Source: §1 · p. 2

2 RELATED WORK

Prior work uses LLMs as judges across domains, while documented biases can undermine their decisions. Feature-scoring approaches generally use predetermined features and often discard rationales rather than using them to discover features.

The LLM-as-a-Judge paradigm

  • LLM-as-a-Judge systems use LLMs to evaluate or rank items, including comparing generated answers; substantial agreement between LLM and human preferences has motivated their use across several domains.Examples include information retrieval, recommendation, alignment training, and automated essay scoring.Source: §2 · p. 2

LLM biases

  • LLM judges can exhibit positional and verbosity biases, which prior work documents as threats to LLM-based decision-making.Examples include favoring the first or second item in a prompt and preferring longer essays.Source: §2 · p. 3

Feature evaluation with LLMs

  • LLM feature-evaluation methods generally score a predetermined feature pool, while this paper treats rationales as data for feature discovery.Prior approaches may discard rationales or use them as qualitative confirmation; one cited approach uses them to create scoring guidelines, but the authors say they know of no approach using them for interpretable feature extraction.Source: §2 · p. 3

3 METHOD

The method collects local pairwise rationales, groups their descriptions into shared features, and fits a choice model to estimate item scores while accounting for judge biases. It then uses diagnostics to flag unreliable feature scores.

Method overview

  • The pipeline compares item pairs along a graph, groups LLM descriptions into shared features, aggregates local observations into per-item scores, and derives diagnostics for poorly pinned-down scores.Local comparisons need little context but can miss dataset-level information; the statistical model uses cited differences and unmentioned features.Source: §3 · p. 3

Rationale collection

  • Each pairwise rationale becomes a sparse verdict tensor recording whether a feature was invoked and, if so, which item was judged stronger.Raw feature descriptions are phrased differently across pairs and are grouped by a second LLM call into K shared features.Source: §3 · p. 3

Statistical choice model

  • The choice model estimates latent item scores from pairwise verdicts, with score differences assumed to drive both feature citation and which item wins when cited.The model is inspired by Bradley–Terry models.Source: §3 · p. 3
  • Feature citation is modeled as more likely when the absolute score difference is larger, and non-citation is informative because it suggests the difference was not salient enough to mention.The citation model includes a feature-specific baseline propensity.Source: §3 · p. 3
  • The model accounts for positional bias with a shared position coefficient and feature-specific citation habits, so these biases do not contaminate estimated scores.The authors report that randomized presentation order makes scores essentially insensitive to this choice.Source: §3 · p. 4

Theoretical guarantees

  • For K = 1 under the stated dense-design assumptions, the likelihood has a local maximizer consistent for every item score as n grows, and that maximizer is unique in a neighborhood and is the local MLE.The theorem gives probability at least 1 − O(n^-c) for every c > 0 when n is sufficiently large.Source: §3 · p. 5
  • The dense-design theorem does not cover the sparse comparison graphs used in practice, where the authors instead report out-of-fold calibration and a well-defined local optimization target near the truth.They numerically optimize the regularized likelihood on sparse graphs.Source: §3 · p. 5

Extracting score reliability

  • Per-feature condition numbers flag poorly identified scores, while reliability coefficients measure the share of observed score spread attributed to true between-item differences; thresholded features are filtered before refitting.Confidence intervals are obtained through a central limit theorem for the estimator.Source: §3 · p. 5

4 EMPIRICAL RESULTS ON INTERPRETABLE FEATURE EXTRACTION

On synthetic cover letters with known features, the method recovers more ground-truth features than direct LLM extraction and outperforms multi-agent alternatives. Ablations show lower attribution and higher leakage without statistical scoring and reliability filtering.

Dataset

  • The test uses 200 cover letters with 18 embedded dials, of which 16 are recovery targets and two are controls: redundant skill coverage and irrelevant prose fluency.Dials are drawn near-orthogonally and with capped correlation to letter length so each is separately recoverable and length cannot proxy for quality.Source: §4 · pp. 5–6

Metrics

  • Evaluation distinguishes dial tracking and semantic recovery from single-dial attribution, cross-dial leakage, and total cost.Leakage measures the average correlation of a dial with attributable features other than the one that best tracks it; lower is better.Source: §4 · p. 6

Results

  • The method tracks all 16 ground-truth features versus 13 for Direct LLM and achieves complete semantic recovery; 16 of 21 features are attributable to one ground-truth feature, versus 13 of 21.Leakage is lower for the method (0.090 versus 0.109 for Direct LLM), while Direct LLM costs less ($0.26 versus $3.45).Source: §4 · pp. 6–7
  • The method outperforms multi-agent alternatives in feature discovery, scoring, or both; gains over Local LLMs are statistically significant for tracking, attribution, and leakage.Multi-agent scoring improves statistical recovery but increases leakage, while multi-agent feature discovery reduces recovery compared with a single agent using the full dataset.Source: §4 · p. 7
  • Without statistical scoring and reliability filtering, attribution falls to 13–14 features versus 16, and leakage rises to 0.103–0.104 versus 0.090.Filtering raised multi-score attribution from 14 of 32 to 14 of 21 while preserving complete tracking and semantic recovery; statistical scoring on the same 21 features then raised attribution to 16.Source: §4 · p. 7

5 APPLICATION TO A REAL DATASET

On ChangeMyView, the method discovers interpretable argumentation features and tests their persuasion signal against direct-LLM, prior, and established feature sets. The reliable discovered set retains strong predictive accuracy with fewer features, while data-derived features span more effective dimensions than data-blind prior features.

The ChangeMyView dataset

  • The task predicts which of two ChangeMyView replies persuaded the original poster, using the poster's awarded delta as the persuasion label.Exchanges pair an original-poster post with a reply.Source: §5 · p. 7

Benchmarks and ablations

  • At k = 16, sample-elicited S is 4.2 points less accurate than Dr + PW; across k, it trails by 3.6 points on average.The 95% paired CI for the average difference is [1.8, 5.4].Source: Figure 2 · p. 8, §5 · p. 9

Extracting interpretable features

  • Among discovered argumentation features, evidence and cases associate more positively with persuasion than tone and appeals; confrontational tone's negative association is not significant.Effectiveness is the fraction of test pairs where the delta-winning reply scores higher on a feature; 0.5 is chance.Source: §5 · p. 8, Figure 3 · p. 9

Benchmarks and ablations

  • The benchmark compares 29 discovered features and their 16-feature reliable subset with sample-elicited S (42 features) and data-blind P (16 features).S is reported only under pointwise scoring because its 42-feature pairwise calls would be prohibitively costly.Source: §5 · p. 8
  • At 16 features, D and Dr span about eight effective dimensions under BT and six to seven under PW, versus about four for P.The data-derived sets are less redundant than the data-blind prior set; P's curve has flattened before 16 features.Source: §5 · p. 8, Figure 2 · p. 8
  • Dr has about half as many features as D, loses at most half a point of accuracy, and its equal-size subsets average one point more accurate over k ≤ 16.The comparison is reported for both pointwise and Bradley–Terry scoring.Source: Figure 2 · p. 8, §5 · p. 9

6 CONCLUSION

The paper concludes that pairwise LLM comparisons and a statistical choice model recover interpretable features that retain predictive signal with fewer, less redundant features. It reports accuracy comparable to established baselines and points to an extension for auditing LLM decisions.

6 CONCLUSION

  • Pairwise LLM comparisons and a statistical choice model recover human-interpretable features, which retain most predictive signal with fewer, less redundant features while matching established baselines' accuracy.Source: §6 · pp. 9–10

REPRODUCIBILITY STATEMENT

  • The paper provides experiment-reproduction code, proves the choice model's mathematical guarantees in Appendix B, and gives full experiment details in Appendices C and D.Source: REPRODUCIBILITY STATEMENT · p. 10

STATEMENT ON LLM USAGE

LLM outputs produced the study's judges, comparisons, pointwise scores, and synthetic cover-letter corpus. The authors also report limited LLM assistance with wording, code, and proof auditing.

STATEMENT ON LLM USAGE

  • LLM API calls produced the analyzed judge outputs, pairwise comparisons, pointwise scores, and synthetic cover-letter corpus.Source: STATEMENT ON LLM USAGE · p. 10
  • The authors used LLMs for occasional rewording and code assistance, and Claude Fable 5 audited proofs, surfacing an error the authors diagnosed and corrected by hand.Source: STATEMENT ON LLM USAGE · p. 10

A EXTENSION: INTERPRETABLE LLM-BASED SCORING

The extension turns an LLM's holistic pairwise verdicts into additive, interpretable per-feature scores. Across two essay corpora, judges largely agree on which constructs matter, while their estimated importance can differ; the decomposed totals also correlate strongly with human holistic scores.

Method

  • The method asks a judge for a holistic winner and feature-based justification, then canonicalizes cited qualities into shared features and records verdicts and citation indicators.The holistic verdict is linked to per-feature score gaps and a position coefficient; feature citations are modeled as more likely when the feature favors the winner by a larger margin.Source: A EXTENSION: INTERPRETABLE LLM-BASED… · p. 15

Statistical model

Interpretability

Application

  • The application uses 100 essays each from ELLIPSE and ASAP 2.0, with four model-family judges and semantic matching of their features.Figure 4 groups shared constructs by meaning and displays constructs found by at least three judges; absent markers denote constructs missing from a judge's reliable basis, not zero estimates.Source: A EXTENSION: INTERPRETABLE LLM-BASED… · p. 16, Figure 4 · p. 16

Results

  • Judges agree more on which constructs structure verdicts than on how much each separates essays; shared constructs differ between ELLIPSE and ASAP 2.0.On ASAP 2.0, thesis control is the construct where all four judges essentially coincide (σk ∈[0.94, 1.01]); sentence-level control is most contested (0.82 to 2.11).Source: A EXTENSION: INTERPRETABLE LLM-BASED… · p. 16, Figure 4 · p. 16
  • Per-judge total scores correlate with human holistic scores on ELLIPSE (Spearman rs = 0.89–0.92) and ASAP 2.0 (rs = 0.77–0.86).The totals decompose exactly into named per-feature scores, enabling feature-by-feature audits.Source: A EXTENSION: INTERPRETABLE LLM-BASED… · p. 16, Figure 5 · p. 17

A.1 DETAILS OF THE EXTENSION

This section develops and evaluates pairwise comparison models for essay and salient-feature judgments, covering model specification, estimation, uncertainty, reliability, and comparisons with tie-based models. It also establishes theoretical guarantees under A1–A3 using curvature, leave-one-out, concentration, and trajectory analyses, including uniform control and local uniqueness of the score-map zero.

A.1 DETAILS OF THE EXTENSION

  • The extension compares 100 band-balanced essays in 2,000 pairs per corpus, elicits winner-plus-feature rationales, then canonicalizes and screens features before meaning-based cross-judge consolidation.Reliable features pass ρ > 0.50 and κ < 100; between six and 16 features survive across the eight judge–corpus cells.Source: §A.1 · p. 17
  • Noise correction shrank ELLIPSE feature spreads by 10.5% on average and reordered 10 of 34 features within their judge.The reported spread subtracts mean sandwich variance from observed score variation, with delta-method standard errors.Source: §A.1 · p. 18, Figure 6 · p. 18
  • Fitted-score Spearman correlation with human holistic scores was 0.89–0.92 on ELLIPSE and 0.77–0.86 on ASAP 2.0, while position bias varied across judges and datasets.The fitted position coefficient β varied in sign and magnitude across judges, reaching |β| up to 2.6, and tracked raw first-position win rates.Source: §A.1 · p. 18
  • At stricter reliability gates, ELLIPSE feature sets shrank from 9/9/9/7 to 6/6/6/7 and constructs shared by at least three judges fell from 7 to 5.Consensus counts can understate agreement when screening removes a judge’s feature; the citation floor also affects apparent sharing, motivating the minimum threshold of three citations.Source: §A.1 · p. 18
  • Winner-gate out-of-fold ECE ranged from 0.012–0.028 on ELLIPSE and 0.010–0.019 on ASAP 2.0; citation-gate ECE rose from 0.003–0.011 in sample to 0.027–0.041 out of fold.Citation-gate overprediction was concentrated in upper probability bins across all four judges and both corpora.Source: §A.1 · p. 19, Figure 8 · p. 21, Figure 10 · p. 23
  • Winner-gate Brier skill was 0.80–0.84 on ELLIPSE and 0.72–0.85 on ASAP 2.0; citation-gate skill against the habit-only submodel was 0.03–0.23 across judges and corpora.The citation gate’s skill indicates that citations track winner-oriented gaps, though its contribution is second-order next to the verdict channel and shrinks out of sample.Source: §A.1 · pp. 19–20, Figure 9 · p. 22

B.1 THE BRADLEY–TERRY WITH TIES LITERATURE

  • Davidson’s model is a single-feature instance of the gated model, which strictly generalizes it, while the Rao–Kupper threshold couples its channels and yields a non-logistic direction conditional on a decisive outcome.A battery of independent Davidson fits is a natural classical baseline within the model class.Source: §B.1 · p. 28
  • The model’s non-even position-biased direction channel and order-invariant gate place its likelihood outside the multinomial-logit family, while joint feature estimation and its inferential targets differ from paired-comparison tie models.The resulting signed curvature can make the likelihood non-concave, unlike the cited tie-model likelihoods; the framework’s gate/direction analogy is to a two-level choice with an outside option, not a formal random-utility foundation.Source: §B.1 · pp. 28–29
  • The framework also applies to raters who report verdicts only on attributes salient enough to mention, with non-citation treated as an outside option.The authors present this as a discrete-choice analogy, not a formal random-utility foundation.Source: §B.1 · p. 29

B.2 WHY A SHARED POSITION BIAS COEFFICIENT?

  • A shared position coefficient is retained because randomized presentation order limits its effect on item scores, even when position bias varies across features.With per-feature coefficients, the median absolute score shift was 1% of the per-feature score spread (90th percentile: 5%); residual shifts correlate with |β̂k| at r = 0.87. Cancellation is first-order, not exact, and the citation channel has no position term.Source: §B.2 · p. 30

B.3.1 CONFIDENCE INTERVALS

  • Under possible model misspecification, sandwich covariance supports asymptotically valid confidence intervals for the pseudo-true parameter, the model-class projection of the judge’s comparison distribution.When the model is correct, the sandwich covariance reduces to inverse Fisher information; intervals concern the score configuration that best summarizes the judge’s comparisons within the model class.Source: §B.3.1 · p. 30

B.4 RELIABILITY DIAGNOSTICS

  • Reliability diagnostics filter features using Hessian-block condition numbers and score spread corrected for analytic estimation variance.The procedure can flag unidentified or unreliable features, permit dropping or fusing them, and refit scores on the retained features.Source: §B.4 · pp. 30–31

B.5.1 MODEL

  • The model represents each comparison with a citation gate and a conditional direction channel, using an even gate link and parity-constrained covariates to preserve item-swap invariance.Item merits are identified only up to an additive constant, so the model fixes their sum to zero.Source: §B.5.1 · p. 31

B.5.4 HESSIAN

  • The merit Hessian block is not positive semidefinite for every comparison, creating the model’s central non-convexity despite positive semidefinite parameter blocks and a vanishing cross block.A cited near-tie can produce negative curvature in the merit block.Source: §B.5.4 · p. 33

B.7 ASSUMPTIONS AND THE POPULATION CURVATURE FLOOR

  • Under the stated compactness and smooth-even-gate assumptions, population merit weights have a positive curvature floor, with the direction channel carrying that floor at a tie.If merit magnitudes are allowed to grow, the citation probability and curvature floor can shrink, degrading the stated rates.Source: §B.7 · pp. 34–35
  • A4 and A5 may use the smaller shared c0, while A6 implies both; the population Hessian’s minimum centered eigenvalue is at least nb0.The floor follows under A1–A3.Source: §B.7.1 · p. 35, §B.7 · p. 35

B.8 THE LEAVE-ONE-OUT CONSTRUCTION

  • The leave-one-out objective replaces comparisons involving item m with their truth-based expectations, making its iterates measurable using only comparisons disjoint from m.This construction strips out item m’s randomness.Source: §B.8 · pp. 35–36

B.9 CONCENTRATION

  • Under A1–A3, score bounds and spectral control of the saboteur hold with polynomially high probability.The score bounds hold with probability at least 1 − O(n^-c) for every c > 0.Source: §B.9 · pp. 36–37
  • For an F−m-measurable point, the leave-one-out perturbation bound holds with high probability when the point is confined to the working region.The accompanying dichotomy lets the coupled induction establish confinement before the score bound is activated.Source: §B.9 · p. 38

B.10 HESSIAN ESTIMATES

  • Under A1–A3, the Hessian is Lipschitz in merit and structural parameters, and cross-Jacobian and cross-block Lipschitz bounds also hold, including for leave-one-out variants.These bounds apply on the specified working region.Source: §B.10 · pp. 39, 41
  • The supporting Hessian results give a structural-block Lipschitz bound, an upper curvature bound for the merit and leave-one-out Hessians, and local positive definiteness near the truth.The local result applies on a shrinking score neighborhood and for structural parameters close to their true values.Source: §B.10 · pp. 42–43
  • The stage-1 result establishes a high-probability solution to the merit equation for structural parameters within a shrinking neighborhood of their truth.Lemma 13 is identified as an ℓ∞ rate result, although the supplied passage omits the rate expression.Source: §B.10 · pp. 43–44
  • Leave-one-out proxies restore per-coordinate concentration that an ℓ2 bound alone cannot provide, enabling a uniform ℓ∞ bound on the iterates.The proof couples each genuine trajectory with a proxy measurable without item m’s data and inductively controls confinement, proxy accuracy, trajectory-proxy closeness, and ℓ∞ error.Source: §B.10 · p. 44
  • The coupled-trajectory result controls the gradient-descent and leave-one-out paths through simultaneous estimates, with constants independent of n and t on a high-probability event.The trajectories start at s∗ and use step size η = (Cmax n)^-1; the leave-one-out event has probability at least 1 − O((T + 1)n^(1−c)).Source: §B.10 · pp. 44, 45, 48
  • For horizons T = O(log n), the leave-one-out guarantees hold simultaneously over the structural-parameter ball on one event with probability at least 1 − O(n^-c).That event is contained in the score and saboteur events and makes the coupled-trajectory result hold throughout the stated parameter balls.Source: §B.10 · p. 48
  • The net-point leave-one-out bound extends to every parameter, yielding a uniform bound of 2K√n log n over parameters, items, and times.The confinement induction also establishes claim (b) for all iterates through horizon T.Source: §B.10 · p. 50
  • After T + 1 algorithm steps, the iterate is near-stationary, with its final residual controlled by the choice of T.This is established for parameters within distance ρn of (γ∗, β∗).Source: §B.10 · pp. 50–51
  • The score map has a unique zero in the small ball around s∗ on the affine slice, established by degree theory and strict monotonicity.The proof first obtains a zero in the ball and then uses strict monotonicity to show uniqueness.Source: §B.10 · p. 51

B.12 STAGE 2: THE PROFILED LIKELIHOOD AND JOINT CONSISTENCY

This stage profiles out item merits to analyze the joint structural parameters, then establishes a unique local maximum-likelihood estimate under dense-design and information assumptions. The appendix also points readers to the experimental details and recovery results collected in Appendix C.

B.12 STAGE 2: THE PROFILED LIKELIHOOD AND JOINT CONSISTENCY

  • Profiling the merit vector yields an objective whose structural gradient is evaluated at the merit-profiled root, coupling the gate and direction parameters through the shared merits.The profile objective minimizes the negative log-likelihood over merits in the specified neighborhood.Source: §B.12 · pp. 53–55
  • Under A1–A3 and A6, a joint stationary point exists with probability at least 1 − O(n^-c), and the estimate is the unique stationary point and maximizer in the stated local set.The result is local and depends on the theorem’s assumptions.Source: §B.12 · pp. 55–56

C EXPERIMENT DETAILS ON INTERPRETABLE FEATURE RECOVERY

  • Appendix C documents the synthetic cover-letter corpus, recovery metrics, baseline configurations, additional feature results, and paired-bootstrap procedure.It also gathers dial-recovery heatmaps and a feature-merging threshold analysis.Source: C EXPERIMENT DETAILS ON INTERPRETABLE… · p. 57

C.1 DATASET

The synthetic dataset uses engineered cover-letter dials to test feature recovery under controlled conditions. Its construction balances and orthogonalizes dial assignments while holding selected nuisance properties constant.

C.1 DATASET

  • The ground truth comprises 18 dials: eight primary attributes, six binary role-skill presences, and four three-level niche attributes, exactly two active per letter.The skills and niche dials are grounded in the job-ad qualifications and responsibilities; Go is mandatory and is not a dial.Source: Figure 11 · p. 58
  • Fixed templates hold paragraph count constant, while skill coverage changes which of six body units describe capabilities rather than filler content.Coverage takes values 2, 4, or 6, limiting the possibility that it acts as a length signal.Source: §C.1 · p. 59, Figure 12 · p. 59
  • The four niche dials are realized in one fixed-position paragraph, with exactly two non-absent per letter to hold the number of extras constant.Fluency is independently varied by injecting 0, 5, or 12 surface spelling errors after assembly.Source: §C.1 · p. 60
  • The corpus contains 200 letters: 184 form a balanced, near-orthogonal pool, while 16 anchor profiles contrast prestige with substance.The anchor letters are excluded from the balance and orthogonality gates.Source: §C.1 · p. 60

C.2.1 METRICS

The metrics assess whether discovered features recover the planted dials statistically and semantically, while also measuring feature attribution and cross-dial leakage.

C.2.1 METRICS

  • Statistical recovery reports the fraction of dials tracked above |r| = 0.4 and the mean maximum absolute correlation across all dials, including misses.The tracking threshold is fixed a priori and results are reported as stable for τ ∈ [0.3, 0.5].Source: §C.2.1 · p. 61
  • Semantic recovery requires a feature to both track a dial and match it semantically; matching uses Hungarian assignment and retains cosine similarities above η = 0.406.The threshold is the 95th percentile of alignments among rejected, unrelated feature–dial pairs.Source: §C.2.1 · pp. 60–61
  • Attributable-feature and leakage metrics measure whether discovered features primarily track one dial and whether their scores correlate with off-target dials.Attribution requires a majority of a feature’s squared correlation to come from its dominant dial.Source: §C.2.1 · p. 61

C.2.2 METHODS DESCRIPTION

The appendix compares pairwise feature discovery with whole-dataset and sample-based rubric baselines, detailing how comparisons, rubrics, and canonical features are elicited and scored.

C.2.2 METHODS DESCRIPTION

  • Our method compares 200 cover letters over 40 balanced perfect matchings, so each item is compared once per matching; 40 matchings were chosen for high precision.Source: §C.2.2 · p. 61
  • Direct LLMs use a rubric elicited from the full dataset, while Local LLMs elicit rubrics from 40-letter samples across 27 runs and fuse the resulting features.Source: §C.2.2 · p. 61
  • To match costs, Direct LLM–Multi-score repeats scoring 20 times and Local LLMs–Multi-score six times, varying instruction wording and feature order.Source: §C.2.2 · p. 62, §C.2.3 · p. 66

C.2.3 PROMPTS

  • The pairwise prompt requests salient differences grounded in letter evidence and identifies which letter shows more, excluding names, identical visa text, and length alone.Source: §C.2.3 · p. 63
  • The taxonomy groups phrasings by shared meaning, keeps distinct co-occurring constructs separate, and sends genuinely idiosyncratic dimensions to a single “Other” group.Source: §C.2.3 · pp. 63–64
  • Open-ended rubric prompts ask for evidence, dimension descriptions, and score levels; scoring prompts assign each letter a dimension-specific score using its listed descriptors.Source: §C.2.3 · pp. 64–66

C.3.1 MODEL CALIBRATION

The section evaluates out-of-fold calibration and discrimination for the model’s citation and direction gates. The direction gate is well calibrated and predictive, while citation forecasts are too extreme and its gap effect is weaker.

C.3.1 MODEL CALIBRATION

  • 0.010 [0.008, 0.014] is the direction gate’s out-of-fold ECE, compared with 0.019 in sample.Source: §C.3.1 · p. 67
  • 0.046 [0.044, 0.049] is the citation gate’s out-of-fold ECE, and its flatter-than-diagonal curve indicates forecasts that are too extreme at both ends.Source: §C.3.1 · p. 67
  • 0.545 [0.536, 0.554] is the direction gate’s out-of-fold Brier skill against the empirical-rate reference, retaining most of its in-sample skill of 0.623.Source: §C.3.1 · p. 67
  • 0.069 [0.063, 0.076] is the citation gate’s out-of-fold Brier skill against the gap-independent empirical-rate submodel, showing that the merit gap carries information about citation.Source: §C.3.1 · p. 67

C.3.2 DISCOVERED FEATURES

The discovered features provide a fine-grained rubric, while reliability diagnostics distinguish features that track item differences from frequently cited features that do not. The rubric also recovers a stated education-level ordering.

C.3.2 DISCOVERED FEATURES

  • The reliability diagnostics identify “Narrative framing and tone” as boilerplate because its citations are inconsistent with its estimated scores, unlike reliable “Employer prestige.”Source: §C.3.2 · p. 67
  • The rubric correctly orders the Education dial as BS < MS < PhD, indicating that it is tailored to the samples seen by the judge.Source: §C.3.2 · p. 67
  • 32 features were surfaced and 21 retained after applying the stated reliability thresholds; the retained set distinguishes side-project existence from project magnitude.“Side-project ambition” nearly passed the thresholds, while “Narrative framing and tone” was filtered despite κ = 4.9 because ρ = 0.72 fell below ρmin.Source: §C.3.2 · p. 67, Table 4 · p. 69

C.4.1 PER-METHOD RUN-VARIANCE BANDS

The paired comparisons assess performance under different resampling designs, and the dial-recovery comparison contrasts the discovered features with Direct LLM features.

C.4.1 PER-METHOD RUN-VARIANCE BANDS

  • Run-variance bands resample each method’s own pipeline unit and are not confidence intervals for the full-data point estimates or comparable across methods.Source: §C.4.1 · pp. 71–72, Table 6 · p. 72

C.4.2 DESIGN OF THE PAIRED COMPARISONS

  • At average degree 25.3 versus the designed 40, our method leads on several metrics; across fusions, Attrib. mean differences span [+0.362, +0.505].The paired-comparison panels use different discovery-variance designs and are not comparable to one another.Source: Table 7 · p. 73, §C.4.2 · p. 73
  • Panel (a) reruns discovery for both methods, whereas panel (b) holds our taxonomy fixed and varies Local LLMs’ fusion, so their intervals represent different sources of variance.Source: §C.4.2 · p. 73

C.4.3 DIALS RECOVERED

  • Our method’s dial-recovery heatmap shows strong diagonal recovery with few features tracking no dial, while Direct LLM misses three dials and assigns several features to some dials.Source: §C.4.3 · p. 73, Figure 16 · p. 74, Figure 17 · p. 75
  • With 27 Local LLM rubric elicitations, recovering every dial requires a high merge threshold that leaves an unusable number of near-duplicate features.Source: §C.4.3 · p. 73, Figure 18 · p. 75

D.1.1 SAMPLES USED

The memorization probe tests whether the judge’s predictive result reflects recall of delta outcomes by comparing named- and unnamed-source framings on identical de-identified exchanges. Naming the source produced no recall signal, and the study’s sample was drawn from topic-matched winning and non-winning replies.

D.1.1 SAMPLES USED

  • The sample comprised topic-matched ∆-winning and non-winning replies; after word-length restrictions, 3,411 training and 800 held-out pairs remained, with 400 training pairs sampled for the study.Source: §D.1.1 · p. 76

D.1.2 MEMORIZATION PROBE

  • The probe tested recall on 1,600 exchanges across four balanced outcome and split cells, comparing unnamed and named-source framings on identical de-identified text.Source: §D.1.2 · p. 76
  • Under P1, accuracy was 69.4% on no-∆ exchanges versus 37.9% on ∆ exchanges; P2 shifted these to 62–64% and 43–46%, respectively.Source: §D.1.2 · p. 76
  • On the very exchanges where it was measured, the judge showed no recall signal under source naming, so outcome recall did not explain the predictive result.Source: §D.1.2 · p. 76

D.2 METHODS DETAILS

This section details the evaluation and analysis procedures for interpretable feature discovery and scoring, including alternative pipelines and analyses of predictive reliability and dimensionality. It also describes the CMV persuasion-scoring prompt, which elicits evidence-linked, calibrated ratings across fixed dimensions.

D.2 METHODS DETAILS

  • The shared evaluation scores 2,400 exchanges and tests feature scores with no-intercept ℓ1 logistic regression, selecting its penalty by grouped five-fold cross-validation on 400 training pairs.The 800 held-out pairs are evaluated once; the feature arms include comparison-derived D/Dr, prior-elicited P, and data-elicited S sets.Source: §D.2 · p. 77
  • Pointwise arms rate every feature 0–10 in one call per exchange, using an anchored scale and anti-central-tendency prompting; reported accuracy averages two scoring runs.This requires 2,400 calls rather than 32,000 for Bradley–Terry scoring; that approach compares exchanges on each feature against frozen anchors, so test items do not influence one another.Source: §D.2 · p. 78

D.3.1 ADDITIONAL MEASURES

  • D + PW and Dr + PW rank among the best methods, surpassed only by full-training opaque embeddings and Tan et al.’s features, while remaining interpretable.Table 9 reports held-out accuracy; the accompanying text gives this ranking without specifying the accuracy values.Source: §D.3.1 · p. 78, Table 9 · p. 80
  • Dr exceeds the full discovered set by 1.0 accuracy point on average across feature counts under both PW and BT scoring, and leads the sample-elicited set by 3.6 points under PW.Against S under BT, Dr’s advantage reaches 3.5 points at k = 16; Figure 19 reports 95% intervals for these comparisons.Source: §D.3.1 · p. 78, Figure 19 · p. 79

D.3.2 DISCOVERED FEATURES AND WINNING ARGUMENTATION TECHNIQUES

  • The reliability filter selects predictive discovered features: 19 of 29 separate winners from losers, all positively, and Dr comprises 16 reliable features.The dimensionality analysis corrects feature-score correlations using independent repeated measurements; Figure 20 says participation ratio and effective rank agree on feature-set ordering.Source: §D.3.1 · p. 79, Figure 20 · p. 80, Table 10 · p. 82

D.3.3 PROMPTS

  • The prompts elicit persuasion dimensions either from cross-topic pair comparisons or from sampled exchanges, then canonicalize repeated phrasings while keeping poster-side and challenger-side constructs distinct.Comparison judges provide evidence phrases and identify which exchange shows more of each dimension; sample-based elicitation instead asks for reply properties that vary within matched pairs.Source: §D.2 · p. 77, §D.3.3 · pp. 85–87
  • The CMV prompt scores each listed persuasion dimension for an OP–challenger exchange, returning a concrete evidence phrase and an integer rating for each.The prompt specifies that dimensions may concern either the challenger’s argument or the OP’s post.Source: §D.3.3 · p. 88
  • Ratings are calibrated against online arguments generally: 0 means absent, 5 typical, and 10 exceptionally strong, with dimensions judged independently across the full range.The prompt excludes topic, agreement or correctness, usernames, and length itself from the ratings.Source: §D.3.3 · p. 88
  • The response format requires one item per dimension in the listed order, naming each dimension exactly and providing its evidence phrase and integer score in JSON.The follow-up instruction supplies the exchange placeholder and requires the JSON object only.Source: §D.3.3 · p. 88
Loading 2610.38082v1…