Same Text, Different Numbers: The Divergence of LLM-Based Measures

1 Introduction

• The mean pairwise rank correlation is 0.52 across measures, ranging from 0.85 for sentiment and 0.76 for innovation to 0.32 for integrity and 0.23 for rule-based specificity.Agreement in transcript rankings varies sharply by construct.Source: §1 · p. 3
• Within-call score differences across providers average 0.75 times the overall standard deviation of the measure, making model-level variation economically large.For a typical construct, disagreement across models evaluating the same transcript is comparable to the measure’s cross-sectional variation.Source: §1 · p. 3
• In variance-components analysis of total score variation, transcript-level variation accounts for 34.4%, provider variation for 33.4%, and residual provider-by-transcript variation for 32.2%, on average.Across constructs, the transcript component is 74% for sentiment and 13% for integrity.Source: §1 · p. 4
• Explicit scoring rules do not ensure consistent implementation: rule-based specificity produces greater, not lower, disagreement across LLMs.The exercise defines specific-language categories and instructs models to count and scale them by transcript length.Source: §1 · p. 4
• Self-reported confidence does not identify more dependable scores, and written justifications have low semantic similarity—even for sentiment, whose scores are highly correlated.Models sometimes cite the same textual feature but map it to opposite assessments.Source: §1 · p. 4
• Observable firm and text characteristics, justification similarity, and fixed effects explain only 20% of cross-provider dispersion; disagreement is greater for larger, higher-growth firms and calls with more forward-looking language.Most variation in disagreement remains unexplained by standard observables.Source: §1 · pp. 4–5
• LLM disagreement is not significantly associated with subsequent analyst forecast dispersion or errors, bid-ask spreads, trading volume, or return variance.This pattern is consistent with a substantial model-specific component rather than disclosure ambiguity also reflected in market participants’ disagreement.Source: §1 · p. 5
• Correlations between ensemble measures and established dictionary-based measures are positive but generally low, from about 0.55 for corporate culture, innovation, and specificity to about 0.3 for political risk and sentiment.LLM measures are neither interchangeable across providers nor simple substitutes for non-LLM measures; agreement across LLMs need not imply convergence with an established operationalization.Source: §1 · p. 5
• Model choice changes estimated relation magnitudes, signs, and significance, so identical transcripts and regression specifications can yield different conclusions across LLM and traditional measures.The differences extend to coefficients estimated using LLM-based versus traditional measures and depend more broadly on how the construct is operationalized.Source: §1 · pp. 5–6
• For sentiment and uncertainty, estimated signs and significance are stable across providers despite magnitude variation up to 60%; other measures can also change sign and significance.Source: §1 · p. 6
• At the 0.80 benchmark, the seven-provider ensemble meets the relative-reliability threshold for ten of thirteen measures but the absolute-dependability threshold for only five.Averaging therefore stabilizes relative rankings for most measures more effectively than absolute score levels.Source: §1 · p. 6
• Newer frontier models narrow cross-model disagreement but leave a substantial gap among LLMs and between LLM-based and traditional measures.Source: §1 · p. 6

2 Related Literature

• The study complements evidence that a single model can match human coders by showing that independently developed models can disagree on the same text.It examines this dependence systematically across thirteen measures and seven developers.Source: §2 · pp. 6–8
• The paper distinguishes its focus from measurement error arising when models transform source text: models may preserve source information yet operationalize the same construct differently.This contrasts the study with Cookson et al. (2026), which focuses on faithful preservation of information in the source text.Source: §2 · p. 8
• Model choice is a distinct dimension of measurement design, separate from comparing prompts or repeated runs of one model.The authors also examine rule-based instructions, provider aggregation, and comparisons with established non-LLM operationalizations.Source: §2 · p. 8
• The paper isolates model choice by holding the transcript, construct definition, prompt, numerical scale, and inference settings fixed across thirteen measures.The measures span a broad set of constructs, allowing the authors to examine how model dependence varies across them.Source: §2 · p. 8
• Cross-model agreement is necessary but not sufficient for validity; validation against human coding can identify the best provider.The paper leaves open which model best recovers the construct, separating that validity question from measurement invariance.Source: §2 · pp. 8–9
• Cross-model disagreement is a measurement property to diagnose because the models are alternative instruments applied to the same fixed text, not measures of an economic object of disagreement.This distinguishes the setting from work that uses forecast dispersion or persona-conditioned responses to measure heterogeneous beliefs.Source: §2 · p. 9

3.1 Sample

• The sample contains 1,946 calls by 501 firms, with management answers in the Q&A serving as the unit of analysis and averaging roughly 3,900 words.Transcripts come from S&P Capital IQ; the sample includes firms that were index members on the call date and had the required financial data.Source: §3.1 · p. 9

3.2 LLM-based textual measures

• Seven independently developed model families score the same texts with fixed prompts, temperature 0, and no external tools, retrieval, or scaffolding.The study constructs 13 measures across 12 constructs, including sentiment, uncertainty, management clarity, climate and political risk, answer specificity, and corporate culture dimensions.Source: §3.2 · pp. 10–11
• The authors add Specificity Ratio, calculated as LLM-counted specific words divided by total words, alongside the broader specificity measure.The specific-information categories include people, organizations, percentages, locations, dates, monetary values, and durations.Source: §3.2 · p. 11
• Anthropic and Qwen assign political-risk scores of exactly 0.00 to more than 99% of calls, while the other five providers find exposure in 12% to 42% of the same calls.Because the Anthropic and Qwen series carry no cross-sectional information, the authors exclude them from all analyses; political risk uses the other five providers, while other constructs use all seven.Source: §3.2 · p. 12
• LLM Dispersion (Z) is the standard deviation across the seven models’ standardized scores for the same observation and construct.Scores are standardized within each model and construct to mean zero and standard deviation one; higher dispersion indicates greater model disagreement.Source: §3.2 · p. 12

3.3 Traditional textual measures

• Traditional dictionary- or rule-based measures provide benchmarks for comparison, but determining which method captures the “true” construct is outside the study’s scope.The objective is to assess agreement and disagreement across measurement approaches; the supplied passage specifies Loughran and McDonald (2011) word lists for sentiment and uncertainty.Source: §3.3 · pp. 12–13
• Management clarity follows Barth et al. (2023), using evasive-language glossary trigram frequency scaled by word count and multiplied by -1 so higher values indicate greater clarity.Source: §3.3 · p. 13
• Climate risk follows Sautner et al. (2023) and uses climate-related bigram frequency, while political risk follows Hassan et al. (2019) and weights political bigrams by training-library frequency.The political measure is scaled by total bigrams.Source: §3.3 · p. 13
• Answer specificity follows Hope et al. (2016), counting specific terms across seven categories per word, and corporate culture follows Li et al. (2021), using word-list frequencies by dimension.The aggregate culture score averages the five dimension-level scores.Source: §3.3 · p. 13
• Additional textual controls include readability, linguistic complexity, and forward-looking-language intensity.Source: §3.3 · p. 13

3.4 Financial data and outcomes

• Calls are matched to Compustat, IBES, and CRSP data to construct firm characteristics, analyst outcomes, announcement returns, and post-call disagreement measures.The primary return outcome is four-factor announcement CAR over the (0, 1) window; Table 1 reports summary statistics for main-analysis variables.Source: §3.4 · p. 14
• IBES supplies analyst forecast dispersion, coverage, and standardized earnings surprises, while CRSP returns support abnormal-return estimates and post-call disagreement measures.Post-call measures include abnormal bid-ask spreads, trading volume, and return variance.Source: §3.4 · p. 14

4.1.2 Correlations of LLMs and traditional measures

• Across constructs, correlations with traditional non-LLM benchmarks ranged from very low to moderately positive; alignment was highest for corporate culture (Spearman rank correlation 0.56), while management clarity was weakest.Source: §4.1.2 · p. 17

4.1.1 Correlations across LLM scores

• Across 13 constructs, mean pairwise correlation was 0.52; sentiment led at 0.85, while specificity ratio was 0.23 and integrity 0.32.Agreement varied substantially by construct; Pearson correlations broadly matched the Spearman patterns.Source: §4.1.1 · pp. 15–16
• Within-call score dispersion averaged 0.75 of total variation across constructs, reaching 0.97 for specificity ratio and 0.95 for integrity.The ratio compares the standard deviation across seven models scoring the same call with total score variation across calls and providers; it was lower for political risk (0.42), climate risk (0.44), and sentiment (0.48).Source: §4.1.1 · p. 16
• Higher self-reported confidence did not signal greater agreement: pairwise correlation fell from 0.52 to 0.48 when both models exceeded their median confidence, then to 0.33 under stricter thresholds.Confidence was also essentially uninformative about a model’s deviation from the consensus.Source: §4.1.1 · p. 16

4.1.3 Similarity of justifications

• Across constructs, justifications showed generally low cosine and Jaccard similarity across models.Cosine similarity captures semantic similarity, while Jaccard similarity captures word overlap.Source: §4.1.3 · p. 18
• Justification similarity ranges from 0.41 (specificity ratio) to 0.73 (political risk), average Jaccard overlap is 0.15 to 0.31, and sentiment cosine similarity is 0.69.These justifications concern the same text, construct, and prompt, yet numerical-score agreement can mask heterogeneity in how models interpret and emphasize textual features.Source: §4.1.3 · p. 18
• In Best Buy’s tariff discussion, OpenAI scored political risk 0.75 and Qwen 0.00 despite acknowledging the same passages, reflecting different implicit definitions of political risk.The remaining providers’ scores spanned nearly the full range between them, from 0.00 to 0.70.Source: §4.1.3 · p. 19

4.1.4 Principal component analysis

• Across constructs, the first principal component explains an average 60% of variation, and five of seven components are needed to explain 90%, so one consensus score loses substantial information.The results indicate that many constructs lack a dominant common factor across providers.Source: §4.1.4 · p. 20
• Sentiment is the exception: its first component explains 85% of variation with 0.36–0.39 loadings, while integrity and specificity-ratio loadings are uneven at 0.20–0.45 and 0.22–0.50.For specificity ratio, the heaviest loading is on Mistral, rather than representing a uniform consensus across models.Source: §4.1.4 · p. 20
• For specificity, median specific-item counts range from 12 (OpenAI) to 100 (Gemini), even though their mean pairwise Spearman correlation is 0.43.Models broadly agree on which texts are more specific but identify very different numbers of specific items.Source: §4.1.4 · p. 21
• Reported word counts range from about one-tenth of the true count (DeepSeek and Qwen) to more than twice (OpenAI) and are nearly uncorrelated (Spearman 0.12); using true counts raises ratio correlation from 0.23 to 0.47.DeepSeek states that it estimates total words by dividing character count by 4.5.Source: §4.1.4 · p. 21
• Specificity-ratio disagreement reflects implementation differences, including entity-recognition thresholds and model-dependent word counts, cautioning against assuming precise definitions ensure model invariance.The authors advise against relying on LLM-generated outcomes that depend on word counts.Source: §4.1.4 · p. 21

4.2 Variance decomposition of LLM disagreement

• Across constructs, 34.4% of score variance came from calls, versus 33.4% from systematic provider differences and 32.2% from residual model-by-call variation.The latter two components together account for 65.6% of total variation.Source: §4.2 · p. 24
• Call-attributable variance was 74% for sentiment, compared with 13% for integrity and 4% for the specificity ratio.Source: §4.2 · p. 24
• The seven-model average had a mean generalizability coefficient of 0.84; only one of 13 constructs fell below 0.70, and sentiment reached G_c(7) = 0.97.These results indicate reliable relative rankings for most constructs.Source: §4.2 · p. 25
• Absolute dependability averaged 0.72; only nine of 13 constructs exceeded 0.70, and five exceeded 0.80.Source: §4.2 · p. 25
• For uncertainty, G_c(7) = 0.90 but Phi_c(7) = 0.65; integrity fell from 0.76 to 0.51 and the specificity ratio from 0.49 to 0.22.The gap between relative and absolute reliability reflects provider-level score differences.Source: §4.2 · p. 25
• Seven-model averages recover a more reliable common component for many constructs, although individual-provider measures remain model-dependent.The average's relatively high generalizability coefficient indicates that provider disagreements do not eliminate the common signal.Source: §4.2 · p. 26
• Gc(7) = 0.49 and Φc(7) = 0.22 for specificity ratio indicate that averaging recovers neither reliable call rankings nor reliable levels.Source: §4.2 · p. 26
• Gc(7) = 0.76 and Φc(7) = 0.51 for integrity show that averaging restores a usable ranking but not reliable levels.Source: §4.2 · p. 26

4.3 Observable determinants of LLM disagreement

• Firm size and Tobin's Q are positively associated with LLM disagreement, while earnings surprise has no significant relation to it.The authors suggest more complex operations and disclosures at larger and higher-growth firms may leave greater scope for model-specific interpretation.Source: §4.3 · p. 27
• Calls with more forward-looking language have significantly higher LLM disagreement, whereas Fog-index readability is not systematically related to disagreement in richer specifications.Forward-looking statements contain expectations, risks, and conditional claims that may leave more room for interpretation.Source: §4.3 · p. 27
• Higher similarity in model justifications is associated with lower score disagreement, although similar rationales do not mechanically imply similar scores.Models can use similar justification language yet assign different scores, or assign similar scores while emphasizing different call aspects.Source: §4.3 · pp. 27–28
• Only 20% of LLM disagreement variation is explained by firm, textual, justification-similarity, construct, industry, and quarter controls, leaving substantial model-by-call variation.The authors interpret this as evidence that much disagreement arises from model-specific responses to particular texts, alongside stable provider-level calibration differences.Source: §4.3 · p. 28

ment?

• LLM dispersion is not significantly related to any of five subsequent outcomes: analyst forecast dispersion or error, bid-ask spreads, trading volume, and return variance.The results provide no evidence that LLM disagreement captures disclosure ambiguity later reflected in analyst or market disagreement, consistent with a substantial model-specific component.Source: ment? · p. 29

4.5 Does model choice affect empirical conclusions?

• Provider-specific coefficients vary in magnitude, sign, and significance for climate risk, management clarity, specificity, and the specificity ratio; climate-risk estimates range from -0.4 to +0.2.Only Qwen's climate-risk measure is significantly associated with the subsequent market reaction.Source: §4.5 · p. 30
• For management clarity, statistical significance is driven by a single provider, while specificity-ratio coefficients are significant for Mistral and OpenAI.These are additional examples of provider-specific differences in the empirical conclusions.Source: §4.5 · p. 30
• Different LLM providers can produce materially different conclusions from the same earnings calls and empirical specification, with estimates varying in magnitude, sign, and statistical significance.The results describe providers reaching significance under some models but not others.Source: §4.5 · p. 30
• Corporate culture, sentiment, and uncertainty show more stable coefficient signs and significance across providers, although magnitudes vary substantially, especially for sentiment and uncertainty.Corporate culture is the clearest case of cross-provider consistency.Source: §4.5 · p. 30
• At 1%, Wald tests reject cross-provider coefficient equality for sentiment (chi^2 = 32.8) and uncertainty (chi^2 = 33.5), but not corporate culture.The tests jointly compare seven provider-specific return regressions using provider interactions and firm-clustered standard errors.Source: §4.5 · p. 30
• LLM sentiment coefficients are positive and significant across providers and uncertainty coefficients negative and significant, while corresponding traditional measures differ greatly in magnitude and are not significant.Thus, agreement among LLMs does not ensure agreement with traditional text-based measures.Source: §4.5 · pp. 30–31

4.6 Is the disagreement an artifact of differing knowledge cutoffs?

• The cutoff analysis treats DeepSeek and Qwen as within-model probes and uses the other five models as calendar controls in a difference-in-differences design.The design removes common monthly patterns because both straddling models' post-cutoff periods fall in late 2024.Source: §4.6 · pp. 32–33
• The cutoff-gap rank correlation with pairwise agreement is ρ = 0.00 (p = 0.99), contrary to the negative relation a cutoff explanation predicts.OpenAI and Mistral's agreement is 0.62, no higher than their average agreement with Google, whose cutoff is fifteen months later.Source: §4.6 · p. 32
• DeepSeek's difference-in-differences estimate is +0.020 (t = 2.9), but its increase concentrates in subjective constructs rather than leakage-prone sentiment.Corporate culture (+0.075), quality (+0.067), and management clarity (+0.057) increase significantly; sentiment's interaction is -0.003 (t = -0.2).Source: §4.6 · p. 33
• After cutoff crossing, within-firm score variance ratios are 1.05 for DeepSeek and 0.94 for Qwen, near the controls' 1.01 and 1.02.The authors also report no visible sharpening of the sentiment–return link for models with access to realized outcomes.Source: §4.6 · p. 34
• The evidence points away from a cutoff explanation: sentiment shows no predicted post-cutoff decline, and overall disagreement does not track cutoff differences.The direct look-ahead test shows no predicted decline in sentiment's correlation with announcement returns after a model crosses its cutoff.Source: §4.6 · pp. 32, 34

5 Do Newer Models Converge?

• The comparison holds transcripts, prompts, scoring scales, and inference settings fixed, but its snapshot generalizability is limited by rapid releases and the cost of rescoring large corpora.The design measures agreement rather than correctness, except that the explicit counting rule makes specificity a partial case where movement toward the benchmark can plausibly indicate accuracy.Source: §5 · pp. 34–35, §0.08 · p. 36
• For sentiment, OpenAI and Google agreement rises only from 0.91 to 0.92, while correlations with the Loughran–McDonald dictionary remain low.The dictionary correlation is essentially flat across vintages: OpenAI 0.36 → 0.33 and Gemini 0.32 → 0.32.Source: §5 · p. 35, §0.32 · p. 35

0.32 \rightarrow 0.32). The dictionary, however, is one operationalization of tone, not ground truth, so

• For management clarity, cross-provider agreement rises modestly from 0.68 to 0.73, yet neither model moves toward the traditional evasive-language benchmark, whose correlation stays near 0.10 and is, if anything, slightly lower for newer models.Source: §0.32 · p. 35

0.08 \rightarrow 0.06; Gemini 0.14 \rightarrow 0.11). As with sentiment, the benchmark is an alternative

• Upgrading the models raises cross-provider agreement on specificity from 0.37 to 0.61, yet substantial disagreement remains even at the frontier.Specificity is the main analysis's least reliable construct; the newer models also move closer to the traditional benchmark.Source: §0.08 · pp. 35–36
• The newer models count specificity items more consistently, with item-count correlation increasing from 0.43 to 0.75, and more accurately estimate transcript length.Length estimates improve from a range of one-tenth to three times the true count to roughly half to one times the true count.Source: §0.08 · p. 36

6 Conclusion

• 0.52 average pairwise cross-model correlation and only about a third of score variation attributable to the text indicate substantial model-specific variation, including for Specificity Ratio.For several constructs, the model matters more than the firm; disagreement is among the largest for the explicitly specified Specificity Ratio.Source: §6 · p. 37
• Provider choice can change downstream inference: researchers using the same announcement-return regression could report a positive effect, a negative effect, or no effect.The conclusion warns that studies claiming to measure the same construct may therefore measure materially different things.Source: §6 · p. 37
• Averaging across providers recovers reliable call rankings for most measures, but remains a partial remedy because score levels are not reliably restored.Aggregation cancels model-specific noise, yet the rankings recovered do not make the levels dependable.Source: §6 · p. 38
• Upgrading to newer OpenAI and Google models raises agreement most for previously divergent constructs, but meaningful disagreement remains for every measure except sentiment.The newer models also do not generally move closer to established benchmarks; the specificity ratio is the exception, with the two frontier models correlating only 0.61.Source: §6 · p. 38
• Greater agreement between newer models generally does not bring them closer to established benchmarks; the specificity ratio is the exception, with the two frontier models correlating only 0.61.For the specificity ratio, higher agreement plausibly reflects movement toward the true value.Source: §6 · p. 38
• The estimates come from fixed-prompt API calls without external tools or extended reasoning, so they need not carry over to consumer chat products or agentic systems.The conclusion notes that added scaffolding could raise or lower agreement across models.Source: §6 · p. 38
• Researchers should disclose and ideally justify the specific model and version, routinely validate across providers, and prefer ensemble measures where feasible.The conclusion also calls for shared benchmarks, routine sensitivity reporting, and external validation against human-coded or otherwise anchored ground truth.Source: §6 · p. 39
• The paper calls for shared benchmarks, reporting sensitivity to model, version, and prompt, and external validation against human-coded or otherwise anchored ground truth.It cautions that model-induced variation may compound as LLMs take on more autonomous roles in research pipelines.Source: §6 · p. 39

Exhibits and Tables

• Figure 1 highlights the constructs with the highest and lowest mean pairwise provider correlations, while Table 2 summarizes pairwise Spearman correlations by construct.Figure 1 also reports each measure’s correlation with its corresponding traditional, non-LLM benchmark; the supplied caption gives no correlation values.Source: Figure 1 · p. 45, Table 2 · p. 47, Exhibits and Tables · p. 47
• The sample comprises 1,946 earnings calls from S&P 500 firms in 2024, scored on 13 constructs by seven LLM providers; Table 1 reports call-by-construct summary statistics.LLM variables are measured at the call-by-construct level, while firm-level and textual variables are measured once per call.Source: Exhibits and Tables · p. 46, Table 1 · p. 46
• Table 3 summarizes PCA-based cross-model commonality, with PC1 variance defined as cross-model agreement and higher values indicating stronger convergent validity.The PCA uses z-standardized LLM scores for each construct; no PC1 values are supplied.Source: Table 3 · p. 48, Exhibits and Tables · p. 48
• Table 4 presents a variance decomposition separating provider calibration from transcript ambiguity, estimated by construct using two-way analysis of variance.Its notes define generalizability and dependability coefficients for provider-level scores and model averages; the supplied passages provide no estimates.Source: Table 4 · p. 49, Exhibits and Tables · p. 49
• Tables 5–7 cover determinants of score dispersion, associations between LLM disagreement and post-call analyst or market disagreement, and links between LLM scores and market reactions.The Table 5 notes describe call-level OLS regressions; Table 6 notes specify disagreement outcomes, while the supplied Table 7 caption gives only its scope.Source: Table 5 · p. 50, Exhibits and Tables · pp. 50–51, Table 6 · p. 51

CAR regressions

• The CAR exhibit notes describe separate OLS regressions of Carhart 4-Factor CAR(0,1) on each provider’s z-scored construct, with firm controls, industry and quarter fixed effects, and firm-clustered standard errors.CAR coefficients represent changes in percentage-point CAR associated with a one-standard-deviation increase in the provider’s score; provider–construct series without variation are excluded, and non-LLM measures are standardized for within-column comparability.Source: Exhibits and Tables · p. 52

Table 8

• Table 8 examines knowledge-cutoff robustness across model cutoffs and calls before and after each cutoff.The sample comprises 1,946 S&P 500 calls scored in 2024, with cutoff dates taken from official documentation when available and otherwise from dated developer statements or the API gateway’s model catalogue.Source: Table 8 · p. 53, Exhibits and Tables · p. 53

• Post-call analyst count measures estimates in the first post-call consensus, while pre-call analyst dispersion and count use the last consensus within 60 days before the call.Source: Appendix Table · p. 56
• Forecast error is the absolute difference between post-call consensus EPS and realized EPS, scaled by the pre-call share price and multiplied by 100.Source: Appendix Table · p. 56
• Market outcomes include abnormal bid-ask spread, trading volume, and return variance over the announcement window, each relative to the firm's estimation-window baseline; textual measures include answer readability.Source: Appendix Table · p. 56
• Textual ratios cover word count, financial jargon, numeric tokens, forward-looking terms, and specific tokens, with the specific-token ratio serving as the non-LLM benchmark for LLM specificity.Source: Appendix Table · p. 56

• Hamid Boustanifar and Sasan Mansouri are listed.Source: LLM-Based Measures · p. 57

A Correlation Heatmaps

• The figure panels cover sentiment, uncertainty, specificity, and specificity ratio.They pair score correlations with textual similarity of score justifications.Source: A Correlation Heatmaps · pp. 58–59
• Additional heatmaps cover innovation, climate risk, political risk, and corporate culture.The panels pair score correlations with textual similarity of score justifications.Source: A Correlation Heatmaps · pp. 59–61
• The remaining figures cover management clarity, teamwork, quality, respect, and integrity scores.They use the same score-correlation and justification-similarity panel labels.Source: A Correlation Heatmaps · pp. 60, 62, 63, 64

B Summary Statistics --- Full Detail

• Table OA.B.1 reports summary statistics for variables used in the main analyses.The table title specifies full detail, including per-construct LLM scores.Source: B Summary Statistics --- Full Detail · pp. 65, 70
• LLM scores are z-standardized within task, while firm-level and textual variables repeat across the 13 constructs.The table notes distinguish call-level LLM-related variables from firm-level and textual variables.Source: B Summary Statistics --- Full Detail · pp. 65, 70
• The table pools the panel, while call-level statistics are reported in Table 1.Source: B Summary Statistics --- Full Detail · p. 70

C Dispersion Regression Tables

• Table OA.C.1 examines determinants of textual-measure dispersion using a range-based measure.Source: C Dispersion Regression Tables · p. 71
• Adjusted R2 values across the four reported columns are 0.196, 0.199, 0.199, and 0.217.Source: C Dispersion Regression Tables · p. 71
• The OLS regressions use 1,894 calls with non-missing controls across 13 constructs and include construct fixed effects.The notes also describe firm-clustered standard errors and control variables.Source: C Dispersion Regression Tables · p. 71

D Self-Reported Confidence

• 0.52 falls to 0.48 in mean pairwise Spearman correlation after screening at or above providers’ own medians, across 11 computable constructs.No construct improves meaningfully under this relative confidence cut.Source: D Self-Reported Confidence · p. 72
• For climate and political risk, high-confidence subsamples are degenerate because models are most confident when assigning zero scores.Almost no provider pair retains score variation in those subsamples.Source: D Self-Reported Confidence · p. 72
• 0.46 at a fixed 0.8 threshold, 0.35 above provider medians, and 0.33 at a top-quartile cut are mean correlations.The top-quartile result covers six computable constructs; the fixed 0.8 cut leaves all thirteen computable.Source: D Self-Reported Confidence · p. 72
• -0.08 is the mean Spearman correlation between a model’s confidence and its deviation from the seven-model consensus.The passage characterizes confidence as essentially uninformative about that deviation.Source: D Self-Reported Confidence · p. 72

E LLM Models

• All models receive identical prompts and decoding settings; six use OpenRouter, while GPT is queried directly through OpenAI’s native API.Source: E LLM Models · p. 74
• Table OA.E.1 lists the models used and identifies their provider, developer, family, model identifier, and access route.Source: E LLM Models · p. 74
• Seven models from independent developers score every earnings-call text, and the same set is held fixed across calls and constructs so disagreement reflects the models rather than differences in coverage.Source: E LLM Models · p. 74

F Prompt Templates

• Templates were used verbatim across providers, with { text} replaced by an earnings-call transcript segment at inference.Source: F Prompt Templates · p. 75

F.1 Sentiment

• Sentiment captures positivity versus negativity about firm performance, prospects, or conditions, scored from -1.00 (very negative) to +1.00 (very positive), with 0.00 neutral.Source: §F.1 · p. 75

F.2 Uncertainty

• Uncertainty measures ambiguity, imprecision, hedging, conditional language, or unresolved future outcomes—not outlook positivity—from 0.00 (very certain / precise) to 1.00 (very uncertain / ambiguous).Source: §F.2 · p. 76

F.3 Climate Risk

• Climate Risk Exposure measures textual presence and salience of physical, transition, or operational climate risks, from 0.00 (none mentioned) to 1.00 (major, explicit concern).Source: §F.3 · pp. 77–78

F.4 Management Clarity

• Management clarity reflects how clear, direct, specific, and internally consistent responses are, scored from 0.00 (very unclear / evasive) to 1.00 (very clear and direct).Source: §F.4 · p. 79

F.5 Political Risk

• Political Risk Exposure reflects textual presence and salience, not risk management quality, and requires a 0.00 score when no explicit political-risk reference appears.Source: §F.5 · p. 80

F.6 Corporate Culture

• The appendix introduces a prompt for quantifying corporate culture signals from a passage of text.Source: §F.6 · p. 81
• The corporate-culture prompt restricts scoring to explicit wording, bars external firm or industry knowledge, and requests scores with wording-based justification.It requests separate dimension scores, an overall score, and confidence.Source: §F.6 · pp. 81–82
• Corporate culture is defined through management-signaled norms and values, scored across innovation, integrity, quality, respect, teamwork, and an overall score from 0.00 to 1.00.Confidence concerns how accurately the scores reflect culture signals, not whether the culture is good.Source: §F.6 · pp. 81–82

F.7 Specificity Ratio

• Specificity Ratio defines specificity as specific words divided by total words, with the resulting score between 0.00 and 1.00.Specific words are drawn from seven categories; named entities and numeric expressions each count as one unit, and only explicit instances count.Source: §F.7 · pp. 83–84
• The Specificity Ratio prompt requires the number of specific words, the score, confidence, and a list of counted instances as outputs.Confidence measures consistency in applying the rules, not whether specificity is high or low.Source: §F.7 · p. 84

F.8 Specificity

• The separate Specificity prompt scores the overall prevalence of concrete, verifiable, precise elements, assigning higher scores to detail-rich text and lower scores to vague or abstract text.Its scale runs from 0.00 for very vague or detail-free text to 1.00 for text dense with concrete details.Source: §F.8 · pp. 85–86
• Specificity scores and confidence are separate outputs; confidence reflects certainty in judging specific elements, and the justification explains what was present or missing.Source: §F.8 · p. 86