PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Contextual Measurement for Recovering Individual and Group-Level Effects: Validation Against Survey Measures and an Occupational Application

Wenxin Jiang, Xuyang Wang, Yuxiao Wu

arXiv:2609.02821v1cs.AIcs.LG

TL;DR

Group-level AI scores cannot reveal within-group differences, raising the question of whether respondent-level AI measures can recover contextual associations as well as survey measures. AICOME decomposes AI measures into group means and individual deviations and tests contextual recovery; AI measures generally reproduce several observed dimensions’ model conclusions when rich information is available, with weekly hours a particularly clear case.

  • Problem

    Group-level AI scores support between-group comparisons but cannot reveal within-group heterogeneity, leaving open whether respondent-level AI measures recover individual- and group-level associations.

    Source: §1 · p. 2

  • Method

    AICOME treats survey and AI measures as alternative indicators of underlying concepts, decomposes respondent-level AI measures into group means and individual deviations, and validates contextual conclusions against observed measures.

    Source: §1 · p. 3

  • Results

    Across four CFPS dimensions, AI measures generally reproduce explanatory power, coefficient directions, and group and individual conclusions; contextual effects are harder to recover except when strong, as for weekly hours.

    Source: §7 · p. 29

  • Takeaways & Limitations

    AICOME is most defensible for recovering a limited number of theoretically important concepts from rich existing respondent information, rather than substituting for direct survey measurement.

    Source: §7 · p. 30

    Performance deteriorates sharply when AI inputs are restricted to occupation and basic demographics, and AI measures can miss subtleties that still require human judgment.

    Source: §7 · pp. 29–30

Abstract

from arXiv · show

Researchers increasingly use artificial intelligence to construct measures of social, organizational, and occupational characteristics that are absent from conventional surveys. We propose AICOME, AI COntextual MEasurement, a framework for evaluating whether AI-derived respondent-level measures can recover individual and group-level effects in contextual models. The key idea is that an AI measure constructed at the respondent level can be used to derive its group-level aggregate and its individual deviation, allowing researchers to estimate both between-group and within-group associations rather than treating AI measurement as response prediction alone. We validate the framework using the 2022 China Family Panel Studies (CFPS), where occupations provide the empirical grouping structure and several job-related survey variables provide validation benchmarks. For computer use, foreign-language use, weekly hours, and management responsibilities, we compare survey measures with AI-derived measures in response-level, model-level, contextual, and boundary-condition validations. The results show that AI contextual measurement can recover much of the contextual-model information contained in observed survey variables when rich respondent and job characteristics are available. Weekly hours provides the strongest validation case, with AI-derived measures reproducing the large negative between- and within-occupation associations with satisfaction observed in CFPS. The framework also identifies clear boundary conditions: performance deteriorates when information is restricted to occupation and basic demographics, and recovery is weaker when several related concepts are treated as simultaneously unobserved. The findings suggest that AICOME is most useful for recovering a limited number of theoretically important constructs from rich existing datasets.

1 Introduction

AICOME asks whether AI-derived respondent-level measures can preserve both group and individual contextual associations, rather than only predict survey responses. It decomposes AI scores into group means and individual deviations and evaluates whether their contextual conclusions resemble those from survey measures.

1 Introduction

  • Group-level AI scores can capture between-group differences but cannot show whether individuals within a group differ or whether those deviations matter for outcomes.Source: §1 · p. 2
  • AI survey prediction has focused on recovering missing or unasked responses, while AICOME asks whether AI measures also recover individual- and group-level associations.Source: §1 · p. 2
  • AICOME decomposes respondent-level AI measures into group means and individual deviations, allowing models to distinguish group-level from individual-level associations.Unlike a respondent-level measure, a standard group-level AI score has no within-group deviation and cannot identify within-group heterogeneity.Source: §1 · p. 2
  • Survey and AI measures are treated as alternative indicators of an underlying concept, not as the concept itself, and can differ in features, prompts, and response scales.Source: §1 · pp. 2–3
  • AICOME validation targets contextual inference, including explanatory power, coefficient directions, and group- and individual-level conclusions, rather than response-level agreement alone.Coefficient-vector correlations and RMSE are descriptive diagnostics, not requirements that AI measures numerically mimic survey measures.Source: §1 · p. 3
  • Both AI prompting strategies recover the large negative between- and within-occupation associations with satisfaction for weekly hours, while occupation-level measures miss the individual signal.Source: §1 · p. 3
  • AICOME performs best with rich respondent information and few missing concepts; performance deteriorates with sparse information or several related constructs missing simultaneously.The framework is not a substitute for direct surveys, and its latent-construct application is not direct proof of latent truth.Source: §1 · pp. 3–4

2 Related Literature and Positioning

Contextual models need measures that vary within groups as well as between them, while many occupational and AI measures assign a single score to each group. AICOME’s validation target is therefore whether respondent-level AI measures preserve substantive contextual inferences, not merely whether they predict survey responses.

2.1 Contextual models and occupational measurement

  • Group-level occupational and AI scores describe between-occupation differences but cannot identify within-occupation heterogeneity because every worker in an occupation receives the same value.The paper’s contribution is to construct respondent-level measures that can be decomposed into occupation means and within-occupation deviations.Source: §2.1 · p. 5

2.2 AI survey prediction, imputation, and synthetic respondents

  • Prior AI survey work primarily evaluates reproduced responses, opinions, attributes, or distributions; this paper instead evaluates recovery of explanatory power, coefficient directions, and substantive between-within conclusions.High response-level similarity need not imply valid downstream inference, since matching marginal averages can coexist with different variation or regression coefficients.Source: §2.2 · pp. 5–6

2.3 Construct validation, generated regressors, and measurement error

  • The validation problem is neither a classical generated-regressor problem nor simple measurement error: both survey and AI indicators may imperfectly measure an underlying job concept.The paper frames validation as construct validation for contextual use, rather than requiring AI and survey coefficients to be numerically equal.Source: §2.3 · p. 6
  • The proxy framework treats the unobserved concept as the target and relates an indicator’s explanatory power to similarity with the concept-based prediction signal, subject to a residual-uncorrelatedness assumption.Under that assumption, competing indicators with larger R² yield fitted values more correlated with those from the underlying concept.Source: §2.3 · pp. 6–7

3 AICOME Framework

AICOME constructs AI measures for individuals and derives group aggregates, enabling contextual analyses that direct group scores cannot support. The framework evaluates these measures through response-level, model-level, contextual, and boundary validations.

3.1 Group-level AI scores

  • Direct group-level AI scores can describe between-group outcome differences, but their within-group deviation is zero and they cannot test within-group heterogeneity.AICOME instead constructs respondent-level measures and derives group aggregates from them.Source: §3.1 · pp. 7–8

3.2 Respondent-level AI measures

  • Respondent-level AI indicators use respondent features and prompting protocols, then aggregate to group means for contextual models that distinguish individual, contextual, and group associations.In the CFPS application, the outcome is job satisfaction and occupations are the groups; βG = βI + βC.Source: §3.2 · p. 8
  • When the β coefficients share a sign, the contextual contrast is smaller in magnitude and harder to detect than the group effect, and its standard error may be larger.Reported coefficients are standardized so βG = βI + βC holds, with standard errors clustered at the occupation level.Source: §3.2 · pp. 8–9

3.3 Validation framework

  • Validation proceeds from correlations and non-contextual explanatory power and coefficient directions to contextual conclusions, then tests deterioration under reduced information and simultaneous missingness.Weekly hours is excluded from controls when it is the focal validated dimension and included otherwise.Source: §3.3 · p. 9, §3.2 · p. 9

4 Data and Measures

The empirical illustration uses 2022 CFPS data, with occupations as the contextual groups and observed work-related survey dimensions as validation benchmarks. It compares respondent-level AI measures under rich and survey-based prompts, while latent dimensions are assessed for robustness rather than truth recovery.

4.1 Data

  • The 2022 CFPS application uses job satisfaction as the outcome and occupation as the contextual unit, with rich respondent and job information supporting AI measurement and survey validation.The framework is presented as applicable beyond occupations to other settings where individuals are nested within larger social units.Source: §4.1 · p. 9

4.2 AI-derived measures

  • For computer use, foreign-language use, weekly hours, and management responsibilities, the study compares survey, rich-prompt AI, and survey-prompt AI measures in non-contextual and contextual validation.Autonomy, people-things, creative-routine, and technology are not directly observed, so their latent application compares prompt strategies rather than validating against survey truth.Source: §4.1 · pp. 9–10, §4.2 · p. 10

5 Validation Using CFPS-Measured Dimensions

Across five validation exercises, AI-derived measures recover much of the contextual-model information in observed CFPS measures, especially when rich respondent information is available. Recovery weakens with sparse information or simultaneous missingness of related constructs.

5.1 Correlation validation of AI-derived measures

  • Correlations between AI-derived and observed CFPS measures range from 0.684 to 0.986; management agreement is especially strong, while foreign language is most challenging.The survey-prompt foreign-language measure reaches 0.684 correlation with the observed measure in deviation form; these correlations are a preliminary benchmark, not the primary contextual validation target.Source: Table 1 · p. 10, §5.1 · p. 11

5.2 Sensitivity to available respondent information

  • Under occupation-and-demographics-only information, average raw-form correlations fall from 0.902 to 0.277 for rich prompts and from 0.879 to 0.251 for survey prompts.Weekly-hours correlation falls from approximately 0.9 to essentially zero; validation deteriorates sharply overall when respondent information is sparse.Source: Table 2 · p. 12, §5.2 · p. 12

5.4 Single-dimension contextual validation

  • In single-dimension contextual models, weekly-hours individual effects are −0.135 for CFPS, −0.136 for rich AI, and −0.129 for survey-prompt AI, with significantly negative contextual contrasts in all three.Management shows positive individual and group effects; computer effects are broadly positive, while foreign language has a weak individual effect and a stronger occupation-level component.Source: Table 4 · p. 13, §5.4 · p. 14

5.5 Joint contextual validation under simultaneous missingness

  • With all four dimensions jointly unobserved, adding the contextual terms raises R2 by 0.0269 for CFPS, 0.0142 for rich-prompt AI, and 0.0167 for survey-prompt AI; no joint-model contextual effect is significant.Weekly hours retains negative between-occupation and within-occupation associations, while management remains positively associated with satisfaction; smaller effects are less stable.Source: §5.5 · p. 16, Table 5 · p. 17
  • Across the eight focal coefficient vectors, deviation-form correlations with CFPS are 0.939 for rich-prompt AI and 0.878 for survey-prompt AI.In the raw parametrization, the corresponding correlations are 0.869 and 0.579, respectively.Source: §5.5 · p. 16, Table 6

6 Application to Latent Dimensions

This section applies prompt strategies to latent occupational dimensions in CFPS, where direct measures are unavailable, to assess whether they yield similar contextual conclusions and substantively plausible results. It examines contextual associations across prompts and model specifications, including People–Things, Autonomy, and Creative–Routine.

6.1.2 Diagnosing the role of time flexibility

  • With the original features, Autonomy closely tracked time flexibility: correlations were 0.955 (rich) and 0.898 (survey), and the flexibility category did not map straightforwardly onto satisfaction.The most flexible category had 3.681 mean satisfaction versus 3.807 for category 2; 42.7% were self-employed or in family businesses, compared with 6.4% in the other categories. Removing time flexibility reduced Autonomy correlations to 0.250 and 0.213 and Creative–Routine correlations to 0.116 and 0.098.Source: §6.1.1 · p. 20, §6.1.2 · p. 20

6.1.3 Autonomy: Results after measurement repair

  • After removing time flexibility, Autonomy showed positive group effects of 0.062 and 0.054 and positive individual effects of 0.047 and 0.054, but no detectable contextual effect.Incremental R2 rose from zero to approximately 0.0024 to 0.0025; the diagnosis showed how outcome analysis and feature inspection could reveal and repair a measurement anomaly.Source: §6.1.3 · p. 21

6.1.4 Face validity for FNoTimeFlex AI measures

  • Ranked occupation means showed generally reasonable, imperfect face validity across all four latent directions under the two prompt designs.Examples included artists and researchers toward the high-Autonomy end, creative occupations toward the creative end, service occupations toward the people-oriented end, and engineering occupations toward the high-Technology end; rankings cannot establish criterion validity.Source: §6.1.4 · pp. 21–22

6.2.4 Summary across the four single-dimension analyses

  • Across single-dimension models, People–Things and revised Autonomy had the clearest positive group and individual associations, while contextual effects were not conclusively established.Technology’s group evidence depended on prompt design, and Creative–Routine’s opposing components and small fit gains left its contextual pattern difficult to interpret.Source: §6.2.4 · pp. 23–24

6.3.1 People–Things

  • People–Things retained the strongest joint group association: 0.131 under the rich prompt and 0.119 under the survey prompt, with positive individual effects but less robust contextual evidence.Its group association remained positive after adjustment for the other dimensions and was not attenuated relative to the single-dimension estimates.Source: §6.3.1 · pp. 24, 26, Table 8 · p. 25

6.3.2 Technology

  • Technology showed positive joint group effects under both prompts, at 0.082 (rich) and 0.117 (survey), while evidence for individual and distinct contextual effects was less consistent.The joint analysis strengthened the group-effect evidence relative to the single-dimension models, but contextual evidence remained more visible under the survey prompt.Source: §6.3.2 · p. 26

6.3.3 Autonomy

  • In joint models, Autonomy’s positive individual effects persisted, but its group coefficients fell to −0.007 and −0.026, indicating no independent positive group association.The positive single-dimension group association was shared with the other latent dimensions; the implied negative contextual estimates were too imprecise for a strong substantive claim.Source: §6.3.3 · pp. 26–27

6.3.4 Creative–Routine

  • Creative–Routine had no independent joint group association under either prompt, and its conflicting individual and contextual estimates did not support a stable satisfaction relationship.The rich-prompt contextual estimate of 0.060 (SE 0.038) coexisted with a negative individual estimate of −0.087 (SE 0.027); the survey-prompt contextual estimate was 0.004 (SE 0.038).Source: §6.3.4 · p. 27
  • Creative–Routine has no stable positive association with job satisfaction after related occupational characteristics are controlled, despite strong face validity and recognizable occupation contrasts.Its single-dimension model captures shared variation with the other dimensions, while its joint model isolates the narrower component orthogonal to them; substantial overlap can shift its coefficient.Source: §6.3.4 · pp. 27–28

6.3.5 Consistency across prompting strategies

  • Rich- and survey-prompt coefficient agreement is higher for group-plus-individual coefficients (correlation 0.864) than contextual-plus-individual coefficients (correlation 0.721).RMSE and mean absolute difference are also lower for the group-plus-individual representation.Source: Table 9 · p. 28, §6.3.5 · p. 28
  • Contextual-effect estimates are more sensitive to prompt differences than gross group effects, so the contextual decomposition warrants greater caution.The gross group effects are generally larger and more stable; contextual effects separate group from individual associations.Source: §6.3.5 · p. 28
  • Despite coefficient differences, rich- and survey-prompt latent-component predicted satisfaction correlates 0.823, and full fitted values correlate 0.967.Prompt variation affects allocation among latent dimensions, especially contextual components, more than overall fitted outcomes.Source: §6.3.5 · pp. 28–29, Table 9 · p. 28

6.3.6 Findings that remain consistent across single and joint models

  • Across both prompts and single- and joint-dimension models, People–Things has the most robust positive group association, while repaired Autonomy consistently has a positive individual association.The paper interprets these as occupational differences in People–Things and within-occupation differences in Autonomy being associated with job satisfaction.Source: §6.3.6 · p. 29

7 Discussion

AICOME recovers contextual associations from AI-derived measures most reliably when rich respondent and job information is available and the regression signal is strong. Its performance weakens with restricted inputs or several jointly unobserved related concepts, while latent-dimension findings vary between single and joint models.

7 Discussion

  • Across four CFPS dimensions, AI-derived measures generally reproduce observed measures’ explanatory power, coefficient directions, and group- and individual-level conclusions when rich information is available and signals are strong.The contextual effect is harder to detect consistently across prompts, but both strategies recover the large negative effect for weekly hours.Source: §7 · p. 29
  • Restricting AI inputs to occupation and basic demographics sharply degrades validation, whereas rich respondent and job features let measures closely track observed survey variables.AICOME draws on information already in the dataset; it does not create information from nothing.Source: §7 · p. 29
  • When several related dimensions are simultaneously unobserved, AI measures retain broad qualitative features but show reduced explanatory power and attenuated or unstable coefficients.The less demanding case—recovering a limited number of concepts while other respondent characteristics remain available—can perform well.Source: §7 · p. 29
  • Latent-dimension results differ across model forms: People-things has a positive occupation-level association with satisfaction, while revised Autonomy has a positive within-occupation association.Single-dimension latent contextual models do not necessarily agree with joint analyses.Source: §7 · pp. 29–30
  • AI measures are not substitutes for direct measurement or ground truth, and satisfactory construction may still require human judgment and correction.Agreement is not guaranteed, and the study tests only two prompting strategies rather than all model, prompt, or time-based variation.Source: §7 · p. 30

Generative AI Disclosure

The authors used language models to generate respondent-level contextual measures through a predefined prompting and scoring procedure, and reviewed AI-generated outputs before taking responsibility for the manuscript.

Generative AI Disclosure

  • Language models generated respondent-level contextual measures from survey and occupational information using a predefined prompting and scoring procedure.The authors reviewed, verified, and revised AI-generated outputs as necessary.Source: Generative AI Disclosure · p. 30
  • Xuyang Wang used LLM (GPT-5.6 Sol) during the research.Source: Generative AI Disclosure · p. 30
  • Wenxin Jiang used generative AI tools, primarily Microsoft Copilot, for coding assistance, drafting and editing, brainstorming, and reference checking.The authors take responsibility for the manuscript’s accuracy, interpretation, and content.Source: Generative AI Disclosure · p. 30

A Variables and Analytic Roles

This appendix defines the analytic roles and controls, target concepts, prompt styles and scales, and feature-set and exclusion procedures used to construct and analyze respondent-level AI measures.

A Variables and Analytic Roles

  • Regression controls are distinct from AI features, which determine information supplied to the model; weekly hours is omitted from controls when it is the focal construct.Occupation titles, rather than numeric codes, serve as the primary contextual signal when available.Source: A Variables and Analytic Roles · pp. 32–33

B AI Feature Sets and Prompt Protocols

B.1 Target Concepts

  • Eight job-related concepts are considered; the first four have CFPS survey counterparts for validation, while the remaining four are latent dimensions.The validated concepts are computer use, foreign-language use, weekly hours, and management responsibility; the latent concepts are autonomy, people orientation, creative work, and technology intensity.Source: §B.1 · p. 33

B.2 Prompt Styles

  • Each concept receives rich-construct and survey-style prompt versions, whose agreement provides evidence about robustness.Rich prompts describe the broader construct; survey prompts predict how a respondent would answer a survey question.Source: §B.2 · pp. 33–34

B.3 Response Scales

  • Most concepts use five-point ratings, whereas weekly hours is estimated as average weekly work hours and analyzed continuously.Source: §B.3 · p. 34

B.4 Feature Sets and Exclusion Rules

  • The reduced feature set retains occupation and basic demographics, while exclusions are concept-specific; the AI4 specification applies the union of exclusions across all four validated prediction tasks.The reduced set removes essentially all job-specific respondent information beyond occupation membership and basic demographics.Source: §B.4 · pp. 34–35

B.5 Concept-Specific Feature Sets

  • Table 10 codes whether variables are included as features, excluded for leakage protection, or unused; its AI4 column marks the unioned exclusion set.Source: §B.5 · p. 35

C CFPS Validation Measures

The validation uses four observed CFPS work-related measures corresponding to the first four AI target concepts, each drawn from an occupational-module survey question.

C CFPS Validation Measures

C.1 Computer Use

This section defines CFPS survey benchmarks and documents the prompts, information conditions, processing rules, and common variable-cleaning and input-construction procedures used to generate respondent-level AI measures. It contrasts broad construct prompts with survey-answer prompts and specifies designs that remove target information or restrict model inputs; the supplied passages report no computer-use validation results.

C.1 Computer Use–C.4 Management Responsibility

  • The four observed benchmarks cover computer use, foreign-language use, weekly work hours, and management responsibility.Management duty is defined as an organizational position with an official management function.Source: §C.1 · p. 35, §C.2 · p. 37, §C.3 · p. 37

D Prompt Definitions and Prompt Templates

E.1 Main AI1 Template

  • The main AI1 template supplied concept-specific respondent profiles, included occupation when available, and instructed the model to assess job characteristics rather than satisfaction or well-being.Weekly hours was requested as a number of hours per week, not a 1–5 score.Source: E Prompt Templates · p. 40, §E.1 · p. 41

E.2 AI4 Simultaneous-Missingness Template

  • AI4 treated computer use, foreign-language use, weekly hours, and management responsibility as simultaneously unavailable, removing their excluded variables from all profile blocks.The model was instructed to generate all four measures from remaining respondent information.Source: §E.2 · pp. 41–42

E.3 Reduced-Information AI4 Template

  • The reduced-information AI4 analysis supplied only occupation and basic demographics, omitting job-specific and other detailed respondent information to assess dependence on richer inputs.The supplied set included age, gender, education degree, marital status, and urban residence.Source: §E.3 · p. 42

E.4 Common Processing Rules–E.5 Common Cleaning Rules

  • Before analysis, weekly hours was generated numerically and other concepts on five-point scales; CFPS values were cleaned by handling missing codes, recoding binary items, and removing invalid occupation codes.Categorical schedule variables were converted to human-readable text before prompting.Source: §E.4 · p. 42, §E.5 · p. 42

E.5 Common Cleaning Rules

  • Several variables are represented as binary indicators, including a yes/no measure coded 1 for yes and 0 for no.The passages also describe human-readable versions of categorical inputs.Source: §E.5 · p. 43
  • For nonnegative income, regressions use log(1 + income), while the language model receives the original income value.Source: §E.5 · p. 43
  • Work-location province uses a valid alternative province code when it differs from residence, subject to specified mainland-China and family-agriculture rules.Otherwise, a valid residential province is used; the variable is not used in regressions.Source: §E.5 · p. 45

F Occupation-Level Comparison Methods

The section describes two occupation-title-only benchmarks for comparison with respondent-level AI measures. Both assign the same score to all workers in an occupation, but use different methods to derive scores.

F Occupation-Level Comparison Methods

F.1 Occupation-Level AI Scores

  • Direct AI scoring rates typical occupational characteristics from the title alone, using five-point scores for ordinal concepts and an estimated typical weekly-hours value.The prompts exclude job satisfaction and respondent-level characteristics; scores are merged to individuals by occupation code.Source: §F.1 · p. 45

F.2 Occupation Embedding Scores

  • The embedding benchmark represents each occupation’s typical work content, responsibilities, and setting, then scores it by projection onto semantic directions for each concept.Higher projections indicate greater similarity to the direction’s positive pole; workers sharing an occupation receive the same score.Source: §F.2 · p. 46

F.3 Relation to Respondent-Level AI Measures

Unlike occupation-level benchmarks, the respondent-level measures combine occupation information with individual characteristics, allowing workers in the same occupation to receive different predicted scores.

F.3 Relation to Respondent-Level AI Measures

  • The respondent-level measures combine occupation information with the specific characteristics of each respondent.Source: §F.3 · p. 46
  • The respondent-level approach uses respondent-specific feature sets Fk, unlike benchmarks based only on occupation titles.Source: §F.3 · p. 46
  • Consequently, workers within the same occupation can receive different respondent-level predicted scores.Source: §F.3 · p. 46
Loading 2609.02821v1…