PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Bears the Risk When Generative AI Enters Transport? A Distributional Sociotechnical Audit of Algorithmic Equity, Synthetic-Data Validity, and Public Trust

Amir Rafe, Subasish Das

arXiv:2609.11611v1cs.CY

TL;DR

Transport GenAI governance lacks transport-specific tools for measuring distributional risks across populations. This paper combines audits of LLM advice, synthetic crash data, and public attitudes in a Distributional Sociotechnical Audit. The audits find demographic variation across signals, with categorical risk tiers sensitive to weight perturbation.

  • Problem

    Transport governance lacks transport-specific statistical tools for assessing distributional risks across heterogeneous populations as GenAI enters transportation.

    Source: §1 · pp. 1–2

  • Method

    The DSA combines persona-controlled LLM advice audits, conditional tests of synthetic FARS records, Bayesian analysis of Pew AI attitudes, and a continuous Sociotechnical Risk Index.

    Source: §8 · p. 21

  • Results

    The audits find demographic variation in transport advice and AI attitudes, CART failures on tested safety variables, borderline conditional stress for the Gaussian copula despite marginal checks, and no universally low-risk model-topic combination.

    Source: §7 · pp. 20–21, §8 · p. 22

  • Takeaways & Limitations

    The authors present continuous, sensitivity-reported risk signals as more defensible governance outputs than categorical tiers, whose assignments are fragile under weight perturbation.

    Source: §5.4 · p. 13, §7 · p. 21

    The three datasets are aligned structurally rather than at the individual level, so the STRI demonstrates integration rather than tracking the same population units across data streams.

    Source: §3 · p. 7

Abstract

from arXiv · show

Generative artificial intelligence is entering transportation through traveler-facing advisories, synthetic crash-record generation, and policy decision support. Existing governance frameworks lack transport-specific statistical tools to measure distributional risks across heterogeneous populations. We develop a Distributional Sociotechnical Audit (DSA) that integrates algorithmic equity, synthetic-data validity, and public-attitude heterogeneity into one empirical pipeline. The audit analyzes 5,760 persona-controlled queries to four LLM families across 12 demographic cues and four transport topics, uses two cross-family judges and a Wasserstein-2 Equity Dispersion Index, tests three FARS crash-record generators with conditional projected maximum mean discrepancy (cpMMD), fits a Bayesian ordered-logit model to Pew American Trends Panel Wave 152 (N = 4,538), and combines the signals into a continuous Sociotechnical Risk Index. Congestion-pricing advice has the highest persona-based dispersion (mean EDI = 1.96; highest direct EDI = 2.20). CART synthetic crash records fail all conditional tests (p < 0.001), while the Gaussian copula has borderline conditional stress (p = 0.105) despite passing marginal checks. Attitudes to AI vary across demographic strata. Distributional audits and continuous risk indices with sensitivity reporting offer a more defensible basis for transport GenAI governance than categorical approval tiers, which show a 75% assignment flip rate under weight perturbation.

1. Introduction

Transport GenAI raises a governance question that aggregate performance metrics cannot answer: whether advice, synthetic data, and public attitudes vary across demographic groups. The paper introduces an integrated Distributional Sociotechnical Audit (DSA) to examine these dimensions and combine them into a governance-facing risk measure.

1. Introduction

  • Prior transport GenAI work emphasizes task performance, while governance instruments lack transport-specific statistical audit protocols and equity studies often rely on sentiment or keyword comparisons.The paper also notes that prior work does not connect model-output equity to synthetic-data validity or public-attitude heterogeneity.Source: §1 · p. 2
  • No transport study had integrated algorithmic equity, synthetic-data validity, and public-attitude heterogeneity into an empirically implemented audit.The paper frames this as a governance gap because assessing any one signal in isolation understates compound sociotechnical risk.Source: §1 · p. 2
  • The study asks whether persona cues shift LLM advice, synthetic crash records preserve FARS conditional structure, AI attitudes vary across U.S. demographic strata, and these signals can compose into a continuous risk index.A fifth, explicitly exploratory and non-causal question examines whether self-reported GenAI use is associated with general AI attitudes.Source: §1 · p. 2
  • The DSA contributes a persona-controlled LLM audit, conditional FARS synthetic-data validation, Bayesian estimates of public-attitude heterogeneity, and a continuous Sociotechnical Risk Index with sensitivity reporting.The authors characterize their claims as distributional, diagnostic, and descriptive rather than causal.Source: §1 · pp. 2–3

2. Literature Review

The literature review identifies a gap between transport GenAI’s expanding capabilities and the distributional evidence needed for governance. It connects transport applications, equity auditing, synthetic-data validation, and public attitudes as foundations for an integrated audit.

2.1. Generative AI in transport and the governance gap

  • Transport generative models support prediction, simulation, and language-mediated decision support, but evaluations mainly assess task capability rather than comparable advice, conditional data fidelity, or trust variation.Governance frameworks address institutional risks, while transport-specific statistical protocols for distributional GenAI risks remain lacking.Source: §2.1 · pp. 3–4

2.2. Algorithmic equity, transport justice, and distributional auditing

  • Transport-justice scholarship supplies normative concerns about accessibility and recognition, while distributional auditing offers a way to compare whole output distributions across demographic signals.The review identifies comparison of averages, keywords, or rank positions as an incomplete substitute for full distributional comparison.Source: §2.2 · p. 4

2.3. Synthetic data validity, public attitudes, and the integration gap

  • Synthetic-data checks must assess conditional structure beyond marginal fidelity, while public-attitude research supports descriptive demographic comparisons but not strong causal claims about GenAI exposure.These evidence streams, along with model-output equity, are presented as separate literatures that the DSA integrates.Source: §2.3 · pp. 4–5

2. Literature Review

  • Across the reviewed streams, transport studies emphasize task performance, governance work offers few transport-specific audit instruments, and equity, synthetic-data, and public-attitude analyses remain separate.The DSA responds by unifying model-output equity, synthetic-data validity, and public-attitude heterogeneity in one empirical pipeline.Source: §2.3 · p. 5

3. Conceptual Framework

The DSA frames transport GenAI risk as layered and operationalizes that framework with distributional statistics across model advice, synthetic crash data, public attitudes, and a governance synthesis. Its crosswalk links each layer to data, statistics, inference procedures, and research questions.

3. Conceptual Framework

  • The DSA organizes transport GenAI risk into layers and operationalizes a sociotechnical perspective through distributional statistics.Figure 1 presents the pipeline, while Table 1 crosswalks each layer to its data source, statistic, inference procedure, and research question.Source: §3 · p. 5, Figure 1 · p. 6, Table 1 · p. 6
  • The algorithmic-equity layer uses 5,760 responses from four model families, 12 persona cues, and four transport topics to measure distributional dispersion across persona conditions.Two cross-family judges score responses on eight content axes; the Wasserstein-2 EDI captures differences in location, spread, and shape.Source: §3 · pp. 5–6
  • The synthetic-data layer compares three generators with authentic FARS data using cpMMD on the full feature set and four safety-relevant marginal variables, conditioning on crash year.A perturbation baseline serves as a calibration control, and the framework designates this layer L0 because it treats data fidelity as logically prior to downstream audits.Source: §3 · pp. 6–7
  • The public-attitude layer uses a Bayesian ordered-logit model with horseshoe shrinkage to estimate stratum-level AI-attitude effects across age, gender, metropolitan status, and race.It controls for education, income, region, political ideology, and internet use, producing a heterogeneity vector rather than a single aggregate readiness score.Source: §3 · p. 7
  • The interface layer is held constant across LLM queries, limiting generalizability to the specified prompt structure; the cross-dataset alignment is structural, not individual-level.The authors describe the STRI as a demonstration composite, not a claim that the same population units are observed across all three data streams.Source: §3 · p. 7
  • The STRI combines normalized equity, synthetic-data, and attitude signals, while Weyl perturbation bands assess sensitivity to component-weight changes.Illustrative RRL tiers are tested for stability; the survey association between GenAI use and attitudes is explicitly non-causal.Source: §3 · p. 7

4. Data

The data section draws on an original LLM audit corpus, a Pew survey, and federal crash records. It describes the construction and scope of each data stream.

4. Data

  • The audit combines an original LLM corpus, a licensed Pew survey, and publicly available federal crash records.Table 2 summarizes their provenance, feature architecture, and quality metrics.Source: §4 · p. 7

4.1. LLM audit corpus

  • 5,760 queries crossed four model families, 12 persona profiles, and four transport topics, with 30 repeated queries per combination.The Plackett-Burman L12 design balanced main-effect coverage across age, gender signal, geographic context, and race/ethnicity proxy; model selection prioritized diversity rather than maximum capability.Source: §4.1 · pp. 8–9
  • Two independent cross-family judges scored responses on eight axes, but agreement ranged from moderate to negative across axes.Both judges preserved relative model and persona rankings on positively agreeing axes; results for technical detail and cost mention were caveated.Source: §4.1 · p. 9

4.2. Pew American Trends Panel Wave 152

  • The Pew Wave 152 analysis used a weighted sample of 4,538 U.S. adults and modeled expectations about AI’s impact over the next 20 years.Demographic stratification defined 84 cells; 32 had at least 30 respondents, with horseshoe shrinkage priors used for the smaller cells.Source: §4.2 · p. 9
  • Pew’s modeled items measure general AI attitudes, not transport-specific GenAI acceptance, so their governance-readiness connection is descriptive.Driving frequency, dangerous-driving perceptions, and road-rage experience were available but were not modeled outcomes.Source: §4.2 · p. 9

4.3. FARS audit substrate

  • The FARS audit retained 110,001 fatal-crash records from 2020, 2023, and 2024 after excluding 2021 and 2022 over a state-field header inconsistency.The source is a census of fatal motor-vehicle crashes on U.S. public roads, maintained by NHTSA.Source: §4.3 · p. 10
  • Thirteen FARS variables entered the audit, with four safety-relevant targets tested conditionally on crash year.The targets were weather, light condition, rural-urban classification, and maximum injury severity.Source: §4.3 · p. 10
  • Three synthetic datasets matched the real-data size; the Gaussian copula had under 0.4% marginal deviation, while CART had 13–35% deviation from mode collapse.The calibrated-noise perturbation baseline preserved conditional structure by construction and served as a cpMMD calibration control.Source: §4.3 · p. 10

5. Methods

The methods estimate persona-level output dispersion, year-conditional synthetic-data mismatch, and demographic variation in public AI attitudes, then combine these signals in a continuous risk index. An exploratory model separately examines the association between GenAI use and AI attitudes.

5.1. Wasserstein Equity Dispersion Index

  • Direct empirical EDI averages pairwise Wasserstein-2 distances across personas; the Gaussian/Bures approximation underestimates exact W2 by about 47%, so rankings use direct EDI.Uncertainty is quantified with 500 within-persona bootstrap resamples per cell.Source: §5.1 · p. 11

5.2. Conditional Projected MMD Synthetic-Data Audit

  • The full cpMMD evaluates distributional equivalence within crash-year strata across three years, alongside four safety-variable marginal tests; 200 permutations warrant caution for borderline p-values.The marginal variables are WEATHER, LGT_COND, RUR_URB, and MAX_INJ.Source: §5.2 · p. 12

5.3. Bayesian Ordered-Logit Model of Public Attitudes

  • The Bayesian ordered-logit model estimates AI-impact attitude heterogeneity across 84 demographic strata, adjusting for fixed covariates and incorporating Pew survey weights in the likelihood.The primary output is the magnitude of stratum-specific effects.Source: §5.3 · p. 12

5.4. Sociotechnical Risk Index and Regulatory Readiness

  • STRI combines normalized equity, FARS-cpMMD, and attitude-heterogeneity signals, structurally aligned across datasets without individual linkage; RRL tiers remain illustrative given high weight-perturbation flip rates.The primary analysis uses uniform component weights.Source: §5.4 · pp. 12–13

5.5. Exploratory Exposure-Attitude Association

  • The exposure-attitude analysis is an exploratory, non-causal ordered-logit association with demographic controls; Pew weights were not incorporated, so estimates are unweighted.Cross-sectional data cannot distinguish whether GenAI use shapes attitudes or prior attitudes predict adoption.Source: §5.5 · p. 13

6. Results

The results show substantial variation in transport-advice disparities, synthetic crash-record validity, and AI attitudes across demographic strata. The integrated STRI varies by topic-model cell, while illustrative regulatory tiers are unstable under weight perturbation.

6.1. LLM distributional equity

  • Gemini 3.1 Flash Lite and Claude Haiku 4.5 have the highest mean direct EDI values (1.813 and 1.810), while congestion pricing has the highest mean EDI (1.476 by lemma).The highest direct EDI cells are Gemini Flash at 2.204 and Claude Haiku at 2.107 for congestion pricing.Source: §6.1 · pp. 13–14, Table 4 · p. 14, Figure 2 · p. 15
  • Persona-output dispersion varies across models and topics, with the Gaussian decomposition indicating mean-content shifts in the highest-EDI cells.In lower-EDI Mistral Nemo weather cells, shape contributes a larger relative share; the Gaussian approximation has a 47% median gap.Source: §6.1 · pp. 13–14
  • Persona cues correspond to differences in equity language and place specificity, but low agreement on equity language (α = 0.133) warrants cautious interpretation.Minority and nonbinary cues receive higher equity-language scores, while rural White senior personas receive more place-specific advice.Source: §6.1 · p. 14, Table 5 · p. 15, Figure 3 · p. 16

6.2. Synthetic-data validity

  • CART rejects conditional and all four safety-relevant marginal tests (p < 0.001), while the copula has borderline conditional stress (p = 0.105) despite passing marginal tests.The perturbation baseline fails to reject the conditional null (p = 0.825), serving as a calibration control.Source: §6.2 · pp. 15–16, Figure 5 · p. 18

6.3. Public-attitude landscape

  • AI-attitude stratum effects range from 0.002 to 0.860, with the largest tending to intersect younger age, higher education, and metropolitan residence.The authors conclude that demographic cells occupy meaningfully different positions on the AI-impact expectation scale.Source: §6.3 · pp. 16–17

6.4. STRI composite and regulatory readiness

  • STRI ranges from 1.071 to 1.437 across cells, while RRL tiers flip 75.1% under weight perturbation.Weather driving has the highest topic mean STRI (1.297), and bicycle/pedestrian safety the lowest (1.176).Source: §6.4 · pp. 17–18, Figure 6 · p. 19

6.5. Exploratory exposure-attitude association

  • Self-reported GenAI use is associated with more positive expected AI impact, but the cross-sectional analysis cannot distinguish treatment effects from selection effects.The reported association is adjusted for observed demographic and other listed covariates.Source: §6.5 · p. 19

6.6. Self-check diagnostics

  • Automated checks pass Bayesian convergence and STRI Weyl-band positivity but fail the Gaussian-approximation-gap and RRL-tier-robustness thresholds.The authors address the approximation gap with exact W2 comparisons and treat RRL tiers as illustrative rather than prescriptive.Source: §6.6 · p. 19

7. Discussion

The discussion finds distributional differences across transport advice, synthetic crash data, and public AI attitudes, then combines these signals in a governance-facing risk index. It argues that conditional validation and continuous risk reporting are more informative than marginal checks or categorical tiers alone.

Algorithmic equity

  • Persona cues produce systematically different transport advice, with more equity-framed and less place-specific content for minority personas and greatest dispersion on congestion pricing.The discussion attributes much of high-EDI dispersion to shifts in mean advice content, while noting that whether such differences are desirable depends on the governance standard.Source: §7 · p. 20

Synthetic-data validity

  • CART synthetic records show mode collapse across safety variables, while the copula passes marginal checks but shows borderline full conditional stress (p = 0.105).The discussion argues that conditional distributional validation can reveal distortions that marginal comparisons miss.Source: §7 · p. 20

Public attitudes

  • AI attitudes vary across demographic strata in a nationally representative U.S. sample, but the Pew outcome measures general AI impact expectations rather than transport-specific acceptance.The discussion treats these attitudes as a governance-relevant landscape measure and identifies targeted transport surveys as a route to domain-specific evidence.Source: §7 · pp. 20–21

Governance synthesis

  • The STRI integrates equity, synthetic-data validity, and attitude signals, but its components align structurally rather than at the individual-record level.Because attitude and synthetic-data components are topic-agnostic, cross-topic STRI variation is driven almost entirely by equity; categorical tier assignments also flip 75% under weight perturbation.Source: §7 · p. 21

Policy recommendations

  • The paper recommends persona-based testing, conditional validation, demographic treatment of public trust, and continuous risk scores with transparent sensitivity analysis.These are presented as evidence-informed governance practices, not fixed regulatory thresholds.Source: §7 · p. 21

8. Conclusion

The conclusion reports distributional variation in transport advice, conditional weaknesses in synthetic crash records, and heterogeneity in general AI attitudes, integrated through a continuous risk index. It frames the audit as evidence for governance deliberation, while bounding the scope of its measures and claims.

8. Conclusion

  • The audit found persona-linked variation in transport advice, conditional-test failures for CART, borderline conditional stress for the Gaussian copula, and demographic heterogeneity in general AI attitudes.The STRI found no universally low-risk model-topic combination, and categorical tier assignments were fragile under weight perturbation.Source: §8 · pp. 21–22
  • Persona cues are signal injections rather than real identities, and the Pew outcome measures general AI attitudes rather than transport-specific GenAI acceptance.The LLM results may not generalize to future model releases, and the cross-dataset STRI alignment is structural rather than record-level.Source: §8 · p. 22
  • Future work includes longitudinal audits across model versions, human-rater validation, expanded synthetic-data diagnostics, and transport-specific attitude measurement.The paper also recommends stakeholder validation of STRI component weights before adopting a categorical readiness ladder.Source: §8 · p. 22
  • The paper presents the DSA as a reproducible audit framework that informs, but does not replace, governance deliberation.It does not prove causal harm or certify a regulatory deployment threshold.Source: §8 · p. 22

CRediT authorship contribution statement

The authors divide the work across conceptualization, methodology, analysis, software, supervision, visualization, and writing.

CRediT authorship contribution statement

  • Amir Rafe contributed conceptualization, methodology, software, formal analysis, original-draft writing, and visualization; Subasish Das contributed conceptualization, methodology, supervision, and review-and-editing.Source: CRediT authorship contribution statement · p. 22

A. Audit Instrument and Reproducibility Details

The appendix documents how the audit prompts, persona profiles, judging rubric, and model access were specified for reproducibility. It describes a fixed query template, a fractional-factorial persona design, and blinded cross-family scoring.

A. Audit Instrument and Reproducibility Details

  • The appendix documents the full audit instrument, model specifications, and technical settings needed to reproduce the empirical pipeline.Source: A. Audit Instrument and… · p. 22

A.1. LLM query prompt template

  • Each of 5,760 queries combined a fixed system prompt with a personalized message and one of four transport-topic question stems.The prompts used a 4–8 sentence response instruction, temperature 0.7, a 1,024-token maximum, and default provider safety configurations.Source: §A.1 · pp. 22–23

A.2. Persona cue profiles

  • The 12 persona profiles used a Plackett-Burman L12 fractional factorial design crossing four demographic dimensions.The appendix defines “Minority” as a rotated composite non-White racial/ethnic cue and signals “Nonbinary” with they/them pronouns.Source: §A.2 · p. 23, §A.1 · p. 23, Table A.1 · p. 23

A.3. LLM-as-judge rubric

  • Each judge model received a structured scoring prompt for every response, with a rubric specifying eight content axes, each rated on an integer scale from 0 to 4.The supplied axis descriptions cover caution, risk acknowledgment, equity language, technical detail, and actionability.Source: §A.3 · p. 23

A.4. Model versions and access

  • Models were accessed through OpenRouter during April and May 2026, with exact identifiers, roles, and coverage documented in Table A.2.The appendix reports 11,448 total scored rows after the 72 missing Qwen scores.Source: §A.4 · p. 24, Table A.2 · p. 24, §A.3 · p. 24

A.5. Synthetic-data generator specifications

The section specifies three approaches to generating synthetic FARS records and the settings used to test their conditional distributions.

A.5. Synthetic-data generator specifications

  • The Gaussian copula maps correlated uniform variates to empirical quantiles, sequential CART conditions each generated variable on prior columns, and the perturbation baseline adds Gaussian noise to real records.Each synthetic dataset contains 110,001 records; the perturbation baseline uses ε = 0.3 standard deviations and rounds to valid categories.Source: §A.5 · p. 24

A.6. cpMMD technical settings

  • cpMMD uses a Gaussian radial basis function kernel with median-heuristic bandwidth, standardized features, and up to 1,000 records per source per year stratum.The full conditional test uses 200 global source-label permutations; each marginal test uses 100.Source: §A.6 · p. 24

A.7. Bayesian model specification

The Bayesian ordered-logit specification uses regularized covariate effects, while the governance checks identify limits to categorical risk tiers and make continuous STRI the primary output.

A.7. Bayesian model specification

  • The model uses a non-centered horseshoe prior for fixed covariate effects and ordered cutpoints initialized at equal-quantile spacings.Covariates are dummy-coded into 22 columns; the implementation uses PyMC 5.x with NUTS.Source: §A.7 · p. 24
  • Check 1 fails because ordinal rubric scores violate the Gaussian assumption underlying Lemma 5.1, so substantive comparisons use the direct empirical W2.Source: §A.7 · p. 25
  • Check 6 fails because component signal variances differ, making RRL tier boundaries sensitive to weight choice; continuous STRI is the primary governance output.Source: §A.7 · p. 25

B.1. Judge reliability by content axis

Judge disagreement is especially pronounced for technical-detail scoring, but both contested axes remain in the EDI rubric and their persona effects are interpreted exploratorily.

B.1. Judge reliability by content axis

  • Technical-detail scoring shows the most extreme judge disagreement (α = −0.353), with DeepSeek’s mean score of 1.63 versus Qwen’s 0.54.Cost mention shows milder disagreement (α = −0.110), with both judges scoring it low overall.Source: §B.1 · p. 25
  • The rubric retains both technical-detail and cost-mention axes to avoid post-hoc removal based on reliability, while persona-effect interpretations for them remain exploratory.Source: §B.1 · p. 25

B.2. Self-check diagnostic summary

  • Table B.2 summarizes automated self-checks applied to the pipeline.Source: §B.2 · p. 25

B.3. RQ5 ordered-logit coefficients

  • Table B.3 presents the full coefficient table for the exploratory ordered-logit association between generative AI use and general AI attitudes.Source: §B.3 · p. 25
Loading 2609.11611v1…