Freeze, Validate, Report: Auditing Urban Station Plans with Common Evidence

1 Introduction

• Reconstructing trips, reporting cells, or candidate blocks from plan outcomes can make favorable scores reflect a different evidence base rather than better access.The paper gives a two-demand-point example of a ranking reversal caused by plan-specific observation selection.Source: §1 · p. 1
• The contract records sensitivity to candidate sets and temporal windows, while the appropriateness of the chosen fairness definition remains a separate normative question.The demonstrations were not used or validated by the Porto or Chicago authorities and do not estimate post-deployment performance or ranking changes.Source: §1 · pp. 1–2
• Common evidence consists of shared observations, reporting partitions, and candidate-block pools; the selection rule is the ordered set of validation metrics.The authority declares the comparison, the analyst applies its inputs, and the auditor checks the record.Source: §1 · p. 1
• Freeze–Validate–Report freezes the comparison inputs, selects one plan using validation evidence, then evaluates only that plan on test data.The authority declares the inputs and rule, the analyst applies them, and the auditor checks the record.Source: §1 · pp. 1–2
• The evaluation compares seven generators across two public origin–destination settings, using observed trip endpoints as demand for hypothetical stations.Porto uses four socioeconomic-status strata; Chicago uses four Hardship Index strata.Source: §1 · p. 2

2 Related Work and Positioning

• The audit formalizes when alternative plans are scored against the same observations and evaluation inputs, rather than auditing every stage of an urban AI system.This is narrower than end-to-end audit frameworks and model-card reporting.Source: §2 · p. 2
• Classical k-center and p-median motivate geometric and demand-weighted baselines, while fairness-oriented methods target group costs or individual service and outliers.The study compares resulting station plans rather than the original algorithm families.Source: §2 · p. 2
• The secondary fairlet diagnostic uses exact proxy-group counts, a physical radius, and a threshold on group mean access; it does not study hierarchy construction or fairlet approximation guarantees [3].Fairlets are treated as indivisible mixed-group blocks, and the diagnostic is not used to select a station plan.Source: §2 · p. 2
• Fixing one reporting grid prevents plans from choosing their own spatial units, but the grid’s scale and zoning remain subject to the modifiable areal unit problem.That sensitivity has also been demonstrated for public-transit reliability equity.Source: §2 · p. 2
• The primary temporal splits preserve order but use one sequence; blocked or rolling-origin validation would require additional windows.The paper distinguishes this concern from the narrower requirement that plans first be scored against the same stated target.Source: §2 · p. 2
• Distance summaries and area-level proxies are partial measures: accessibility also involves land use, opportunities, time, and individual constraints, and an operational proxy may differ from its intended construct.The paper treats these measures as declared decision inputs, not neutral descriptions of a city.Source: §2 · p. 2

3.1 Common evidence and rank reversal

• Plan-specific observation selection can reverse the declared target ordering: the example reports S1 ahead of S2 although the target mean ranks S2 first.Each reported value is correct for its retained point, but represents a minimum rather than the declared mean; the reversal’s direction depends on the metric and outcomes.Source: §3.1 · pp. 2–3

3 Problem and Evaluation Contract

• Common evidence makes compared scores represent the same target, allowing their difference to be attributed to plan outcome vectors, but not establishing causal or normative fairness claims.The selected plan is evaluated once on test data after the selection record is written; test metrics for the other plans are not computed.Source: §3.1 · p. 3, §3.4 · p. 4

3.2 OD access and fixed proxy groups

• Endpoint distance sums each trip’s origin-to-nearest-station and destination-to-nearest-station distances; group summaries include worst-group p90 and the gap between highest and lowest group means.Individual fairness would require a separate criterion for each trip.Source: §3.2 · p. 3

3.3 Fixed grid constructed without plan outcomes

• Trips are assigned to fixed grid cells using origin coordinates, and cell membership is invariant across plans.Cell summaries use auditable cells; the 90th–10th percentile cell-mean gap reduces, but does not remove, the influence of extreme cells.Source: §3.3 · p. 3

3.4 Selection rule fixed before validation

• The predeclared rule minimizes worst-group p90 first, then breaks ties by overall p90, cell gap, share within L, and group mean gap.It uses point estimates without a positive indifference threshold, so it does not establish statistical or practical superiority; PMF is secondary to selection.Source: §3.4 · p. 4

4.1 Seven methods for generating station plans

• All seven generators receive the same training trips, 300 candidate sites, and station budget k=200, while the audit evaluates every resulting plan on common validation evidence.The candidate sites are drawn from observed trip endpoints, restricting siting to observed demand locations rather than independently authorized sites.Source: §4.1 · p. 4
• The portfolio spans farthest-first, bounded k-medoids and demand-weighted p-median search, group-fair mean, IFkCO, Priority, and a grid control.Group-fair mean prioritizes the highest proxy-group mean; IFkCO adapts individually fair k-center with outliers; Priority uses a group maximum-to-baseline-p90 ratio; Grid uses endpoint counts and geometry.Source: §4.1 · p. 4

4.2 Secondary proportional mean fairlet diagnostic

• The secondary proportional mean fairlet diagnostic builds one geometry-and-label proposal pool, then uses plan-specific origin-access outcomes to determine eligible blocks; it does not select the station plan.A fairlet is a small mixed-group block treated as one unit by later clustering; the same set-packing model selects disjoint eligible blocks, and origin-only equal means can reflect leveling down as well as improvement.Source: §4.2 · p. 4
• The diagnostic’s inclusion floors, exclusion-rate-gap limit, and packing constraints jointly bound admissible block counts; meeting the minimum inclusion count alone is insufficient.Solver incumbents and upper bounds concern the eligible subset induced by the generated pool, and “optimal” is used only relative to that subset.Source: §4.2 · p. 5

5 Experimental Setup

• The retrospective comparison uses fixed time splits, selection criteria, and deterministic search budgets across Porto taxi trips and Chicago ride-hail trips.Proxy groups are area-level socioeconomic labels, not traveler attributes or verified residence data.Source: §5 · p. 5, §5.1 · p. 5

5.1 Temporal data and proxy groups

• The primary Chicago train, validation, and test windows are all on June 1, 2022, so clock time is confounded with split role.The periods were fixed before plan comparison, with a gap before the later test window.Source: §5.1 · p. 5

5.2 Selection and PMF parameters fixed before validation

• Porto-SES uses 3 km grid cells and a 2 km endpoint-distance limit; Chicago-Hardship uses 5 km cells and a 5 km limit, with 25-trip minimum cell support, at least two cells, and 90% auditable trips.Source: §5.2 · p. 5

5.3 Ablation with plans and data held fixed

• The ablation holds validation plans, trips, constraints, and solver budget fixed while comparing a shared geometry-and-label pool with separate plan-dependent pools.Its diagnostic ranking prioritizes admissibility, then inclusion, exclusion-rate gap, and threshold-meeting proposals; that order does not select the station plan.Source: §5.3 · p. 6

5.4 Post hoc sensitivity checks

• Post hoc checks expand candidate sites from 300 to 600 and compare Chicago’s same 07:00–08:00 hour across three dates; these checks are not confirmatory and do not estimate sampling uncertainty.A separate bootstrap resamples validation trips within proxy groups while keeping plans, group sizes, and the original selection fixed; its ranges are neither simultaneous nor selection-adjusted.Source: §5.4 · p. 6

5.5 Reproducibility

• Saved hashes, inputs, decision logs, proposal pools, and solver results are checked by separate verifiers, which pass the reported main, ablation, and sensitivity-artifact checks.The results have not been independently replicated by an external group.Source: §5.5 · pp. 6–7

6.1 Validation comparison and selection

• Validation selects IFkCO in Porto on worst-group p90 at 885 m, while Chicago selects Grid by a 22 m first-priority advantage despite IFkCO’s lower overall p90.Chicago’s selected Grid plan has a 1,735 m worst-group p90; its 22 m advantage does not establish statistically reliable or practically meaningful superiority.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 Test results without reselection

• 805 m/872 m are Porto’s selected-plan test overall/worst-group p90s; Chicago’s are 1,404 m/1,656 m, after validation selects IFkCO and Grid, respectively.Test data do not affect selection.Source: §6.2 · p. 7, §6.1 · p. 7

6.3 Post hoc sensitivity checks

• Expanding the candidate set changes selection in both cities; same-hour Chicago validation across dates selects Priority, while resampling spans both Chicago pairwise orderings.Porto’s resampled IFkCO-minus-Grid W90 difference remains below zero; the paired sensitivity result is not a portfolio-wide superiority claim.Source: §6.3 · p. 7

6.4 Secondary PMF results, bounds, and pool ablation

• Porto’s 10,000-node search finds 174 disjoint blocks against an upper bound of 184, leaving a 5.7% gap; Chicago’s sparse pool yields no admissible PMF solution.The Porto result meets fixed requirements but leaves optimality unresolved; Chicago’s inadmissibility applies only to the sampled pool.Source: §6.4 · p. 7
• Plan-specific proposal pools alter diagnostic ranks: Porto retains IFkCO as its unique top-ranked admissible result, while all Chicago results remain inadmissible.The pool ablation’s solver certifies maxima only within generated pools; those maxima omit inclusion and exclusion-rate constraints, which are checked separately for admissibility.Source: §6.4 · pp. 8–9

7.1 Validity controls

• Shared observations, reporting cells, and proposal pools make comparisons traceable; separating validation selection from test reporting prevents test outcomes from changing the recorded choice.PMF conclusions remain limited to sampled pools.Source: §7.1 · p. 9

7.2 Prospective use in urban decision support

• A prospective contract specifies target data, proxies, authorized sites, reporting rules, metrics, validation period, and proposal pools; analysts publish common audit outputs, while authorities decide and version later changes.The workflow has not received municipal or stakeholder validation for its current data, limits, or priorities.Source: §7.2 · p. 9

7.3 Comparative evaluation for urban decision intelligence

• Fixed generators isolate the comparison question; the study offers no empirical claim about learned generators, although the contract is framed for classical, learned, and hybrid alternatives.Source: §7.3 · p. 9

7.4 Remaining limitations

• Proxy definitions, thresholds, and metric priorities remain authority choices; observed endpoint trips and straight-line distances do not represent full-population station demand or broader accessibility.The proxy strata describe origin areas, not travelers’ SES.Source: §7.4 · p. 9
• The bootstrap assumes independent, exchangeable trips within proxy groups and does not capture spatial or temporal dependence, changing group shares, training uncertainty, or rolling-origin variation.The primary Chicago windows also confound split role with clock time; the same-hour check does not resolve that confound.Source: §7.4 · pp. 9–10
• A zero solver gap would establish optimality only within the sampled PMF pool; Porto-SES has a nonzero gap, while Chicago-Hardship fails pool requirements.Source: §7.4 · p. 10

7.2 Prospective use in urban decision support

• Hashes and phase logs document execution, not prior commitment: the chronology lacks an external timestamp, and the original test was seen before later analyses.The discussion recommends externally timestamping and publishing the contract before comparison.Source: §7.2 · p. 9, §7.4 · p. 10

• Changing candidate sites or temporal design changes the selected plan in both retrospective cases.Source: §8 · p. 10
• The protocol’s value is making populations, sites, reporting cells, thresholds, and metric priorities explicit in the decision record.Source: §8 · p. 10
• PMF conclusions remain limited to sampled pools, and the authority should publish the contract and start a new evaluation whenever an input changes.Source: §8 · p. 10