PaperlayerView exact revision on arXiv

Source-linked AI summary

Caption-Mediated Perceived-Safety Estimation for Pedestrian Routing

Simon Parkinson, Paloma Liu, Wei Zheng, Mohammadreza Sheikhfathollahi

arXiv:2609.38479v1cs.CVcs.AI

TL;DR

Perceived safety affects pedestrian mobility, but many routing systems rely on reported crime and image-based estimates can be difficult to audit. This paper derives risk classes from stored, inspectable captions and finds caption-mediated text reaches parity with direct image embeddings. In routing, safety weighting produces a journey-length-dependent exchange between detour and low-risk route length.

  • Problem

    The paper addresses whether caption mediation can predict perceived safety and how it compares with alternatives, given that text is explainable but narrower than learned image embeddings.

    Source: §I · p. 1

  • Method

    The system stores vision–language captions and derives perceived-risk classes from structured text features, then evaluates captioning conditions and image embeddings under a common pipeline.

    Source: §VIII · p. 12

  • Results

    Caption-mediated text reached parity with direct CLIP image embeddings on USID and Place Pulse under the reported benchmarks.

    Source: §VIII · p. 12

  • Takeaways & Limitations

    On 3-to-6-km journeys, safety weighting returned 12.78% more low-risk route length for a 2.73% median detour in Manchester, while its effect was little below 1 km.

    Source: §VIII · p. 13

    The supervised benchmark rests on 49 images, yielding wide confidence intervals and unstable rankings that cannot support strong claims about relative captioning-model quality.

    Source: A. Limitations · p. 13

Abstract

from arXiv · show

This paper presents an explainable approach to pedestrian routing, in which perceived safety is estimated from street-level imagery through an explicit natural-language intermediate representation. A vision--language model caption is generated and stored before any scoring is undertaken, and the perceived-risk class is derived entirely from structured features of that stored text, so that every segment score remains inspectable by the user. Nine captioning conditions across five model families are benchmarked against a direct Contrastive Language--Image Pre-training (CLIP) image-embedding baseline under an identical downstream pipeline, and the caption-mediated representation is found to reach parity with the image embedding rather than to trail it. The approach was deployed over 654,115 images covering 36 electoral wards in two locations in Northern England (Manchester and Huddersfield). Independent field validation against 3,669 locally collected ratings of 494 images across 70 participant sessions established agreement that is statistically significant but modest, at $r=0.262$, against a measured noise ceiling of 0.737 imposed by disagreement between raters. A single-use confirmatory test then found that a pipeline 44\% stronger on the supervised benchmark did not produce measurable improvement in the field ($r=0.250$, $p=0.84$), so the benchmark gains did not predict the deployment gains in this case. Routing behaviour varies systematically with journey length. There is negligible change below 1\,km, reaching a median increase of 12.78\% in low-risk route length for a median detour of 2.73\% on journeys of 3 to 6 km.

I. INTRODUCTION

Perceived safety matters for pedestrian mobility, yet existing safety-routing inputs and image-based estimates leave important gaps in representativeness, auditability, and independent validation. The paper proposes a caption-mediated routing system and evaluates its benchmark performance, routing trade-offs, and agreement with local human ratings.

I. INTRODUCTION

  • The system targets city-scale, legible routing because perceived fear can alter travel choices beyond recorded incidents, including route avoidance and avoiding journeys.The paper presents a field validation study as part of addressing the identified auditability and validation gaps.Source: §I · p. 1
  • Existing systems often use reported crime, which records reported offences rather than where people feel unsafe and underrepresents offence categories relevant to VAWG.Manual environmental audits measure the environment directly but are labour intensive and do not scale citywide.Source: §I · p. 1
  • Prior image-based systems commonly produce scores that are difficult to interrogate, while independent deployment-area validation and measurement of human-reference reliability remain rare.These gaps limit auditability and make reported agreement difficult to compare across studies.Source: §I · p. 1
  • The study evaluates safety-weighted routing against distance-optimal routing by measuring detours and changes in exposure, and tests whether benchmark gains transfer to field agreement.It also asks how deployed predictions compare with local judgements and the reliability ceiling imposed by rater disagreement.Source: §I · p. 2
  • The proposed system stores a vision–language caption before scoring and derives perceived-risk classes from structured text features, making individual judgements inspectable.The paper compares nine captioning conditions across five model families under a shared downstream pipeline and benchmarks against CLIP image embeddings.Source: §I · p. 2

II. RELATED WORK

Prior work establishes that street imagery can support scalable perceived-safety estimates, while leaving questions about score auditability, route inputs, and validation practice. This paper uses readable captions for safety estimation and retains a conventional weighted shortest-path search for routing.

A. Perceived Safety from Street-Level Imagery

  • Place Pulse and Streetscore demonstrate that perceived-safety judgements can be recovered from street imagery at scale, but the resulting judgements remain opaque,,.Streetscore used support vector regression, while later work used convolutional networks trained on Place Pulse 2.0.Source: A. Perceived Safety from Street-Level… · p. 2
  • Unlike approaches that attach text after scoring, this work stores captions first and derives safety labels from caption features, making the representation inspectable.Free-form captions also retain compositional and contextual information that discrete object vocabularies can lose.Source: A. Perceived Safety from Street-Level… · pp. 2–3

B. Safety-Aware Route Planning

  • Most safety-aware route-finding weights routes using crime or crash statistics, whereas this study derives a safety cost from street-level perception,.Its routing component is a conventional weighted shortest-path search; the contribution lies in deriving the weight rather than the search itself.Source: B. Safety-Aware Route Planning · p. 3

C. Validation Practice and the Reliability of Human References

The method stores captions as the basis for later scoring and routing, and evaluates the approach using distinct datasets rather than pooling their labels. The related validation discussion motivates measuring the reliability ceiling of human references.

C. Validation Practice and the Reliability of Human References

  • Agreement with noisy human judgements is bounded by reference reliability, yet reliability is rarely reported alongside model quality in perceived-safety studies.The authors note that noise ceilings had not been reported for street-view perceived-safety systems to their knowledge.Source: C. Validation Practice and the… · p. 3

III. METHOD

  • The pipeline collects street imagery, stores a natural-language caption, extracts interpretable features, classifies perceived risk, maps scores to network edges, and computes two routes.The caption is the stored stage-two artefact from which subsequent judgements derive, rather than a post-hoc explanation.Source: §III · p. 3, Figure 1 · p. 3

A. Image Collection

  • Street imagery is sampled along merged OpenStreetMap pedestrian and drivable networks, with panorama deduplication and four headings per retained panorama.The study obtained 61,981 distinct panoramas in Huddersfield and 80,889 in Manchester; the authors report a Huddersfield deduplication defect.Source: A. Image Collection · pp. 3–4

B. Caption Generation

  • Nine captioning conditions across five model families were evaluated, and every subsequent stage operates on the stored caption rather than the source image.Qwen2.5-VL prompts explicitly directed attention to perceived safety, while Florence-2 used detailed-captioning prompts.Source: B. Caption Generation · p. 4

C. Supervision and Evaluation Data

  • USID supplies per-image fear-of-crime supervision, while Place Pulse supplies an independent 99-image extreme-groups evaluation with 50 safe and 49 unsafe images.The two datasets index different constructs, locations, capture protocols, and label types, so benchmarks are reported separately and robustness requires agreement across them.Source: C. Supervision and Evaluation Data · p. 4
  • The datasets are not pooled because their labels index different constructs and scales, which would make a pooled classifier fit a blend corresponding to neither.Pooling is noted as a possible future direction given the small size of each dataset.Source: C. Supervision and Evaluation Data · p. 4

D. Caption-Derived Fear-of-Crime Classification

Caption text is filtered and represented with interpretable topic and sentiment features before supervised risk classification. Classified locations are then aggregated to pedestrian-network edges and used to compare distance-only and safety-weighted routes.

D. Caption-Derived Fear-of-Crime Classification

  • Captions are lower-cased, tokenised, and filtered to remove generic caption phrasing so downstream models attend to substantive environmental descriptors.Examples include graffiti, parked, dark, alley, and shops.Source: D. Caption-Derived Fear-of-Crime… · p. 4
  • Each image is represented by a sentiment-derived safety score and five LDA topic-membership values, with high-weight topic words retained for tracing classifications to captions.The small topic count is chosen because each captioning condition has only 49 supervised images, limiting stable estimation of larger topic models.Source: D. Caption-Derived Fear-of-Crime… · p. 4
  • USID scores are divided into low, neutral, and high classes using boundaries half a standard deviation below and above the sample mean.Ten classifier families are evaluated with stratified cross-validation; macro-F1 is the primary ranking metric because it is less sensitive to class imbalance.Source: D. Caption-Derived Fear-of-Crime… · p. 4

E. Safety-Weighted Routing

  • Classified locations are matched to pedestrian-graph edges using nearby imagery, with samples spaced 20 m in Huddersfield and 100 m in Manchester.Severity classes map to 0, 1, and 2, respectively, for low, neutral, and high.Source: E. Safety-Weighted Routing · pp. 4–5
  • Edges without nearby imagery receive an Unknown weight of 1.5, while route costs compare distance alone with distance multiplied by the edge safety weight.The positive offset prevents zero-weight edges from becoming free to traverse; the reported routing results are not sensitive to using weight 1.0 for unclassified edges.Source: E. Safety-Weighted Routing · p. 5

IV. CITY AND TOWN DEPLOYMENT

The pipeline was deployed across Huddersfield and Manchester, with sampling density varying by area. The benchmark section compares caption and image representations under a shared evaluation pipeline and cautions against treating single-split results as stable.

IV. CITY AND TOWN DEPLOYMENT

  • Huddersfield’s eight wards and Manchester’s 28 wards formed the two deployment areas.The pipeline was first developed in Huddersfield and then applied in Manchester.Source: §IV · p. 5
  • Huddersfield was sampled every 20 m; Manchester used 20 m in eight inner-city wards and 100 m in twenty outer wards.The coarser outer-ward interval kept acquisition costs reasonable across an area roughly five times larger than Huddersfield; inner-city wards retained finer sampling because of denser pedestrian activity.Source: §IV · p. 5
  • The sampled regimes separated by observed density: 1,247–3,390 locations per km² in inner-city wards versus 209–973 in outer wards.Source: §IV · p. 5

V. REPRESENTATION AND CAPTIONING BENCHMARKS

  • The benchmark evaluates captioning models and text representations against a direct image-embedding reference using the same classification task.This design separates the effects of caption model and text representation.Source: §V · p. 5
  • Repeated cross-validation gave the deployed configuration 0.584 ± 0.052 macro-F1, versus 0.638 on a single split; selecting the best classifier inflated scores by a median 0.045.The authors treat single-split rankings as provisional because split and classifier-selection variability can affect close orderings.Source: §V · p. 5

A. Direct-Embedding Baseline

Caption-mediated representations reached parity with direct image embeddings in the reported benchmarks, while results varied with text encoding and captioning condition. The section also illustrates how stored captions and their features make individual classifications inspectable.

A. Direct-Embedding Baseline

  • The direct-image reference uses CLIP ViT-B/32 embeddings, with full 512-dimensional and six-dimensional PCA variants to distinguish representation quality from size.Source: A. Direct-Embedding Baseline · p. 6

B. Caption Trade-off

  • On USID, the best text representation reached 0.830 macro-F1 versus 0.796 for image embeddings; parity, not superiority, was claimed.The 0.034 margin was smaller than the text condition’s standard deviation, and best-of-ten classifier selection inflated rows by a median 0.045.Source: Table II · p. 6, B. Caption Trade-off · p. 6
  • With identical captions and six dimensions, TF-IDF reached 0.830 macro-F1 versus 0.698 for topic-and-sentiment features, showing encoding—not text use—conditioned performance.The authors report that topic-and-sentiment features discard caption content retained by term-frequency weighting.Source: B. Caption Trade-off · p. 6, Table II · p. 6

C. Captioning Models Under a Fixed Representation

  • At fixed sentence-embedding representation, Qwen2.5-VL-7B scored 0.829 on USID and Qwen2.5-VL-3B scored 0.710 on Place Pulse, comparable to CLIP’s 0.796 and 0.696.The margins over the respective CLIP baselines were smaller than the standard deviations.Source: Table III · p. 7, C. Captioning Models Under a Fixed… · p. 7

B. Caption Trade-off

C. Captioning Models Under a Fixed Representation

  • A worked high-risk example was driven by the enclosure-and-emptiness topic at weight 0.883, while graffiti contributed 0.092.The figure presents the source image, caption, extracted topic weights, and resulting class, allowing the basis of a classification to be inspected.Source: C. Captioning Models Under a Fixed… · p. 7

D. Sensitivity to Design Choices

The deployed topic-and-sentiment encoding remains behind term-frequency weighting across tested topic counts and class boundaries, while the text-versus-image ordering changes with the cut points. Routing conclusions are stable across alternative ordinal cost scales, but probability-weighted severity produces little route discrimination.

D. Sensitivity to Design Choices

  • Across K = 3, 5, and 10, deployed topic-and-sentiment macro-F1 varies by just 0.006 on USID, and remains below same-dimensional term-frequency weighting.The reported ranges are 0.696–0.702 on USID and 0.685–0.700 on Place Pulse.Source: D. Sensitivity to Design Choices · p. 7, Table IV · p. 9
  • Across all four class-boundary schemes, topic-and-sentiment encoding remains last, 0.13–0.15 macro-F1 behind alternatives; text-versus-CLIP ordering reverses under tertile and quartile cuts.Term-frequency weighting leads CLIP at 0.830 against 0.796 under deployed cuts, but trails at 0.790 against 0.801 under tertiles and 0.703 against 0.731 under quartiles.Source: D. Sensitivity to Design Choices · p. 7, Figure 3 · p. 8
  • Under all three alternative ordinal severity scales, routing remains inert below 1 km and gains grow with journey length in both cities.In Manchester, long-journey low-risk gains range from 7.92% to 9.49%, compared with 7.68% under the deployed scale; penalising High more increases gains and detours.Source: D. Sensitivity to Design Choices · pp. 8–9, Table V · p. 9
  • Probability-weighted severity changes no route composition at any journey length, compressing edge-weight variation and leaving 22.3% of long routes identical to distance-optimal routes.The variant was available for Manchester alone because Huddersfield records did not retain class probabilities.Source: D. Sensitivity to Design Choices · p. 9

VI. ROUTING

Safety-weighted routing has little effect on journeys under 1 km, while longer journeys exchange a modest detour for more low-risk route length. The effect varies with journey length and city, and mostly substitutes Low for Neutral segments rather than avoiding High-risk streets.

VI. ROUTING

  • The evaluation sampled 3,000 origin–destination pairs across short, medium, and long true network-distance bands and solved both route costs to optimality.The upper band ends at 6 km to exclude routes over 9 km that are not walking journeys.Source: §VI · p. 9
  • On Manchester journeys of 3 to 6 km, a median detour of 2.73% gains 12.78% more route length in Low-risk segments; mean severity falls for 97.4% of pairs.For journeys under 1 km, median detour and Low-risk-length gain are both zero in both cities; detour and gain correlate positively with route length in each city (p < 0.001).Source: §VI · pp. 9–10, Table VI · p. 9
  • Manchester’s less homogeneous network yields roughly three times Huddersfield’s Low-risk gain on 3-to-6-km journeys for about twice the detour.The cities also differ in unclassified network share, but recomputing with unknown-edge weight 1.0 has negligible effect on the results.Source: §VI · pp. 9–10
  • The routing exchange grows with journey length, while High-risk length is unchanged at the median in five of six conditions because High segments are rare.High segments make up 0.7% of Huddersfield and 1.8% of Manchester networks, so the router principally substitutes Low for Neutral segments.Source: §VI · pp. 9–10
  • Some sampled routes use only unclassified edges and are excluded from severity statistics, while the graded trade-off lets users exchange longer walks for proportionally better routes.These pairs number 2 in Huddersfield and 74 in Manchester; their mean severity is undefined rather than zero.Source: §VI · p. 10

VII. VALIDATION

The validation asks whether the deployed classifier tracks local perceptions, how rater disagreement bounds agreement, and whether benchmark improvement transfers to the field. An independent local rating study found statistically significant but weak agreement, with Low separated from Neutral and High more clearly than those latter classes are separated from each other.

A. Study Design

  • Field ratings were reserved for one confirmatory use and were not used to select captioning models or representations.This preserves the ratings as an independent test of transfer to the deployment area.Source: A. Study Design · p. 10, §VII · p. 10
  • A two-tier stratified sample used 16 shared anchor locations and 480 pool locations, with equal allocation across eight districts and three predicted classes.Equal allocation prevents the roughly 5% High class from being critically under-represented; sampled scenes were separated by at least 120 m.Source: A. Study Design · p. 10
  • The study collected 3,669 ratings across 494 images and 70 participant sessions, mapping predicted Low, Neutral, and High classes to rating-scale directions 3, 2, and 1.Each participant rated perceived safety from 1 (lowest) to 5 (highest) without seeing the classifier prediction.Source: B. Model–Human Agreement · p. 10, A. Study Design · p. 10

B. Model–Human Agreement

  • Per-image predicted class correlates with mean field rating at Pearson r = 0.262 (p < 10^-8), a statistically significant but weak relationship.Mean ratings are 3.38, 3.00, and 2.94 for Low, Neutral, and High respectively; Low is separated more clearly than Neutral from High.Source: B. Model–Human Agreement · pp. 10–11, Figure 5 · p. 11

C. Inter-Rater Disagreement and the Noise Ceiling

Human disagreement limits measurable model–human agreement: at the study’s rating density, the estimated ceiling is 0.737. The section estimates that ceiling from split-half reliability and notes that it depends on the reference population and sampling design.

C. Inter-Rater Disagreement and the Noise Ceiling

  • Raters disagree substantially: the mean within-image standard deviation is 1.006 points, and 73.8% of rater pairs differ by at least one point.On the anchor set, the mean within-image standard deviation is 1.067; two raters differ by 1.17 points on average.Source: C. Inter-Rater Disagreement and the… · p. 11
  • Split-half reliability estimated from 2,000 random partitions gives rel = 0.525 and a ceiling derived from the Spearman–Brown correction.The mean raw split-half correlation was r1/2 = 0.356, with SD 0.030 across partitions.Source: C. Inter-Rater Disagreement and the… · p. 11
  • r = 0.262 reaches 35.0% of the estimated reliability ceiling, indicating a real but minority share of the measurable signal.The attenuation-corrected correlation is 0.355.Source: C. Inter-Rater Disagreement and the… · p. 11
  • 0.737 is the estimated maximum model–human correlation at k = 7.4 ratings per image; the ceiling rises with rating count.Figure 6 shows ceilings of 0.366 for one rating, 0.780 for ten, and 0.870 for twenty ratings per image.Source: Figure 6 · p. 11, C. Inter-Rater Disagreement and the… · p. 11
  • The ceiling is an approximation dominated by pool images with about five ratings, and it bounds agreement with this participant population, not a universal perceiver.The single-rating back-projection assumes a common k, although image-level k ranges from 5 to 65.Source: C. Inter-Rater Disagreement and the… · pp. 11–12

D. Confirmatory Transfer Test

The confirmatory field test found no measurable gain from a pipeline that performed substantially better on the supervised benchmark. The discussion places this result alongside exploratory subgroup findings, benchmark comparisons, routing behavior, and limits on deployment validity.

D. Confirmatory Transfer Test

  • A 44% supervised-benchmark improvement produced no measurable field improvement: the candidate reached r = +0.250 versus the deployed system’s r = +0.262 (p = 0.840).The difference was −0.012 and indistinguishable from zero on the same images.Source: D. Confirmatory Transfer Test · p. 12
  • The systems agreed weakly (r = 0.155); the candidate’s Low and Neutral class means were inverted and statistically indistinguishable, while 66.6% of predictions were Low.The class distribution was consistent with domain shift between USID imagery and UK Street View panoramas.Source: D. Confirmatory Transfer Test · p. 12

E. Exploratory Women and Girls Subgroup Analysis

  • Women and girls’ sessions had r = 0.261 agreement versus r = 0.138 in other sessions, but the difference was not significant and the comparison was exploratory.The Fisher r-to-z test gave p = 0.087 and the permutation test p = 0.081; the comparison group was small.Source: E. Exploratory Women and Girls… · p. 12

VIII. DISCUSSION

  • Caption-mediated text reached parity with direct CLIP embeddings: macro-F1 was 0.830 versus 0.796 on USID and 0.728 versus 0.696 on Place Pulse.The margins were smaller than text-condition standard deviations, and USID rankings reversed under two of four class-boundary schemes.Source: §VIII · p. 12
  • Topic-and-sentiment encoding reached 0.698 macro-F1 versus 0.830 for term-frequency weighting on the same captions, implicating encoding rather than text itself.The encoding can be replaced without regenerating captions; it ranked last under all four class-boundary schemes.Source: §VIII · p. 12
  • The stored caption makes the classification basis inspectable and contestable, unlike construct scores or a 512-dimensional embedding.The discussion distinguishes legibility from interpretability alone, noting that a ten-dimensional CLIP probe recovered 97% of opaque embedding performance.Source: §VIII · pp. 12–13
  • On 3–6 km journeys, a 2.73% median detour returned 12.78% more low-risk route length in Manchester; below 1 km, weighting had little effect.In Huddersfield, a 1.42% median detour returned 4.27% more low-risk length.Source: §VIII · p. 13
  • The pipeline was applied to 654,115 images across 36 wards, but perceptual validity was evaluated only in Huddersfield and cannot yet be assumed for Manchester.High-classified segments account for only 0.7% and 1.8% of routable length in the two study areas; distinguishing Neutral from High requires in-domain labelled data.Source: §VIII · p. 13

A. Limitations

The paper’s conclusions are bounded by a small, unstable supervised benchmark and by limitations in the evaluation procedures. The deployed system also has temporal, sampling, and representation constraints that narrow what its scores can establish.

A. Limitations

  • 49 benchmark images yield wide cross-validation confidence intervals and unstable model rankings, and the apparent 44% improvement did not transfer to field data.The authors therefore use independent field validation to check extrapolation from those images to 654,115.Source: A. Limitations · p. 13
  • Selecting the best of ten classifier families inflates per-condition macro-F1 by a median of 0.045, while shared selection and estimation procedures make reported values comparative rather than unbiased generalisation estimates.The CLIP baseline is one ViT-B/32 architecture and scale, not an upper bound on embedding methods.Source: A. Limitations · p. 13
  • The CPTED prompts were not independently validated, and the confirmatory test compares complete pipelines rather than isolating any component.The probe is exploratory and does not support a deployment decision; identifying responsible components requires a fresh sample.Source: A. Limitations · p. 14
  • The safety weighting is static despite time-varying perceptions, and validation in one city by self-selected volunteers is not representative of the pedestrian population.Nearest-node linear scanning is adequate at study-area scale but would require a spatial index nationally.Source: A. Limitations · p. 14
  • Sentiment comes from a consumer-review model and was not validated against safety judgements; the bag-of-words topic representation loses word order and negation scope.The approach estimates safety from visible environmental cues, not crime.Source: A. Limitations · p. 14

B. Future Work

The conclusion reports field validation and length-dependent routing effects, while future work prioritizes in-domain ratings, independent comparisons, and broader validation. The authors frame the system as complementary to practitioner expertise.

B. Future Work

  • Rating 20–30 people per image on a smaller set could raise the estimated ceiling to approximately 0.87–0.91 and sharpen model-quality estimates; repeating the study in Manchester would test transfer.The current field validation is designed for depth rather than breadth.Source: B. Future Work · p. 14
  • Cross-validation within the field-rated images can test whether in-domain training raises agreement above 0.25, distinguishing out-of-domain supervision limits from construct reliability.This diagnostic requires no further imagery.Source: B. Future Work · p. 14
  • Fresh independent ratings are needed to compare re-encoding stored captions with substituting the captioner; the priorities are in-domain recalibration, Manchester validation, and time-varying route weights.Re-encoding requires no re-captioning, whereas substituting the captioner does.Source: B. Future Work · p. 14

IX. CONCLUSION

  • 12.78% more low-risk route length came with a 2.73% median detour on 3–6 km journeys, with little effect below 1 km; field agreement was modest at r = 0.262.The 0.737 ceiling reflects disagreement among residents, and a pipeline 44% stronger on the benchmark did not improve field agreement (r = 0.250, p = 0.84).Source: §IX · p. 14
  • 0.737 is the measured correlation ceiling, and single-rating reliability is 0.138; the authors recommend reporting rater density, dispersion, and an estimated ceiling alongside agreement.They argue agreement statistics are not comparable across studies unless human-reference reliability is reported.Source: §IX · pp. 14–15
  • The authors intend routing capabilities to complement, not replace, practitioners’ expertise in environmental design and community safety.They identify in-domain labelled data and time-varying safety weightings as next directions.Source: §IX · p. 15
Loading 2609.38479v1…