A Vision-Language Framework for Measuring Social Life on Sidewalks

1 Introduction

• Pedestrian counts miss whether people are sitting, standing, or walking and whether nearby people are actually interacting.In-person observation provides richer accounts of social life but is labor-intensive to apply at scale.Source: §1 · p. 2
• The framework combines revised social-group detection, per-person observable-attribute queries, and composite indicators to measure grouping, dwelling, activity, and spatial context.Its activity system uses ten independent observable dimensions to avoid conflating visible states with contextual inferences.Source: §1 · p. 2
• Across 102,514 New York City sideviews, pedestrian volume and sidewalk social activity are only weakly associated.Streets with the highest pedestrian counts are not necessarily where people most often gather or linger.Source: §1 · p. 2

2.3 Detecting Human Behavior with Computer Vision

• The framework overview links group detection and per-person activity coding to refined outputs and city-scale social indicators.It depicts timestamped sidewalk-facing sideviews as the unit of analysis and shows the two detection branches feeding cross-branch refinement.Source: Figure 1 · p. 3

2.1 Observational Studies of Urban Public Space

• Field observation has documented sitting, congregating, and lingering, but rich behavioral studies have been local and labor-intensive.Whyte’s New York plaza research required more than a decade; examining social behavior across hundreds of streets calls for a different approach.Source: §2.1 · pp. 2, 4

2.2 Street-Level Imagery in Urban Research

• Street-level imagery supports computational urban analysis, but prior imagery studies often count pedestrians without distinguishing whether they are alone, together, moving, or stationary.A review of more than 300 computer-vision urban studies found that over 80% focused on simple scene-level metrics with limited connection to human behavior.Source: §2.2 · p. 4

2.3 Detecting Human Behavior with Computer Vision

• Existing computer-vision approaches locate, count, or attribute individuals but do not observe them in relation to one another or spatial context.MINGLE combines person detection, pairwise social judgment, and graph-based grouping; this paper extends that approach to per-person activity coding.Source: §2.3 · p. 5
• VLMs prompted with high-level social categories can conflate observable physical states with contextual inferences.Restricting outputs to directly observable attributes is described as the design principle that resolves this problem and supports social-indicator construction.Source: §2.3 · p. 5

3.1 Detecting Groups: a revised three-step pipeline

• The group pipeline detects people, judges pairwise social relationships, then integrates those judgments through graph-based clustering after validity filtering.SAM3 replaces the prior person detector to improve recall for occluded and small-scale figures; PatchFusion replaces the prior depth estimator for higher-resolution, more geometrically continuous interpersonal-distance estimates.Source: §3.1 · p. 6
• Pairwise social-judgment F1 rose from 72% in the original MINGLE framework to 80.64%, while the 3B-parameter model achieved comparable performance at lower inference cost.The 3B model is used for the study’s pairwise social-interaction judgments.Source: §3.1 · p. 6

3.2 Detecting Activities: per-person VLM query

• High-level action prompts caused VLMs to infer plausible situations from misclassified physical states, and models also returned only one label when behavior combined several.The example of “waiting for public transport” followed from omitting walking as a posture option.Source: §3.2 · p. 6
• The redesigned activity prompt codes ten independent observable dimensions; on a 999-image held-out benchmark, Qwen3-VL-30B-A3B-Instruct achieved mean field accuracy of 88.5%.Perceived sex is provisional and excluded from primary analyses; Tier 1 fields exceed 85% accuracy, while Tier 2 fields are reliable on models with at least 8B parameters.Source: §3.2 · p. 7

3.3 Cross-branch Refinement and Fusion

• Cross-branch refinement uses per-person validity labels to remove spurious detections before clustering, then propagates a shared locomotion label across each detected group.The refinement stage integrates activity and group outputs before indicator construction.Source: §3.3 · p. 7

4 Social Measurement System

• The system aggregates group membership and VLM-coded attributes into an image-level SDI, while independent binary flags record population groups and behavioral types.These measures make raw per-person outputs comparable across images or locations.Source: §4 · p. 8

4.1 Constructing the Social Dwelling Index

• WGI weights group size (solo 1, dyad 2, groups of three or more 3), WDI weights dwelling (active 1, stationary 2, seated non-riders 3), and SDI multiplies the weights per person.The weights are ordinal heuristics rather than empirically estimated parameters; SDI is normalized by the number of detected people and does not scale mechanically with pedestrian volume.Source: §4.1 · pp. 8–10, Figure 2 · p. 8, Figure 3 · p. 9
• Spearman ρ = 0.949–0.998 for SDI ranking stability, while the weak SDI–count association persists across specifications (r = 0.163–0.212).The sensitivity analyses test three alternative grouping and dwelling-intensity weighting specifications.Source: §4.1 · p. 10

4.2 Binary Flags

• Perceived-sex coding is structurally unreliable in U.S. street imagery because provider face blurring removes a primary visual cue, so it is excluded from primary analyses.The attribute is retained for exploratory use where blurring rates are known to be low.Source: §4.2 · p. 10
• Five conservative Vulnerable Street User Flags identify infants, children, wheelchair users, walker users, and older adults using combined age, posture, and object conditions.All flags also require the person's bounding box to exceed a minimum pixel-area threshold.Source: §4.2 · p. 10, Table 2 · p. 13
• Twelve primary activity labels use directly observable, visually distinct cues; chatting requires a stationary group with a non-upright member, and overlaps favor higher-priority labels.Object-anchored behaviors take priority over posture- or zone-inferred states because object detection is described as more reliable.Source: §4.2 · pp. 13–14, Table 3 · p. 14
• Independent Spatial Context Flags record locations such as crosswalks or building entrances alongside, rather than instead of, a person's primary activity label.A person labeled passing may also be flagged as in a crosswalk.Source: §4.2 · pp. 14, 16, Table 4 · p. 16

5.1 SDI and Pedestrian Volume

• Pedestrian count correlates weakly with SDI (r = 0.168), more with WGI (r = 0.264) than WDI (r = −0.007), while WGI and WDI are nearly orthogonal (r = −0.017).These pairwise correlations indicate that SDI is not reducible to crowd size and that grouping and dwelling are complementary dimensions.Source: §5.1 · p. 16, Table 5 · p. 18
• At similar headcounts (N ≈9–10), solo-moving scenes have SDI = 1.11, versus SDI = 4.22–4.89 for scenes with grouped and dwelling people.The image pairs show an approximately fourfold difference in SDI at near-constant pedestrian volume.Source: §5.1 · p. 18, Figure 8 · p. 19

5.2 Mapping Decoupling Between SDI and Pedestrian Count in NYC

• Across NYC, SDI and pedestrian count form spatial clusters: high-SDI, low-count areas include Harlem, while high-count, low-SDI areas include major transit corridors.The map aggregates image-level zSDI − zcount scores by H3 hexagonal cell; its inset reports an image-level correlation of r = 0.168.Source: §5.2 · p. 18, Figure 9 · p. 20

5.3 Mapping Activity Entropy in NYC

• Neighborhood activity entropy uses an 18-bin mass vector of 12 primary activity categories, a pooled other category, and five spatial-context counts; flags may overlap.Shannon entropy is computed over the normalized vector, so higher values indicate a more even mix of activity types.Source: §5.3 · p. 21, Figure 10 · p. 21
• Activity entropy shows a center–periphery gradient, with higher values in Manhattan and inner Brooklyn and Queens, but variation within Manhattan and across peripheral areas.Higher entropy reflects multiple activity types co-present in a street environment; peripheral areas have fewer observations.Source: §5.3 · p. 21

• Static imagery captures sidewalk conditions at one moment, so the indicators do not directly measure duration, frequency, or temporal dynamics.Source: §6 · p. 22
• VLM query uncertainty can persist under occlusion, small person scale, motion blur, or image degradation, and propagates into composite indicators.Source: §6 · p. 22
• Sparse pedestrian counts in many sideviews can destabilize image-level activity distributions, while aggregation across larger areas produces more robust patterns.Source: §6 · p. 22
• The framework measures observable sidewalk presence, not internal states, motivations, social meaning, or social experience itself.Source: §6 · p. 22
• Future work can examine activity-demographic links, zone-type variation, opposite sidewalk faces, and repeated time windows.Source: §6 · p. 22
• Street-view evidence brings observational questions to city scale, making convergence or divergence with fieldwork a productive research question.Source: §6 · p. 22

• The framework replaces pedestrian counting with observational measurement of how people use sidewalks.Source: §7 · p. 23
• The integrated framework combines revised social group detection with per-person observable queries on timestamped, sidewalk-facing sideviews.Source: §7 · p. 23
• A ten-dimension observable taxonomy addresses VLM conflation of visible states with contextual inference, and benchmarking across six models selected Qwen3-VL-30B-A3B-Instruct for city-scale deployment.Source: §7 · p. 23
• The indicator system provides SDI for image-level grouping and dwelling, accessibility-sensitive population flags, and activity and spatial-context flags.Source: §7 · p. 23
• Pedestrian volume and SDI were weakly associated (r = 0.168), and activity entropy was similarly decoupled from foot traffic (r = 0.24).Streets with high pedestrian counts were not where people most often lingered, gathered, or showed the greatest behavioral diversity.Source: §7 · p. 23

• The study did not involve human subjects research requiring institutional review board approval.Source: Ethical Statement · p. 23
• The street-level imagery was obtained from third-party providers.Source: Ethical Statement · p. 23
• Providers blurred identifiable facial information before access, and the images were used for observational analysis in compliance with platform terms of service.Source: Ethical Statement · p. 24

• The framework is independent of a specific imagery dataset and applies to panoramic street-level imagery with capture timestamps.Source: Data Availability · p. 24
• The authors cannot redistribute the third-party source imagery; Apple Lookaround, Bing Streetside, and Mapillary imagery can be retrieved using the described procedure.Source: Data Availability · p. 24
• Derived New York City data tables across 102,514 sideviews will be deposited in a public repository upon publication; intermediate model outputs and benchmarking annotations are available from the corresponding author upon request.Source: Data Availability · p. 24

A1.1 Source Requirements and Sampling

• Timestamped panoramic street-level imagery is required because capture time is needed for temporal stratification.Google Street View is unsuitable for this purpose because its API does not expose capture timestamps; timestamped platforms such as Apple Lookaround, Bing Streetside, and Mapillary are compatible.Source: A1.1 Source Requirements and Sampling · p. 29
• The sampling design targets likely activity periods because pedestrian activity varies with time-of-day and day-of-week rhythms.The paper notes that lingering versus transit and the social character of a space differ across work, social, and recreational schedules.Source: A1.1 Source Requirements and Sampling · p. 29
• Images are sampled in four periods: weekday and weekend midday (12:00–14:00) and evening (17:00–19:00).These windows are aligned with peak travel and social activity and enable comparisons beyond static pedestrian counts.Source: A1.1 Source Requirements and Sampling · p. 29

A1.2 Image Availability and Spatial Coverage

• The NYC sample comprises 102,514 Apple Lookaround sideviews from midday and evening peak hours, but coverage is patchy and not exhaustive.The appendix figure maps the sideview locations across the city.Source: A1.2 Image Availability and Spatial… · p. 29, Figure A1 · p. 30

A1.3 Sideview Projection

• A 150° sideview FOV provides continuous lateral pedestrian-zone coverage across 100 m capture intervals without gaps, while wider angles degraded VLM coding accuracy.The resulting 1600×800 image is vertically cropped to remove sky and road regions.Source: A1.3 Sideview Projection · p. 29, Figure A2 · p. 30, Figure A3 · p. 31

A3 Adaptive Individual Cropping

• Adaptive cropping pads the tightest box around target entities, then recenters and shifts a square crop within image boundaries to retain context without truncation.The illustrated output uses standardized square focal patches.Source: A3 Adaptive Individual Cropping · p. 31, Figure A8 · p. 34

A2 Supplementary Figures: Revised Group Detection Framework

• The revised group detector’s 3B-parameter Update 4 model reached F1 = 80.64% at substantially lower inference cost and supplies all pairwise interaction judgments.It was trained on the 76k-image corpus.Source: Figure A6 · p. 33

A4 Prompt Design for Individual Activity Detection

• The decomposed ten-dimension prompt records directly observable states rather than inferred social meaning, including support surfaces for seated people and validity labels to filter false positives.The prompt instructs the model to use only visible information and not infer high-level intent or social meaning.Source: A4 Prompt Design for Individual… · pp. 34–35

A5 Model Selection and Validation

• Qwen3-VL-30B-A3B-Instruct was selected for activity detection after evaluation against 999 manually annotated pedestrian samples.The authors report that it had the strongest instruction-following capabilities and classification accuracy among evaluated models.Source: A5 Model Selection and Validation · p. 37

A6 Shannon Entropy over Activity Labels

• Per H3 cell, the activity-label entropy uses an 18-bin mass vector combining 13 primary-activity bins and five spatial-context flags, which may overlap.Entropy ranges up to log2 18 ≈ 4.17 bits; a cell dominated by one activity type approaches zero.Source: A6 Shannon Entropy over Activity Labels · pp. 37–38

A7 Sensitivity of the Social Dwelling Index to Alternative Weighting Schemes

• Across alternative SDI weightings, its association with pedestrian count remains weak (r = 0.163, 0.165, and 0.212 versus r = 0.168 at baseline).Baseline rank agreement is high under all alternatives (ρ = 0.998, 0.992, and 0.949); the dwelling-adjusted scheme departs most, reflecting its shifted balance between grouping and dwelling.Source: A7 Sensitivity of the Social Dwelling… · p. 39