PaperlayerView exact revision on arXiv

Source-linked AI summary

PlaceSeek: Human-Centered Geospatial Retrieval of Urban Outdoor Places via Semantic Grounding and Affective Alignment

Ziqi Cui, Shangyu Lou

arXiv:2608.24133v1cs.CVcs.AIcs.IR

TL;DR

Existing geospatial retrieval is poorly suited to open-ended urban outdoor-place needs involving activities, physical evidence, and affective experience. PlaceSeek maps natural-language queries to geolocated street-view imagery through physical grounding and affective alignment, and its Milan evaluation outperforms the cited baselines while ablations distinguish their roles.

  • Problem

    Existing geospatial retrieval remains largely POI-centric, making unnamed outdoor places and queries involving activity, physical evidence, or affective expectations difficult to represent.

  • Method

    PlaceSeek uses intent-aware retrieval with Semantic Grounding to verify required visual evidence and Affective Alignment to re-rank physically valid street-view candidates.

  • Results

    PlaceSeek outperforms the evaluated baselines in human-annotated Milan street-view retrieval, while ablations show physical grounding supports validity and affective alignment improves ranking among valid candidates.

  • Takeaways & Limitations

    Complex urban spatial queries require both verifiable visual evidence and human perceptual preferences.

  • Takeaways & Limitations

    The evaluation covers one city, ten query tasks, and five annotators, while six Place Pulse dimensions do not fully capture richer affective semantics.

Abstract

from arXiv · show

People search for urban outdoor places not only by category or function, but also by what activities a place can support and how it is perceived. Existing geospatial retrieval remains largely POIcentric and metadata-driven, making it difficult to satisfy openended, affective, or activity-oriented needs. We present PlaceSeek, a human-centered outdoor place retrieval framework that maps natural-language queries to geolocated street-view imagery. PlaceSeek introduces an intent-aware retrieval mechanism that decomposes user queries into functional and affective sub-intents. A Semantic Grounding Module verifies whether candidate street-view results contain the physical evidence needed to support the intended activity, while an Affective Alignment Module re-ranks physically valid candidates using a LoRA-adapted vision-language model trained on human urban perception judgments. We evaluate PlaceSeek on 31,956 street-view locations in Milan across 10 naturallanguage queries annotated by five human evaluators. PlaceSeek achieves 88.0% Precision@5, a mean match score of 3.39/4.0, and 0.920 nDCG@5, outperforming CLIP, fine-tuned CLIP, SigLIP, and a VQA-based baseline. Ablation results show that physical grounding is essential for retrieval validity, while affective alignment improves ranking quality among physically valid candidates. These findings highlight that complex urban spatial queries require modeling both verifiable visual evidence and human perceptual preferences. PlaceSeek provides a potential framework for human-centered nextgeneration geospatial retrieval systems.

1 Introduction

Urban outdoor-place search often involves activities, physical facilities, and affective expectations that POI-centric systems cannot represent well. PlaceSeek addresses this gap by grounding visible evidence and aligning retrieved scenes with human perception.

  • Urban users may seek places supporting activities and experiences rather than named categories such as parks or cafes.Such queries can combine activity support, physical facilities, and affective expectations.
  • Unnamed outdoor spaces and limited metadata make structured databases poorly suited to abstract activity- and affect-related queries.The retrieval target may be an informal outdoor place rather than an explicitly indexed POI.
  • Street-view imagery offers broad coverage and visual information, but global text-image similarity misses affective preferences and required small physical elements.A visually dominant concept such as greenery can overshadow whether a necessary bench is present.
  • PlaceSeek requires both visible physical evidence and affective alignment to retrieve places suited to intended activities and experiences.The framework is designed for natural-language queries over geolocated street-view imagery.
  • PlaceSeek combines Semantic Grounding and Affective Alignment Modules and evaluates them through human-annotated Milan street-view retrieval experiments.The evaluation compares PlaceSeek with baseline methods and includes ablation analysis.

2 Related Work

Related geospatial retrieval and vision-language methods provide scalable retrieval foundations but remain limited for unnamed outdoor places, localized physical evidence, and subjective urban perception.

  • Traditional geospatial retrieval targets structured entities such as POIs, roads, and buildings, limiting coverage of subjective and difficult-to-predefine spatial needs.Recent LLM- and RAG-based methods still mainly retrieve structured map entities.
  • Existing methods do not directly retrieve unnamed or weakly indexed spaces such as pocket parks, street corners, and informal resting areas.These places are often absent from spatial databases.
  • Dual-encoder models such as CLIP, OpenCLIP, and SigLIP enable efficient open-vocabulary retrieval through shared image-text embeddings.Precomputed image embeddings support retrieval over large image collections.
  • CLIP-style global similarity cannot guarantee that query-required physical elements are visibly present, whereas grounding models can partially localize candidate objects.Global matching may miss fine-grained evidence needed for a place to satisfy the query.
  • CLIP-style models do not explicitly model human affective preferences, making qualities such as safety, quietness, comfort, and romance difficult to retrieve reliably.These qualities are not reducible to ordinary object categories.
  • Image-by-image multimodal assessment can improve semantic judgment but is computationally expensive for city-scale retrieval.The challenge is to preserve dual-encoder efficiency while improving physical and perceptual top-k match quality.

3 Problem Formulation

PlaceSeek formulates outdoor-place retrieval as ranking geolocated street-view images against a query’s semantic, physical, and affective requirements. Its staged pipeline first narrows candidates, then verifies evidence, and finally refines their ordering.

  • Problem Formulation: The task returns ranked geolocated street-view results for a natural-language query describing a desired outdoor place.Unlike POI search, the target satisfies intended activity, physical evidence, and affective expectations.
  • Inputs and Outputs: Queries may combine concrete visual requirements, affective preferences, and intended activities, such as greenery, safety, quietness, and outdoor reading.The search space consists of street-view images associated with geographic locations and viewing directions.
  • Ranking Criteria: The ranking function considers semantic relevance, explicit physical evidence, and affective or perceptual alignment.These criteria correspond to complementary aspects of whether an image satisfies the query.
  • Pipeline: PlaceSeek stages retrieval by narrowing candidates semantically, verifying necessary visual elements, and re-ranking verified candidates affectively.This ordering separates candidate generation, physical validation, and perceptual ranking.
  • Core Challenges: The formulation addresses compositional intents, unreliable global visual similarity, and perceptual terms that generic embeddings do not capture well.Examples include inferred seating for outdoor reading and pedestrian or heritage evidence for a walkable historic street.

4 The PlaceSeek Framework

PlaceSeek decomposes natural-language queries into physical evidence and affective preferences, then retrieves and verifies street-view candidates before perceptual reranking. Its pipeline combines intent parsing, coarse-to-fine physical grounding, and multimodal verification.

  • 4.1 Intent Parsing: Intent parsing separates physical evidence requirements, affective preferences, intended activities, and query constraints.The parser preserves explicit visual evidence, infers activity-supporting affordances, and distinguishes subjective needs for downstream modules.
  • Framework Overview: The pipeline first applies physical grounding and then uses affective alignment to rerank the candidate set by perceptual alignment.Figure 2 summarizes the staged relationship between the LLM parser, SGM, and AAM.
  • 4.2 Semantic Grounding: The Semantic Grounding Module uses coarse OpenCLIP retrieval, GroundingDINO localization, and Qwen3-VL verification to construct physically verified candidates.OpenCLIP retains the top 1% of candidates, GroundingDINO localizes evidence, and Qwen3-VL supplies semantic verification labels and confidence values.
  • 4.2 Semantic Grounding: Qwen3-VL verification checks whether detected regions are semantically correct and functionally suitable, addressing limitations of detector confidence alone.Its structured output includes yes, no, or uncertain labels, confidence, and a short visual rationale.
  • 4.2 Semantic Grounding: Multiple physical evidence terms are merged at the panorama or view level so candidates can be assessed against the query’s full physical requirements.Each term is constructed independently before rule-aware view-level assessment.

5 end

The grounding procedure removes views that fail required physical evidence or violate forbidden evidence.

  • Views failing the physical gate are removed before final candidate ranking.

10 end

PlaceSeek combines hard physical validity with affective alignment and coarse semantic similarity to rank candidates. LoRA adaptation improves the model’s representation of human urban perception, while the final ranking preserves the physical gate.

  • Physical Evidence Re-ranking: Must-have and not-exist terms define hard validity constraints, whereas more-better and less-better terms act as soft ranking preferences.Candidates must satisfy all required evidence and avoid forbidden evidence before soft preferences affect ordering.
  • Affective Alignment: LoRA adaptation fine-tunes the OpenCLIP image encoder on Place Pulse 2.0 human judgments while keeping the text encoder frozen.The supervision covers safer, livelier, more beautiful, wealthier, more depressing, and more boring dimensions.
  • Affective Alignment: 65.7% macro-average win rate was achieved by the LoRA-adapted model, compared with approximately 52% for unmodified CLIP-style models.The adapted model gained 7–12.6 percentage points over the baselines on six perceptual dimensions.
  • Affective Alignment: Affective terms are represented through both projected Place Pulse perceptual scores and direct prompt-bank similarity.The projected signal aligns with human perceptual dimensions, while the direct signal preserves the query expression’s semantic meaning.
  • Final Physical-Affective Re-ranking: The final score combines physical, affective, and coarse CLIP signals, with the physical score carrying the main evidence from grounding.Affective alignment refines ordering, while normalized CLIP similarity retains a weak global semantic prior.
  • Final Physical-Affective Re-ranking: Final ranking preserves the physical gate, so affective and CLIP signals only order candidates that already satisfy physical requirements.Candidates violating required or forbidden evidence are excluded before sorting by the final score.

5 end

The final reranking stage removes candidates failing physical validity and produces the ranked result list from the remaining views.

  • Views that fail the physical gate are removed before the remaining candidates are ranked.The procedure then produces the final ranked list R_k(q).

5 Experimental Setup

The evaluation compares PlaceSeek with four retrieval baselines on ten natural-language queries over Milan street-view imagery, using human relevance judgments and complementary ranking metrics.

  • Dataset and queries: PlaceSeek is evaluated on ten natural-language queries over 31,956 Milan street-view locations sampled in four viewing directions.The imagery comprises 127,824 georeferenced images collected at 100 m intervals.
  • Dataset and queries: The ten queries cover activity-oriented, object-oriented, perception-oriented, and mixed outdoor-place intents.These query types combine intended uses, visible evidence, and affective or experiential preferences.
  • Comparison methods: The comparison includes CLIP, affectively fine-tuned CLIP, SigLIP, and a VQA-based Qwen3 baseline.The baselines test global similarity, affective adaptation alone, a stronger general-purpose retriever, and image-by-image visual question answering after coarse filtering.
  • Annotation protocol: Each top-20 result was independently rated by five annotators for overall, physical, and affective match on a four-point Likert scale.Annotators viewed only the query text and street-view image.
  • Ground truth and metrics: Aggregated relevance uses majority voting across five annotators, with scores of 3–4 treated as matches and mean scores retained for graded metrics.Precision@k counts successful matches, while mean match and nDCG@k use the aggregated scores.
  • Annotation protocol: Physical match achieves κ= 0.631, compared with κ= 0.430 for affective match and κ= 0.522 for overall match.Unanimous binary agreement occurs on 51.7% of overall-match items.

6 Results

PlaceSeek achieves the strongest overall retrieval performance, with gains spanning precision, graded relevance, physical evidence matching, and affective alignment. Ablations show that grounding is essential for validity, while affective alignment improves ranking among grounded candidates.

  • 6.1 Main Results: 88.0% Precision@5, 3.39/4.0 mean match score, and 0.920 nDCG@5 are PlaceSeek’s top-5 results across ten queries.PlaceSeek remains ahead of baselines including VQA (Qwen3) and SigLIP across these metrics.
  • 6.1 Main Results: 89.5% Precision@20 exceeds SigLIP’s 66.0%, VQA’s 61.5%, and CLIP’s 50.5%.PlaceSeek maintains the highest cumulative precision through most of the top-20 list, improving the broader candidate set for map-based exploration.
  • 6.1 Main Results: 3.63/4.0 physical and 3.41/4.0 affective match scores are PlaceSeek’s highest dimension-specific top-5 scores.Compared with the strongest baseline in each dimension, physical match improves by 0.29 points and affective match by 0.43 points.
  • 6.2 Ablation Study: 38.0% Precision@5 and 46.5% Precision@20 result when SGM is removed, showing that affective re-ranking alone cannot ensure required visual elements.Removing SGM causes the largest ablation degradation, particularly for queries with explicit physical evidence requirements.
  • 6.2 Ablation Study: 74.5% Precision@20 without AAM shows that affective alignment mainly improves rank-sensitive ordering and candidate-set stability among physically valid candidates.The full model matches the no-AAM variant in Precision@5, but achieves higher nDCG and broader-cutoff precision.
  • 6.2 Ablation Study: 88.9% P@5 and 87.2% P@20 are achieved by adding MLLM verification to CLIP+GroundingDINO in the element-level SGM task.The results indicate that each SGM stage contributes to more reliable physical evidence confirmation.
  • 6.3 Performance Across Query Tasks: 100% Precision@5 occurs on six tasks, and PlaceSeek is best or tied for best on eight of ten tasks.The full pipeline remains consistent across activity-, object-, perception-, and mixed-intent queries.

7 Conclusion and Future Work

PlaceSeek frames outdoor place retrieval as intent-aware geospatial search that jointly verifies physical evidence and aligns affective expectations. The authors report strong Milan results but identify limits in geographic, perceptual, visual, and deployment scope.

  • Conclusion: PlaceSeek maps natural-language queries to geolocated street-view results by jointly verifying physical evidence and aligning affective expectations.The framework targets human-centered retrieval of urban outdoor places rather than only named categories or establishments.
  • Conclusion: Ablations show that physical grounding is essential for retrieval validity, while affective alignment refines ranking quality among grounded candidates.This conclusion follows the reported component-specific ablation results.
  • Future Work: Evaluation is limited to one city, ten query tasks, and five annotators, so generalizability across cities, cultures, and user intents remains untested.The authors call for broader studies.
  • Future Work: Affective alignment is constrained by six Place Pulse 2.0 dimensions that cannot fully capture richer semantics such as romantic or cozy.Future work should incorporate broader human perception datasets.
  • Future Work: Street-view appearance alone cannot determine whether a place is actually safe or accessible.The authors suggest adding socioeconomic indicators, crime statistics, or POI context, alongside user-specific spatial constraints.

A.1 Term-Level Presence Score

The term-level presence score combines detector evidence using fixed weights and gives verification greater influence in fusion. Forbidden-term violations are triggered by verification or a fused score above a stated threshold.

  • Term-Level Presence Score: The rule-aware physical reranker uses fixed weights selected from pilot inspection and unchanged across evaluation queries.These weights are design parameters rather than query-adaptive values.
  • Term-Level Presence Score: (α1, α2, α3) = (0.45, 0.25, 0.30) weights maximum box confidence most heavily, with box count and visible scale as supporting evidence.The detector-based presence score uses n_box for detected-box count and a_box for the largest detected box-area ratio.
  • Term-Level Presence Score: η = 0.65 gives the Qwen3-VL verification score greater weight than the detector-only score in term-level fusion.For not-exist rules, Qwen3-VL verification or a fused presence score above 0.45 counts as a violation.

A.2 Physical Rule Score

PlaceSeek computes a physical rule score by aggregating normalized, rule-specific visual evidence available for each query, then combines physical validity with affective alignment and a coarse CLIP prior. Candidate views are physically gated and ranked using physical, affective, and semantic signals.

  • Physical Rule Score: Physical reranking aggregates term-level presence scores according to the rule types active for each candidate query.The score is computed for candidate view v from the visual evidence associated with the query’s physical requirements.
  • Physical Rule Score: The physical rule score is a normalized weighted sum over active must-have, more-better, less-better, and not-exist components.M, M+, M−, and F denote the corresponding term sets; normalized CLIP scores and available-component weights are used in the calculation.
  • Final Fusion: Affective scores are min-max normalized over the physically verified candidate set, with missing affective scores filled with zero before normalization.The normalized coarse-retrieval CLIP score is carried forward from the physical stage.
  • Final Fusion: Physically verified candidates are sorted first by final fused score, then by raw affective score and normalized CLIP score.The fusion design keeps physical verification dominant, uses affective alignment for perceptual refinement, and retains CLIP as a weak global semantic prior.
  • Intent Representation: The intent schema passes physical requirements, affective preferences, activities, and query-specific constraints from intent parsing to the grounding and alignment modules.Physical requirements include rule type, requiredness, and visual rationale, while affective preferences include polarity, perceptual family, and rationale.
  • Semantic Grounding: Grounding verification accepts a region only when it matches the target, is usable for the intended activity, and has adequate overall quality and condition.It rejects unsuitable or inaccessible regions and returns uncertainty when image evidence is too ambiguous to judge confidently.
Loading 2608.24133v1…