PaperlayerView exact revision on arXiv

Source-linked AI summary

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

Chenyang Yu, Zhaomin Xiao, Yan Huang, Eduardo Blanco

arXiv:2610.38755v1cs.AIcs.IR

TL;DR

Inferring how long people stay at geotagged locations matters for mobility and travel applications, but social-media posts rarely state duration explicitly. The paper builds a human–LLM co-annotated tweet corpus and evaluates duration inference models. Advanced reasoning models improve over standard proprietary models, yet performance plateaus around 0.86 F1 amid ambiguous and culturally nuanced cues.

  • Problem

    Stay duration is rarely explicit in social-media posts, so inferring it from noisy textual, visual, and contextual cues remains challenging despite its relevance to transportation planning and personalized recommendations.

    Source: §I · p. 1

  • Method

    The authors create a corpus of 4,312 geotagged tweets with human–LLM co-annotations and evaluate pretrained language models, text-only LLMs, and vision-language models on the task.

    Source: §I · pp. 1–2

  • Results

    Advanced reasoning models substantially outperform standard proprietary models, but F1 scores plateau around 0.86 even with few-shot examples and multi-turn Chain-of-Thought prompting.

    Source: C. Results · p. 7

  • Takeaways & Limitations

    Accurate inference requires handling ambiguous, culturally nuanced cues and maintaining logical consistency across visual and textual signals.

    Source: §VII · p. 9

    The corpus contains only X tweets geotagged at the US-city level and includes only COVID-related tweets, so its generalizability to other locales is uncertain.

    Source: §VI · p. 8

Abstract

from arXiv · show

Gardening in the backyard implies living there, while checking into an Airbnb more likely means staying for a few days. Such duration judgments are important for understanding human mobility, travel behavior, and social context. However, duration is rarely stated explicitly in social media posts and must instead be inferred from noisy textual, visual, and contextual cues. In this paper, we investigate whether vision-language models possess such duration commonsense. We create a new corpus of geotagged tweets co-annotated by humans and LLMs to serve as a testbed for this task. We find that state-of-the-art models struggle with duration inference, even when guided by multi-turn Chain-of-Thought prompting. Error analyses further reveal that models often fail to integrate critical contextual information, or they overly rely on prominent visual details while overlooking subtle hashtags and semantic nuances. These findings underscore the significant challenges remaining in context-aware multimodal understanding.

I. INTRODUCTION

The paper studies how to infer how long people will stay at geotagged locations from noisy social-media cues, a socially nuanced problem with applications such as estimating travel demand. It introduces a human–LLM co-annotated tweet corpus and reports that models still struggle with ambiguous and culturally nuanced duration cues.

I. INTRODUCTION

  • The task differs from event-duration prediction because social-media posts rarely state stay duration and require reasoning across informal text, hashtags, and visual cues.Prior work has focused on event durations or human-centric duration in formal text; this socially nuanced task remains underexplored.Source: §I · p. 1
  • The examples contrast a brief Morro Bay kayaking trip with a long New Orleans stay suggested by indoor and pet cues and references to sheltering in place.The examples illustrate how text and image context can jointly inform stay-duration judgments.Source: Figure 1 · p. 1, §I · p. 1
  • The paper frames duration inference as determining how long tweet authors will stay at tagged locations from noisy user-generated content.Such judgments can support applications including transportation planning and estimating travel demand.Source: §I · p. 1
  • The authors introduce a corpus of 4,312 geotagged tweets with duration annotations, combining human judgment with LLM consistency and scalability.The annotations indicate how long authors are likely to stay at the locations mentioned in their posts.Source: §I · p. 1
  • The paper evaluates pretrained language models, text-only LLMs, and vision-language models, finding that multi-turn Chain-of-Thought and advanced reasoning improve performance but do not resolve difficult cues.Models frequently overlook contextual constraints or conflate disjoint visual and textual clues.Source: §I · p. 2

II. RELATED WORK

Prior duration research largely addresses events or formal text, leaving informal, multimodal social-media inference about people’s stays comparatively underexplored. The paper addresses this gap with a hybrid human–LLM annotation strategy for a new tweet corpus.

II. RELATED WORK

  • Prior work predicts event durations in formal text and social media, but event duration can differ from people’s stays; extracts where people are located and for how long from formal text.Source: §II · p. 2
  • The paper’s task differs by using informal social-media text and images to infer people’s stay durations, rather than relying on formal text or text alone.The task is challenging because informal posts can contain noisy, socially nuanced cues.Source: §II · p. 2
  • The authors describe their work as the first to study people’s duration commonsense from social-media text and images.They note that few prior studies address duration related to people, including work on location duration in formal text.Source: §II · p. 2

III. A CORPUS OF DURATION COMMONSENSE IN TWEETS

  • Because no existing dataset supports the task, the authors combine crowdworker judgments with LLM annotations to build a cohesive corpus.The strategy draws on human contextual sensitivity and LLM consistency and scalability.Source: §III · p. 2

A. Collecting Tweets

The corpus is drawn from pandemic-era tweets with city-level geotags, text, and images, and excludes explicit duration phrases. Human annotators and LLMs judge how long authors will remain in the geotagged city.

A. Collecting Tweets

  • The authors selected GEOCOV19TWEETS, a dataset of approximately 420,000 tweets, for its precise geotags and range of stay durations.The source captures brief outings as well as prolonged home stays, although pandemic-era data may exhibit behavioral biases.Source: A. Collecting Tweets · p. 2
  • Tweets were retained only if they had US city-level geotags, both text and images, and no signs of corporate or organizational accounts.The filtering used username keywords and poster-like image detection via CRAFT.Source: A. Collecting Tweets · p. 2
  • Tweets with explicit duration phrases, such as “5 days” or “one year,” were removed so the task tests inference rather than extraction.Source: A. Collecting Tweets · p. 2
  • The annotation examples span a 0–24-hour departure, a 1–30-day road-trip stop, and a more-than-30-day residence inferred from everyday home-life cues.The residence example combines a home-cooked meal with hashtags such as #whatsfordinner and #onogrindz.Source: Figure 2 · p. 3

B. Annotation Guidelines

  • Crowdworkers and six LLMs were asked how long the author would stay continuously in the geotagged city after posting.Annotators chose among leaving, staying, living, and cannot determine; roughly 2,500 crowdworkers participated, and each tweet retained an average of 2.5 high-quality annotations after adjudication and filtering.Source: B. Annotation Guidelines · p. 3
  • LLMs received each tweet’s text and image(s), a task-specific system prompt, and the same four-option duration question.The prompt examples are referenced in Table VI.Source: B. Annotation Guidelines · p. 3

C. Annotation Quality Control of Crowdworkers and LLMs

The authors use iterative agreement-based filtering and competence-weighted aggregation to improve the quality of human and LLM annotations. Low-agreement LLM annotators are excluded before final label aggregation.

C. Annotation Quality Control of Crowdworkers and LLMs

  • Crowdworker quality control required approval rates above 95% and at least five seconds per task, with five human annotations collected per task.These filters were based on pilot observations.Source: C. Annotation Quality Control of… · p. 3
  • After iteratively removing the lowest-competence crowdworker’s annotations and recomputing Krippendorff’s α, the final average α was 0.68 across all human annotations.MACE weighted the remaining annotators by inferred competence to aggregate human labels, avoiding simple majority voting.Source: C. Annotation Quality Control of… · p. 3
  • LLMs with Fleiss’ κ below 0.2—Qwen 2.5 and GPT5-mini—were excluded, and MACE aggregated predictions from the remaining four models.Source: C. Annotation Quality Control of… · p. 3

D. LLM and Human Annotation Alignment

Human and LLM annotators agree most on stays longer than 30 days, but disagreements expose recurring ambiguities that the authors resolve through rejection or discarding.

  • The strongest human–LLM agreement is for stays longer than 30 days, while brief-versus-long predictions show substantial disagreement.The confusion matrix reports percentage values, with diagonal cells indicating consensus.Source: D. LLM and Human Annotation Alignment · p. 4, Table II · p. 4
  • Disagreements arise from COVID-19 quarantine duration, confusing the tagged city with a specific location, unclear visit intent, or uncertain residency status.The first two reasons are described as the most common; intent and residency cases are more ambiguous.Source: D. LLM and Human Annotation Alignment · p. 4
  • The authors reject predictions within 30 days for the first two error types, but discard cases with unclear intent or residency.Remaining human or LLM annotations are kept as final labels.Source: D. LLM and Human Annotation Alignment · p. 4

IV. CORPUS ANALYSIS

The filtered corpus contains 4,312 geotagged tweets with adjudicated duration labels, dominated by stays longer than 30 days.

  • The final corpus contains 4,312 geotagged tweets with adjudicated duration labels.Instances pair a tweet with its geotagged US city.Source: §IV · p. 4
  • Roughly 70% of tweets are labeled as stays longer than 30 days.The remaining labels are nearly evenly split between 1–30 days and 0–24 hours.Source: §IV · p. 4
  • The under-1% “I cannot determine” label is excluded from analyses and experiments because it is not a usable duration judgment.Source: §IV · p. 4

V. EXPERIMENTS AND RESULTS

The experiments compare pretrained encoders, text-only LLMs, and VLMs on duration classification, including evaluations that use multi-turn Chain-of-Thought prompting.

  • The task is evaluated as classification from a tweet-and-city pair, with PTLMs fine-tuned and LLMs and VLMs tested without fine-tuning.PTLM baselines include text-only, image-only, and multimodal variants; the latter combine text and image representations.Source: §V · p. 4
  • The evaluation includes text-only LLMs and VLMs with and without deep thinking, while Table III covers common annotator errors and Table IV analyzes prediction pairs.Text-only LLMs receive tweet text, an image caption, and the geotagged city.Source: §V · p. 5, Table III · p. 5, Table IV · p. 5
  • Multi-turn Chain-of-Thought prompting first describes visual elements and their relation to textual cues before the final duration prediction.The intermediate reasoning trace is intended to ground the final prediction.Source: §V · p. 5

A. Experiment setting for pretrained language models

The experimental setup combines trained encoder-based baselines with evaluations of recent LLMs and VLMs, including prompts that separate visual analysis from duration inference.

A. Experiment setting for pretrained language models

  • RoBERTa-Large and Vision Transformer representations are fused by concatenation, addition, or multiplication before a two-layer classifier.The encoders are trainable; models use dropout and are trained with Adam and categorical cross-entropy.Source: A. Experiment setting for pretrained… · p. 5

language models

  • The evaluation covers text-only Qwen 2.5 and Llama 3.2 alongside open and closed VLMs, including models with at least 1 billion parameters.Table V reports weighted-average F1 for model families and evaluation settings.Source: B. Experiment setting for large… · p. 6, Table V · p. 6
  • Multi-turn prompting separates visual description and its relation to text and hashtags from the subsequent duration-inference question.Few-shot examples walk through how visual and textual hints lead to an inference.Source: B. Experiment setting for large… · p. 6

C. Results

Few-shot examples and multi-turn Chain-of-Thought improve duration inference, with deep-thinking VLMs achieving the strongest reported performance. Yet scores plateau, and models still struggle with ambiguous or culturally nuanced cues.

C. Results

  • GLM-4V achieves the highest overall F1 score, 0.86, in the few-shot with Multiturn CoT setting.The reported performance still plateaus around 0.86 despite few-shot exemplars and Multi-turn CoT.Source: C. Results · pp. 6–7
  • Gemini 1.5 Flash’s F1 score rises from 0.33 in zero-shot single-turn to 0.47 in few-shot multi-turn.Few-shot settings consistently improve performance across virtually all models, and Multi-turn CoT provides an additional boost.Source: C. Results · p. 6
  • VLM settings receive original images, whereas text-only settings use image descriptions; the authors report that advanced deep-thinking VLMs outperform caption-reliant LLMs.The paper suggests direct visual grounding helps with nuanced semantic duration information.Source: C. Results · p. 6, Table VI · p. 7
  • Deep-thinking models substantially outperform standard proprietary models, but their F1 scores plateau around 0.86 despite few-shot exemplars and Multi-turn CoT.The authors attribute the remaining difficulty to highly ambiguous or culturally nuanced commonsense cues.Source: C. Results · p. 7

D. Error Analysis

Errors arise from missed contextual and visual cues as well as semantic reasoning failures; deep-thinking VLMs reduce contextual and visual mistakes, but semantic errors remain high. The discussion also limits the corpus and labels, and describes privacy safeguards.

D. Error Analysis

  • Models miss context such as residency-indicating hashtags or geotags, overlook subtle visual cues, and can confuse a visitor’s stay with the user’s residency.Examples include underestimating duration from activity alone, missing a beach walk in an image, and conflating a visitor’s duration with the author’s residency.Source: D. Error Analysis · p. 7, Table VII · p. 8
  • Visual errors dominate text-only LLMs, while non-deep-thinking VLMs resolve many visual errors but primarily struggle to synthesize images with textual hints.Deep-thinking VLMs minimize contextual and visual mistakes, but semantic errors remain high.Source: D. Error Analysis · pp. 7–8, Figure 4 · p. 8
  • Semantic errors remain high even for deep-thinking VLMs, making semantic reasoning the primary bottleneck for duration inference.Non-deep-thinking VLMs do not reduce semantic errors compared with text-only LLMs.Source: D. Error Analysis · pp. 7–8, Figure 4 · p. 8

VI. DISCUSSION

  • The corpus contains US-city-geotagged tweets, is primarily from X, and includes only COVID-related tweets, so it may not generalize to other locales.The authors say their focus is location–stay-duration relationships and believe findings will likely generalize to other domains.Source: §VI · pp. 8–9
  • Duration labels are inferred rather than observed future movement; sparse, ambiguous intent makes fine-grained annotation unreliable, and co-annotation does not eliminate bias.The study uses three coarse labels: 0-24 hours, 1-30 days, and more than 30 days; the authors suggest future work on more objective or uncertainty-aware labels.Source: §VI · p. 9
  • The corpus omits direct user identifiers and supports takedown requests, while worker demographic data are restricted or reported only in aggregate.The paper also states that crowdworkers are compensated according to platform norms and requirements.Source: §VI · p. 9

VII. CONCLUSIONS

The paper introduces duration inference for tweet authors at tagged locations and a corpus of 4,312 geotagged tweets. Its experiments and error analysis identify ambiguous, culturally nuanced cues and logical consistency across visual and textual signals as challenges, with complex semantic reasoning remaining a fundamental limitation.

VII. CONCLUSIONS

  • The authors introduce a task of inferring how long tweet authors stay at tagged locations and create a corpus of 4,312 geotagged tweets.They report that accurate inference requires handling ambiguous, culturally nuanced commonsense cues and maintaining logical consistency across disjoint visual and textual signals.Source: §VII · p. 9
  • Even deep-thinking VLMs face a fundamental limitation in complex semantic reasoning for duration inference.The authors give a flight departure implying hours and a Gateway Arch visit implying days as examples of the commonsense required.Source: §VII · p. 9

Related papers

Loading 2610.38755v1…