PaperlayerView exact revision on arXiv

Source-linked AI summary

RobotEQ: Towards Social Proactive Intelligence in Embodied Agents

Kuofei Fang, Zheng Lian, Xinyi Che, Haomin Ouyang, Shufan Zhang, Xuehao Wang, Qi Liu, Liyi Liu, Chenqi Zhang, Wenxi Cai, Wenyu Dai, Jinyang Wu, Fan Zhang, Haoyu Chen, Xiaobai Li, Bin He

arXiv:2605.06234v3cs.ROcs.HC

TL;DR

Existing proactive assistance research is limited to narrow scenarios and task execution, while embodied agents also need to respect social expectations across diverse settings. The paper introduces SPI and RobotEQ, a dataset and benchmark with behavior-judgment and spatial-grounding tasks. Its analysis finds that current models have notable SPI limitations and lag behind human performance.

  • Problem

    Proactive-assistance research remains confined to limited scenarios and task execution, while embodied agents must also interpret social expectations.

    Source: §1 · pp. 1–2

  • Method

    The paper defines SPI and introduces RobotEQ, with behavior judgment and spatial grounding supported by curated data and human annotations.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Current models exhibit notable SPI limitations and lag behind human performance.

    Source: §7 · p. 10

  • Takeaways & Limitations

    RobotEQ provides a dataset and benchmark for studying socially appropriate embodied-agent behavior across diverse scenarios.

    Source: §1 · p. 2, §7 · p. 10

    The work focuses exclusively on images, and its annotation pool lacks cultural diversity, so results may reflect annotators’ decision boundaries rather than generalizable social principles.

    Source: Limitation · pp. 10–11

Abstract

from arXiv · show

Embodied agents represent a prominent research focus across both academia and industry. The prevailing paradigm has gradually shifted from reactive assistance, which requires explicit user queries, to proactive assistance, capable of recognizing human needs and offering support without explicit instructions. Nevertheless, existing studies on proactive assistance remain confined to narrow scenarios and primarily emphasize task completeness, whereas real-world agents must operate in open-domain environments while adhering to social expectations. To bridge this gap, we extend the concept of proactive assistance to Social Proactive Intelligence (SPI), characterized by diverse scenarios, social understanding, and robot-centric behaviors. We further introduce RobotEQ, a dedicated benchmark for SPI. We first define two tasks: behavior judgment, emphasizing global contextual understanding, and spatial grounding, focusing on local perceptual details. Building on these tasks, we construct RobotEQ-Data, a dataset comprising 1,812 synthetic and 223 real-world scenarios, 7 social facets, 22K+ human annotations, 3K+ behavior judgment questions, and 3K+ spatial grounding questions. Furthermore, we establish RobotEQ-Bench to evaluate the performance of representative models. Experimental results demonstrate that current models fall short of achieving reliable SPI. Further analysis reveals that incorporating external social knowledge yields consistent improvements. This work aims to advance the development of socially desirable embodied agents in open-domain environments.

1 Introduction

Proactive assistance can infer human needs, but existing work is limited to narrow scenarios and task execution; socially appropriate behavior across diverse settings remains a challenge. The paper introduces SPI and RobotEQ to address this gap through two evaluation tasks and a benchmark.

1 Introduction

  • Proactive assistance can infer human needs without explicit instructions, but existing research remains confined to limited scenarios and emphasizes task execution.A task can succeed while violating social expectations, such as using someone else’s glass to fetch water.Source: §1 · pp. 1–2
  • RobotEQ evaluates SPI through behavior judgment and spatial grounding, with 22K+ human annotations and 3K+ questions for each task.Behavior judgment targets global context; spatial grounding targets local perceptual details.Source: §1 · p. 2
  • Current models show notable limitations in SPI and lag behind human performance.The paper also reports comprehensive evaluation and error analysis of representative models.Source: §1 · p. 2
  • SPI extends proactive assistance toward understanding social expectations across diverse scenarios, beyond task success alone.The paper frames this as realizing socially desirable embodied agents in open-domain environments.Source: §1 · p. 2

2 Related Work

Related work spans reactive assistance, proactive assistance, and social intelligence, but the paper identifies gaps in scenario breadth, social dimensions, and embodied-agent perspective. It positions SPI and RobotEQ as addressing these gaps.

2 Related Work

  • Reactive agents follow human commands through perception, planning, and action, while proactive agents anticipate needs without explicit instructions.Proactive assistance is described as actively perceiving environments, inferring human needs, and offering support.Source: §2 · pp. 2–3
  • LLM-enabled embodied models can handle open-ended instructions but remain fundamentally reactive, motivating a shift toward anticipating user intent.The passage links this shift to limited user-friendly interaction.Source: §2 · p. 3
  • Prior proactive benchmarks confine evaluation to narrow predefined scenarios or limited dimensions, motivating SPI’s focus on diverse embodied scenarios and social facets.Examples include safety- or navigation-centered benchmarks and long-horizon household collaboration.Source: §2 · p. 3
  • Most social-intelligence research centers on human behavior and interaction, whereas embodied agents have distinct viewpoints and action spaces.RobotEQ targets social intelligence for embodied agents through a dataset and benchmark.Source: §2 · p. 3

3 Social Proactive Intelligence

SPI concerns socially appropriate behavior by embodied agents across diverse scenarios. Its properties span social reasoning and multimodal context as well as robot-centered inputs, actions, and hardware-independent planning.

3 Social Proactive Intelligence

  • SPI aims for embodied agents to adhere to social expectations and act appropriately across diverse scenarios.The paper characterizes it through six properties.Source: §3 · p. 3
  • SPI requires open-domain scenarios, socially desirable reasoning, and multimodal judgments based on events, human states, and environmental context.Examples of social standards include respecting privacy and ensuring safety.Source: §3 · p. 3
  • Inputs and behaviors are defined from the robot’s perspective, while high-level semantic planning is decoupled from hardware-specific actions.The paper describes this design as aligned with SPI’s generalization objective.Source: §3 · pp. 3–4

4 Dataset Construction

RobotEQ-Data is constructed to support behavior judgment and spatial grounding across varied embodied scenarios, using real and synthetic images and human-verified annotations. Its statistics describe scenario and facet coverage, while annotator agreement is used to assess label reliability.

4.1 Scenario Design

  • The scenario taxonomy has 10 major categories and 56 fine-grained subcategories, with LLM-generated scenarios organized hierarchically.An expert model removes duplicates after multiple generation rounds.Source: §4.1 · p. 4

4.2 Image Collection

  • Real-image curation filters candidates for clarity, robot-centric perspective, scenario suitability, and duplication, leaving about 200 images.Human experts review the curated images; synthetic images are generated from scenario-specific prompts and refined through expert-model and human checks.Source: §4.2 · p. 4

4.3 Behavior Judgment

  • Candidate behaviors receive proper, improper, or invalid labels from multiple annotators, with majority voting and expert review for low agreement.Invalid behaviors and behaviors inferable without visual input are excluded.Source: §4.3 · pp. 4–5

4.4 Spatial Grounding

  • Spatial-grounding questions use edited images with four candidate regions; three annotators select all applicable answers, allowing multiple valid regions.Invalid images are excluded, and majority vote determines the final answers.Source: §4.4 · p. 5

4 Dataset Construction

  • RobotEQ-Data pairs each scenario with behavior-judgment and spatial-grounding questions across seven evaluation facets.The facets include safety, privacy, proxemics, cooperation, politeness, priority, and sensitivity.Source: §4 · p. 4, §4.5 · p. 6

4.5 Dataset Statistics

  • Retail and Hospitality & Consumer Services is the largest scenario category, while sensitivity and cooperation account for the largest facet shares.Each scenario is paired with both task types, and behaviors may map to multiple facets.Source: §4.5 · p. 6
  • Annotators are screened and trained before labeling, and a domain expert calibrates provisional majority-vote answers into final ground truth.The paper reports high inter-annotator agreement as evidence of label reliability.Source: §4.5 · p. 6

5 Experimental Setup

The evaluation compares behavior-judgment performance across synthetic and real subsets using majority-vote and annotator-aware metrics, alongside three categories of VLM baselines.

5 Experimental Setup

  • Macro-F1 is the primary subjective-unaware metric because proper and improper behaviors are imbalanced; Micro-F1 is a secondary diagnostic.Table 1 reports results on synthetic and real subsets using subjective-unaware and subjective-aware metrics.Source: §5 · p. 6, Table 1 · p. 7
  • Probability of Agreement (PA) measures expected agreement between a model and a randomly sampled human rater.Evaluation units are behavior-judgment items or individual candidate options in spatial-grounding questions.Source: §5 · p. 6
  • Consensus-Weighted Probability of Agreement (CWPA) divides per-question agreement by human agreement rate, penalizing mismatches more on high-consensus items and discounting them on ambiguous items.The weighting is intended to prevent ambiguous items from diluting error signals attributed to model misunderstanding rather than rater disagreement.Source: §5 · p. 7
  • The benchmark evaluates closed-source VLMs, locally run open-source general-purpose VLMs, and open-source task-specialized VLMs.The categories represent an upper-bound reference, readily available resources under constrained compute, and whether specialized visual skills transfer to SPI, respectively.Source: §5 · p. 7

6 RobotEQ-Bench

RobotEQ-Bench finds current VLMs remain below human performance, with results varying by task and analyses revealing challenges in robustness, implicit social norms, and inference cost.

Behavior Judgment and Spatial Grounding

  • Models lag human judgment on both tasks; closed-source models lead on behavior judgment but show no clear advantage on spatial grounding.Spatial grounding requires joint reasoning over visual details, spatial layout, and social norms.Source: §6 · p. 8, Figure 5 · p. 8

Synthetic Data Quality Assessment

  • Synthetic-only training transfers strongly to real-domain evaluation, supporting the authors’ conclusion that the synthetic data generalize to real-world scenarios.The evaluation trains on synthetic data and tests on the real subset.Source: §6 · p. 8, Figure 7 · p. 8

Annotation Quality Analysis

  • Human annotators achieve consistently strong results on both tasks, supporting the reliability of the annotations.The annotators held advanced degrees and AI backgrounds; the authors also report high inter-annotator agreement.Source: §6 · p. 9

Robustness to Visual Perturbation

  • Nearly all evaluated VLMs lose performance under visual perturbations, with fisheye distortion causing particularly severe drops.The tested perturbations are blur, fisheye, and fisheye combined with water; the authors attribute the fisheye effect to an obstructed field of view.Source: §6 · p. 9, Figure 8 · p. 9

Performance Comparison across Social Facets

  • GPT-5.5 is closest to human performance across seven facets, but sensitivity remains difficult because it requires understanding unstated constraints and human emotional states.The authors describe frontier models as capturing a substantial portion of explicit, commonly observed norms while still struggling with implicit social knowledge.Source: §6 · p. 9, Figure 9 · p. 9

RAG for Social Knowledge Augmentation

  • Retrieval-augmented social knowledge improves performance for most models, especially open-source ones, but raises inference time or API costs.The added context increases input length; the authors say practical use requires weighing cost against performance.Source: §6 · p. 10, Table 2 · p. 10

Error Analysis

  • GPT-5.5 errors include overly aggressive or cautious actions, insufficient social commonsense, and spatial choices that ignore social appropriateness.The analysis argues that SPI requires balancing task goals, human states, social norms, and spatial norms in a unified decision process.Source: §6 · p. 10

7 Conclusion

The conclusion presents RobotEQ as a dataset and benchmark for evaluating socially aware embodied agents, and reports that current models lag human performance.

7 Conclusion

  • RobotEQ combines behavior-judgment and spatial-grounding data with 22K+ human annotations to evaluate representative VLMs, whose performance still lags behind humans.The paper also reports annotation-quality checks, visual-clue analysis, perturbation robustness analysis, and RAG as a social-knowledge augmentation strategy.Source: §7 · p. 10

Limitation

The authors identify image-only inputs and limited cultural diversity in the annotation pool as scope boundaries. RobotEQ’s scenario taxonomy covers 10 major categories and 56 fine-grained subcategories.

Limitation

  • Image-only inputs leave video and audio cues underexplored, and the authors plan to extend RobotEQ to multimodal inputs.They identify temporal dynamics in video and accompanying audio as useful cues for proactive intelligence.Source: Limitation · p. 10
  • Figure 10 identifies four representative failure modes: overly aggressive or cautious behavior, limited basic social knowledge, and spatial grounding errors.Source: Figure 10 · p. 11
  • Limited cultural diversity means results may reflect annotators’ decision boundaries rather than generalizable social principles.The authors plan to broaden annotator diversity in future work.Source: Limitation · p. 11

A.1 Scenario Classification

  • RobotEQ’s scenario classification comprises 10 major categories and 56 subcategories, refined to capture distinct social reasoning challenges with broad environmental coverage.The taxonomy draws on surveys and structured expert discussion.Source: §A.1 · p. 17, Figure 11 · p. 17

A.2 Scenario Generation

RobotEQ builds scenario and image pools, then constructs behavior-judgment and spatial-grounding questions with human review and annotation. The pipeline combines generation, filtering, refinement, and human-established labels.

A.2 Scenario Generation

  • The beam-merge strategy generates roughly 100 scenario candidates per subcategory, then deduplicates them and uses prompting rules to target socially meaningful decisions.Ten requests each yield at least ten candidates; a separate expert model removes near-duplicates.Source: §A.2 · pp. 18–19

B.1 Real Images

  • The real-image pipeline creates 1,344 viewpoint-diverse search queries, crawls multiple sources, and filters candidates through model screening, image-quality checks, diversity selection, and expert curation.Crawling yields approximately 44,428 raw images.Source: §B.1 · p. 20

B.2 Synthetic Images

  • Synthetic-image construction translates scenario descriptions into first-person visual prompts specifying spatial layout and socially salient cues, then generates one candidate image per prompt.Directly using scenario text as an image prompt can omit social cues or distort spatial relations.Source: §B.2 · p. 20
  • An expert model reviews generated images against seven quality criteria, flags hard-constraint failures or four-or-more failed criteria for revision, and sends images through refinement and human verification.The revision process uses expert feedback and final human checks for robot-centric perspective and photorealism.Source: §B.2 · pp. 20, 21, 23, Table 3 · p. 22

C Behavior Generation

  • Behavior generation uses scenario text and robot-centric images to propose five proper and five improper scene-grounded behaviors, with human majority votes establishing final labels.Model-assigned propriety labels are removed and candidate order is shuffled before annotation.Source: C Behavior Generation · p. 23, Figure 17 · p. 23, Figure 18 · p. 24

D Human Annotation for Behavior Judgment

  • Behavior labels are established through recruitment and training, a pilot study, and full-scale annotation; three annotators label each behavior, with majority voting and expert review.Seven annotators are selected for full-scale labeling; invalid behaviors are excluded.Source: D Human Annotation for Behavior Judgment · p. 24

E Spatial Grounding Questions Generation and Annotation

  • Spatial questions begin with image-only human proposals, then a two-stage process generates question titles and annotated images; annotators independently determine valid answer regions.The single-stage variant more often produces misplaced, overlapping, or missing annotations.Source: E Spatial Grounding Questions… · pp. 24–25

F Models Used in Data Construction

RobotEQ-Data uses commercial models at distinct construction stages, alternating model families between consecutive quality-critical steps to reduce systematic bias. RobotEQ-Bench evaluates closed-source, general-purpose open-source, and task-specialized open-source vision-language models.

F Models Used in Data Construction

  • The data-construction pipeline employs frontier commercial LLMs at different stages and alternates model families between consecutive quality-critical steps to reduce systematic bias.Table 4 lists each model’s pipeline role and API documentation.Source: F Models Used in Data Construction · p. 26, Table 4 · p. 26

G Evaluated Model Details

  • The evaluated model groups comprise closed-source API models, open-source general-purpose VLMs, and open-source task-specialized VLMs.The grouping compares frontier systems, broadly usable open models, and models specialized for fine-grained grounding or document understanding.Source: G Evaluated Model Details · p. 26

G.1 Closed-Source VLMs via API

  • Closed-source baselines include Gemini 2.5 Pro, GPT-5.4, GPT-5.5, Claude Sonnet 4.6, and Gemini 3.1 Pro Preview.Source: §G.1 · p. 27

G.2 Open-Source General-Purpose VLMs

  • The evaluated models included Qwen2.5-VL-7B-Instruct for practical open-source deployment constraints, Aya-Vision-8B for multilingual multimodal training, InternVL3-8B for scale comparison, GLM-4.1V-9B-Thinking [24] for visual reasoning, and LLaVA-OneVision-7B [30] as a strong open-source baseline.Source: §G.2 · p. 27

G.3 Open-Source Task-Specialized Vision Models

The benchmark includes task-specialized visual models alongside general-purpose systems and standardizes evaluation settings. Its facet-level analysis reports imbalanced scenario-label frequencies, with Sensitivity most frequent and Privacy less frequent.

G.3 Open-Source Task-Specialized Vision Models

  • Task-specialized baselines include GUI grounding, GUI action grounding, region-level grounding, and document-understanding models.The models test whether grounding- or document-focused visual skills transfer to RobotEQ tasks.Source: §G.3 · p. 28

G.4 Model Summarization; G.5 Experiment Settings

  • Open-source models use Hugging Face checkpoints and model-specific settings where available; local inference uses greedy decoding and images resized to at most 768 pixels.The local setup uses vLLM for 17 models and Transformers for five unsupported architectures; API models use temperature zero and structured JSON output.Source: §G.4 · p. 28, §G.5 · p. 29

H Facet Details

  • Facet labels can overlap, and human annotators review and correct LLM-generated labels against the predefined taxonomy.Gemini 3.1 Pro Preview assigns one or more labels before human calibration.Source: H Facet Details · pp. 29–30
  • Sensitivity has 1,442 labels, Cooperation 1,403, Safety 983, and Privacy 325, showing an imbalanced facet distribution.Privacy is less frequent in the collected scenario pool but remains relevant to evaluating respect for personal boundaries.Source: H Facet Details · p. 30, Table 6 · p. 30

I Spatial Grounding Experiment

The spatial-grounding evaluation reports results for representative VLMs across three metrics, with human performance as an upper bound. The accompanying passages describe role-specific social-knowledge references and RobotEQ’s ethical scope.

I Spatial Grounding Experiment

J RAG Knowledge Base Construction

K Ethics Statement

  • RobotEQ uses synthetic images without real individuals and limits content to prosocial service, excluding violent, discriminatory, or harmful material; results do not establish deployment readiness.Source: K Ethics Statement · p. 31

Related papers

Loading 2605.06234v3…