Multimodal Rapport Estimation in Real-World HRI

Akihiro Sakuramoto; Takato Hayashi; Ryo Miyoshi; Yuki Okafuji; Shogo Okada

Source-linked AI summary

Multimodal Rapport Estimation in Real-World HRI

Akihiro Sakuramoto, Takato Hayashi, Ryo Miyoshi, Yuki Okafuji, Shogo Okada

arXiv:2608.18401v1Aug 19, 2026cs.HCcs.CLcs.RO

TL;DR

Real-world HRI involves unconstrained disengagement and naturally occurring multi-party participation, making it unclear whether laboratory-developed interaction-quality methods generalize. This study evaluates third-party rapport estimation from multimodal recordings in a Japanese drugstore and finds that Gemini text predictions fused with HuBERT and V-JEPA perform best overall.

Problem
It remains unclear whether interaction-quality modeling developed in controlled laboratories generalizes to real-world HRI with unconstrained disengagement and multi-party participation.
Method
The study builds a rapport-annotated dataset from uncontrolled Japanese retail HRI and compares multimodal predictors, including zero-shot LLMs, embedding-based models, and prediction-level fusion.
Results
Gemini 2.5 Flash performs strongly alone, while fusing its text predictions with HuBERT and V-JEPA achieves the best overall performance.
Takeaways & Limitations
Rapport estimation in real-world HRI should account for variation in interaction duration and multi-party interaction.
Takeaways & Limitations
Because the data come from one Japanese drugstore and a Wizard-of-Oz robot, generalizing the findings to other contexts or autonomous HRI requires caution.

Abstract

from arXiv · show

Evaluating interaction quality in real-world HRI is an important challenge. If interaction quality can be estimated reliably, the results can be used to improve dialogue strategies and ultimately enable robots to adapt their behavior autonomously. However, existing automatic evaluation methods have been developed primarily in controlled laboratory settings, and it remains unclear whether they can be directly applied to real-world environments, where users are free to disengage and multi-party participation may arise naturally. In this study, we investigate the automatic estimation of third-party-rated rapport scores using 62 sessions of multimodal recordings collected in a Japanese drugstore. We compare zero-shot LLMs, pretrained text, audio, and visual models, and their prediction-level fusion. The results show that, in real-world HRI, zero-shot LLMs achieve strong performance, while audio and visual models tend to provide complementary information. In particular, Gemini 2.5 Flash performs strongly as a single model, and a fusion model combining Gemini (text) with HuBERT and V-JEPA performs best overall. Further analyses showed that estimation performance varied across interaction-duration and group-size conditions. These findings suggest that rapport estimation in real-world HRI requires evaluation and model design that account for contextual variability beyond that assumed in laboratory settings.

1 Introduction

This study addresses the challenge of evaluating interaction quality in uncontrolled real-world HRI, where users may disengage freely and multiple participants may join. It focuses on rapport, introducing a real-world annotated dataset and multimodal estimation baselines for WoZ-mediated interactions.

Uncontrolled real-world HRI differs fundamentally from laboratory interaction because users can leave freely, conversation timing is unconstrained, and multiple participants may join.
Reliable interaction-quality estimation could improve dialogue strategies and enable robots to adapt their behavior autonomously, but laboratory-based findings may not generalize to real-world settings.
The study focuses on rapport as relational interaction quality and uses the Connection-Coordination Rapport (CCR) Scale, proposed specifically for HRI [12].
The study contributes a rapport-annotated dataset from an uncontrolled Japanese retail environment, a multimodal baseline, and evidence that fusing zero-shot LLM predictions with embedding-based models performs best on the dataset.

2 Related Work

Related work frames rapport as a dynamic, dyadic property central to interaction quality, while noting that most assessments rely on first-person judgments. Prior HRI and human–agent studies use subjective criteria and multimodal or LLM-based estimation, but largely assume controlled interaction settings rather than real-world variability.

Rapport Conceptualization: Rapport is a dynamic dyadic property comprising mutual attentiveness, positivity, and coordination.It emerges through interaction rather than constituting a stable individual personality trait.
Rapport Measurement: Most previous rapport scales rely on first-person evaluation, whereas Lin et al. [12] introduced a third-person HRI scale with Connection and Coordination factors.The factors distinguish interpersonal warmth and positivity from mutual attentiveness and coordination.
Interaction-Quality Evaluation: Rapport and related subjective criteria are used to evaluate interaction quality because objective measures such as task success or dialogue length can be insufficient.Rapport has also been linked to relationship-building outcomes including improved learning and successful negotiation.
Automatic Estimation: Prior studies have modeled satisfaction, rapport, enjoyment, and related judgments using sentiment, audiovisual social cues, personality measures, and LLMs.However, most work was conducted in laboratory or online settings with relatively controlled timing, participant roles, and conversational structure, unlike real-world HRI.

3 Dataset with Rapport Annotation

The study collected real-world HRI sessions in a Japanese drugstore and filtered them to 62 sessions with 97 analyzable participants. Rapport was measured using mean third-party CCR-8 ratings, which showed strong reliability and varied across interactions.

Dataset construction: The dataset comprised 131 sessions recorded during 32 hours across six days in a Japanese drugstore, where a teleoperated robot provided guidance and casual conversation.Data collection received institutional ethics committee approval.
Dataset construction: Filtering removed sessions with four or more participants or preschool-aged children, leaving 62 sessions, 101 participants, and 97 analyzable speech-producing participants.Analyzed participants produced 7.06 (SD=5.83) utterances on average; sessions averaged 11.85 (SD=9.08) utterances and 54.23 (SD=42.42) seconds of video.
Rapport annotation: Three trained Japanese-speaking third-party annotators rated each participant’s rapport toward the robot using the eight-item, five-point CCR-8 scale.The scale contains Connection and Coordination factors, and individual-level ratings were used in the experiments.
Rapport annotation: Individual-level rapport averaged 3.72 (SD=0.80), ranged from 1.38 to 4.88, and reflected interactions from disengagement to sustained questioning.Low-rapport examples included ignoring prompts or walking away, whereas high-rapport examples involved actively asking questions and sustaining the exchange.

4 Rapport Estimation Model · 4.1 Task Definition

The task estimates each participant’s scalar rapport score from time-series features in text, audio, or visual modalities. Training uses CCC to align predictions with targets in correlation, mean, and variance.

4.1 Task Definition: Each interaction is decomposed into participant-level instances containing observed features and the corresponding rapport score.Interactions may involve one or more participants.
4.1 Task Definition: For each participant, the model receives a time series of feature vectors from one modality: text, audio, or visual.The sequence length is denoted by T, and embedding-based models use a single-modality input.
4.1 Task Definition: The estimation target is a scalar rapport score y ∈ R, with a parameterized function learning predictions ỹ that approximate y.The model is parameterized by θ.
4.1 Task Definition: Because annotator rating styles and scale usage introduce noise, the objective emphasizes agreement between predicted and target scores beyond raw association.The formulation addresses annotator-specific variation in absolute score values.
4.1 Task Definition: CCC jointly captures Pearson correlation with agreement in prediction and target means and variances.CCC ranges from −1 to 1, with 1 indicating perfect agreement.
4.1 Task Definition: The training loss minimizes disagreement so predictions become maximally concordant with ground-truth rapport scores.CCC is used as the basis for the training objective.

4.2 Pretrained Embedding-Based Feature Extraction

The study extracts fixed, L2-normalized representations for each participant using frozen pretrained models across text, audio, and visual modalities. Text and audio operate at the utterance level, whereas visual features operate on video clips with target-user-focused pooling when available.

Preprocessing: Preprocessing combines automatic annotation with manual correction, including manual participant-ID assignment linking transcripts, audio segments, and video clips.The authors identify fully automatic participant tracking and identification as future work rather than the focus of rapport estimation.
Feature extraction: Frozen pretrained models provide fixed participant representations across text, audio, and visual modalities, with text and audio processed per utterance and vision per clip.These representations support multimodal rapport estimation while keeping modality-specific temporal units.
Text and audio: Text utterances use 768-dimensional Sentence-T5-large embeddings, while utterance-segmented speech uses 1024-dimensional HuBERT-large-ll60k embeddings [5].Both representation types are L2-normalized after encoding.
Visual: Visual video is sampled at approximately 8 fps into non-overlapping 64-frame clips, producing 1664-dimensional V-JEPA 2.1 features [13].Features are L2-normalized; available target-user bounding boxes enable weighted spatial pooling before temporal aggregation.

4.3 Embedding-Based Models

Embedding-based rapport models separately process text, audio, and visual streams using additive attention to form interaction embeddings, followed by an MLP that predicts rapport scores.

4.3 Embedding-Based Models: Each modality uses additive attention pooling to aggregate variable-length features into a single interaction embedding, followed by a prediction head that regresses rapport.Separate models are trained for text, audio, and visual modalities.
4.3.1 Additive Attention Pooling: The attention mechanism computes modality-specific scores over sequence steps, masks padding positions, and uses softmax weights to form a weighted-sum interaction embedding.
4.3 Embedding-Based Models: The interaction embedding is passed to a two-layer MLP with dropout rate 0.2, producing a scalar rapport prediction on the z-score scale that is inverse-transformed before evaluation.

4.4 Zero-Shot LLMs · 4.5 Late Fusion · 4.6 Evaluation Procedure

The study evaluated zero-shot LLMs as third-party CCR-8 annotators, combined model predictions through late fusion, and used session-level cross-validation with leakage prevention and complementary regression metrics. Experimental and coding procedures were explicitly documented, including generative-AI assistance for parts of the implementation.

4.4 Zero-Shot LLMs: Three zero-shot LLMs rated eight CCR-8 items as third-party annotators on the human annotators’ Japanese 5-point scale.The evaluated models were GPT-5.4, Claude Sonnet 4.6, and Gemini 2.5 Flash; prompts included item definitions and behavioral examples, with structured JSON outputs.
4.5 Late Fusion: Late fusion combined model prediction scores by unweighted averaging, because weighted averaging performed similarly while requiring additional parameters.The equal-weight scheme was adopted to avoid introducing extra parameters.
4.6 Evaluation Procedure: Evaluation used a predefined 30-fold train/validation/test split for embedding models and the Random Baseline.Splits were constructed at the session level.
4.6.1 Data Splits: Session-level splitting kept participants from each session in one set, excluded test data from training decisions, and pooled out-of-fold predictions for final metrics.Validation data came from remaining non-test data, while model selection and early stopping excluded the held-out test set.
4.6.1 Data Splits: Performance was reported with MAE, PCC, and CCC, with CCC serving as the primary metric aligned with the training objective.MAE measures error magnitude, PCC measures linear association, and CCC accounts jointly for correlation and mean/variance agreement.
4.6.2 Metrics: Generative-AI coding tools assisted parts of the experiments, analyses, and figure-generation code.The tools were Claude by Anthropic and Codex by OpenAI.

5 Experimental Results

Zero-shot LLMs performed strongly for real-world rapport estimation, while audio and visual models provided complementary information. Fusing Gemini 2.5 Flash (T) with HuBERT and V-JEPA achieved the best overall reported performance.

5.1 Overall comparison: Gemini 2.5 Flash (T+A+V) achieved the highest CCC at 0.618, while Gemini 2.5 Flash (T) achieved the highest PCC at 0.665.GPT-5.4 (T) attained the second-highest PCC at 0.644, indicating that text input alone carries substantial useful information.
5.1 Overall comparison: Late-fusion embedding models did not surpass zero-shot LLMs in PCC or CCC, but ST5+HuBERT+V-JEPA (T+A+V) achieved the table’s lowest MAE.Thus, fused embedding models showed an advantage specifically for MAE despite lower correlation-based performance than zero-shot LLMs.
5.1 Overall comparison: HuBERT (A) outperformed all other supervised single-modality models across the three metrics, whereas ST5 (T) achieved the lowest PCC and CCC among supervised models.These results show considerable variation among embedding-based models across modalities.
5.2 Complementarity analysis: HuBERT+V-JEPA (A+V) showed complementary predictive information beyond Gemini (T), with partial correlation 0.359 and incremental ΔR2 0.072.The model was selected because it occupied the informative, non-redundant region of the utility–redundancy map relative to Gemini (T).
5.3 Prediction-level fusion: Gemini (T) + HuBERT+V-JEPA (T+A+V) achieved the best fusion results, with MAE 0.471, PCC 0.717, and CCC 0.656.The fusion configuration averages Gemini (T) with HuBERT and V-JEPA predictions, using effective weights of 1/2, 1/4, and 1/4, respectively.

6 Discussion

Rapport in real-world HRI can be estimated meaningfully with zero-shot LLMs and embedding-based models, with text-only LLMs effective alone and strongest overall when fused with audio-visual embeddings. Performance varied with interaction duration and group size, while comparisons were constrained by differences from prior laboratory work and several dataset, labeling, and evaluation limitations.

6.1 Overall interpretation: Text-only LLMs estimated rapport effectively, and fusing the LLM with audio-visual embedding models achieved the best overall performance.In the real-world setting, the LLM-based predictor outperformed embedding-based late fusion, while text embeddings alone provided little useful signal.
6.1 Overall interpretation: Compared with Speech-to-Joy, the findings differ: the present LLM-based predictor outperformed embedding late fusion, whereas favored text-plus-audio fusion on PCC and CCC.Direct quantitative comparison is limited because studied enjoyment using self-reported labels, while this study estimated third-party-rated rapport.
6.2 Interaction duration: Gemini 2.5 Flash (T) achieved CCCs of 0.563 and 0.551 in short and long interactions, whereas ST5 (T) increased from 0.168 to 0.411.HuBERT (A) declined from 0.463 to 0.420 and V-JEPA (V) declined from 0.330 to 0.295 across the same conditions.
6.3 Group size: Gemini 2.5 Flash (T) maintained high CCC across group sizes and peaked at CCC = 0.721 in three-person interactions, while most supervised unimodal predictors declined.The results indicate greater performance variation for supervised embedding-based predictors, especially across group-size conditions.
6.4 Limitations: The conclusions are exploratory because data came from one Japanese drugstore setting, 97 individual cases, and interactions with available audio and transcripts, limiting generalizability.Additional analyses used smaller subsets, and the target was third-party-rated observable rapport rather than participants’ own experienced rapport, enjoyment, or satisfaction.
6.4 Limitations: Model comparisons were practical baselines rather than evidence that LLMs are inherently superior because inputs and inference frameworks were not strictly identical.Zero-shot performance may also vary with prompts, model versions, API specifications, and input formats.

7 Conclusion

The study examined automatic estimation of third-party-rated rapport in real-world HRI, finding strong text-only performance from Gemini 2.5 Flash and the best overall performance from fusing Gemini with HuBERT and V-JEPA. The results suggest complementary roles for LLMs and embedding-based predictors and motivate robustness to interaction-duration and multi-party variability.

7 Conclusion: Fusing Gemini (T) with HuBERT and V-JEPA yielded the best overall performance for estimating third-party-rated rapport scores in real-world HRI.The fusion result suggests that LLMs and embedding-based predictors are complementary rather than competing components.
7 Conclusion: Gemini 2.5 Flash (T) achieved strong predictive performance using text-only input.
7 Conclusion: Future work should develop models robust to variation in interaction duration and multi-party interaction.

8 Safe and Responsible Innovation Statement

Because the study estimates third-party-rated rapport from multimodal real-world HRI data, it requires privacy protection and responsible use. Rapport estimation does not directly measure internal states and should not alone determine evaluations or decisions about individuals.

8 Safe and Responsible Innovation Statement: Rapport estimation from multimodal real-world HRI data requires privacy protection and responsible deployment practices.Deployments should restrict raw-data access, minimize retention, support de-identification, and provide clear notice with appropriate consent procedures.
8 Safe and Responsible Innovation Statement: Rapport estimates do not directly measure an individual’s internal state and should not be the sole basis for evaluating or making decisions about individuals.