PaperlayerView exact revision on arXiv

Source-linked AI summary

FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation

Hang Ren, Peidong Wang, Yichen Gao, Wen Zhang, Jiayang Yu, Xiaocui Yang, Shi Feng

arXiv:2610.20051v1cs.CL

TL;DR

MESC systems can provide weak semantic feedback, allow reward dimensions to compensate for poor support, and lack stage-specific credit. FACES combines multi-source rewards with non-compensatory gating and token-level CoT supervision; under a controlled 15-word response cap, it improves evaluation scores and preference over FIRES, with evidence of transfer across external datasets.

  • Problem

    Existing MESC reinforcement-learning objectives provide weak semantic feedback, allow understanding scores to mask poor support, and lack stage-specific credit for reasoning and response generation.

    Source: §1 · p. 2

  • Method

    FACES uses multi-source semantic rewards, non-compensatory gated fusion, and reasoning-stage markers to align dimension-specific token advantages with CoT and response spans.

    Source: §1 · p. 2

  • Results

    3.61 to 4.04 average MLLM-based evaluation score, 80.22% pairwise win rate, and 75.3% human preference rate over FIRES under the 15-word response cap.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations on three external datasets provide evidence of transfer without additional training, and FACES remains preferred under an independent GLM judge.

    Source: §5 · p. 8

    FACES models instance-level alignment rather than user-specific preferences and remains a research prototype, not for clinical diagnosis, psychotherapy, or crisis intervention without professional supervision.

    Source: Limitations · p. 9, Ethical Considerations · p. 9

Abstract

from arXiv · show

In recent years, with advances in multimodal interaction technologies and the growing demand for intelligent emotional support, Multimodal Emotional Support Conversation (MESC) has emerged as an important task. However, existing systems still suffer from shallow responses and inappropriate support strategies due to weak semantic feedback, cross-dimension reward compensation, and coarse reasoning supervision. To address these challenges, we propose FACES, a fine-grained gated semantic alignment framework for MESC. Specifically, FACES integrates multi-source rewards to construct multidimensional semantic and functional alignment signals, employs a non-compensatory gated fusion mechanism to reduce cross-dimension reward offset, and leverages structured Chain-of-Thought (CoT) anchors to perform token-level credit assignment. Under a controlled response-length protocol, FACES improves the average MLLM-based evaluation score from 3.61 to 4.04 and achieves an 80.22% pairwise win rate over SOTA. Human evaluation under the same protocol yields a 75.3% preference rate. Results on three external datasets further support transferability. The code and dataset are available at https://github.com/OCT-ARR/arr.

1 Introduction

MESC requires responses to connect multimodal understanding with appropriate support, yet existing reinforcement-learning objectives provide weak semantic feedback, allow reward compensation, and assign coarse credit. FACES combines multidimensional rewards, gated fusion, and token-level supervision; its controlled evaluation reports improved scores and preferences over FIRES.

1 Introduction

  • Existing MESC reinforcement-learning objectives use weak semantic feedback, allow strong dimensions to mask poor support, and provide coarse completion-level credit.Correct emotion recognition alone does not ensure an appropriate supportive response.Source: §1 · pp. 1–2
  • FACES combines multi-source semantic rewards, non-compensatory gated fusion, and reasoning-stage markers that align dimension-specific token advantages with CoT and response spans.Its reward sources include CLIP for visual perception, MiniLM for emotion understanding, and an MLLM judge for situation, strategy, and outcome.Source: §1 · p. 2
  • 3.61 to 4.04 average MLLM-based evaluation score, 80.22% pairwise win rate, and 75.3% human win rate for FACES over FIRES under a 15-word response cap on MESC-FT.Results on MELD-ESC300, StickerConv, and AvaMERG also support cross-dataset transferability.Source: §1 · p. 2

2 Related Work

Prior MESC work includes multimodal modeling and explicit reasoning, while reward evaluation and fine-grained optimization address complementary challenges. FACES applies non-compensatory fusion and routes dimension-level advantages to aligned token spans.

2.1 MESC Research

  • MESC research includes multimodal emotion-to-support modeling and explicit reasoning over visual context, emotion, situation, and strategy, but still faces weak semantic feedback and coarse reasoning supervision.Flexible reasoning for MESC was explored by Wang et al. (2025a), while appraisal-guided reasoning and retrieval is described by Chu et al. (2026).Source: §2.1 · p. 2

2.2 Reward Evaluation and Fusion

  • Linear reward fusion can let strong dimensions mask weak ones, motivating non-compensatory fusion for multimodal emotional support.Preference-based and model-based reward evaluation are among prior approaches discussed.Source: §2.2 · pp. 2–3

2.3 Fine-grained Reward Optimization

  • FACES combines heterogeneous dimension-level signals with non-compensatory fusion and routes group-relative advantages to aligned reasoning and response spans.Prior fine-grained methods supervise different units, including structured reasoning, dialogue prefixes, and process–outcome signals.Source: §2.3 · p. 3

3 Methodology

FACES scores structured candidates across semantic and functional dimensions, fuses rewards so support-critical scores can constrain the total, and uses span-aligned token advantages to optimize reasoning and responses.

3 Methodology

  • FACES optimizes structured candidates using CLIP, MiniLM, and an MLLM judge to score five dimensions, then assigns the resulting reward-driven advantages to aligned reasoning and response spans.The five dimensions are visual perception, emotion understanding, situation understanding, support strategy, and response outcome.Source: §3 · p. 3, §3.1 · p. 3

3.1 Multi-source Semantic Rewards

  • Five reward dimensions combine visual and emotion signals with reference-conditioned situation, strategy, and outcome scores; the visual score uses a validation-selected CLIP threshold.The MLLM judge evaluates semantic consistency, functional consistency, and evidence-aware contextual fit across three semantically equivalent prompts.Source: §3.1 · p. 3

3.2 Non-compensatory Gated Reward Fusion

  • After a format-and-outcome warm-up, FACES averages cognitive scores and multiplicatively gates them by strategy and outcome scores, preventing either support-critical dimension from being compensated by stronger cognitive scores.Format-invalid candidates receive zero reward and zero token advantages; the warm-up aims to establish structural control and basic response quality.Source: §3.2 · p. 4

3.3 Fine-grained Rewards

  • FACES converts dimension scores into within-group rank advantages, aligns them to CoT and response spans, and applies gated token-level advantages in clipped GRPO updates.The last five tokens of each segment receive full advantage, earlier tokens receive a decayed advantage, and format-invalid candidates receive zero token advantages.Source: §3.3 · p. 5

4 Experiments

Experiments compare FACES with prior systems and prompted MLLMs, evaluate it under controlled response length, ablate its components, and test transfer. FACES outperforms FIRES in cap-15 MLLM and human preferences, while external evaluations and matched optimization comparisons further assess its performance.

4.1 Experimental Setup

  • Experiments train and evaluate in-domain on MESC-FT and test zero-shot transfer on MELD-ESC300, StickerConv, and AvaMERG.MESC-FT uses an approximately 8:1:1 training, validation, and test split; the external datasets are evaluation-only.Source: §4.1 · p. 5

4.2.1 Automatic Evaluation

  • FACES leads on METEOR, Distinct-3, and DIFF-EPITOME, while FIRES leads on BLEU-2 and ROUGE-L and Gemini-2.5-Flash-Lite leads on BERTScore-F1.FACES also improves over FIRES on BERTScore-F1, despite not leading that metric overall.Source: Table 1 · p. 6, §4.2.1 · p. 6

4.2.2 MLLM Evaluation

  • Under cap-15, the average MLLM evaluation score rises from 3.61 to 4.04, with an 80.22% pairwise win rate over FIRES.Table 2(a) reports gains across all three pointwise dimensions; the pairwise result is significant under a two-sided sign test with ties excluded (p < 0.001).Source: Table 2 · p. 7, §4.2.2 · p. 7

4.2.3 Human Preference Evaluation

  • On MESC-FT and MELD-ESC300, human preference rates for FACES are 75.3% and 70.3%, versus FIRES rates of 14.0% and 14.7%, respectively.Both advantages are statistically significant under two-sided sign tests excluding ties (p < 0.001); annotators also prefer FACES to GPT-4o in 76.0% of MESC-FT and 72.0% of MELD-ESC300 cases.Source: §4.2.3 · p. 7

4.3 Ablation Study

  • Full FACES leads on METEOR, BERTScore-F1, Distinct-3, and DIFF-EPITOME; removing fine-grained credit assignment broadly degrades metrics.Without multi-source rewards, BLEU-2 and ROUGE-L rise while semantic and support-oriented metrics fall; additive fusion allows cross-dimension compensation.Source: §4.3 · pp. 7–8, Table 5 · p. 8

4.4 Generalization Analysis

  • 62.00% Qwen win rate on MELD-ESC300 accompanies significant semantic-metric gains; cap-15 Qwen wins reach 63.7% on StickerConv and 79.3% on AvaMERG.External evaluation is without fine-tuning; on MELD-ESC300, human preference is 70.3% for FACES versus 14.7% for FIRES.Source: §4.1 · p. 5, Table 6 · p. 8, §4.4 · p. 8

4.5 Optimization

  • Under matched backbone, SFT initialization, prompts, and length-controlled evaluation, FACES exceeds DPO, SimPO, and RLOO.The comparison uses the same evaluation protocol for all methods.Source: Table 7 · p. 8, §4.5 · p. 8

5 Conclusion

FACES addresses weak semantic supervision, compensatory reward fusion, and coarse credit assignment in MESC through multidimensional rewards, non-compensatory gating, and span-aligned token-level advantages. It improves evaluations over FIRES under matched length-controlled settings, with results also supporting transfer without additional training.

5 Conclusion

  • FACES addresses three MESC challenges with multidimensional rewards, non-compensatory gating, and span-aligned token-level advantages.The challenges are weak semantic supervision, compensatory reward fusion, and coarse credit assignment.Source: §5 · p. 8
  • On MESC-FT, FACES is preferred to FIRES in matched cap-15 MLLM and human evaluations, and the preference advantage persists under an independent GLM judge.Evaluations on three external datasets provide further evidence of transfer without additional training.Source: §5 · p. 8
  • The findings identify fine-grained semantic alignment with non-compensatory gating as a promising direction for MESC generation.Source: §5 · p. 9

Limitations

The study identifies personalization and evaluation as areas for further extension. Its present scope uses instance-level semantic alignment and keyframe-based visual inputs.

Limitations

  • FACES focuses on instance-level semantic alignment and does not explicitly model user-specific preferences.Source: Limitations · p. 9
  • The evaluation prioritizes reproducibility and controlled comparison, relying on automatic metrics, MLLM assessment, pairwise human preferences, and keyframe-based visual inputs.Source: Limitations · p. 9
  • The authors identify finer-grained human evaluation, more diverse annotators, and richer signals such as vocal prosody and physiological measures as future extensions.These extensions are proposed for realistic support settings.Source: Limitations · p. 9

Ethical Considerations

The paper frames FACES as a research prototype and describes data-use safeguards alongside the experimental and training setup. The model is not intended for unsupervised clinical or crisis use.

Ethical Considerations

  • The authors use MESC-FT and construct MELD-ESC300 from public MELD/EMOTyDA resources while complying with source licensing and redistribution requirements.Raw media are not redistributed without authorization; the paper instead provides acquisition guidance, processing scripts, and reproduction instructions.Source: Ethical Considerations · p. 9
  • FACES remains a research prototype, not a standalone tool for clinical diagnosis, psychotherapy, or crisis intervention.Real-world responses to acute crises or safety risks require human oversight, safety screening, and professional referral protocols.Source: Ethical Considerations · p. 9

A Experimental Setup

  • The experimental setup reports core model, training, decoding, evaluation, reward, and hardware settings while omitting unchanged optimizer defaults and other non-critical details.Source: A Experimental Setup · p. 12

A.1 Training Configuration

  • Training proceeds from supervised fine-tuning to a format-and-outcome GRPO warm-up, then full FACES training with gated rewards and clipped GRPO updates.Table 8 reports the core configuration and GRPO-specific hyperparameters.Source: §A.1 · p. 12, Table 8 · p. 12
  • DPO, SimPO, and RLOO share the backbone, SFT initialization, prompts, and evaluation protocol, but retain their own optimization objectives and update rules.Source: §A.1 · p. 12

A.2 Inference and Evaluation Settings

The appendices specify inference, reward, prompt, and controlled-evaluation procedures. They report reliable reward alignment and persistent FACES preference under length-controlled evaluation, including when reasoning traces are hidden.

A.2 Inference and Evaluation Settings

  • Test-time generation is greedy, while GRPO rollouts sample at temperature 1.3 and top-p 0.95; both judges receive the same context, visual evidence, length-controlled responses, and candidate ordering.Judges can receive up to eight images, and bootstrap analyses use 5,000 resamples with seed 42.Source: §A.2 · p. 12, Table 9 · p. 13

A.3 Reward Configuration

  • FACES uses non-compensatory reward fusion: continuous reward components are normalized to [0, 1], while visual grounding, emotion recognition, and situation understanding aggregate into cognitive reward, and format, strategy, and outcome gate the final reward.Source: §A.3 · p. 13

A.4 Reward Reliability and Cost

  • Three-prompt averaging raises ICC from 0.74–0.79 to 0.90–0.92, gated reward correlates with human Overall Quality at ρ = 0.72, and FACES uses 2.45 times as many GPU hours as FIRES.Source: §A.4 · p. 13

B Prompt Templates

  • Training-time reward construction and post-hoc evaluation use separate prompt families, each with three semantically equivalent variants and machine-readable JSON outputs.The variants preserve input fields, scoring criteria, score range, and output schema while changing surface wording.Source: B Prompt Templates · p. 14

B.1 Training-time Reward Prompts

  • Training-time MLLM prompts score situation understanding, support strategy, and response outcome; CLIP and MiniLM separately score visual perception and emotion understanding.The three prompt variants’ normalized scores are averaged before reward fusion.Source: §B.1 · p. 14

B.2 MLLM Evaluation Prompts

  • Post-hoc evaluation combines pointwise scores for empathy, multimodal appropriateness, and context–response consistency with blind pairwise judgments, without reference annotations.For length-controlled evaluations, judges see only final responses, not explicit reasoning traces; three prompt variants, candidate swapping, and majority voting reduce prompt and order sensitivity.Source: §B.2 · p. 14

C Controlled Evaluation

  • The controlled views compare response-only evaluation with cap-30, cap-25, and cap-15 views, which prefix-truncate each response to the specified word limit.The pairwise significance analysis uses two-sided exact binomial sign tests on decisive comparisons and bootstrap confidence intervals for FACES preference share.Source: C Controlled Evaluation · p. 17, §C.1 · p. 17, §C.3 · p. 17

C.3 Statistical Significance

  • 87.2% on MESC-FT and 86.5% on MELD-ESC300 are FACES’s decisive preference shares under cap-15, where reasoning traces are hidden; FACES remains preferred across reported caps on both datasets.The cap-15 condition is described as the most conservative evidence because it hides explicit reasoning and restricts visible response length.Source: §C.3 · p. 17, §C.2 · p. 17

C.4 Cross-Judge Robustness

With an independent GLM judge, FACES maintains a cap-15 preference advantage over FIRES across datasets. In comparisons with directly prompted MLLMs, both automatic judges and human annotators prefer FACES, though human comparisons cover GPT-4o specifically.

C.4 Cross-Judge Robustness

  • 84.9% cap-15 win rate on MESC-FT, 64.3% on StickerConv, and 65.7% on AvaMERG; GLM’s tie-excluded preferences significantly exceeded chance across all three datasets.GLM’s cap-25 and unrestricted MESC-FT win rates were 97.1% and 99.6%, respectively.Source: §C.4 · p. 17, Table 20 · p. 18

C.5 Strong Direct-Prompted MLLMs

  • FACES beat all six directly prompted MLLMs under both judges, with lower win rates on MELD-ESC300 than MESC-FT but the same preference direction for every opponent.Across opponents, win rates ranged from 71.6%–76.7% on MESC-FT and 65.0%–70.0% on MELD-ESC300 for Qwen; GLM ranges were 69.3%–74.0% and 62.0%–67.0%.Source: §C.5 · p. 18, Table 21 · p. 18
  • Human annotators preferred FACES to GPT-4o in 76.0% of MESC-FT instances and 72.0% of MELD-ESC300 instances, agreeing in direction with both judges.The human comparison included GPT-4o only; comparisons with the other five MLLMs rely on automatic judges.Source: §C.5 · p. 18, Table 22 · p. 18

D Detailed Preference Evaluation

Human preference labels were collected through independent, blinded pairwise judgments and majority voting, with adjudication for three-way disagreements. The agreement analysis compares those labels with Qwen judgments while retaining ties, and its scope is limited to Qwen on FACES–FIRES pairs.

D.1 Human Annotation Protocol

  • Three annotators independently judged anonymized, randomized cap-15 response pairs using dialogue context and visual evidence; majority vote set the label, with a fourth expert adjudicating three-way disagreements.Fleiss’ κ was calculated from the primary annotators’ labels before adjudication.Source: §D.1 · p. 19, Table 23 · p. 19

D.2 Human Preference Counts

  • Human preference counts include adjudicated decisions where applicable, use all evaluated instances including ties as the denominator, and report Fleiss’ κ before adjudication.The comparisons cover FACES versus FIRES on MESC-FT and MELD-ESC300.Source: §D.2 · p. 19, Table 24 · p. 19, Table 25 · p. 19

D.3 Human–MLLM Agreement

  • Human and Qwen labels show agreement beyond chance on both datasets, but substantial sample-level disagreement remains; this supports consistency with human preferences without establishing that Qwen can replace human evaluation.MESC-FT has higher exact agreement but lower Cohen’s κ, with expected chance agreement of approximately 0.629 versus 0.493 on MELD-ESC300.Source: §D.3 · p. 19, Table 26 · p. 20
  • The agreement analysis concerns only the post-hoc Qwen judge on FACES–FIRES pairs, not GLM or FACES–GPT-4o comparisons.Figure 3 presents normalized dimension-level ablations and describes FACES as most balanced across metrics.Source: §D.3 · p. 19, Figure 3 · p. 20

E Additional Ablation Analysis

The additional ablations show that reward dimensions and format control contribute to a balanced metric profile, while gated aggregation avoids critical failures seen with linear aggregation. A supplementary diagnostic also finds lower composite scores when feedback is detached from its corresponding semantic spans.

E Additional Ablation Analysis

  • FACES’ full model achieves the best METEOR, BERTScore-F1, Distinct-3, and DIFF-EPITOME profile; removing format control raises lexical overlap but harms semantic quality and diversity.Removing emotion or situation rewards worsens DIFF-EPITOME more, while removing strategy reward lowers semantic metrics.Source: E Additional Ablation Analysis · p. 20

E.1 Reward Aggregation Diagnostics

  • 0.0% critical failure rates for geometric mean, Tchebycheff, and gated aggregation versus 2.4% for linear aggregation; gated aggregation also has the highest mean Strategy and Outcome scores.The offline comparison selects the top 25% of candidates after excluding those with format scores below 0.5.Source: §E.1 · p. 20, Table 29 · p. 21
  • 103.4% of the global mean reward in the high-risk region under linear aggregation versus 27.6% under gated aggregation; gated aggregation reduces, but does not eliminate, critical-failure selection within prompt groups.The share of groups where a critical failure receives maximum advantage falls from 14.9% to 11.1%.Source: §E.1 · p. 21, Table 30 · p. 21

E.2 Fine-grained Credit Diagnostic

  • 10.7% lower composite score when feedback is decoupled from its corresponding spans, falling from 50.6 to 45.2 in the supplementary five-dimensional diagnostic.The diagnostic used 100 fixed MESC-FT validation instances.Source: §E.2 · pp. 21–22, Table 31 · p. 22

F Hyperparameter Sensitivity Analysis

Sensitivity analyses identify trade-offs around the default visual threshold, gate floor, and endpoint-anchored credit parameters. In comparisons of token-credit mechanisms, endpoint anchoring leads on overall validation reward and test-set MLLM scores; external datasets are reserved for evaluation without additional training.

F Hyperparameter Sensitivity Analysis

  • The default visual threshold τv = 0.60 gives the best DIFF-E. and MLLM average, while lower thresholds slightly improve lexical overlap and higher thresholds reduce semantic and MLLM performance.The analysis reports moderate robustness to changes in τv.Source: F Hyperparameter Sensitivity Analysis · p. 22, Table 32 · p. 23
  • Setting γ = 1.00 improves B-2 and R-L but worsens DIFF-E. and the MLLM average; the reported analysis supports using the gate to limit positive updates for relatively low-reward candidates.When γ is too small, positive local advantages are overly suppressed.Source: F Hyperparameter Sensitivity Analysis · p. 22, Table 33 · p. 23

F.1 Credit Assignment Alternatives

  • Endpoint anchoring reaches 0.0413 Overall reward, exceeding linear and exponential decay by 14.7% and 19.0%; on the test set, its average is 4.04 versus 3.94 and 3.89.Exponential decay scores higher on Format, Emotion, and Situation rewards, but not Overall reward; endpoint anchoring has the highest observed test scores across all three evaluated dimensions.Source: §F.1 · pp. 22, 24, Table 35 · p. 23, Table 36 · p. 23

G External Dataset

  • MESC-FT is the sole training set; MELD-ESC300, StickerConv, and AvaMERG are reserved for cross-dataset evaluation without additional training.The paper states that adding training datasets would change both training-data amount and distribution, making method effects harder to distinguish from additional-data effects.Source: G External Dataset · p. 24

G.1 Benchmark Construction

MELD-ESC300 provides a filtered, audited external benchmark from scripted dialogue, and FACES shows higher MLLM scores and pairwise preference than FIRES under the tested cap-15 protocol. Additional evaluations examine automatic baselines, zero-shot transfer, and qualitative successes and failures.

G.1 Benchmark Construction

  • MELD-ESC300 contains 300 filtered samples, using negative-affect context, non-negative target replies, and five frames from the immediately preceding utterance.The benchmark includes 208 strict and 92 relaxed samples; the target speaker differs from the preceding speaker, and targets are 4–40 words.Source: Table 37 · p. 24, §G.1 · pp. 24–25
  • Expert audit found 274 of 300 samples valid overall, with validity rates of 92.8% for the strict subset and 88.0% for the relaxed subset.The benchmark centers on mildly supportive or non-negative replies, and its target-emotion filter is not a proxy for empathy.Source: §G.1 · p. 25, Table 38 · p. 25
  • MELD-ESC300 is scripted TV dialogue, intended as a controlled external generalization test rather than a replacement for dedicated emotional-support benchmarks.It combines recent negative-affect context, neutral-to-positive target replies, and visual evidence from the preceding turn.Source: §G.1 · p. 25

G.2 MELD-ESC300 MLLM Evaluation

  • 3.35 for FIRES versus 3.73 for FACES in pointwise average scores, with FACES preferred in 62.0% versus 9.7% of pairwise instances on MELD-ESC300.The pairwise difference is significant with ties excluded (p < 0.001); evaluation used cap-15 without additional training.Source: §G.2 · p. 25, Table 39 · p. 25

G.3 Additional Automatic Baselines

  • Table 40 provides additional automatic evaluations of FACES and six directly prompted MLLM baselines on MELD-ESC300.The supplied passage identifies the comparison scope but does not state the results.Source: §G.3 · p. 26, Table 40 · p. 26

G.4 Additional External Generalization

  • On StickerConv and AvaMERG, transfer evaluation uses 300 instances per dataset without additional training; automatic results are complementary rather than uniformly one-sided.FACES mainly improves supportive semantics, diversity, and DIFF-EPITOME, while FIRES remains competitive on some lexical or embedding-based metrics.Source: §G.4 · p. 26, Table 42 · p. 27

H Case Study

  • In the qualitative failure, FACES mistakes Mona’s photograph for a phone, producing a fluent but visually ungrounded follow-up.The authors note that a visual misread can propagate into situation inference and strategy selection, and gated reward fusion cannot fully repair incorrect upstream perception.Source: H Case Study · p. 26, Figure 4 · p. 27

Related papers

Loading 2610.20051v1…