EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration

• VEC models infer sentiment polarity or recognize emotions in images, while emotion perception can be subjective across viewers.The introduction frames VEC as a component in developing and evaluating visual emotional intelligence in agents.Source: §1 · pp. 1–2
• EmoCaliber verbalizes confidence alongside emotion predictions so users can selectively rely on outputs, including when cues are ambiguous or beyond the model’s competence.Clear emotional cues are associated with high confidence, while emotionally ambiguous or out-of-competence responses receive low confidence.Source: §1 · pp. 2–3
• The three-stage framework develops structured reasoning, teaches confidence verbalization, then calibrates confidence expression with reinforcement learning.The stages use structured reasoning via supervised fine-tuning, confidence-augmented supervised fine-tuning, and reinforcement learning to calibrate confidence while incentivizing more precise reasoning.Source: Figure 2 · p. 2, §1 · p. 3
• EmoCaliber shows advantages in self-evaluated confidence and consistent superiority in in-domain and out-of-domain VEC performance on VECBench.The benchmark groups six VEC datasets into training and test splits.Source: §1 · pp. 3–4

2.1.1 SMALL-SCALE SPECIALIZED VEC MODELS

• Small-scale specialized VEC models are tied to their training objectives, with generalization degrading across emotion taxonomies and image domains.Their progression includes hand-crafted features, pretrained models, and vision–language pretraining.Source: §2.1.1 · p. 4

2.1.2 VEC SYSTEMS BASED ON MULTIMODAL LARGE LANGUAGE MODELS

• MLLM-based VEC offers flexibility across task formulations, with tuning-based and training-free approaches exploring complementary ways to improve emotion comprehension.Examples include emotion-focused instruction tuning and methods that guide affective reasoning through prompts or intermediate states.Source: §2.1.2 · p. 4

2.1.3 HANDLING SUBJECTIVITY IN EMOTION PERCEPTION

• Emotion subjectivity can leave plausible interpretations unrepresented by majority-voted labels, while existing responses include statement-judgment benchmarks and multiple-label video annotation.Confidence verbalization is proposed as an alternative that provides a quantitative sense of possible subjective interpretations and aligns with established frameworks.Source: §2.1.3 · p. 5

2.2 CONFIDENCE VERBALIZATION AND CALIBRATION

• Confidence-estimation approaches range from prompting and sampled-answer consistency to group-based, binary, and reinforcement-learning methods, but fine-grained sample-wise uncertainty remains a challenge.The discussion identifies modest calibration effectiveness or increased latency for early methods and notes the use of Brier score in an RL reward by RLCR.Source: §2.2 · p. 5

3.1 STAGE 1: ENDOWING WITH STRUCTURED REASONING

• Stage 1 structures emotion reasoning into five steps and SFT-trains on 143,446 quality-screened VEC-CoT triplets split 6:3:1.The pipeline synthesizes CoTs from image–label pairs and retains those passing quality evaluation; the steps span salient elements, people, context, interactions, and comprehensive analysis.Source: §3.1 · pp. 5–6, Figure 4 · p. 6, Table 1 · p. 6

3.2 STAGE 2: TEACHING TO VERBALIZE CONFIDENCE

• Stage 2 estimates confidence using normalized VAD emotion-label distance and a probability-based prior, then appends the estimate to the response for SFT.Incorrect predictions receive confidence values in [0, 0.5].Source: §3.2 · pp. 7–8

3.3 STAGE 3: CALIBRATING CONFIDENCE EXPRESSION

• Stage 3 uses GRPO with confidence, correctness, and format rewards, penalizing confidence misaligned with prediction correctness.The confidence reward encourages high confidence for correct predictions and low confidence for incorrect ones.Source: §3.3 · p. 8

4.1 IMPLEMENTATION DETAILS

• Training uses batch size 8 and learning rates of 1e-5, 4e-6, and 1e-6 across the three stages; stage one freezes the vision encoder, while stage three tunes all parameters.Stage three uses a KL coefficient of 0.01.Source: §4.1 · p. 8
• Table 2 compares advanced MLLMs on VECBench’s in-domain Visual Emotion Recognition tasks, marking the best and second-best results.The experiment section presents implementation details, benchmark setup, performance comparisons, and analytical studies.Source: §4 · p. 8, Table 2 · p. 9

4.2 EVALUATION ON VECBENCH

• VECBench consolidates datasets and reserves held-out data for testing, mitigating potential leakage across fragmented VEC datasets.Its test split includes in-domain and out-of-domain samples.Source: §4.2 · p. 9
• VECBench combines six datasets, with training drawn from FI, WebEmo, and EmoSet and tests covering corresponding in-domain tasks plus Abstract, Artphoto, and UnbiasedEmo.The listed training datasets cover both Visual Emotion Recognition and Visual Sentiment Analysis.Source: §4.2 · p. 9

4.3 BASELINES AND EVALUATION METRICS

• Baselines include Qwen2.5-VL, InternVL3.5, Qwen3-VL, GPT-5, EmoVIT, and Emotion-LLaMA; reported EmoVIT results are reproduced using Qwen2.5-VL.AffectGPT is excluded from the main comparisons because it targets video and preliminary tests found it substantially underperformed image-targeted baselines on VEC.Source: §4.3 · pp. 9–10
• Tables 3 and 4 compare MLLM emotion-prediction performance on in-domain Visual Sentiment Analysis and out-of-domain Visual Emotion Recognition tasks, respectively.Source: Table 3 · p. 10, Table 4 · p. 10
• Emotion prediction is evaluated with Accuracy and macro-F1, while confidence is assessed using ECE, Brier score, and AUC.Lower ECE and Brier scores are preferable; AUC measures discrimination between correct and incorrect predictions.Source: §4.3 · p. 10

4.4 COMPARISON ON EMOTION PREDICTION

• 53.70 average accuracy on ID VER exceeds second-best Qwen3-VL by 10.05, and EmoCaliber significantly outperforms all competitors on those tasks.The authors attribute the gains to structured reasoning that captures fine-grained visual cues and affective correlations.Source: §4.4 · p. 10
• On ID VSA, EmoCaliber has the highest average accuracy and ranks in the top two across all subtasks, though its gains are less pronounced.The authors suggest simpler VSA tasks may offset some advantages of stronger affective reasoning.Source: §4.4 · p. 11
• 50.37 average accuracy on OOD VER ranks second, slightly behind GPT-5 at 50.55, with test samples entirely outside training domains.The authors report this as evidence that performance gains are not merely due to memorizing in-domain patterns.Source: §4.4 · p. 11

4.5 COMPARISON ON CONFIDENCE ESTIMATION

• Baseline confidence estimates use verbalized scores, the probability of the first decisive answer token, or average token probability across the response.The response-wide average includes both reasoning and the final answer.Source: §4.5 · p. 11
• 13.63 ECE on ID VER is nearly half the 26.31 ECE of second-best Qwen3-VL (Ans), and EmoCaliber significantly outperforms competitors on ID and OOD VER confidence estimation.On ID VSA, EmoCaliber performs slightly below GPT-5 and broadly comparably to other state-of-the-art MLLMs.Source: §4.5 · p. 11
• The authors characterize EmoCaliber as a strong confidence-estimation model and a practical baseline for reliable VEC systems.Source: §4.5 · p. 12

4.6 ANALYSIS OF MODEL EVOLVEMENT

• The third stage reverses the second-stage prediction decline, surpasses the scaffold model, and improves all reported confidence metrics.The results support the intended contribution of each training stage.Source: §4.6 · p. 12

4.7 ABLATION OF LEARNING RATE

• A first-stage learning rate of 1e-5 achieves the highest average accuracy across tasks, balancing in-domain performance with out-of-domain generalization.Lower rates slightly underperform on ID VER and ID VSA; higher rates degrade OOD VER, possibly through overfitting.Source: §4.7 · p. 12
• A second-stage learning rate of 4e-6 achieves top-2 performance on most tasks, with only minimal prediction-accuracy decline despite optimizing confidence-related tokens.This rate is lower than the first-stage 1e-5 because fewer tokens are updated.Source: §4.7 · p. 13

4.8 ABLATION OF REWARD DESIGN

• Minor RL configuration changes produce divergent optimization outcomes, making confidence calibration notably unstable in VEC.Table 9 compares the effects of different RL setups.Source: §4.8 · p. 13
• The initial setup uses the Rconf formulation from RLCR (Damani et al., 2025) with standard GRPO, but yields severe overconfidence, concentrating estimates above 0.9.The reward is described as aligned with calibration metrics but imposes relatively smooth constraints across confidence levels.Source: §4.8 · p. 13
• The authors jointly remove advantage normalization and replace the reward with a log-likelihood-based objective to mitigate overconfidence.Either modification in isolation is insufficient to achieve the intended calibration behavior.Source: §4.8 · pp. 13–14

4.9 CASE STUDY

• In ID VER, ID VSA, and OOD VER examples, EmoCaliber reasons over subjects, fine-grained objects, contextual backgrounds, and artistic or stylistic cues.The cases illustrate diverse visual cues being considered together.Source: §4.9 · p. 14
• The examples show higher confidence for relatively clear cases and lower confidence for ambiguous inputs such as abstract artwork.The estimates reflect both emotion-perception subjectivity and the model’s self-assessed reliability.Source: §4.9 · p. 14

5 CONCLUSION

• The authors unify six widely used VEC datasets into VECBench for fair, systematic evaluation and hope it will help establish baselines for future research.The conclusion also presents the VEC-CoT dataset and three-stage training framework as components of the work.Source: §5 · pp. 14–15
• EmoCaliber verbalizes confidence alongside emotion predictions, and the authors report superior performance in both emotion prediction and confidence estimation.The conclusion presents this as a practical solution toward more reliable VEC systems.Source: §5 · pp. 14–15