PaperlayerView exact revision on arXiv

Source-linked AI summary

MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

Zijuan Zhao, Zheren Fu, Hou Xia, Licheng Zhang, Yi Liu, Zhendong Mao

arXiv:2608.18096v1cs.CLcs.LG

TL;DR

Evaluating macro-societal values in multimodal content is urgent. MAVEN introduces a hierarchical framework, benchmark, metric, and compact evaluator optimization strategy. Its 2B evaluator matches its 8B counterpart and approaches frontier closed-source VLMs.

  • Problem

    Evaluating macro-societal values in multimodal content has become an urgent challenge.

  • Method

    MAVEN combines hierarchical value scoring, a human-verified multimodal benchmark with VSMS, SA-MDPO distillation, and multi-role consensus.

  • Results

    On MacroValue-Bench, the compact 2B evaluator matches its 8B counterpart and approaches frontier closed-source baselines across primary dimensions and secondary indicators.

  • Takeaways & Limitations

    The results support using compact evaluators for scalable macro-societal value evaluation.

  • Takeaways & Limitations

    MAVEN and MacroValue-Bench encode specific normative positions and inherit regional skew from predominantly Chinese-culture sources and East Asian human verifiers.

Abstract

from arXiv · show

Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge. Existing frameworks are largely confined to safety-oriented taxonomies, text-only psychometric probes, or single-label classification. Therefore, we propose MAVEN, a hierarchical framework for macro-societal value evaluation of multimodal content, grounded in international human-rights instruments and cultural value theory. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi-level quantitative scoring. Building on MAVEN, we construct a human-verified multimodal benchmark and a soft-match metric to evaluate VLMs' assessments across value dimensions. For evaluator optimization, we propose a span-adaptive variant of multi-level preference optimization for evaluator distillation, together with a training-free multi-role consensus strategy at inference time. We evaluate existing open- and closed-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro-societal value judgments. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed-source VLMs, offering a practical path toward scalable macro-societal value evaluation. Our SA-MDPO implementation and MacroValue-Bench are available at https://github.com/zzzzzzzzjj/MAVEN.

1 Introduction

MAVEN addresses the underexplored problem of quantitatively evaluating macro-societal value alignment in multimodal content by providing a hierarchical framework and benchmark. It also distills frontier VLM judgment into a compact evaluator using SA-MDPO and MRC, which approaches frontier closed-source performance.

  • Prior frameworks inadequately support VLM-based macro-societal evaluation because they focus on microethical safety, decouple evaluation from real-world multimodal content, or use single-label and open-ended outputs.The resulting gap is multimodal, quantitative assessment for macro-societal value governance.
  • MAVEN is a hierarchical framework for quantitatively scoring macro-societal values in multimodal content, grounded in international human-rights instruments and cultural value theory.It organizes values into 6 primary dimensions and 72 secondary indicators.
  • MacroValue-Bench is a 1,157-item multimodal benchmark for real-world evaluation paired with the VSMS metric.The benchmark is constructed on top of MAVEN’s value framework.
  • SA-MDPO distills value-judgment capability from frontier teacher VLMs into a compact 2B student, while MRC provides training-free multi-role consensus at inference.These methods target scalable evaluation because frontier VLMs are capable but prohibitively large.
  • On MacroValue-Bench, the compact 2B evaluator matches its 8B same-family counterpart and approaches frontier closed-source baselines across primary dimensions and secondary indicators.Ablations show measurable gains from span-adaptive Multi-level DPO and MRC on their corresponding evaluation axes.

2 Related Work

Prior work grounds value analysis in cross-cultural theories and international human-rights instruments, but existing evaluation largely focuses on text-only value expression or safety-oriented multimodal alignment. These approaches do not provide a unified quantitative framework for assessing macro-societal values embedded in multimodal content.

  • Theoretical foundations: Cross-cultural value research models values as interdependent motivational types, while Moral Foundations Theory proposes universal moral concerns and the World Values Survey documents cross-cultural value variation.Schwartz’s theory identifies ten motivationally distinct value types validated across cultures.
  • Theoretical foundations: International human-rights instruments operationalize macro-societal values through principles of peace, cooperation, and universal civil, political, economic, and cultural entitlements.The relevant foundations include the UN Charter, UDHR, ICCPR, and ICESCR.
  • Text-only value evaluation: Text-only benchmarks such as ValueBench, CLAVE, and ValueCompass evaluate value orientations expressed in text rather than values embedded in multimodal content.Their methods include rephrased psychometric questionnaires, concept extraction with classification, and contextual alignment under specified scenarios.
  • Multimodal value evaluation and alignment: Multimodal alignment methods including SPA-VL, Safe RLHF-V, and Ch3Ef primarily target harmlessness or helpful-honest-harmless behavior rather than macro-societal values.HumaniBench covers seven human-centered AI principles but uses heterogeneous task-specific metrics, while VIVA frames value assessment as multiple-choice action selection.

3 MAVEN Framework

MAVEN defines a two-layer macro-societal value framework with six primary dimensions and collaboratively curated secondary indicators, paired with structured scoring and MacroValue-Bench. Its benchmark uses model-then-human verification, while Value-Aware Soft Match accounts for semantic coupling among indicators.

  • Theoretical foundation: MAVEN organizes macro-societal values into six primary dimensions and collaboratively constructed secondary indicators spanning macro- to micro-level contexts.Experts filter redundant or off-topic proposals and iteratively re-elicit candidates until consensus; Peace includes World, Social, Cyber, and Inner Peace.
  • Theoretical foundation: The two-layer structure supports low-latency screening through primary dimensions and fine-grained diagnosis through secondary indicators.
  • Quantitative scoring schema: Each primary dimension uses a 5-point score {−2, −1, 0, +1, +2}, while its 12 secondary indicators use a 3-point scale {−1, 0, +1}.+1 or −1 at the primary level requires at least one secondary indicator with a non-zero matching sign, preserving consistency between layers.
  • MacroValue-Bench: MacroValue-Bench contains 1,157 image–text items from four sources, covering value-laden, ambiguous, and neutral-control content.Three frontier VLMs annotate each item three times; per-dimension and per-indicator majority voting produces labels that human checkers verify against the original content.
  • Value-aware soft matching: Indicator embeddings show semantic coupling, with Intergenerational Equity and Intergenerational Justice reaching cosine similarity as high as 0.86; VSMS therefore thresholds matches at τ = 0.65.The threshold suppresses weakly related pairs while retaining genuine cross-dimension neighbors, addressing limitations of hard-match metrics.

4 Compact Evaluator Optimization

MAVEN’s compact evaluator combines span-adaptive multi-level preference distillation during training with Multi-Role Consensus during inference. The approach adapts MDPO through rank-gap-aware regularization and semantically grounded preference data, then aggregates 14 stakeholder perspectives without additional training.

  • Overall framework: MAVEN combines training-time distillation with inference-time aggregation to obtain a compact yet capable evaluator.Frontier value-judgment capability is distilled into a compact student via Span-Adaptive Multi-level DPO, followed by Multi-Role Consensus at inference.
  • Span-Adaptive Multi-level DPO: The MDPO adaptation uses a rank-gap-aware β schedule and semantically grounded preference-data construction.These two extensions replace fixed regularization behavior while adapting preference optimization to value-evaluator distillation.
  • Span-Adaptive Multi-level DPO: Preference tuples are created by progressively degrading single seed samples while holding non-zero indicator counts constant within ±1.The procedure uses K = 4 and isolates supervision to indicator-level correctness rather than annotation sparsity.
  • Multi-Role Consensus: Multi-Role Consensus is a training-free inference strategy that uses 14 stakeholder personas to approximate human consensus.The roles include a Value Alignment Analyst, Educator, Diplomat, and Labor Rights Advocate.

5 Experiments

Experiments on the 1,157-item human-verified MacroValue-Bench show that value-evaluation performance varies across models and dimensions, while SA-MDPO and MRC substantially improve compact evaluators. The 2B evaluator reaches frontier-level performance, with gains supported by rank-gap-aware optimization and multi-role consensus.

  • Evaluation Settings: Evaluation uses QWK, Accuracy, and macro-F1 for primary dimensions, plus VSMS, Recall, and Precision for secondary indicators.QWK is the primary metric because it captures ordinal agreement and penalizes large mis-rankings more than adjacent ones.
  • Main Results: Doubao-Seed-1.6-V leads commercial models on every metric, while GPT-4o reaches 0.56 QWK despite its general capability.Within Qwen3-VL, QWK rises monotonically with model scale; the result suggests macro-societal value evaluation does not emerge uniformly from general-purpose capability or scale.
  • Main Results: Democracy has the highest Accuracy but lowest QWK, while Doubao-Seed-1.6-V is the only model exceeding 0.5 QWK on every dimension.Closed-source models achieve their highest QWK on Peace and Development, highlighting uneven per-dimension difficulty.
  • Alignment and Distillation: SA-MDPO raises the 2B baseline from 0.39 to 0.60 QWK, 0.50 to 0.71 Accuracy, and 0.80 to 0.95 VSMS.The resulting student surpasses Qwen3-VL-8B on QWK and Accuracy at 4× fewer parameters; MRC further raises QWK to 0.624.
  • Ablation Studies: SA-MDPO consistently outperforms the baseline, with balanced data producing the best 2B configuration and validating its rank-gap-aware β schedule.DPO underperforms the baseline, while MDPO improves overall but drops under balanced data.
  • Ablation Studies: Multi-Role Consensus consistently improves QWK across Qwen3-VL-Instruct scales, with the largest gain on the smallest model.The ablation compares 14-persona consensus against 14 independent runs of the original Value Alignment Analyst persona.

6 Conclusion

MAVEN combines a hierarchical evaluation framework, benchmark, evaluator-distillation method, and inference strategy for multimodal macro-societal values. Its compact 2B evaluator approaches frontier closed-source VLMs and advances deployable content-level value governance.

  • 6 Conclusion: MAVEN integrates the MacroValue-Bench, SA-MDPO distillation method, and MRC inference strategy within a hierarchical framework for multimodal macro-societal value evaluation.Together, these components form the paper’s proposed evaluation system.
  • 6 Conclusion: Distilled with SA-MDPO and inferred under MRC, the compact 2B evaluator approaches frontier closed-source VLMs.The result highlights a compact evaluator for scalable macro-societal value assessment.
  • 6 Conclusion: The work contributes to content-level value governance and moves macro-societal value assessment toward practical deployment in AI systems.This positions the framework as a step toward making value assessment a deployable capability.

Limitations

MAVEN’s framework and benchmark have limited cultural coverage, while its Multi-Role Consensus strategy imposes substantial inference-time overhead. The framework’s 72 indicators encode potentially contested normative positions, and the benchmark is regionally skewed toward Chinese-culture sources.

  • Cultural coverage of the framework and data: MAVEN’s 72 secondary indicators inevitably encode specific normative positions, some of which remain politically contested across societies.The six primary dimensions are grounded in international human-rights instruments and Schwartz’s cross-cultural value theory.
  • Cultural coverage of the framework and data: MacroValue-Bench inherits a regional skew because most items are drawn from Chinese-culture sources.The passage specifically identifies Weibo and Global Tim… among the sources contributing to this skew.
  • Inference-time cost of Multi-Role Consensus: 14 distinct stakeholder personas make Multi-Role Consensus roughly 14× more computationally expensive than a single-persona pass.MRC aggregates per-indicator judgments from the same VLM; this improves ordinal agreement but may be prohibitive for high-throughput moderation pipelines.

Ethics Statement

MAVEN treats macro-societal value evaluation as inherently normative and identifies three ethical concerns: privacy and safety, inherited evaluator biases, and potential misuse for censorship. The framework is intended for diagnostic evaluation, with consequential automated decisions requiring human review.

  • Data privacy and content safety: MAVEN’s data sources are publicly accessible, and the dataset retains only post text and task-relevant images without user-level metadata.Sources include Weibo, Global Times Online, People’s Daily Online, and the publicly released Ch3Ef-harmless subset.
  • Teacher-side priors: The model-then-human pipeline uses three frontier VLMs whose pretraining and alignment may introduce residual value priors into labels and evaluations.The initial labeling models are GLM-4.5V, Doubao-Seed-1.6-V, and Gemini-2.5-Pro; human verifiers reviewed aggregated labels.
  • Potential misuse: MAVEN could be repurposed to censor content or suppress dissent, contrary to its intended diagnostic purpose.The framework is designed for AI-generated or AI-curated content, not ranking, filtering, or sanctioning human expression.
  • Safeguards: Automated decisions with material consequences should always involve human review, while MAVEN’s releases aim to enable scrutiny and counter-argumentation.The authors explicitly reject providing a turnkey mechanism for consequential enforcement.

A Full MAVEN Indicators · B Annotation Prompt and Scoring Schema

MAVEN enumerates 72 secondary indicators across six primary dimensions, with concise definitions supporting annotation. Its shared prompt scores multimodal inputs at both secondary and primary levels using signed ordinal scales and structured rules.

  • A Full MAVEN Indicators: MAVEN lists 72 secondary indicators grouped by primary dimension, with short English definitions for annotation and full bilingual definitions released with the dataset.The indicators cover Peace, Development, Equity, Justice, Democracy, and Freedom.
  • A Full MAVEN Indicators: The appendix tabulates secondary indicators for Peace, Development, Equity, and Justice.These dimensions are presented in Tables 7–10.
  • A Full MAVEN Indicators: The full indicator system also includes secondary indicators for Democracy and Freedom.These dimensions are presented in Tables 11–12.
  • B Annotation Prompt and Scoring Schema: The same system prompt is used in both the model-then-human annotation pipeline and inference-time evaluation, replacing {rule} with all 72 indicator definitions.This connects annotation and evaluation through a shared value-definition specification.
  • B Annotation Prompt and Scoring Schema: Primary dimensions receive integer representation scores from −2 to 2, while 72 secondary indicators receive integer scores from −1 to 1.For primary dimensions, −2 means strongly violates, 0 means unrelated, and +2 means strongly supports or embodies; secondary scores denote violation, absence, or positive reflection.
  • B Annotation Prompt and Scoring Schema: Scoring assigns negative values to opposition or undermining, positive values to active support or opposition to violating content, and 0 to unrelated input.The rules distinguish the content being represented from the input’s stance toward that content.
  • B Annotation Prompt and Scoring Schema: A non-zero primary score requires at least one matching-sign non-zero secondary indicator, whereas zero primary scores impose no secondary constraint.The prescribed output is a valid JSON object containing indicator scores for all six dimensions.

C Annotation Quality Analysis

MAVEN’s annotation pipeline is internally stable but shows only moderate agreement across VLMs at the primary-dimension level, making majority voting and human verification necessary. Human reviewers modified 15.6% of aggregated labels, confirming that verification materially improves annotation quality.

  • Intra-VLM consistency: Average pairwise QWK across three independent runs was 0.71, ranging from 0.64 for GLM-4.5V to 0.78 for Gemini-2.5-Pro.This supports using VLM annotations as a stable foundation for the pipeline.
  • Inter-VLM agreement: At the primary-dimension level, the three VLMs achieved a moderate average pairwise QWK of 0.52.The result indicates that independent VLM judgments do not reliably converge at the dimension level.
  • Inter-VLM agreement: The three VLMs agreed exactly on 84.3% of indicator assignments, despite diverging when aggregating dimension-level magnitudes.This contrast motivates majority voting and subsequent human verification.
  • Human modification rate: Human verifiers modified 15.6% of aggregated dimension-level labels, with rates from 3.9% on Democracy to 25.1% on Peace.Eight annotators independently reviewed all 1,157 samples over five days, making verification a substantive pipeline step.

D Multi-Level Preference Data Construction · E Multi-Role Consensus

MAVEN constructs four quality levels by degrading indicator configurations while preserving non-zero counts, then uses 14 complementary personas to provide multi-role value judgments. The construction combines relocation, suppression, and adversarial corruption with score rederivation and stakeholder-specific evaluation lenses.

  • D Multi-Level Preference Data Construction: D Multi-Level Preference Data Construction: Each seed yields a K = 4 tuple ⟨r0, r1, r2, r3⟩, with r0 highest quality and r1–r3 progressively degraded.Across levels, the total number of non-zero indicators remains within ±1 of the seed count, grounding quality differences in activated indicators rather than annotation sparsity.
  • D Multi-Level Preference Data Construction: D Multi-Level Preference Data Construction: Level 1 relocates values from non-zero indicators to related zero-valued indicators while keeping scores identical and preserving the non-zero count exactly.Each non-zero indicator participates in at most one similarity-thresholded swap.
  • D Multi-Level Preference Data Construction: D Multi-Level Preference Data Construction: Level 2 randomly suppresses half the non-zero indicators, activates related zeros, rederives modified scores, and balances the final non-zero count back to N.The procedure starts from Level 0 and uses similarity to surviving non-zero indicators for activation.
  • D Multi-Level Preference Data Construction: D Multi-Level Preference Data Construction: Level 3 negates non-zero scores, flips or zeros indicator halves, randomly activates replacements with uniformly chosen ±1 values, and reassigns originally zero dimension scores.New scores for affected zero-score dimensions are derived from the corrupted indicators using Eq. (8).
  • E Multi-Role Consensus: E Multi-Role Consensus: The representative Judge persona evaluates content through jurisprudence, procedural fairness, constitutional rights, due process, equality before the law, and fundamental-rights protection.It assigns severe negative scores in Justice and Democracy to content promoting arbitrary power, vigilantism, or systemic bias.
  • E Multi-Role Consensus: E Multi-Role Consensus: MRC uses 14 personas, with Persona 1 as an impartial baseline and Personas 2–14 contributing complementary stakeholder-specific perspectives over MAVEN secondary indicators.The full system prompts are released with the dataset and code.
Loading 2608.18096v1…