PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao

arXiv:2610.11316v1cs.CLcs.AI

TL;DR

Structured request schemas are precise but rigid, motivating a natural-language interface for constrained decision-making. MetaEncoder turns a multimodal decoder into a contrastively trained bi-encoder, outperforming multimodal embedding models on decision tasks and reaching near-SOTA retrieval, while complex reasoning remains a limit.

  • Problem

    Task-specific structured schemas are precise but rigid, motivating a general-purpose decision interface using natural-language requests and candidates.

    Source: §1 · p. 1

  • Method

    MetaEncoder fine-tunes a pre-trained 30B multimodal decoder into a bi-encoder that aligns requests and candidates through unidirectional contrastive learning.

    Source: §1 · p. 2

  • Results

    MetaEncoder outperforms state-of-the-art multimodal embedding models on all decision-making tasks and achieves near-SOTA results on multimodal retrieval benchmarks, but bi-encoders remain limited on complex reasoning tasks.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    MetaEncoder performs strongly on reading, perception, classification, and single-step judgment, while complex comprehension and adaptation remain weaker areas.

    Source: §5 · p. 11

    Complex decision-making is constrained by the lack of explicit reasoning capabilities in the bi-encoder.

    Source: §5 · p. 11

Abstract

from arXiv · show

System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.

1 Introduction

The paper revisits natural-language decision interfaces and asks how far a scalable bi-encoder can go. MetaEncoder performs strongly on decision-making and multimodal retrieval, while reasoning-intensive tasks remain a limitation.

1 Introduction

  • A System One model returns a constrained choice and probability distribution over candidates rather than generating free-form text.Such scores are useful for decisions including routing, classification, ranking, and retrieval.Source: §1 · p. 1
  • The proposed interface replaces task-specific structured schemas with free-form requests and natural-language candidates, while retaining constrained candidate selection.Requests may include images or video, and the goal is instruction-following decision-making similar to prompting a decoder-only model.Source: §1 · pp. 1–2
  • Bi-encoders scale from fewer than 256 options to millions through independent candidate encoding, but request-candidate interaction passes through a single vector.For closed-set decisions, adding candidates to the request prompt supplies information equivalent to a constrained decoder.Source: §1 · p. 2
  • MetaEncoder fine-tunes a 30B multimodal decoder into a bi-encoder using unidirectional contrastive learning to align requests with candidates, handling closed sets of fewer than 256 options and open corpora with millions of candidates.Source: §1 · p. 2
  • MetaEncoder outperforms state-of-the-art multimodal embedding models on all decision-making tasks and achieves near-SOTA multimodal retrieval results.The paper also identifies weaker performance on complex decision-making tasks where explicit reasoning is lacking.Source: §1 · p. 2
  • The evaluation spans 11 benchmark suites and 190 tasks, testing where bi-encoders generalize as instruction-following encoders and where they do not.The evaluated areas include multimodal decision-making, understanding, and open-corpus retrieval.Source: §1 · p. 2

2 Related Work

Related work contrasts candidate-scoring architectures by how the prompt interacts with candidates, trading off accuracy, cost, scale, and reuse. MetaEncoder adopts a cacheable bi-encoder while exploring its decision-making limits.

Background

  • Universal embedders expanded from text retrieval and semantic similarity to instruction-following and multimodal inputs, with classification, VQA, and grounding cast as candidate ranking.The section frames bi-encoders, cross-encoders, and constrained decoding as differing in prompt-candidate interaction and its effects on cost, scale, and expressiveness.Source: §2 · p. 3

Bi-encoders

  • Bi-encoders encode prompts and candidates independently, enabling precomputed candidate embeddings and scoring over millions of candidates, but their single dot product limits expressiveness.This bottleneck is especially limiting for fine-grained or compositional distinctions and prompt-defined labels.Source: §2 · p. 3

Cross-encoders

  • Cross-encoders jointly process each prompt-candidate pair and are consistently more accurate than bi-encoders, but require K full forward passes and cannot cache representations.Scoring candidates in isolation also prevents direct use of option-set structure.Source: §2 · p. 3

One-step constrained decoding

  • One-step constrained decoding places all candidates in one prompt, enabling joint comparison in a single pass, but is sensitive to option order and bounded by the context window.Its output is a distribution over positions rather than reusable candidate representations.Source: §2 · pp. 3–4

Summary

  • The three designs trade off interaction: bi-encoders support indexing and scale, cross-encoders favor accuracy at O(K) cost, and constrained generation compares candidates jointly but lacks reusable embeddings.MetaEncoder adopts the cacheable bi-encoder design to explore its limits in general-purpose decision-making.Source: §2 · p. 4

3 MetaEncoder

MetaEncoder represents requests and candidates in a shared natural-language interface, optionally incorporating images or video. Its bi-encoder architecture and unidirectional contrastive training support both small decision sets and large retrieval pools.

3.1 Natural-language interface

  • Requests and candidates are natural-language descriptions, optionally paired with images or video, in both decision making and retrieval.A small candidate set can be included in the request; large open-set pools are indexed in advance because they cannot fit in the request context.Source: §3.1 · p. 4

3.2 Architecture

  • The same Muse-Glimmer-based network encodes requests and candidates, using the backbone’s native vision tower for images and video.For the request representation, the final-layer hidden state of the last token is normalized; under causal attention, that token attends to the full input.Source: §3.2 · p. 4
  • For closed sets, randomly dropping the option list during training discourages the model from relying only on option text in the prompt.Candidate embeddings are independent of the request and can be cached; cosine scores yield a probability distribution over candidates.Source: §3.2 · p. 4

3.3 Unidirectional Contrastive Training

  • Closed-set training contrasts each request against its own options, matching evaluation over the task’s offered candidates.Open-set training instead uses candidates from other requests in the global batch, plus hard negatives where available.Source: §3.3 · p. 5

4.1 Benchmarks

The evaluation spans 11 benchmark suites and 190 tasks across closed-set decision making and multimodal understanding, as well as open-set retrieval. The suites cover text, images, video, and visual documents.

Benchmark scope

  • The benchmarks span closed sets of fewer than 256 options and open sets scored against a shared, larger candidate pool.The evaluation groups tasks into decision making, multimodal understanding, and retrieval.Source: §4.1 · p. 5, Table 1 · p. 6

Decision making

  • JEVBench and ImaJEV-Bench test decisions with row-specific label criteria, including multimodal states where answers may require combining text rules with image values.JEVBench has 231 text-only rows with 2–6 options each; ImaJEV-Bench has 173 dev entries across visual, text, and joint subsets.Source: §4.1 · p. 5

Multimodal understanding

  • Understanding benchmarks cast multiple-choice answers as candidates, covering text knowledge, college-level image-text questions, lecture videos, adversarial natural-image questions, and fine-grained video timing.The supplied suites include MMLU, MMMU, Video-MMMU, NaturalBench, and TempCompass.Source: §4.1 · p. 5

Open-corpus retrieval

  • Retrieval evaluation includes 13 multilingual NanoBEIR text tasks and MMEB-V3 image, video, and visual-document partitions.MMEB-V3 covers multimodal classification, question answering, grounding, and retrieval, all framed as ranking candidates for a multimodal query.Source: §4.1 · pp. 5–6

4.2 Evaluation

MetaEncoder outperforms leading multimodal retrieval models on decision-making and multimodal multiple-choice understanding, while slightly lagging OpenJev-27B on text-only decisions. Evaluation encodes requests and candidates separately and ranks candidates by cosine similarity.

Evaluation protocol

  • Evaluation encodes each natural-language request once and each candidate independently, then ranks candidates by cosine similarity.Closed-set prompts include the options; open-set prompts contain only the state and task description, with candidates drawn from the full task pool.Source: §4.2 · p. 6

Baselines and metrics

  • The reported baselines include OpenJev-27B for decision making and WeMM-Embedding-9B and Qwen3-VL-Embedding-8B for multimodal retrieval.Closed-set performance uses accuracy (Hit@1); open-set reporting uses NanoBEIR nDCG@10, MMEB-V3 image and video Hit@1, and visual-document nDCG@5.Source: §4.2 · p. 6

Results

  • MetaEncoder outperforms current SOTA retrieval models on decision-making and multimodal multiple-choice understanding, but slightly lags OpenJev-27B on text-only decision-making.The authors attribute the gap to the tradeoff between bi-encoder caching efficiency and full-attention ranking accuracy; OpenJev-27B lacks retrieval and multimodal capabilities.Source: §4.2 · p. 6

4.3 Analysis

These analyses show that option-list and candidate formatting improve closed-set decisions, while option-order changes have little effect. Video gains depend on frame count and task type, and reasoning-intensive tasks expose a remaining limitation of single-step scoring.

4.3.1 Whether to Include Options in Prompt

  • Appending option lists improves performance across 56 of 57 MMLU subjects and 27 of 30 MMMU subjects, with larger gains on text-only benchmarks.Table 3 reports unweighted subject means for MMLU and MMMU and Hit@1 for MMMU-Pro.Source: §4.3.1 · p. 7, Table 3 · p. 7

4.3.2 Candidate Format in Target

  • C2, which includes both candidate labels and descriptions, works best; label-only C1 regresses across benchmarks and its MMMU deficit grows as the option set expands.C1 reaches 0.6512 on MMLU against a 0.2500 chance floor, but its MMMU deficit versus C0 grows from +0.0281 at three options to −0.1725 at ten.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.3.3 Sensitivity to candidate order

  • Across ten random option-order permutations, none of the three candidate variants has a standard deviation above 0.003, indicating little sensitivity to order.Figure 1 reports the permutation results for MMLU.Source: §4.3.3 · p. 8, Figure 1 · p. 8

4.3.4 Video understanding

  • More frames improve Video-MMMU mainly on perception and comprehension, while adaptation does not benefit; at 32 frames, request and candidate rendering still matter.Accuracy rises from 0.5322 at 8 frames to 0.5800 at 32 and 0.5900 at 64, while the 64-frame setting costs roughly eight times as much as 8 frames for a 0.058 gain. At 32 frames, the best candidate uses both its letter and text, and removing the option list causes a large regression.Source: §4.3.4 · p. 9, Figure 2 · p. 9, Table 5 · p. 9

4.3.5 Encoder versus Constrained Decoder

  • MetaEncoder and the constrained decoder perform comparably on MMLU, while MetaEncoder outperforms the decoder on MMMU, MMMU-Pro, and Video-MMMU.The comparison uses the pre-trained Muse-Glimmer backbone as a constrained decoder baseline.Source: §4.3.5 · p. 10, Table 6 · p. 10

4.3.6 Missing Reasoning Ability

  • On MMMU, disabling constrained decoding raises Muse-Glimmer decoder accuracy from 0.4962 to 0.7202; separately, prepending Muse-Glimmer-generated CoT raises MetaEncoder accuracy from 0.5730 to 0.7155.With final labels truncated from the CoT to eliminate direct label leakage, MetaEncoder reaches 0.6871 accuracy.Source: §4.3.6 · p. 10

5 Conclusion

MetaEncoder provides a multimodal bi-encoder interface for closed-set decision-making and understanding, and open-set retrieval. Its evaluation shows competitive performance, while complex comprehension and adaptation remain challenging.

5 Conclusion

  • Evaluation across 11 benchmarks shows competitive performance for MetaEncoder on multimodal decision-making, understanding, and retrieval.Source: §5 · p. 11
  • Appending candidate options to the prompt lets the bi-encoder handle decision-making effectively, narrowing its performance gap with cross-encoders and constrained decoders.Source: §5 · p. 11
  • MetaEncoder excels at reading, perception, classification, and single-step judgment, but is weaker on complex comprehension and adaptation.The paper identifies reasoning as a bottleneck and points to supplied decoder reasoning traces or direct encoder reasoning as possible routes to address it.Source: §5 · p. 11

A Prompts

The appendix documents prompts used to evaluate MetaEncoder’s natural-language interface. It describes a shared chat template and benchmark-specific ways to render task instructions, candidates, media, and retrieval inputs.

A Prompts

  • The appendix provides the exact prompts used for each evaluation suite, aligning them with MetaEncoder’s natural-language interface.Source: A Prompts · p. 16

A.1 Common template

  • A shared two-turn chat template encodes requests and candidates; request media is attached as input, and last-token pooling reads the sequence ending at the user turn.Source: §A.1 · p. 16

A.2 Decision making

  • Decision-making prompts render task definitions and label meanings as criteria, making rows answerable even when their label set is unseen.Boolean criteria use descriptive yes/no labels rather than opaque True/False values.Source: §A.2 · p. 16
  • S1MB prompts use each decision’s instructions and state, with natural-language candidate descriptions; one row is emitted per case-decision pair.Source criterion identifiers are used to resolve the correct candidate but are not shown to the model.Source: §A.2 · p. 16

A.3 Multimodal understanding

  • Video multiple-choice prompts include the full answer text as candidates, while TempCompass uses sentence-only candidates and eight frames sampled from 64 per clip.Video-MMMU Adaptation also supplies its still image separately from the video frames.Source: §A.3 · p. 17
  • NaturalBench pairs two images with two yes/no questions whose answers flip across images; MMLU and MMMU prompts can render options directly in the request.Source: §A.3 · p. 17

A.4 Retrieval

  • NanoBEIR prompts supply task-specific retrieval instructions, while MMEB-V3 builds requests from task instructions and query text and candidates from target text and images.Source: §A.4 · p. 18

Related papers

Loading 2610.11316v1…