PaperlayerView exact revision on arXiv

Source-linked AI summary

Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

Tong Zhu, Zhe Xu, Zefeng He, Yunkai Chai, Xuyang Hu, Dehong Zheng, Yun Luo, Yafu Li, Qi Zhang, Yu Cheng

arXiv:2610.22131v1cs.CL

TL;DR

Interactive agents often need finite choices, but free-form responses add decoding overhead and complicate model-to-controller interfaces. Intern-Decision uses masked decision fields and a causal multimodal language model to return structured choices and candidate probabilities. Its 4B model achieves 90.02% unweighted mean accuracy across seven suites, compared with 88.74% for Jev.

  • Problem

    Many agent operations require choices from predefined alternatives, while free-form generation adds decoding overhead and complicates interfaces with downstream controllers.

    Source: §1 · pp. 1–2

  • Method

    Intern-Decision uses autoregressive masked prediction over decision fields to produce structured outcomes and candidate probabilities in one causal forward pass.

    Source: §1 · p. 2

  • Results

    90.02% unweighted mean accuracy across seven suites for Intern-Decision-4B, compared with 88.74% for Jev.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Intern-Decision can serve as a practical component of larger agent systems, with separate calibration and handoff studies illustrating probabilistic outputs for uncertainty analysis and selective use of a larger model.

    Source: §7 · p. 13

    Longer states, more image tokens, and larger schemas still increase computation and memory; generalization to arbitrarily large schemas and complex environments remains open.

    Source: §3.5 · p. 5, §7 · p. 13

Abstract

from arXiv · show

Interactive agents repeatedly choose among a finite set of actions, labels, or ordinal outcomes. Generating a free-form response for each choice adds latency and complicates the interface between a model and its environment. We present Intern-Decision, a family of 0.8B, 2B, and 4B multimodal decision models that return structured choices and candidate probabilities from text and optional images. Our autoregressive masked language modeling objective places one decision placeholder per field in a complete assistant JSON skeleton. Ground-truth symbols appear only in supervision, while all fields are evaluated in a single causal forward pass using the backbone's native language-model head. The 4B model achieves a 90.02% unweighted average accuracy across seven evaluation suites, compared with 88.74% for Jev v1.13.0. For a fixed 289-token request with three fields on one RTX 4090, mean latency is 44.16 ms for the 4B model and approximately 33--34 ms for the smaller models. On a separate 96-case known-distribution pilot, temperature scaling reduces expected multiclass Brier from 0.628 to 0.550 and expected calibration error from 0.213 to 0.089. A confidence-based handoff study on 111 public Jevbench-Hard items improves accuracy from 82/111 (73.87%) to 97/111 (87.39%) by routing 42 items to Atria-Dawn-Preview. We also describe qualitative game, visual interface, and browser demonstrations. These results support compact decision models as a practical component of larger agent systems.

1. Introduction

Many agent operations are closed-set decisions, yet agents often use free-form generation that adds decoding overhead and complicates controller interfaces. Intern-Decision scores multimodal decision fields in one causal pass, and its 4B model leads the reported seven-suite comparison.

1. Introduction

  • Closed-set agent operations include tool invocation, model routing, and actions, but free-form generation adds decoding overhead and complicates downstream controller interfaces [19] [25] [30].Practical multimodal agents also need multiple decisions over shared states, combining visual observations, task instructions, and changing interface states [28] [29].Source: §1 · pp. 1–2
  • Intern-Decision recasts structured decisions as next-token prediction over masked fields, returning predictions and candidate probabilities in one causal forward pass.It retains the pretrained causal backbone and native language-model head, avoiding iterative text generation.Source: §1 · p. 2
  • The decision corpus combines teacher-constructed challenging decisions with public task distributions across text, visual judgments, interface actions, and interactive control.The 0.8B, 2B, and 4B Qwen3.5 backbones are fine-tuned with the vision tower and projector frozen; temperature scaling uses a separate calibration split.Source: §1 · p. 2
  • 90.02% unweighted mean accuracy across seven suites for Intern-Decision-4B compares with 88.74% for Jev.For a fixed 289-token request with three fields, the 4B model averages 44.16 ms on one RTX 4090; the paper also reports separate calibration and handoff studies.Source: §1 · p. 2
  • The paper presents confidence-based handoff to a larger model and qualitative demonstrations of visual interaction as ways compact decision models can complement larger agent systems.Its three evaluation perspectives are decision accuracy, inference efficiency, and probability quality.Source: §1 · p. 2

2. Related Work

Prior structured decision systems provide typed choices, scores, and binary judgments, while differing in architecture and inference strategy. Intern-Decision places multimodal decisions within this paradigm and uses autoregressive masked prediction with candidate probabilities.

Decision Models

  • Jev, JevK5, Kev, SemIf-OpenJev, and Laya use distinct interfaces or architectures for structured decisions; Intern-Decision focuses on multimodal inputs.Examples include Jev-style next-token logits, a learned pointer head, direct candidate scoring, and a non-autoregressive encoder.Source: §2 · p. 2

Autoregressive and Masked Prediction

  • Intern-Decision combines answer placeholders with next-token-aligned prediction in a causal multimodal model, scoring all requested fields in one pass without iterative refinement or answer feedback.This differs from masked-token methods that iteratively refine predictions and prompt-based classification approaches using label words [12] [11].Source: §2 · p. 3

Calibration and selective computation

  • The evaluation separates hard-label calibration from an exact-distribution pilot and reports one escalation operating point rather than an optimized routing policy.Temperature scaling adjusts confidence while preserving the most likely label [13]; Brier scoring evaluates the full predictive distribution.Source: §2 · p. 3

3. Autoregressive Masked Decision Modeling

Intern-Decision represents requests as ordered multimodal states and decision fields, then predicts each field’s answer symbol from a masked assistant schema. Its objective and inference path score structured choices in a shared causal pass, with computational scope still dependent on input and schema size.

3.1. Problem formulation

  • A request comprises a state, optional ordered images, and ordered questions; each field returns a categorical distribution and selected label over its candidate set.The interface supports choice, Boolean, and ordinal score types; the release supports 62 candidate symbols and the HTTP service permits up to 16 fields per request.Source: §3.1 · p. 3

3.2. Input construction and causal readout

  • The compiler appends an assistant JSON skeleton with one decision placeholder per field, and training supervises mapped answer symbols without inserting them into the input.The causal backbone reads logits from the position immediately before each placeholder; all requested logits are collected in the same forward pass, with no answer feedback. Figure 1 depicts visible evidence and masked context separately from target symbols.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.3. Training objective

  • Training uses full-vocabulary cross-entropy at shifted decision positions; JSON punctuation, state tokens, and schema descriptions are context rather than supervised targets.The training path uses hard labels, not full-JSON generation loss or full-token continued pretraining.Source: §3.3 · p. 4

3.4. Candidate probabilities and structured outputs

  • At inference, legal candidate logits are normalized per question, then symbols are mapped back to original labels to create schema-conforming responses.Temperature changes probability spread without changing the argmax; candidate probabilities remain available to downstream controllers.Source: §3.4 · p. 5

3.5. Computational scope

  • The shared pass avoids iterative answer decoding and separate field-by-field passes, but longer states, more image tokens, and larger schemas still increase computation and memory.Causal ordering also means fields can have different contextual prefixes; the method does not enforce permutation invariance or a jointly constrained output distribution.Source: §3.5 · p. 5

4. Data Construction

The paper builds a shared structured-decision corpus and an evaluation-only test set for checking probability estimates against known distributions.

4.1. Decision-oriented data construction

  • The corpus combines synthetic textual decisions, public task distributions, and synthetic visual and interactive tasks under a shared state, optional-image, typed-question, and target-label interface.Sources include news classification, tool routing, typed decisions, safety judgments, rendered games, VizDoom, and shapes-based pointer actions.Source: §4.1 · pp. 5–6
  • 17,880 samples, 23,500 decisions, and 5,500 image-bearing samples make up the identical training input used for all three released checkpoints.Multi-question records can contribute several decisions while counting once as a training sample.Source: §4.1 · p. 5
  • Visual sources cover recognition, comparison, localization, and discrete action selection, mapping shapes attributes and spatial relations to pointer moves and game frames to control decisions.These capabilities overlap within the recorded sources.Source: §4.1 · p. 6

4.2. Calibration test set construction

  • The evaluation-only set contains 96 cases across 24 mechanism families, six categories, and paired option orders, yielding 48 paired settings rather than 96 independent trials.Each family has two settings; canonical and reversed option orders share the same state, question, and reference distribution.Source: §4.2 · p. 6
  • A deterministic generator computes exact reference distributions from specified sample spaces and assumptions, with checks for valid probabilities, matching options, and equivalent reversed-order pairs.The references are not Monte Carlo estimates or model-generated answers; inputs omit the reference probabilities, and the set is not used for training, temperature fitting, or checkpoint selection.Source: §4.2 · p. 6
  • The exact-content check found no overlap with temperature-fitting records, but does not establish absence of semantic overlap with every training source.Source: §4.2 · p. 7
  • The Monty Hall cases ask for the prize location among the doors, not whether switching is the best action.The informed-host example assigns reference probabilities of 1/3 to the initial door and 2/3 to the other unopened door.Source: §4.2 · p. 7

5. Evaluation Results

Across seven evaluation suites, Intern-Decision-4B leads or ties on five, while the evaluation also examines latency, probability calibration, and confidence-based handoff. The results are scoped to the stated benchmark and measurement protocols.

5.1. Benchmark scope and scoring

  • The seven-suite average weights suites equally, while accuracy is counted per decision; public Jevbench results are diagnostic, not independent held-out generalization evidence.The public subset excludes private items and the separate judge tier, and its accuracy averages do not reproduce the official multi-axis leaderboard composite.Source: §5.1 · p. 7

5.2. Decision accuracy

  • 90.02% is Intern-Decision-4B’s unweighted average across seven suites, exceeding evaluated baselines and leading or tying on five suites.The comparison is restricted to the evaluated systems and protocols.Source: §5.1 · p. 7, §5.2 · p. 8, Table 3 · p. 8
  • The suite average rises from 79.38% at 0.8B to 84.68% at 2B and 90.02% at 4B, but the trend cannot be attributed solely to parameter count.The evaluation lacks matched-data and training-compute disclosures or ablations to isolate the source of the pattern.Source: §5.2 · p. 8
  • On Jevbench-Hard, temperature scaling leaves the 4B model’s 82/111 correct decisions unchanged while reducing ECE from 0.132613 to 0.065267.Jev answers 80/111 correctly and has ECE 0.094685; the temperature was fitted on separate calibration data.Source: §5.2 · p. 8, Table 4 · p. 8

5.3. Local inference latency

  • For a fixed 289-token, three-field request on one RTX 4090, reciprocal mean latencies imply 29.43, 30.05, and 22.64 serial requests per second for 0.8B, 2B, and 4B, respectively.These are derived rates assuming negligible loop overhead, not measured saturated throughput or interactive-agent frame rates; long contexts and screenshots were not tested.Source: §5.3 · p. 9, Table 5 · p. 9

5.4. Known-distribution calibration pilot

  • On the 96-case known-distribution pilot, temperature scaling reduces overall expected Brier by 0.078 and expected ECE by 0.124 while preserving every argmax.The calibrated model improves on its uncalibrated result in both metrics across all six categories.Source: §5.4 · pp. 9–10, Table 6 · p. 10

5.5. Confidence-based model handoff

  • Routing 42 of 111 Hard items yields a hybrid that answers 15 more correctly than the small model and has an estimated time 53.29% below Atria alone.This is one potentially useful operating point, not an established optimal policy.Source: §5.5 · p. 10, Table 7 · p. 10

6. Multimodal Interaction and Model Collaboration

Intern-Decision supplies local choices within caller-managed closed loops, illustrated through game, visual-selection, and browser interactions. The demonstrations are selected qualitative trajectories, not controlled benchmark measurements.

6.1. A closed-loop decision interface

  • The caller defines the action space and when to request another decision, while Intern-Decision selects locally within an observe–evaluate–execute loop.The demonstrations are selected qualitative trajectories rather than controlled benchmark measurements.Source: §6.1 · pp. 10–11

6.2. Visual game control

  • Five game demonstrations span navigation, hazard avoidance, platform movement, health search, and enemy targeting from visual observations.The Doom health-gathering trajectory shows the agent changing its view after contacting a wall, but does not establish a learned exploration objective or general exploration algorithm.Source: §6.2 · p. 11, Figure 5 · p. 11

6.3. Visual selection and mouse actions

  • Four visual-selection demonstrations show screenshot-guided pointer movement and clicking to complete tasks without human takeover or correction.The tasks include selecting objects by size or color, identifying an odd shape, and choosing an object by spatial order.Source: §6.3 · pp. 11–12, Figure 6 · p. 12

6.4. Hierarchical browser assistance

  • Browser assistance divides work between a planner that maintains cross-page context and Intern-Decision, which selects actions from the live interface.Three selected traces reach requested evidence in 6, 6, and 4 local decisions; these are qualitative demonstrations, and inference latencies exclude planning, network, and browser execution.Source: §6.4 · pp. 12–13, Figure 7 · p. 13

7. Conclusion

Intern-Decision makes finite multimodal choices in a single causal forward pass and provides candidate probabilities through a typed interface. The 4B model has the highest seven-suite average in the presented comparison, while generalization to arbitrarily large schemas and complex environments remains open.

7. Conclusion

  • The 4B model achieves the highest seven-suite average in the presented comparison, with a 44.16 ms mean for a fixed short request on an RTX 4090.Separate calibration and handoff studies show how probabilistic outputs can support uncertainty analysis and selective use of a larger model.Source: §7 · p. 13
  • Generalization to arbitrarily large schemas and complex environments remains an open problem.The conclusion presents Intern-Decision as a practical component of larger agent systems, not as a solution to that open problem.Source: §7 · p. 13

A. Release and Reproduction Details

  • The release procedure verifies test hashes and checkpoint-specific temperature presets, replays calibration on saved probabilities, and scores the pilot separately.Reproducing Intern-Decision rows does not automatically reproduce every baseline row because external-baseline adapters and historical responses are not included.Source: A. Release and Reproduction Details · p. 16, Table 8 · p. 16

Recovered Training Settings

The released 0.8B, 2B, and 4B checkpoints share recovered training settings and input files, with model-size-specific Qwen3.5 initialization.

Recovered Training Settings

  • The three runs share audited input files, and the audit records 17,880 rows, 23,500 fields, and 5,500 image rows with no truncation.Published weight-shard hashes bind each model to its corresponding completed run.Source: Recovered Training Settings · p. 16
  • The released 0.8B, 2B, and 4B checkpoints use matching Qwen3.5-0.8B, Qwen3.5-2B, and Qwen3.5-4B backbones, with 2 epochs / 156 optimizer updates.The peak and minimum learning rates are 2 × 10−6 and 2 × 10−7, respectively.Source: Recovered Training Settings · p. 16
  • Training uses BF16 on four GPUs with seed 42, global batch size 16 packed sequences, and sequence length 8,192 tokens.The maximum gradient norm is 1.0, and the vision tower and projector are frozen.Source: Recovered Training Settings · p. 16, Table 9 · p. 16

Composition Overview

The frozen training mixture combines synthetic text decisions, public task distributions, and synthetic visual tasks, with separate counts and provenance notes for its components.

Composition Overview

  • The frozen mixture contains 17,880 samples, 23,500 decisions, and 5,500 image-bearing samples.Multi-question records can contribute more than one decision.Source: Composition Overview · p. 16, Table 10 · p. 17
  • Synthetic text includes 4,066 general-category records and 2,645 difficult-category records generated by deepseek-v4.1-flash.The difficult-category total combines 1,762 hard-batch and 883 patch-batch records; the reporting categories reflect synthesis purpose rather than each record’s difficulty annotation.Source: Composition Overview · p. 17
  • Public sources cover news, tool routing, typed decisions, and safety, while synthetic visual sources include games, VizDoom, and shapes or CAPTCHA-style tasks.The nine named games and VizDoom supply synthetic training data, not evaluation suites in the quantitative comparison.Source: Composition Overview · p. 17

Text Decision: Probability Judgment

The examples span text and visual decisions, from exact probability judgments and tool routing to game moves, interface actions, and structured random-experiment predictions. They specify task rules, candidate options, and, in several cases, the mechanisms and references used to assess predictions.

Text Decision: Probability Judgment

  • The probability-judgment example asks whether drawing at least one gold token from five draws exceeds 0.45, with equality counting as no; it requires exact computation rather than keyword matching.The bin contains 40 tokens, exactly three gold, and draws are without replacement.Source: Text Decision: Probability Judgment · p. 17

Text Decision: Tool Routing

  • The tool-routing example asks the model to choose between getFarm and addCropToFarm for a request to retrieve farm information by ID.It tests selection from a small candidate catalog.Source: Text Decision: Tool Routing · p. 17

Visual Decision: Snake

  • The Snake example maps a game frame to a movement choice, targeting the apple while avoiding walls and the snake’s body; the training-frame figure illustrates the visual input.The target move is left, and the snake cannot reverse direction.Source: Visual Decision: Snake · p. 18, Figure 8 · p. 18

Visual Decision: VizDoom (Two Questions per Frame)

  • VizDoom assigns turn and attack decisions to the same frame, illustrating multi-field supervision from one visual state.The example targets the largest visible enemy and fires only when the screen center lies within its silhouette span.Source: Visual Decision: VizDoom (Two… · p. 18, Figure 9 · p. 18

Visual Decision: Captcha-Style Shapes

Mechanism coverage and reference construction

  • The 96-record pilot covers six categories of synthetic random mechanisms, with exact-rational oracles and validation for normalization, option support, and order invariance.The categories include randomness and support, composed events, hidden state, observation bias, selective disclosure, and sequential processes.Source: Mechanism coverage and reference… · p. 19

Support and conditioning: two further actual cases

  • The sock example requires updating the remaining inventory after three red socks are removed, yielding blue 4/7 and red 3/7; the die example tests equal mass on possible faces and zero on impossible ones.For the die, even a perfect prediction has expected Brier 5/6 but zero excess expected Brier.Source: Support and conditioning: two further… · p. 19

Monty Hall: prize location as the outcome

  • The Monty Hall example specifies a knowledgeable host’s reveal and switch protocol, then frames the prediction as prize location among the final two unopened doors.The accompanying text labels a TVD comparison between the initial door and the other unopened door, without supplying values here.Source: Monty Hall: prize location as the… · p. 20

Related papers

Loading 2610.22131v1…