PaperlayerView exact revision on arXiv

Source-linked AI summary

To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation

Haojie Hu, Senda Chen, Ying Shen, Lin Zhang

arXiv:2610.21613v1cs.MM

TL;DR

Ref-AVS systems often assume that longer chain-of-thought improves grounding, although clear queries can be harmed and ambiguous ones can benefit. This paper uses counterfactual budget labels to test pre-generation prediction of reasoning need and routes from the generation-onset state. The controller reduces reasoning tokens while retaining most always-long performance.

  • Problem

    Ref-AVS systems insert chain-of-thought before grounding on the premise that more reasoning always improves segmentation, despite query-dependent benefits and harms.

    Source: §1 · p. 1

  • Method

    The paper uses counterfactual budget labels to train and evaluate prediction of reasoning need from the pre-generation multimodal onset state, then routes with a lightweight controller.

    Source: §1 · pp. 2–3

  • Results

    The controller reduces reasoning-token consumption by 60% while retaining about 96% of always-long J &F.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The generation-onset state provides a useful decision point for assigning reasoning budgets in the evaluated Ref-AVS pipelines.

    Source: §7 · p. 10

    Counterfactual labels require target masks and pipeline-specific outcomes, so material changes to the backbone, prompts, or input distribution may require recalibration.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on $R^2$-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.

1 Introduction

Ref-AVS systems commonly use chain-of-thought reasoning before grounding, but the value of additional reasoning varies by query: it can hurt clear cases and help ambiguous ones. The paper studies predicting reasoning need from the multimodal model state before generation and uses that signal to allocate budgets.

1 Introduction

  • The study challenges systems that insert chain-of-thought before spatial grounding on the premise that more reasoning always improves segmentation.Ref-AVS localizes sounding objects in dynamic scenes from natural-language queries.Source: §1 · p. 1
  • No fixed reasoning budget is preferred for every sample, and an idealized per-sample policy substantially outperforms always-long reasoning.Clear queries can suffer phrase and localization drift from lengthy reasoning, while genuinely ambiguous references can benefit from it.Source: §1 · pp. 1–2
  • A linear classifier on the generation-onset state predicts whether additional reasoning will help, outperforming text-only and uncertainty-based baselines.The state is captured after multimodal inputs are processed but before the first reasoning token is generated.Source: §1 · p. 2
  • The single forward-pass controller cuts reasoning-token consumption by 60% while preserving about 96% of always-long J &F.It assigns each sample ZERO, SHORT, or LONG reasoning from the generation-onset state.Source: §1 · pp. 2–3

2 Related Work

Related work spans Ref-AVS grounding and reasoning methods, representation probing, and adaptive-compute approaches. This paper combines a fused pre-generation readout, counterfactual mask-utility supervision, and budget selection before the first reasoning token.

Reasoning in Ref-AVS

  • Ref-AVS methods range from perception-centric pipelines to structured reasoning, reflection, multi-agent approaches, and certainty-based adaptive routing.MAR3 [23] reports an empirical case where three reflection steps underperform two; this paper examines sample-level outcomes across matched budgets.Source: §2 · p. 3

Probing and Mechanistic Interpretability

  • Linear probes reveal information in representations, while mechanistic interpretability methods such as causal tracing and activation patching study functional associations.This paper uses representation probing for pre-generation budget routing; within-generation causal mechanisms are outside its evaluation.Source: §2 · p. 3

Adaptive Compute

  • Adaptive-compute methods include early exits, expert routing, and reasoning-length controllers, which often use incremental confidence, external difficulty criteria, or surface features.In contrast, this controller routes once from the generation-onset state, before any reasoning token is generated.Source: §2 · p. 3

3 Preliminaries

The preliminaries define Ref-AVS as mask prediction for a sounding target and establish a controlled comparison of three reasoning budgets. Counterfactual labels select the lowest-cost budget within a performance tolerance.

Ref-AVS Task Formulation

  • Ref-AVS predicts a spatiotemporal mask for the sounding target from an untrimmed video, synchronized audio, and natural-language referring expression.Performance uses region similarity J, contour accuracy F, and their average J &F.Source: §3 · pp. 3–4

Discrete Reasoning Budgets

  • ZERO uses the raw reference, SHORT produces a compact grounding phrase after one reasoning step, and LONG uses a full TGS-style multimodal chain of thought.The same GroundingDINO detector and SAM2 segmenter follow each policy, while prompts differ as well as realized lengths.Source: §3 · p. 4

Counterfactual Budget Labeling

  • Counterfactual Budget Labeling assigns the cheapest budget whose empirical J &F is within 0.02 of the best evaluated budget.All-zero cases are assigned ZERO by cost tie-break, excluded from supervision under the original protocol, and retained in full-raw evaluation.Source: §3 · p. 4

The Generation-Onset Representation

  • The generation-onset representation is the final-layer state at the last input token, after multimodal evidence is integrated but before reasoning text is generated.It can be reused for routing without an additional model forward pass.Source: §3 · p. 4

4 Methodology

The methodology tests whether reasoning-budget utility is predictable from the generation-onset representation and whether that information can support routing. It compares the full state with hand-crafted signals and uses a lightweight controller for budget selection.

4 Methodology

  • The methodology centers on reusing the generation-onset representation for probing, signal analysis, and adaptive routing.The state is captured before generation begins.Source: §4 · p. 4

4.1 The Pre-Decisional Budget Signal

  • A linear classifier reads the onset state to predict the counterfactual budget label, with binary detection distinguishing ZERO from SHORT or LONG.The labels represent whether reasoning is needed under the evaluated CBL rule.Source: §4.1 · p. 4

4.2 Hand-Crafted Signals: A Proxy Comparison

  • The proxy comparison tests visual and auditory attention entropy, query-induced contextual shift, and first-token decoding uncertainty as hand-crafted signals of reasoning need.The entropy measures aggregate attention over layers, heads, and reference tokens; the shift compares onset state with the state before reference-text tokens.Source: §4.2 · p. 5
  • Comparing the joint four-signal vector with the full onset state measures information missed by those evaluated summaries, not by every possible compact feature family.A performance gap does not rule out other compact features.Source: §4.2 · pp. 5–6

4.3 Projection Removal Intervention

  • The projection-removal ablation removes the training-fold budget-predictive direction from train and held-out onset features, then refits a probe.It measures readout degradation after refitting without modifying the frozen MLLM’s generation activations.Source: §4.3 · p. 6

4.4 The Single Forward-Pass Controller

  • A two-layer MLP maps the onset state to ZERO, SHORT, or LONG, and reuses the backbone’s encoding pass for its routing decision.The query is dispatched by the highest softmax probability; the controller adds near-zero latency.Source: §4.4 · p. 6

5 Experiments

The experiments find heterogeneous reasoning needs, show that generation-onset states predict those needs, and evaluate onset-based routing against proxy signals and other controllers. The controller offers a quality–cost trade-off, while the evidence is bounded by the evaluated protocols and settings.

5.1 Experimental Setup

  • The evaluations use Llama-2-7B with LoRA adapters on RefAVSBench and source-only transfer to R2-AVSBench, with readout and router metrics estimated out of fold.The RefAVSBench fold estimates are not evaluations on an untouched official test split.Source: §5.1 · p. 6

5.2 Quantifying Reasoning-Budget Heterogeneity

  • The cost-aware CBL oracle exceeds Always-Long by +5.3 to +7.4 J &F points, and all three budgets receive substantial label mass across splits.The 42.4% need-zero share on RefAVSBench test_s identifies cases where direct grounding is within τ of the best evaluated outcome, not cases where long reasoning always harms quality.Source: Table 1 · p. 6, §5.2 · p. 7

5.3 Pre-Decisional Readout of Reasoning Need

  • A linear probe on honset reaches 70.1% binary accuracy against a 57.7% majority baseline, with readability peaking at L19 and remaining informative at the final layer.The dimensional recovery curve retains substantial readout performance in reduced PCA space without identifying a unique encoding mechanism.Source: §5.3 · p. 7, Figure 3 · p. 7, Table 2 · p. 8

5.4 PDBS Compared with Interpretable Proxies

  • Evaluated hand-crafted proxies narrow but do not close the gap to the onset MLP, whose accuracy is 0.689 versus 0.586 for the combined stronger features.These comparisons support a claim limited to evaluated proxies, not irreducibility or a causal mechanism.Source: §5.4 · p. 8

5.5 Functional Evidence for the Routing Subspace via Projection Removal

  • Removing the budget-predictive direction lowers retrained-probe accuracy from 70.1% to 61.2% and routed J &F from 0.528 to 0.480.The ablation modifies readout features rather than the MLLM’s generation activations, so it does not establish causal self-assessment.Source: §5.5 · p. 9

5.6 Single Forward-Pass Controller Performance

  • Onset routing reduces reasoning tokens by 39.6% versus Short-then-Decide with a 0.007 decrease in J &F, while source-only transfer to R2-AVSBench yields 0.458 J &F versus Always-Long’s 0.474.The controller lies on the evaluated Pareto frontier; calibration and larger action-set experiments extend the tested policy family but do not establish arbitrary early-exit optimality.Source: §5.6 · pp. 9–10, Table 4 · p. 9

6 Discussion & Limitations

The discussion supports onset-state readout for external Ref-AVS routing, but not causal self-assessment by the frozen MLLM. Calibration and policy choices constrain where the approach is established.

6 Discussion & Limitations

  • Readout and projection-removal results concern external routing features, not the frozen MLLM’s generation activations, so they do not establish causal self-assessment.The supported claim is utility for external Ref-AVS routing.Source: §6 · p. 10
  • CBL requires target masks and pipeline-specific outcomes, so material changes to the backbone, prompts, or input distribution may require recalibration.Offline costs must be amortized over repeated queries.Source: §6 · p. 10
  • The three-policy action set is an auditable discretization, not an established optimum over arbitrary token- or layer-level exits.Wider task transfer and within-generation causal mechanisms remain open questions.Source: §6 · p. 10

7 Conclusion

The conclusion presents generation-onset routing as a way to allocate reasoning budgets in Ref-AVS. On RefAVSBench Seen, the adaptive controller uses 40% of the Always-Long reasoning-token budget, while its overall J &F is lower than the reported TGS-Agent baseline.

7 Conclusion

  • Generation-onset hidden states provide predictive information for pre-generation budget routing, and the original three-action controller retains approximately 96% of Always-Long J &F while reducing reasoning tokens by 60%.Source-only transfer provides a further cost–quality trade-off.Source: §7 · p. 10

A Full RefAVSBench Comparison

  • At 40% of the Always-Long reasoning-token budget, the original adaptive controller achieves 52.8 J &F on the Seen split.The reported component scores show a trade-off: the controller has higher J on Seen and Mix and slightly higher F on Unseen, but lower overall J &F than TGS-Agent.Source: A Full RefAVSBench Comparison · p. 13
  • The aggregate differences from TGS-Agent do not identify the mechanism responsible for the component-score trade-off.The Mix summaries are not sample-count-weighted averages of the displayed Seen and Unseen scores.Source: A Full RefAVSBench Comparison · p. 13

B Implementation Details

The implementation details describe feature extraction, training, budget prompts, and out-of-fold evaluation for the routing experiments. The CBL label distribution remains stable across the evaluated tolerance settings.

B Implementation Details

  • The LoRA merge check found a 0.030 relative feature difference in one sample, which documents that instance rather than population-wide equivalence.The primary probing and routing experiments use merged LoRA weights.Source: B Implementation Details · p. 13
  • Generation-onset extraction uses manually validated boundaries at the final audio/video context and reference-text tokens to capture the state before decoding.The validation covered samples with highly variable reference lengths.Source: B Implementation Details · p. 14
  • Probes and MLP controllers use Adam at 10^-3, batch size 32, and up to 50 epochs with validation-loss early stopping; the study used about 85 GPU-hours and 10 CPU-hours.The MLP maps 4096 to 256 to 3 with ReLU and 0.1 dropout; class-balanced weights address dataset imbalance.Source: B Implementation Details · pp. 14–15
  • ZERO skips autoregressive decoding and sends the raw query to grounding, while SHORT and LONG use distinct reasoning prompts with the same multimodal inputs and stop-token template.Reported token counts are realized averages, and choosing ZERO still requires multimodal input encoding.Source: B Implementation Details · p. 14
  • The five-fold evaluation trains routers and selects stopping points using training-fold labels, reserving held-out labels for metric reporting.RefAVSBench estimates are out-of-fold within benchmark populations rather than training on an official split and testing on an untouched population; original R2-AVSBench evaluation uses no target-label fitting or tuning.Source: B Implementation Details · p. 14
  • Adaptive baselines share the five-fold protocol, including a mean-pool feature controller, a GroundingDINO confidence router, and Short-then-Decide.Short-then-Decide generates SHORT for every sample, then routes using its hidden and logit features.Source: B Implementation Details · p. 15
  • Cross-backbone Qwen-VL probe accuracies measure agreement with transferred TGS-Agent labels, not native optimal-budget accuracy.Audio-derived information is supplied through an audio-summary prompt and feature interface when raw audio is not natively tokenized.Source: B Implementation Details · p. 15

C CBL Threshold Sweep

  • Across τ settings from 0 to 0.03, budget-label distributions remain stable and Cohen’s κ is at least 0.79 relative to τ = 0.02.This supports persistence of the observed heterogeneity across the evaluated tolerances.Source: C CBL Threshold Sweep · p. 15

D Additional Robustness Checks

Additional robustness checks examine attention summaries, audio input, and supervision for samples where every budget fails. The reported supervision comparison finds little change in full-raw J &F while reducing token use.

D Additional Robustness Checks

  • The tested last-four-layer and five-most-variable-head attention entropy aggregations each yield binary AUC below 0.53, showing limited discrimination for those aggregations.The result does not rule out other attention summaries.Source: D Additional Robustness Checks · p. 15
  • Replacing raw audio with Whisper-generated ASR transcripts reduces J &F by 0.006 on test_s and 0.008 on test_u.The comparison does not identify which acoustic cues account for the difference.Source: D Additional Robustness Checks · p. 15
  • The E1–E0 full-raw J &F difference is −0.0002 (95% CI [−0.0018, 0.0014]), and the token difference is −4.7 (95% CI [−5.6, −3.8]).E1 includes cost-tie ZERO labels for all-fail cases; E0 excludes those cases from training and validation supervision, while both are evaluated on all 2,287 samples.Source: D Additional Robustness Checks · pp. 15–16, Table 6 · p. 16
  • The 165 all-fail test_s cases reduce population mean token use by 4.69 tokens because every evaluated action scores zero on them.Their cost-tie labels affect cost but not J &F, and are not interpreted as successful-budget routing accuracy.Source: D Additional Robustness Checks · p. 16

E Additional Diagnostic Tables

Additional diagnostic tables assess binary reasoning-need prediction, errors near budget-label boundaries, and projection uncertainty. They distinguish predictive performance from what can be concluded about ambiguity or the model’s own generation.

E Additional Diagnostic Tables

  • The separately trained binary MLP obtains 0.689 accuracy for distinguishing need-zero from need-reasoning, while a same-dimensional Gaussian control remains near the reported majority result.The three-way router collapsed to binary and the separately trained binary MLP are distinct predictors.Source: E Additional Diagnostic Tables · p. 16, Table 7 · p. 16
  • Need-short errors concentrate near weakly separated action outcomes among the 2,122 informative cases, supporting label-boundary ambiguity without establishing stochastic label noise or uniformly low regret.The median split compares high-separation with near-tie cases; Table 8 reports binary accuracy from the original three-way controller’s predictions.Source: E Additional Diagnostic Tables · p. 16, Table 8 · p. 17
  • The overthinking-trap audit covers 200 need-zero examples where ZERO exceeds LONG by more than τ, rather than the entire need-zero class or test population.This scopes the selected harmful-long subset.Source: Table 9 · p. 17
  • Removing the selected projection direction changes retrained-probe accuracy by −0.089 (95% CI [−0.103, −0.075]) and routed J &F by −0.048, versus −0.003 for random-direction removal.The results support the direction’s usefulness for external readout and routing, but do not establish that the frozen MLLM uses it during generation.Source: E Additional Diagnostic Tables · p. 17

F Hand-Crafted Signal Results

Evaluated hand-crafted proxy summaries retain less predictive information than the onset-state readout, while the computational analysis distinguishes measured reasoning costs from offline calibration costs.

F Hand-Crafted Signal Results

  • Stronger proxy features narrow the gap to the onset MLP from 14.8 to 10.3 percentage points, but do not recover all of its predictive information.The four-signal MLP reaches 0.541 binary accuracy versus 0.689 for the onset-state MLP; the combined stronger features reach 0.586.Source: F Hand-Crafted Signal Results · p. 17, Table 11 · p. 18

G Computational Cost Analysis

  • The cost analysis measures reasoning tokens and runtime in the stated configuration, not energy, memory, or throughput savings across other serving setups.The direct paired latency benchmark separately distinguishes routing-decision time from completion of the MLLM and segmentation pipeline.Source: G Computational Cost Analysis · p. 17, Table 10 · p. 18
  • The historical 7.2k–127.5k-query amortization range is an accounting bracket, not a universal deployment guarantee.Its bounds use study-wide versus MLLM-only accounting; the narrower calculation excludes detector/segmenter evaluation and other setup work.Source: G Computational Cost Analysis · p. 17

G.1 Direct paired latency benchmark

The onset router reaches a lower measured latency and token-cost point than Short-then-Decide while retaining nearly its quality. Additional figures and calibration analyses characterize its operating points, transfer, and calibration requirements.

G.1 Direct paired latency benchmark

  • With cache reuse, onset routing remains 0.61 s faster in mean MLLM latency than Short-then-Decide, while their time-to-decision difference is unchanged.Complete-pipeline mean/p95 latencies are 6.42/8.96 s for onset routing, 7.02/9.42 s for cached Short-then-Decide, and 7.28/10.10 s without cache reuse.Source: §G.1 · pp. 18–19
  • 39.6% fewer reasoning tokens with a 0.007 decrease in J &F; onset routing reduces mean decision time by 61.5% and model latency by 27.1% versus Short-then-Decide.The onset decision occurs first on 99.8% of samples and retains 98.7% of Short-then-Decide’s quality.Source: §G.1 · p. 18, Table 14 · p. 19

H Additional Figures

  • The qualitative examples contrast long-reasoning drift on a simple query with successful resolution of an ambiguous instrument reference.The proxy diagnostics show a 14.8-point accuracy gap for four signals, narrowed to 10.3 points by stronger evaluated proxies; the per-label figure reports lowest recall for Need-short.Source: Figure 4 · p. 19, Figure 5 · p. 20, Figure 7 · p. 21

I.1 Calibration protocol and source-domain sensitivity

  • One-quarter of the source calibration pool preserves the retrained full-data operating point within the specified margins, whereas 5% and 10% fall outside the quality margin.At 25%, the quality difference is −0.003 and token difference +1; the smaller conditions have quality differences of −0.017 and −0.010.Source: §I.1 · p. 21, Table 15 · p. 21

I.2 Group-disjoint distribution adaptation

  • Most of the full-refresh gain on R2-AVSBench is recovered by 20% native calibration under the group-disjoint source initialization.R20 differs from R100 by −0.001 J &F and zero tokens, and exceeds Short-then-Decide by 0.009 J &F while using 75 fewer tokens.Source: §I.2 · p. 22, Table 16 · p. 22

I.3 Native backbone calibration and label accounting

Native calibration and finer semantic-budget grids provide distinct ways to improve routing outcomes, with results bounded by their supervision, action-set, and diagnostic-oracle settings.

I.3 Native backbone calibration and label accounting

  • Native-only Q20-N obtains 0.536 J &F at 107 tokens versus 0.538 at 106 tokens for Q100-N, satisfying both non-inferiority criteria.The native Qwen Always-Short reference obtains 0.535 J &F at 91 tokens, exceeding the transferred-label router’s quality while using fewer tokens.Source: §I.3 · p. 22, Table 18 · p. 23
  • Native Qwen outcomes change which cases are informative: 2,145 of 2,287 source examples are informative, and 140 of the original 165 all-fail cases become informative.Native labels therefore cannot be obtained by preserving the source informative subset.Source: §I.3 · p. 23, Table 19 · p. 23

I.4 Complete setup and conditional amortization

  • Charging construction of the retained source table makes Q20-R setup exceed the full-native reference, so its inexpensive-refresh result depends on that table genuinely predating migration.Q20-R costs 5,960 seconds with the table treated as sunk, versus 24,140 seconds when its construction is charged; abrupt pipeline or domain changes can require a new calibration study.Source: §I.4 · p. 24, Table 21 · p. 24

J.1 Semantic policies and action-dependent populations

  • The finer semantic policies have at most 0.4% cutoff incidence and at least 99.6% parse validity, but adjacent policies can cross in realized token cost.Content-dependent generation means named policy depth does not impose strict per-example token ordering.Source: §J.1 · p. 25, Table 23 · p. 25
  • Finer policies improve the tolerance-based oracle by 0.015 J &F while reducing average cost by five tokens, with 0.0141 of the gain from original informative examples.Another 0.0009 comes from original all-fail cases rescued by an intermediate action, so the gain is not created by changing the evaluation denominator.Source: §J.1 · p. 25, Table 24 · p. 25

J.2 Population quality–cost frontiers

  • The population frontiers are outcome-aware diagnostics, not deployable routers; the tolerance-based CBL policy instead selects the cheapest sample-level action within tolerance of its best score.The exact deterministic frontier chooses one evaluated action per sample, whereas the relaxed frontier permits convex mixtures.Source: §J.2 · p. 25
  • Returns diminish as the tested policy grid densifies: at 110 tokens, the five-action expansion adds 0.0102 J &F and seven actions add a further 0.0053.The second difference is −0.0049 with interval [−0.0064, −0.0034], while the positive seven-minus-three difference indicates remaining headroom over three actions.Source: §J.2 · p. 26
  • At matched three-action latency budgets, seven actions gain 0.0146 J &F under MLLM time and 0.0134 under full-pipeline time; at 110 tokens, the seven-action gain is 0.0155.The average seven-versus-three gain across 0–275 tokens is 0.0091 J &F, with interval [0.0076, 0.0106].Source: §J.2 · pp. 25–26, Table 27 · p. 26, Table 26 · p. 26

J.3 Matched controllers and action masking

This section covers matched action-space extensions and complete-stage exits for learned Ref-AVS controllers, as well as the MLLM implementation and cited assets. It notes calibration, reproducibility, and license-reporting limits, and cautions that token or runtime savings alone do not establish energy or carbon savings or validate safe deployment.

J.3 Matched controllers and action masking

  • Utility-7 improves over Utility-3 by 0.009 J &F (95% interval [0.0054, 0.0126]) across all five seeds, for two additional tokens.The quality contrast remains within the stated five-token and 0.10-second latency margins.Source: §J.3 · p. 26, Table 28 · p. 27
  • The full τ-CBL-7 controller gains 0.007 J &F [0.0040, 0.0100] and uses two fewer average tokens than τ-CBL-3.Utility-7’s point estimate is 0.540 at 112 tokens versus 0.535 at 182 for Short-then-Decide; this is not a separate paired significance test.Source: §J.3 · p. 27
  • Utility-7’s no-refit action-mask contrast raises J &F by 0.006 [0.0031, 0.0089] at two additional tokens, with added actions selected on 34.7% of cases.The checkpoint and trade-off coefficient are fixed; quality contributions from the added policies sum to 0.0060.Source: §J.3 · p. 27, Table 29 · p. 27, Table 30 · p. 27

J.4 Bounded complete-stage exit sensitivity

  • At 25% calibration, Utility-7 obtains 0.538 J &F at 111 tokens, while E7 obtains 0.537 J &F at 111 tokens; their respective setup costs are reduced.Utility-7 setup falls from 52,310 to 13,320 seconds; E7 setup is 5,620 seconds and uses shared-rollout reuse, with original three-action setup as reference.Source: §J.3 · pp. 27–28, §J.4 · p. 28
  • Complete-stage E7 exceeds E3 by 0.011 J &F at 110 tokens; its learned controller rises from 0.532 to 0.539 at 112 tokens, with a 0.005 no-refit mask gain.The experiment tests six complete grounding exits plus zero reasoning, not arbitrary token checkpoints or layer-level early exit.Source: §J.4 · p. 28, Table 31 · p. 28

K Broader Impacts

  • The paper identifies unwanted surveillance, inappropriate recording use, and incorrect grounding as potential harms, and does not equate token/runtime savings with verified energy or carbon savings.It calls for respecting recording and model permissions and human verification where incorrect masks could affect people; benchmark evaluation does not validate privacy, demographic fairness, or safety-critical deployment.Source: K Broader Impacts · p. 28, §10 · p. 33

NeurIPS Paper Checklist

  • The checklist notes that complete action prompts and Utility-K’s exact training objective are missing, no public repository provides full reproduction instructions, and the resource ledger is incomplete.The authors describe the work as empirical and methodological, without theoretical results requiring proofs.Source: NeurIPS Paper Checklist · pp. 30–32

10. Broader impacts

  • The principal datasets, models, and software sources are cited, but the paper does not explicitly enumerate every asset’s license and applicable terms.The authors therefore do not claim that the full asset-credit and licensing requirement is satisfied.Source: §10 · p. 34
  • The paper describes the MLLM’s backbone, LoRA setup, onset-state extraction, reasoning policies, and use of outputs for grounding and segmentation.The controller reads the MLLM’s pre-generation state to select a policy; the policies are not hard token caps.Source: §10 · p. 35

Related papers

Loading 2610.21613v1…