PaperlayerView exact revision on arXiv

Source-linked AI summary

Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

Zishan Ahmad, Vishal Vaddina

arXiv:2608.18591v1cs.AIcs.CL

TL;DR

Uniform reasoning budgets can waste compute or trigger over-thinking penalties in visually complex document tasks. BudgetDoc and its lightweight multimodal estimator, DRB, predict performance across reasoning budgets and enable dynamic selection that matches or improves maximum-budget baselines in 9 of 15 configurations while reducing cost.

  • Problem

    It is unclear whether a lightweight multimodal model can estimate larger LLMs’ per-document reasoning performance across budgets before inference.

  • Method

    The paper builds BudgetDoc and trains DRB to predict ordinal performance for document, prompt, model, and reasoning-budget configurations.

  • Results

    DRB-guided budget selection matches or improves F1 over always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost.

  • Takeaways & Limitations

    The results support lightweight external pre-flight estimation for selecting reasoning budgets on document-centric tasks.

  • Takeaways & Limitations

    Deployment on new providers or architectures requires calibration data, and the evaluation is limited to document-centric visual tasks.

Abstract

from arXiv · show

Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB (Document-Reasoning Balancer), an approx. 1B-parameter pre-flight estimator (SigLIP-2 + Qwen3-0.6B) that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.

1 Introduction

The introduction argues that more test-time reasoning is only conditionally beneficial for document tasks, where over-thinking can reduce performance or waste cost. It presents BudgetDoc and DRB as a lightweight multimodal approach to estimate per-sample reasoning performance and optimize budgets.

  • Motivation: Across three benchmarks and five frontier models, the over-thinking penalty is systematic and significant.Additional test-time compute can degrade performance or waste money relative to a lower-cost configuration.
  • Contributions: BudgetDoc exhaustively evaluates 360 base samples across 25 model-budget configurations, producing 9,000 labeled document-performance pairs across three tasks.The benchmark is designed for supervised estimation of how model accuracy varies with compute.
  • Contributions: DRB is a ∼1B-parameter multimodal estimator that predicts 7-class ordinal performance labels and achieves a weighted F1 of 0.753 on the held-out BudgetDoc test set.Its architecture combines SigLIP-2, Qwen3-0.6B, cross-page fusion, and a lightweight MLP head.
  • Contributions: DRB-driven budget optimization delivers upto 99% cost reductions across five frontier models while targeting the minimum budget expected to achieve near-peak performance.The introduction also identifies model selection as an exploratory application with encouraging cross-provider generalization results.
  • Motivation: Maximum reasoning budgets can be suboptimal in both accuracy and cost, motivating external pre-inference estimation of model performance by budget.The performance surface varies with document complexity, query type, and model inductive biases.

2 Related Work

Related work establishes that test-time reasoning can improve performance but may become wasteful when over-allocated, motivating inference-efficiency methods for document understanding. BudgetDoc and DRB differ from prior evaluation and routing approaches by predicting performance before execution and selecting reasoning budgets within fixed models.

  • Test-time compute scaling: Test-time compute scaling improves LLM accuracy on sufficiently complex tasks, while over-allocated reasoning can be harmful or wasteful for many inputs.This work frames over-allocation as a complementary regime to conventional reasoning-token scaling.
  • In-flight reasoning controls and pricing reversals: 32% of cost pathologies persisted across 8 reasoning models and 12 tasks despite dynamic thinking and reasoning-effort controls.Gemini 2.5 Flash and GPT-5 expose mechanisms intended to scale or regulate reasoning expenditure.
  • LLM performance prediction and evaluation: DRB is a pre-hoc estimator that predicts performance before inference, conditioned on an unexecuted model-and-reasoning-budget configuration.Prior work primarily emphasized posthoc evaluation, self-consistency scoring, or learned reward models.
  • LLM routing and model cascades: Unlike model-routing cascades that choose between strong expensive and weak cheap models, this work studies performance estimation and budget selection within a fixed model.Model selection is additionally probed as a secondary application.
  • Document understanding with VLMs: Document understanding provides a visually rich, multi-page setting where vision-language models jointly process rendered page images and text alongside structured or open-ended queries.DRB uses SigLIP-2 and Qwen3, with Transformer-based cross-page fusion.

3 Methodology

BudgetDoc exhaustively labels model–budget configurations across three document tasks, while DRB predicts seven ordinal performance classes from document, prompt, model, and budget representations. At inference, hierarchical scanning selects the highest predicted-performing budget, favoring lower cost when tied.

  • Data collection and labeling: 9,000 total labeled pairs span 360 base contexts, five models, and five reasoning budgets.The dataset is built through exhaustive evaluation of each document configuration, with prompts paraphrased using gemini-2.5-flash-lite.
  • Source benchmarks: BudgetDoc covers RVL-CDIP multi-class classification, TAT-QA arithmetic reasoning, and CheckboxQA form-checkbox extraction.RVL-CDIP concatenates three category PDFs into one multi-page document; TAT-QA uses semi-structured financial tables and text; CheckboxQA extracts checkbox states from scanned forms.
  • Data collection and labeling: Seven ordinal performance classes discretize per-sample F1 scores from total failure to perfect output.Class 0 denotes f1 < 0.5, Classes 1–5 represent thresholded partial successes, and Class 6 denotes f1 ≥0.99; document-level splitting prevents leakage.
  • DRB architecture: DRB maps each document, prompt, model, and budget configuration to a predicted performance class using four modality-specific encoders and late-fusion MLP classification.Visual pages use SigLIP-2-Large-patch16-512 with cross-page fusion, prompts use Qwen3-0.6B, and learned embeddings represent model and budget.
  • Inference-time budget selection: DRB scans budgets from lowest to highest, stops on predicted Class 6, and otherwise selects the highest predicted class with lower cost breaking ties.It jointly scores model–budget pairs in an exploratory model-selection case study, extending beyond its training distribution.

4 Experiments

Experiments on 100 held-out documents across three document tasks show that DRB estimates budget-dependent performance effectively and usually reduces inference cost while preserving or improving F1. Exploratory model-selection results retain substantial savings but reveal larger accuracy losses, especially across GPT and provider families.

  • Estimation quality: DRB achieves a weighted F1 of 0.753 on the BudgetDoc test set (n = 2500), showing that a ∼1B-parameter model estimates frontier-LLM performance across reasoning budgets.Class 6 recall reaches 0.932, identifying samples that succeed without extended reasoning.
  • Budget optimization: 13 of 15 model-dataset configurations reduce cost under DRB-guided budget selection, with reductions ranging from 5.2% to 99% versus maximum-budget baselines.The evaluation covers five models across RVL-CDIP, TAT-DQA, and CheckboxQA using held-out documents.
  • Budget optimization: 9 of 15 configurations match or improve baseline F1, including gains of +5.7% on CDIP, +5.2% on TAT-DQA, and +5.1% on CheckboxQA for gemini-2.5-flash.TAT-DQA remains the hardest transfer target, with F1 losses of up to −5.7% in some configurations.
  • Budget optimization: GPT-5.2 costs $0.318/sample on CheckboxQA at maximum budget, while DRB achieves 98.5% cost reduction with F1 changes of ≤3.2%.On CDIP, GPT-5.2 costs $0.096/sample and DRB achieves a 99.0% reduction.
  • Cross-model selection: Cross-family selection yields F1 drops of 6.4% on CDIP and 11.2% on TAT-DQA while retaining 59–94% adjusted cost savings, exposing calibration limits across providers.This case study is exploratory because DRB was trained for fixed-model budget estimation rather than model routing.

5 Analysis

DRB shows that a lightweight multimodal estimator can guide reasoning-budget allocation effectively, despite systematic over-thinking penalties and irregular hard-task performance surfaces. Its benefits are greatest for expensive models and complex documents, while early stopping keeps estimation overhead modest.

  • Small models can estimate LLM performance: 0.753 weighted F1 shows DRB can estimate ordinal frontier-model performance, reliably identifying Class 6 with F1 = 0.860 and recall 0.932.Class 6 represents tasks completely solvable at every budget level.
  • When estimation is most valuable?: $0.318 vs. $0.004 per sample indicates why estimation-guided selection is most valuable for high-cost models on complex documents.For cheap Gemini-Lite models on short documents, DRB latency can instead dominate because the absolute cost gap is small.
  • Estimation fails gracefully on hard tasks: 40–98% cost savings show that DRB fails gracefully on TAT-DQA, even when irregular budget-performance surfaces make estimation difficult.The estimator can still avoid extremely expensive configurations despite unreliable mappings between chain length and accuracy.
  • Early stopping reflects estimation confidence: 1.0–3.9 of 5 budget levels are scanned on average, and all Gemini models on CDIP achieve a 100% early-stop rate.DRB terminates at the first budget level on CDIP when sufficiently confident, keeping operational overhead modest relative to downstream API savings.

6 Conclusion

The conclusion finds that DRB can estimate frontier-LLM document-task reasoning performance, enabling large cost reductions while maintaining or improving F1. It also reports partial transfer to model selection and motivates external pre-flight estimation despite built-in reasoning controls.

  • Conclusion: 0.753 weighted F1 and 0.932 recall on the perfect-performance class show that DRB estimates frontier-LLM reasoning performance on BudgetDoc.DRB is an approximately 1B-parameter lightweight multimodal estimator.
  • Conclusion: Up to 99% cost reductions were achieved across five frontier models and three datasets while maintaining or improving F1 in the reported configurations.The savings came from per-sample reasoning-budget selection using DRB’s estimates.
  • Conclusion: Model-selection transfer within the Gemini family incurred −2.5% to −9.9% F1 losses and yielded 10–76% cost savings, while broader GPT and cross-family settings degraded in F1.Cost savings remained 59–99% in the broader settings by avoiding expensive high-effort configurations.
  • Conclusion: 32% of nominally cheaper models still cost more in practice because of thinking-token overruns, supporting external pre-flight estimation beyond built-in dynamic thinking controls.The cited comparison concerns Gemini auto-thinking and GPT reasoning-effort tiers.

Limitations

The framework performs robustly across Gemini and GPT families but requires calibration when adapted to new providers or unobserved architectures. This deployment limitation reflects the need to collect configuration–outcome pairs for target-domain fine-tuning.

  • Deployment limitations: Adapting the architecture to new model providers or unobserved architectures requires a small calibration sample collected through standard evaluation and used to fine-tune the estimator.The passage identifies this data-gathering requirement as a deployment limitation.

Ethics Statement

The study uses public benchmarks and frontier-model APIs under standard commercial terms, without collecting personal data. Its estimation-guided budget selection reduces unnecessary reasoning-token expenditure, with positive environmental impact.

  • The work uses publicly available RVL-CDIP, TAT-QA, and CheckboxQA benchmarks.
  • Frontier model APIs were accessed under standard commercial terms, and no personal data was collected or used.
  • Estimation-guided budget selection reduces unnecessary reasoning-token expenditure, producing a positive environmental impact.

A DRB Per-Class Estimation Performance on BudgetDoc

DRB’s BudgetDoc test-set estimates span seven F1-score bins, from total task failure to perfect output. Class 6 recall is strongest and most consequential for assigning minimal budgets, while Class 1 recall remains difficult to distinguish from total failure.

  • Evaluation setup: Seven F1-score bins (Classes 0–6) define DRB’s per-class estimation evaluation on the BudgetDoc test set (n = 2500).Class 0 represents total task failure, while Class 6 represents perfect output.
  • Class-wise performance: 0.932 Class 6 recall most consequentially indicates reliable minimal-budget assignment when any budget would produce correct output without sacrificing accuracy.Class 6 denotes perfect performance and has the highest F1, driven by strong recall.
  • Class-wise performance: 0.361 Class 1 recall reflects the difficulty of distinguishing near-total failures from total failures, though both receive low-cost budget assignments.This fine-grained distinction matters less in practice because both classes map to low-cost budgets.

B Implementation Details

DRB combines SigLIP-2 and Qwen3 components with a 12-layer fusion transformer, and its deployment uses dynamic early-stopping scans whose full computational cost is charged against downstream savings.

  • Model and training: DRB uses a SigLIP-2-large-patch16-512 vision backbone, Qwen3-0.6B prompt encoder, and 12-layer fusion transformer with hidden dimension 1152.Training ran for 20 epochs on an NVIDIA A100 (80GB) with AdamW, learning rate 5 × 10−5, and batch size 4.
  • Compute overhead: $0.54/hr total deployment cost on GCP n1-standard-4 includes one NVIDIA T4 GPU and reduces to $0.00015 per second of compute.Hierarchical budget scanning uses early stopping, so execution latency varies with document-query complexity.
  • Cost accounting: Adjusted DRB Cost fully charges the pre-flight estimator’s local computational burden against downstream API savings.The adjustment uses empirical mean scanning latency for each model-dataset group, represented by ∆tconfig.

C GPT Effort-Level Sensitivity

Table 4 compares DRB with GPT models across all four reasoning-effort tiers, while accounting for dynamic configuration-level compute overhead through adjusted cost factors.

  • GPT Effort-Level Sensitivity: Table 4 extends the comparison to all four GPT reasoning-effort tiers: none, low, medium, and high/xhigh.The comparison is between DRB and GPT models.
  • GPT Effort-Level Sensitivity: Adjusted ∆Cost factors in the dynamic configuration-level compute overhead.The table marks improvements in green and degradations in red.
Loading 2608.18591v1…