PaperlayerView exact revision on arXiv

Source-linked AI summary

From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model

Xunlan Zhou, Xianliang Yang, Li Zhao

arXiv:2609.29283v1cs.AIcs.LG

TL;DR

Visual software often needs a choice among supplied alternatives rather than generated text. PixelJev supplies a native-image, runtime-candidate decision interface and evaluates frozen and adapted small multimodal models; adaptation improves source recognition and selected transfers, while broader reliability requirements remain open.

  • Problem

    Visual software may need a category or a choice from changing options, while fixed classifiers expose a fixed label vocabulary and conversational VLMs generate text.

    Source: §1 · p. 1

  • Method

    PixelJev maps native image input, an instruction, and runtime candidate schema to a structured choice and candidate-conditioned probabilities.

    Source: §1 · p. 1

  • Results

    Few-shot adaptation improves source recognition and selected transfers, but does not universally replace larger models or establish reliable confidence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The working interface supports recognition and question answering, while general-purpose use still requires competence and reliability as tasks and visual evidence change.

    Source: §8 · p. 8

    Transfer evidence is limited by one adaptation split, three optimization seeds, one VQA adapter seed, and unknown foundation-model pretraining overlap.

    Source: §6 · p. 7

Abstract

from arXiv · show

Visual software often needs a decision over supplied alternatives rather than a generated explanation. We present PixelJev, a native-image decision interface that maps an image, a task instruction, and a runtime candidate set to a structured choice and candidate-conditioned probabilities using small open multimodal models. Its initial realization unifies recognition and multiplechoice visual question answering through an existing language-model readout, with separately evaluated options for frozen inference, language-side adaptation, and held-out calibration. Across seven benchmark evaluations, 64-shot source adaptation raises Pets accuracy from 60.13% to 92.40% across optimization seeds and transfers to natural resampling, new texture labels, and A-OKVQA without target fitting, while frozen inference already supports both VQA tasks. A matched prompt-only follow-up on Pets and ScienceQA attributes the large Pets gain to adaptation and identifies a narrower output validity benefit of candidate readout in adapted VQA. Specialist DINOv2 probes remain stronger on source recognition, frozen 4B is stronger than adapted 2B on DTD and ScienceQA, and accuracy gains do not ensure calibrated target probabilities. These findings establish a working starting point for general-purpose visual decision models and identify the remaining requirements: schema robustness, cross-family transfer, and reliable use of visual evidence.

1. Introduction

Visual software often needs a choice among supplied alternatives, while fixed classifiers and generated text offer different interfaces. PixelJev makes the decision space runtime-supplied and evaluates how a small multimodal model reuses that interface across tasks, while identifying broader requirements that remain open.

1. Introduction

  • A visual decision model lets callers supply the decision space at runtime and returns a valid identifier with scores conditioned on that space.This sits between a fixed classifier vocabulary and a VLM that generates text.Source: §1 · p. 1
  • PixelJev uses native image input and small open multimodal backbones to map an image, instruction, and candidate schema to a choice and candidate-conditioned probabilities.The image enters the vision branch directly, without a captioning intermediary.Source: §1 · p. 1
  • General-purpose visual decisions require interface reuse, held-out task-family competence, and reliability as visual evidence and decision schemas change; this paper establishes reuse and measures early transfer.Broader competence and reliability remain open requirements.Source: §1 · p. 1
  • The implementation reuses an existing language-model readout and evaluates frozen inference, language-side LoRA, and scalar calibration as separate choices.The paper asks how classification-only few-shot adaptation transfers and when frozen scale or specialist representations are better alternatives.Source: §1 · p. 1
  • PixelJev specifies a reusable contract with native pixels, runtime candidate descriptions, stable return identifiers, and explicit probability semantics.Its evaluations span source recognition, natural resampling, new texture labels, and direct VQA, alongside larger-model, specialist, and matched prompt-only alternatives.Source: §1 · p. 2

2. Related Work

PixelJev builds on program-facing decision interfaces, candidate-logit readout, and established visual representations rather than claiming a new learning algorithm. Its empirical focus is how small-model recognition adaptation transfers across tasks and compares with scale and specialist alternatives.

2. Related Work

  • Visual Jev combines language-model-head readout, answer-supervised adaptation, and shared-context execution; PixelJev instead studies recognition-label adaptation and classification-to-VQA transfer.The papers differ in backbone and evaluation choices, defining scope rather than measured superiority.Source: §2 · p. 2
  • PixelJev assembles established components: CLIP image-text alignment, DINOv2 image features, and LoRA adaptation, around a programmatic contract.Candidate scoring is close to ordinary multiple-choice VLM inference, not evidence of a distinct learning algorithm or visual representation.Source: §2 · p. 2
  • The evaluations separate interface reuse, source adaptation, and target probability quality across natural resampling, texture semantics, and question-specific choices.The settings do not establish exclusion from unknown foundation-model pretraining.Source: §2 · p. 2

3. PixelJev: A Reusable Visual Decision Interface

PixelJev formalizes a runtime-supplied candidate interface, then realizes it with existing multimodal-model components and separately evaluates frozen inference, adaptation, and calibration. Its diagnostics test finite interface behaviors rather than universal schema invariance or visual evidence dependence.

3.1. The Decision Contract

  • A request combines an RGB image, instruction or question, and ordered candidate schema; the model returns a distribution over candidates and program-supplied identifiers, with provenance recorded.Recognition fixes the instruction and descriptions, while multiple-choice VQA changes them per question; new-class evaluation uses descriptions outside adapter supervision without fitting a new head.Source: §3.1 · pp. 2–3

3.2. An Initial Realization with Open Multimodal Models

  • The implementation validates candidate slots and reads probabilities from the existing model rather than introducing a new architecture.The authors do not claim an intrinsic speedup or universal task-accuracy advantage over conventional VLM prompting.Source: §3.2 · p. 3

3.3. Three Separately Evaluated Operating Regimes

  • Frozen inference needs no benchmark fitting labels or extra decision head; adaptation optionally minimizes candidate-conditioned cross-entropy with one shared LoRA adapter across three source recognition tasks.Calibration separately fits a positive scalar temperature on held-out labels, leaving the backbone and argmax unchanged.Source: §3.3 · p. 3

3.4. Interface Diagnostics

  • Finite diagnostics include candidate reversal, an alternate instruction, and substitution of a different-class image, but do not establish arbitrary schema invariance or visual evidence dependence on every task.The reported controls are in the appendix.Source: §3.4 · p. 3

4. Evaluating Reuse, Adaptation, and Transfer

The evaluation progresses from frozen interface use to restricted source adaptation and transfer without target fitting, treating recognition, shifts, and VQA as distinct questions. It compares practical alternatives and uses explicit selection and uncertainty protocols rather than collapsing performance into one generality score.

4. Evaluating Reuse, Adaptation, and Transfer

  • Evaluation separates source recognition, natural shift, new texture semantics, and per-question VQA choices rather than combining them into a single generality score.The source tasks are CIFAR-10, Oxford-IIIT Pets, and EuroSAT, with full official test partitions retained.Source: §4 · pp. 3–4, §4.1 · p. 4

4.1. Source Tasks and Supervision

  • The 64-shot condition uses 3,648 fitting images across source tasks and is repeated with optimization seeds 7, 8, and 9 on the same split.The 16-shot set is a strict subset; development and calibration sets remain unchanged across budgets, and the models are not seed ensembles.Source: §4.1 · p. 4

4.2. Alternatives to Adapting the Interface

  • Comparisons include frozen Qwen checkpoints, CLIP zero-shot scores, and CLIP/DINOv2 feature probes, but differ in total supervision, trainable parameters, and compute.They are practical alternatives, not compute-matched architectural ablations; prompt-only controls are separately examined.Source: §4.2 · p. 4

4.3. Matched Prompt-Only Follow-up

  • The prompt-only follow-up crosses frozen 2B or the seed-7 adapter with ordinary generation or candidate readout on Pets and ScienceQA, using identical inputs and a fixed format cue.The targeted test does not establish readout effects on the other five benchmarks.Source: §4.3 · p. 4

4.4. Transfer Without Target Fitting

  • Transfer without target fitting includes CIFAR-10.1 natural resampling, DTD's new texture labels, and image-containing A-OKVQA and ScienceQA questions.DTD simultaneously changes visual content, semantics, and candidate cardinality; VQA evaluations use no fitting, demonstration selection, prompt search, temperature estimation, or checkpoint selection.Source: §4.4 · p. 4

4.5. Selection and Uncertainty

  • Accuracy is primary, with NLL, Brier score, ECE, and selective risk also retained; uncertainty uses paired group bootstrap intervals without multiplicity correction.LoRA spreads are sample standard deviations across optimization seeds, not training-split uncertainty.Source: §4.5 · p. 5

5. What Transfers Through a Shared Interface?

The shared decision interface works across source recognition, shifted and new-label evaluations, and per-question VQA, but adaptation does not improve every workload or guarantee reliable probabilities.

5.1. Source Fitting Improves the Small Model, Not Every Model Ranking

  • Pets accuracy rises from 60.13% frozen 2B to 92.40 ± 0.26% adapted, while EuroSAT rises from 49.63% to 88.31 ± 1.57%.CIFAR-10 changes from 96.04% to 97.04 ± 0.11%.Source: §5.1 · p. 5
  • DINOv2’s 64-shot probe has the highest point estimate on all three source tasks, while frozen 4B narrows the adapted model’s advantage only on CIFAR-10.The source-task DINOv2 estimates are 95.67%, 97.62%, and 92.13%; frozen 4B estimates are 70.65%, 96.92%, and 56.81%.Source: §5.1 · p. 5

5.2. Source Adaptation Extends Beyond Its Label Vocabulary

  • On DTD, adapted 2B scores 45.85%, below frozen 4B at 64.79%; every seed improves over the smaller frozen base, but adaptation does not uniformly replace frozen model scale.The specified adaptation-overlap check flagged no target images on CIFAR-10.1 or DTD; this is a result of one particular check, not proof of unrestricted data independence.Source: §5.2 · p. 6

5.3. Per-Question Schemas Work Without VQA Fitting

  • Without VQA fitting, frozen 4B scores 85.07% on A-OKVQA and 89.34% on ScienceQA, while the classification adapter scores 83.32% and 82.35%.On A-OKVQA, adaptation improves over frozen 2B by 2.71 pp, with interval [1.13, 4.30]; on ScienceQA, its +0.84 pp change has interval [−2.14, 3.33].Source: §5.3 · p. 6
  • Frozen CLIP question-answer matching scores 59.21% on A-OKVQA and 41.99% on ScienceQA, below every evaluated Qwen variant on both datasets.The comparisons use one fixed adapter seed, and preprocessing and shortened ScienceQA contexts prevent attributing the entire gap to reasoning ability.Source: §5.3 · p. 6

5.4. Prompt-Only Controls Separate Adaptation from Output Validity

  • On Pets, both matched readout paths rise from 56.36% to 91.63%, a +35.27 pp change; on ScienceQA, adapted direct readout scores 80.81% versus 78.93% with generation.The ScienceQA direct-minus-generated difference is +1.88 pp ([1.13, 2.86]) and reflects output validity in these records; ordinary generation takes 1.32–1.44 times direct-request time.Source: §5.4 · p. 6

5.5. Probability-Valued Outputs Still Need Reliability Evidence

  • On DTD, adaptation lowers mean raw NLL from 2.6985 to 1.9253, while mean raw ECE changes from 0.1905 to 0.2040, so accuracy gains do not ensure calibrated target probabilities.Calibration does not improve every metric: frozen CIFAR-10 and adapted EuroSAT worsen in ECE despite improved NLL.Source: §5.5 · p. 6
  • The adapter has 7,382,016 trainable parameters, and 64-shot optimizer-step intervals span 3,220.25–3,373.94 seconds, excluding loading and several evaluation and saving costs.Appendix B.1 separates this training cost from inference latency.Source: §5.5 · p. 6

6. Discussion: What Does “General-Purpose” Require?

PixelJev demonstrates reuse of one decision contract across recognition and changing VQA options, but this does not establish general visual competence. The discussion also limits what the transfer results show about visual evidence, confidence, and the scope of evaluation.

6. Discussion: What Does “General-Purpose” Require?

  • The same contract and candidate-token readout serve fixed recognition vocabularies and changing VQA options, with source-trained adapters reused without target fitting.This demonstrates concrete forms of reuse, not competence on arbitrary tasks or detection, segmentation, action selection, and open-world rejection.Source: §6 · p. 7
  • PixelJev is not claimed as the first visual decision architecture, and no matched experiment against Visual Jev is reported.The evidence instead concerns classification-to-decision transfer, frozen model scale, and specialist alternatives.Source: §6 · p. 7
  • Accuracy gains do not demonstrate improved visual representations because the visual tower is frozen, and the prompt-only factorial tests inference paths rather than training objectives.It separates the fixed adapter from its inference path, not candidate-conditioned training from ordinary answer SFT.Source: §6 · p. 7
  • Transfer evidence is bounded by limited seeds and splits, unknown pretraining overlap, and benchmark settings that change several factors together.For example, DTD changes images, semantics, candidate count, and supervised slot coverage together; VQA accuracy with images does not prove visual evidence use.Source: §6 · p. 7
  • Candidate confidence excludes unlisted alternatives and changes with the candidate set, so scores need not compare across schemas or establish deployment risk.Low ECE alone is insufficient, and detecting missing visual evidence differs from predicting answer error.Source: §6 · p. 8

7. A Research Agenda for Visual Decision Models

The agenda proposes tests for schema robustness, cross-family transfer, evidence-aware decisions, and deployment value. These are directions for future work, not additional results.

7. A Research Agenda for Visual Decision Models

  • Test schema robustness with semantic-ID-aligned permutations, paraphrases, and controlled distractor changes to separate slot or token artifacts from task difficulty.The proposed tests target whether decisions remain robust when candidate schemas change.Source: §7 · p. 8
  • Evaluate cross-family transfer by withholding complete decision families and image groups, with matched ordinary VLM and specialist baselines.Training should span multiple decision families.Source: §7 · p. 8
  • Test evidence-aware decisions with relevant-region interventions and irrelevant-region controls, separating answer correctness from the need for a better observation.These are proposed experiments, not findings of the present work.Source: §7 · p. 8
  • Measure deployment value through end-to-end risk, latency, and memory on concrete workloads, comparing shared-context execution with batched and cached baselines.The agenda cautions against inferring shared-context execution benefits from one-token readout alone.Source: §7 · p. 8

8. Conclusion

PixelJev offers a working native-image interface with runtime-supplied candidate schemas rather than task-specific output heads. The conclusion frames general-purpose visual decisions as requiring reliable performance as tasks, schemas, and visual evidence change.

8. Conclusion

  • PixelJev provides a working native-image decision interface whose candidate schema is supplied at runtime rather than fixed by a task-specific output head.Small frozen multimodal models already instantiate this interface.Source: §8 · p. 8
  • Few-shot language-side adaptation improves source recognition and selected transfers without VQA-specific fitting, while specialist encoders and larger frozen models expose limits of this realization.Held-out calibration also reveals remaining limitations.Source: §8 · p. 8
  • Progress toward general-purpose visual decisions requires preserving useful decisions as task semantics, candidate schemas, and visual evidence change.The conclusion cautions that improving one classification score alone is not the whole path.Source: §8 · p. 8

Impact Statement

The work evaluates bounded visual decisions on existing research benchmarks and does not validate PixelJev for high-stakes deployment. Pretraining overlap, benchmark bias, and demographic performance remain unresolved.

Impact Statement

  • The study examines bounded visual decisions on existing research benchmarks, not a validated decisionmaker for medical, legal, safety-critical, or surveillance applications.A valid choice can still be incorrect, and confidence over supplied alternatives does not account for an omitted correct answer.Source: Impact Statement · p. 8
  • The experiments do not resolve pretraining overlap, benchmark bias, or performance across demographic groups.Model and dataset use remains subject to original licenses and access conditions.Source: Impact Statement · p. 8

A.1. Sample Ledger

The sample ledger separates fitting, development, calibration, and test data, while the optimization settings specify which model components are updated and how probe inputs are prepared.

A.1. Sample Ledger

  • 912 or 3,648 fitting images are shared across the adapter’s source tasks, while 684 separate calibration labels are used only for probability scaling.Development labels select checkpoints; the 16-shot fitting set is a strict subset of the 64-shot set, and development, calibration, and test selections remain identical across budgets.Source: §A.1 · p. 10, Table 3 · p. 10
  • Near-similar images are grouped before fitting; training groups touching test data are excluded, but the official test denominator is retained.The grouping is conservative and does not establish that every connected image pair is a true duplicate.Source: §A.1 · p. 10

A.2. Optimization and Scoring Settings

  • Only allowlisted language-side projections are trainable; the original backbone tensors remain frozen, and the three selected 64-shot checkpoints are from epoch 3.The higher-resolution frozen control does not materially improve Pets accuracy.Source: §A.2 · p. 10
  • CLIP and DINOv2 probe features are L2-normalized, with DINOv2 using its final CLS representation; native preprocessing is retained, so inputs need not have equal resolution or crops.Each probe fits task-specific labels, unlike the shared multitask Qwen adapter.Source: §A.2 · pp. 10–11

B. Additional Probability and Budget Results

The section evaluates held-out probability quality and describes additional probability, budget, cost, and VQA checks. Calibration does not change accuracy and is not uniformly beneficial across probability metrics.

B. Additional Probability and Budget Results

  • Calibration does not change accuracy and is not uniformly beneficial: Pets DINOv2 worsens in both NLL and ECE, while CIFAR-10 and EuroSAT ECE worsen despite improved NLL.NLL and ECE are unitless, lower-is-better measures within each dataset; Figure 3 presents calibrated Pets reliability and risk-coverage curves.Source: Table 5 · p. 11, Figure 3 · p. 11
  • Figure 3 evaluates Pets reliability in occupied equal-width confidence bins and selective risk as error among predictions retained in descending confidence order.Methods use separate source-fitted temperatures; connecting segments are guides, not smoothed fits or confidence intervals.Source: Figure 3 · p. 11

C. Direct Multiple-Choice VQA

  • A-OKVQA scoring accepts every slot containing the correct answer text, whereas ScienceQA scores the official answer index.All questions and slots are retained, including repeated correct option text and an empty distractor in some A-OKVQA questions.Source: C. Direct Multiple-Choice VQA · p. 12
  • The frozen CLIP text baseline has no trained multimodal fusion head and is not intended as an upper bound on CLIP-based VQA.Its fixed text window preserves complete candidate answers, but 424 ScienceQA contexts are shortened.Source: C. Direct Multiple-Choice VQA · p. 12
  • Ordinary DINOv2 cannot directly match arbitrary text choices because it has no aligned text encoder; text-aligned variants were not evaluated.ScienceQA’s image-similarity bootstrap uses 1,016 groups for 1,799 distinct pixel images.Source: C. Direct Multiple-Choice VQA · p. 12

D. Instrument Checks and Readout Semantics

Instrument checks test whether candidate readout behaves as intended, and a matched follow-up compares adaptation and readout paths. The appendix outlines further experiments to address transfer, evidence sensitivity, and deployment boundaries.

D. Instrument Checks and Readout Semantics

  • The adapted-interface diagnostic reached 97.30% accuracy; reversing candidates changed some decisions, while substituting different-class images reduced accuracy to zero.An alternate instruction preserved every decision, and direct and constrained one-token candidate logits matched exactly.Source: D. Instrument Checks and Readout… · p. 12

E. Staged Research Milestones

  • The appendix proposes five staged follow-ups rather than completed experiments.They extend matched interface controls, isolate schema and visual-evidence sensitivity, test held-out decision families, connect reliability to actions, and evaluate one bounded deployment workload.Source: E. Staged Research Milestones · pp. 12–13

F.1. What the Factorial Does and Does Not Identify

  • The factorial estimates applying the existing adapter under either readout and changing readout at fixed weights, but not matched candidate-conditioned versus full-vocabulary training.It also cannot attribute all Jev-style effects to one algorithm or extrapolate the selected adapter to training-seed uncertainty.Source: §F.1 · p. 14

F.2. Development-Only Format Selection

  • The format cue was selected on eight preselected Pets development images using validity and token-count criteria, not correctness.Both the system cue and assistant prefill met the criterion, so the predeclared system-first rule determined the choice.Source: §F.2 · p. 14

F.3. Unconstrained Generation and Parsing

  • 55 adapted-ScienceQA disagreements were exactly invalid generated responses, while every valid generated choice matched the direct choice.The invalid outputs comprised 52 out-of-range letters and three non-letter formats; direct inference answered 38 invalid cases correctly.Source: §F.3 · p. 15

F.4. Matched Warm Timing

  • Warm request times include image loading through answer parsing or candidate-probability construction, but exclude model loading, network, and serving queues.Measurements are implementation-specific on the pinned shared runtime, not exclusive-device service latencies or an optimized generation-engine benchmark.Source: §F.4 · p. 15, Table 11 · p. 15

Related papers

Loading 2609.29283v1…