PaperlayerView exact revision on arXiv

Source-linked AI summary

Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

Cong Xu, Ravi Sankar

arXiv:2609.29601v1cs.ROcs.CLcs.CV

TL;DR

End-to-end VLMs couple visual grounding and language reasoning as the language model shrinks. This paper instead serializes a frozen perception stack’s output for a text-only reader; zero-shot, the interface beats same-scale VLMs, while matched supervision brings the interfaces together.

  • Problem

    Prior comparisons between visual input and text descriptions were confounded by privileged scene content and possible answer-string leakage.

    Source: §I · p. 1

  • Method

    A frozen perception stack feeds a deterministic semantic serializer, which presents the perceived state to an unmodified text-only LLM that answers without seeing pixels or visual tokens.

    Source: §I · p. 1

  • Results

    Zero-shot, the serialized interface beats same-scale VLMs at both co-primary scales and holds down to 1.5B; under matched supervision, the interfaces converge without a resolved difference.

    Source: §X · p. 8

  • Takeaways & Limitations

    For closed-set, object-level embodied QA, vision can live outside the reader while its parameters remain available for language, with the zero-shot trade winning between 1.5B and 7B.

    Source: §X · p. 8

    Evidence comes from one domain, with the confirmatory bank temporally held out within the same recordings rather than externally validated.

    Source: §IX · p. 8

Abstract

from arXiv · show

End-to-end vision-language models (VLMs) bind visual competence to the scale of their language model: as the language model shrinks, perception and reasoning degrade together. We study an embodied scene question-answering (QA) interface in which vision never enters the language model. A frozen perception stack detects and ranges objects; a deterministic semantic serializer compiles the perceived state, errors included, into decision-aligned text; an unmodified text-only large language model (LLM) answers. On a visible-scope-matched, occlusion-audited campus-robot benchmark, under a prospectively frozen criterion, the serialized interface, using detectors fine-tuned in-domain within each fold, outperforms a zero-shot VLM whose language model has the same 7B scale (0.7892 vs 0.7462), with a larger margin at 3B (0.7673 vs 0.6913). Preregistered decoupling experiments show the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope. The advantage grows as the reader shrinks to 1.5B and reverses at 0.5B, and a ground-truth oracle locates the reader-capability floor. Under matched task supervision the interfaces converge: a VLM fine-tuned with low-rank adaptation (LoRA) overtakes the zero-shot system but only ties an equally supervised text reader (0.8441 vs 0.8396, no statistically resolved difference), and both routes remain perception-bound. Reported perception parameters are comparable to those of the VLM's vision tower, and total compute is not smaller.

I. INTRODUCTION

The introduction proposes separating visual perception from language-model reading and reports a criterion-frozen comparison favoring the serialized interface. It frames the benchmark and vocabulary assumptions and supplies conventions for interpreting reported numbers.

I. INTRODUCTION

  • End-to-end VLMs couple visual grounding and language reasoning in shared weights, so both degrade as the decoder shrinks—a cost especially relevant to small embodied platforms.The proposed alternative keeps vision outside the language model and uses a deterministic serializer to provide a physically grounded textual state to a text-only reader.Source: §I · p. 1
  • The benchmark addresses privileged scene descriptions and answer-string leakage through visible-scope matching and preregistered probes, while assuming deployment decision vocabularies are known.The contribution list describes the CODa benchmark, two frame-disjoint banks, and frozen paraphrase and novel-judgment probes.Source: §I · pp. 1–2
  • +0.0429 accuracy at 7B and +0.0761 at 3B: the serialized interface exceeds same-size VLMs in the criterion-frozen zero-shot comparison.The contribution summary reports confidence intervals of [+0.0073, +0.0690] and [+0.0403, +0.0996], respectively.Source: §I · p. 2

II. ABBREVIATIONS, TERMINOLOGY, AND NUMERICAL CONVENTIONS

  • Accuracy is the fraction correct on the named bank; paired differences use cluster-bootstrap confidence intervals, and percentage-point differences equal 100 times accuracy differences.Confirmatory accuracies and contrasts are reported to four decimals; model sizes are in billions of parameters.Source: §II · p. 2

III. RELATED WORK

Prior work spans learned visual interfaces, symbolic vision-language pipelines, and embodied scene-text systems. This paper distinguishes its approach through deterministic decision-aligned serialization and comparisons designed to match scope and retain real perception errors.

A. HOW MAINSTREAM MLLMS ENCODE VISION

  • Mainstream multimodal LLMs typically connect a ViT to a language model through learned bridges, leaving the LLM to decode perception from continuous embeddings.The paper contrasts these learned interfaces with deterministic symbolic text supplied to an unmodified LLM.Source: A. HOW MAINSTREAM MLLMS ENCODE VISION · p. 2

B. NEURO-SYMBOLIC AND PROGRAMMATIC VISION

  • Unlike prior neuro-symbolic systems, this work uses deterministic code rather than generated programs and aligns fields to the decision vocabulary.Its comparison also matches scope, includes real perception errors, and uses a frozen criterion.Source: B. NEURO-SYMBOLIC AND PROGRAMMATIC… · pp. 2–3

C. VLMS FOR EMBODIED PLATFORMS; SCENE-AS-TEXT

  • Embodied VLMs and scene-text systems establish feasibility, but prior evaluations leave open whether text wins through format or privileged content.The paper positions its interface-level comparison across wording, reader scale, and supervision against prior- and current-generation baselines.Source: C. VLMS FOR EMBODIED PLATFORMS;… · p. 3

A. DATA AND QUESTION GENERATION

  • CODa supplies campus-robot front-camera images and ego-frame 3D boxes; questions cover five families and only objects in view within 40 m.Ground-truth controls serialize that same object set; nearest-distance is three-band classification, not general metric estimation.Source: A. DATA AND QUESTION GENERATION · p. 3

B. AUDIT GATES

The study gates potential wording and visibility confounds, freezes probe banks before evaluation, and compares pixel and serialized-text arms. An exploratory ground-truth comparison also indicates that serialization format matters when scene content is fixed.

B. AUDIT GATES

  • 0.4142 for the 3B reader and 0.4089 for the 7B reader: the blind arm’s confirmatory accuracy is modestly above the 1/3 chance level.The audit gate checks option-length, gold-position, and length biases before either bank is frozen.Source: B. AUDIT GATES · p. 3
  • The LiDAR occlusion audit affects 2.3% of question-relevant objects; primary contrasts remain essentially unchanged on the 1,298-item occlusion-clean subset.The full-bank numbers remain primary.Source: B. AUDIT GATES · p. 3

C. FROZEN PROBE SETS

  • Two probe banks were frozen before any arm ran: N1 paraphrases stems and options while preserving boundaries and answers, and N2 tests unfamiliar judgment logic.N2 contains 3,974 questions whose judgment boundaries do not appear in serializer vocabularies.Source: C. FROZEN PROBE SETS · p. 4

D. ARMS

  • The pixel arm receives the raw image, while the typed arm gives a text-only Qwen2.5 reader the serialized state; both are tested at 7B and 3B.A development-bank ground-truth-typed oracle and caption and blind controls complete the comparison ladder.Source: D. ARMS · p. 4
  • 0.955 vs 0.857 vs 0.804: at fixed ground-truth content, Flat beats an older serializer and a prose caption on the exploratory development bank.The reported 21-percentage-point oracle margin over the pixel arm is what real perception must earn.Source: D. ARMS · p. 4

V. PERCEPTION-TO-TEXT SYSTEM

The perception stack combines detector outputs, estimates object ranges, and produces a confidence-tagged scene state. Its implementation also includes small-object recovery and limits on accepted detections.

A. PERCEPTION STACK (FRONT STEREO PAIR; FROZEN AT INFERENCE)

  • Two YOLO11 detectors are fine-tuned per fold on training sequences and ensembled, with all vision components frozen at inference.Slicing-aided tiling recovers small objects; the open-vocabulary detector is restricted to low-confidence absence recovery after direct fusion produced phantom objects.Source: A. PERCEPTION STACK (FRONT STEREO… · p. 4
  • Four range estimates—ground-plane geometry, stereo disparity, class-height prior, and monocular metric depth—are fused by their median.The median limits the influence of any single-cue failure, though a gross outlier can still shift the mid-pair mean.Source: A. PERCEPTION STACK (FRONT STEREO… · p. 4
  • Detections are deduplicated only when class, bearing, and range meet the stated agreement conditions; objects beyond 42 m are dropped.The resulting perceived state records class, bearing, range, and a confidence tag, with no component retrained between system versions.Source: A. PERCEPTION STACK (FRONT STEREO… · p. 4

B. SEMANTIC SERIALIZER: COMPUTE WHAT CAN BE COMPUTED

The serializer precomputes decision-aligned fields from perceived object states and leaves ambiguous judgments to the text-only reader. The reader and confirmatory protocol are fixed, with evaluation organized to limit sequence leakage and instrument faults.

B. SEMANTIC SERIALIZER: COMPUTE WHAT CAN BE COMPUTED

  • The serializer computes distance bands and bearing sectors from fused range and bearing, while leaving genuinely ambiguous aggregation to the reader.Borderline flags use a width derived from the measured per-band range-error distribution; selections include a runner-up so the reader can assess when a selection is wrong.Source: B. SEMANTIC SERIALIZER: COMPUTE WHAT… · p. 4
  • Scoped serialization supplies family-specific streams, reducing the median length from 1842 to about 720 characters for four families, while counting retains per-object rows at about 1860 characters.The question family may guide the stream, but the serializer does not read the answer or option set.Source: B. SEMANTIC SERIALIZER: COMPUTE WHAT… · pp. 4–5

C. READER

  • The frozen, unmodified Qwen2.5-7B text reader uses a fixed answer-format prompt, with no fine-tuning, few-shot examples, or logit surgery.A different-family reader at the same scale is evaluated separately.Source: C. READER · p. 5

VI. PROSPECTIVELY FROZEN PROTOCOL

  • Both 7B and 3B were prespecified as co-primary endpoints, and the headline claim requires both to pass the intersection–union test.The criterion was frozen before confirmatory runs; the confirmatory bank was generated afterward, and no version, threshold, or prompt was changed after its generation.Source: §VI · p. 5
  • Four sequence-rotating folds ensure each question is answered by a detector and calibration fit without its recording sequence; all system numbers are out-of-fold.Cluster resampling respects near-duplicates within recordings, and pairing removes between-scene variance.Source: §VI · p. 5
  • Repeated runs bound same-machine variation at ±0.0015, while instrument checks caught serving faults that otherwise would have inflated the 3B pixel-arm effect to +0.601.Long contiguous runs of unparsed outputs indicate serving-state faults rather than format non-compliance.Source: §VI · p. 5

A. MAIN RESULT: THE ZERO-SHOT REGIME

On the confirmatory benchmark, the serialized interface outperforms the same-scale zero-shot VLM at both primary reader sizes. Probes indicate that the advantage survives paraphrase, is concentrated in decision-aligned computation, and is bounded by the decision vocabulary.

A. MAIN RESULT: THE ZERO-SHOT REGIME

  • 0.7892 vs 0.7462 at 7B and 0.7673 vs 0.6913 at 3B: Scoped beats pixel on the confirmatory bank under the frozen criterion.The corresponding paired differences are +0.0429 (95% CI [+0.0073, +0.0690]) and +0.0761 (95% CI [+0.0403, +0.0996]).Source: A. MAIN RESULT: THE ZERO-SHOT REGIME · p. 5
  • Nearest-distance gains are +0.2362 at 3B and +0.1890 at 7B; counting gains +0.1358 at both sizes, while nearest-class is a resolved 7B loss of −0.0617.Direction and path-object contrasts are unresolved; the gain is concentrated in the families reducible to deterministic arithmetic over the state.Source: A. MAIN RESULT: THE ZERO-SHOT REGIME · p. 6
  • Qwen3-4B reading serialized state beats Qwen3-VL-4B by +0.0595, while the 8B contrast is unresolved at +0.0233 (95% CI [−0.0246, +0.0553]).The result replicates under identical decoding on a current-generation model family.Source: A. MAIN RESULT: THE ZERO-SHOT REGIME · p. 6
  • Controls locate the gain in the serialization layer: the same VL weights gain +0.0377 at 7B and +0.0949 at 3B when reading serialized text instead of their image.A decoder swap on identical text is null, and an image containing object IDs, classes, and ranges still leaves the VLM 0.124 below those facts as text.Source: A. MAIN RESULT: THE ZERO-SHOT REGIME · p. 6

B. WHAT THE ADVANTAGE IS MADE OF

  • On N1, after every stem and option was reworded, the advantage persisted at +0.0414 at 3B (attenuation 0.456) and +0.0738 at 7B, with all 21 leave-one-sequence-out refits significant.A wording-identity account predicts collapse, but it did not occur.Source: B. WHAT THE ADVANTAGE IS MADE OF · p. 6
  • Replacing aligned band strings with raw numeric ranges removes most of the advantage: the band vocabulary alone is worth +0.0542/+0.0354, or 71%/83% of the total.Counting retains gains of +0.1424/+0.1325; misaligned discretization also helps a 3B reader by +0.0158 versus raw ranges.Source: B. WHAT THE ADVANTAGE IS MADE OF · p. 6
  • On questions with novel judgment logic, the aligned serializer retains a 7B advantage of +0.0413 but has no resolved 3B advantage (+0.0118); the numeric variant loses at 3B (−0.0473).This bounds the task-aware interface to decision vocabularies known at deployment.Source: B. WHAT THE ADVANTAGE IS MADE OF · p. 6

C. READER SCALE: ADVANTAGE, FLOOR, AND CEILING

The reader-scale sweep identifies a capability floor at 0.5B, while matched supervision removes the accuracy distinction between the text and vision interfaces. Both routes remain constrained by perception.

C. READER SCALE: ADVANTAGE, FLOOR, AND CEILING

D. THE SUPERVISION REGIME

  • 0.8441 (FT-3B) and 0.8577 (FT-7B) exceed the zero-shot serialized system at both sizes; the scale step is small and unresolved.The FT-3B and FT-7B results are out-of-fold confirmatory-bank accuracies.Source: D. THE SUPERVISION REGIME · p. 7, Table 5 · p. 7
  • 0.8396 vs 0.8441: FT-Text and FT-VLM show no statistically resolved accuracy difference under matched supervision.FT-Text − FT-VLM = −0.0045, with 95% CI [−0.0368, +0.0177].Source: D. THE SUPERVISION REGIME · p. 7, Table 5 · p. 7
  • Counting remains the weakest family for every route, while both interfaces remain perception-bound.The 7B ground-truth oracle scores 0.9699 versus 0.8577 for FT-7B; compiled counting reaches the reported family level with zero gradient samples.Source: D. THE SUPERVISION REGIME · p. 7

E. WHAT THE READER BUYS

The reader’s advantage over fixed option-aware scorers persists on reworded and novel judgments, while measured compute is redistributed rather than reduced. Error attribution identifies reader mistakes on correctly perceived states as a basis for compiling decision-relevant quantities.

E. WHAT THE READER BUYS

  • The option-aware scorer reaches 0.8245 on the confirmatory bank, above the deterministic option-blind answerer at 0.619 and below the zero-shot 7B reader at 0.7892.The comparison shows the LLM is not the accuracy ceiling on fixed families.Source: E. WHAT THE READER BUYS · p. 7
  • 0.3479: all three option-aware learned baselines fall to chance on paraphrased N1 when evaluated without refitting.On N2 they score 0.3183 to 0.3510; the zero-shot 7B reader reaches 0.5860 on N2.Source: E. WHAT THE READER BUYS · p. 7, Table 6 · p. 7

F. COMPUTE

  • Total compute is not smaller: the ≈0.63B-parameter perception stack adds ≈0.95 s per frame, while reader latency is lower at the query stage.For ten questions per frame, the text readers take 2.79–3.27 s versus 3.70 s for the VLM with its visual prefix cached.Source: F. COMPUTE · p. 7

VIII. ERROR ATTRIBUTION: THE METHOD BEHIND THE SERIALIZER

  • The error ledger separates state-rule disagreement, reader error, and reader repairs, while cautioning that state-rule disagreement is not a causal perception error.The reader repairs 310 items where the option-blind rule fails on the perceived state.Source: §VIII · p. 7
  • Bearing calibration caused only 1 sector flip in 309; losses instead included 43 undetected classes and 39 ordering errors, with 54 of 98 nearest-distance errors attributable to the reader.The operator ladder supports a single-operator effect of +0.258∗.Source: §VIII · pp. 7–8

IX. DISCUSSION AND LIMITATIONS

The discussion distinguishes zero-shot, supervised, and ceiling regimes: zero-shot serialization wins within the tested range, while matched supervision brings the interfaces to a tie. The conclusion limits the practical claim to closed-set, object-level embodied QA.

IX. DISCUSSION AND LIMITATIONS

  • Under matched supervision, the tuned VLM overtakes the zero-shot system, but an equally supervised text reader ties it; both routes remain perception-bound.A 3B reader on ground-truth states scores 0.9247, exceeding a 7B reader on real states at 0.7892.Source: §IX · p. 8, §X · p. 8
  • The zero-shot advantage is strongest at smaller tested scales, but narrows with VLM generation: the 4B margin is +0.0595∗ and the 8B comparison is unresolved.Auditability, a swappable perception stack, and a language-only reader do not depend on that margin.Source: §IX · p. 8

X. CONCLUSION

  • For closed-set, object-level embodied QA, vision can sit outside the reader; between 1.5B and 7B, the zero-shot trade wins outright.Under matched supervision, the interface choice instead concerns engineering trade-offs such as auditability, amortized latency, and single-question latency.Source: §IX · p. 8, §X · p. 8

IX. DISCUSSION AND LIMITATIONS

  • The evidence covers one domain and a temporal holdout within the same recordings, not external validation.Open-ended queries remain future work beyond the known-vocabulary regime.Source: §IX · p. 8

APPENDIX A ARM GLOSSARY

The appendix reports per-family contrasts and blind-arm results, then illustrates serialized states containing missed detections and incorrect fields. It also identifies how counting items expose absent queried classes.

APPENDIX A ARM GLOSSARY

  • Table 7 collects the evaluation-arm definitions; all arms decode greedily at T=0 on the forced three-choice track unless noted otherwise.The amendment ledger records each run’s full configuration.Source: Appendix A · p. 8

APPENDIX B PER-FAMILY CONTRASTS AND BLIND-ARM PRIOR

  • Distance-band and counting show resolved gains, nearest-class a resolved loss at 7B, and direction and path-object remain unresolved at both scales.The contrasts use the same 21-cluster bootstrap as the primary endpoints.Source: Appendix B · p. 8
  • No question family’s blind-arm accuracy reaches the weakest scene-conditioned arm, bounding what wording alone can achieve after the audit gate.Table 9 gives the blind-arm breakdown by family.Source: Appendix B · p. 9, Table 9 · p. 9

APPENDIX C A SERIALIZED STATE, VERBATIM

  • A nearest-Railing question has no Railing in the perceived state, so the reader must answer despite a real detection miss.The example is presented as part of the paper’s “errors included” evaluation regime.Source: Appendix C · p. 9
  • A second Railing example has a present field that says “on the left side” when the gold answer is “in the center”; direction-field accuracy is 0.553.The disagreement reflects instance selection and the serializer’s bearing rule.Source: Appendix C · p. 9
  • A counting example asks about a Trash Can absent from both the count header and object rows, while only counting receives per-object rows.The example illustrates the scoping principle for serialization.Source: Appendix C · p. 9

APPENDIX D PER-MODULE COMPUTE

The appendix reports measured compute for the perception stack and readers, audits confirmatory-bank composition and frame separation, and describes released materials for reproducing the analyses.

APPENDIX D PER-MODULE COMPUTE

  • ≈0.95 s is the conservative serial sum of perception-stack module latencies, with peak per-module GPU memory of 4.2 GiB.Stereo runs on the CPU and can overlap GPU modules; reader and VLM per-question latencies were also measured.Source: Appendix D · p. 10

APPENDIX E BANK COMPOSITION AND SEPARATION AUDIT

  • The confirmatory bank contains 1,328 questions, nearly balanced across five families, with one question per frame and 21 recording-sequence bootstrap clusters.Sequences range from 1 to 254 questions (median 43); every confirmatory frame is at least 1.5 s from every development frame.Source: Appendix E · p. 10, Table 12 · p. 10

DATA AND CODE AVAILABILITY

  • Per-item logs and analysis scripts are released, and every reported statistic can be recomputed; CODa images are not redistributed.Trained detector weights are available on reasonable request.Source: DATA AND CODE AVAILABILITY · p. 10

Related papers

Loading 2609.29601v1…