Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

Haiming Li; Yingsheng Liu; Jingmin Zhu; Siyuan Yan; Xieji Li; Jiajun Sun; Zhen Yu; Zongyuan Ge

Source-linked AI summary

Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs

Haiming Li, Yingsheng Liu, Jingmin Zhu, Siyuan Yan, Xieji Li, Jiajun Sun, Zhen Yu, Zongyuan Ge

arXiv:2608.20999v1Aug 21, 2026cs.CV

TL;DR

The paper asks whether MLLMs convert internally available ordinal evidence into appropriately ordered digit-token outputs. It introduces frozen-backbone Ordinal Lens Alignment, which uses lightweight hidden-state lenses and online digit-logit correction. OLA outperforms LoRA-tuned OrderChain in most settings and improves over an offline lens in every setting.

Problem
Native digit-token generation may expose internally recoverable ordinal evidence weakly, leaving a mismatch between hidden-state evidence and ordered outputs.
Method
OLA trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses their predictions, and corrects only digit-token logits during generation.
Results
OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings and improves over OLA-Offline in all 16 settings.
Takeaways & Limitations
Target-restricted online alignment connects latent ordinal evidence to the MLLM generation process without modifying the frozen MLLM.
Takeaways & Limitations
The evaluation covers four open-source backbones and excludes closed-source MLLMs because OLA requires hidden-state and digit-token row-submatrix access.

Abstract

from arXiv · show

Multimodal LLMs apply the language model interface to visual inputs, where ordinal regression tasks such as age estimation, image quality assessment, and disease grading require autoregressive decisions over ordered class labels. We ask whether MLLMs reliably convert internal ordinal evidence into ordered digit-token outputs. Across four ordinal benchmarks and four MLLM backbones, ordinal labels are linearly recoverable from hidden states with Spearman correlation up to 0.938, and a task-designed prompt further sharpens this structure. Yet native digit-token outputs weakly expose it: the unembedding matrix filters the ordinal direction, and the digit-token row space retains below 1.15% across all 16 model-dataset combinations, with a 16 to 77 absolute-point accuracy gap between linear-probe and native outputs. We introduce Ordinal Lens Alignment (OLA), a frozen-backbone inference-time method that trains lightweight W_S-anchored lenses on mid-to-deep decoder layers, fuses them into an ordinal distribution, and corrects only digit-token logits at generation. OLA outperforms the SOTA LoRA-tuned OrderChain baseline in most settings while keeping the MLLM frozen, surpasses discriminative ordinal baselines in most cells, and improves over an offline lens in every setting.

1 Introduction

Ordinal tasks require outputs that respect label distance, but prompted digit-token generation does not guarantee ordinally appropriate decisions. The paper diagnoses this interface mismatch and introduces OLA to reconnect latent ordinal evidence to generation.

Ordinal errors carry different distances, making age estimation, disease grading, image quality assessment, and aesthetic rating distinct from nominal classification.
Digit-token generation preserves the language-model interface but provides no architectural guarantee that emitted tokens respect the underlying ordinal scale.
OLA trains lightweight W_S-anchored lenses on mid-to-deep decoder layers and applies target-restricted corrections to digit-token logits while keeping the backbone components frozen.
SRCC reaches 0.938 while digit-token row-space retention remains below 1.15% across all 16 model-dataset combinations.
OLA improves over prompt-only, hidden-steering, capacity-matched, and offline-lens alternatives, and outperforms LoRA-tuned OrderChain in most settings.

2 Task Setup & Diagnostic Findings

MLLM hidden states contain strong, prompt-enhanced ordinal structure, but native digit-token outputs expose only a small fraction of it. The resulting interface mismatch aligns with large gaps between probe and native prediction accuracy.

Latent ordinal evidence: Every model-dataset cell reaches SRCC ≥0.756, with peaks of 0.938 on Adience, 0.919 on Aesthetic, and 0.843 on HCI.These results use last-token hidden states with task-designed prompts.
Latent ordinal evidence: Across all 12 backbone-dataset combinations, ordered labels are linearly recoverable from frozen hidden states.Thus, the reported downstream errors are not attributable to absent internal ordinal evidence.
Prompt effects: On DR, task-designed prompts raise SRCC from 0.255–0.365 to 0.756–0.758, while HCI rises from 0.551–0.586 to 0.773–0.843.The largest gains occur where neutral probes are weakest.
Output-interface diagnostics: The unembedding spectrum reweights ordinal evidence: the top 10% singular-value bucket contributes 53% of output energy despite containing 13% of raw probe-direction mass.The bottom 25% carries 43% of raw mass but only 14% after σ2 weighting.
Output-interface diagnostics: Digit-token row-space retention averages 1.00%, aligning with 16 to 77 absolute-point accuracy gaps between hidden-state linear readouts and native outputs.Cell means span 0.88%–1.14%, with the largest gaps on DR and HCI.

3 Ordinal Lens Alignment

OLA connects probe-readable ordinal evidence to digit-token generation without changing the frozen MLLM or unembedding head. It learns lightweight layer-wise lenses, fuses their outputs, and applies a confidence-gated correction only to digit-token logits.

Method overview: OLA is a frozen-backbone inference-time procedure whose only learned parameters are lightweight lens and fusion weights.It does not modify the vision encoder, language-model backbone, or unembedding head.
Layer-wise lenses: Forward hooks capture last-token states from K mid-to-deep decoder layers, which WS-anchored lenses map into digit-token scores.The frozen digit-token row submatrix WS anchors each lens readout.
Design rationale: The design separates latent evidence readout, multi-layer aggregation, and output-side alignment without repairing WS or replacing the classifier.This distinguishes OLA from hidden-state steering and LoRA-based MLLM adaptation.
Aggregation: OLA fuses per-layer scores with learned softmax weights into an instance-specific ordinal distribution P OLA.The aggregation adds only K scalar fusion parameters, independent of C and d.
Online alignment: Before greedy decoding, OLA applies a discrepancy-driven, sign-aware correction to digit-token logits using the native distribution and a confidence gate.Non-digit logits, including EOS, remain unchanged, and λ⋆ is selected on the validation split.

4 Experiments

The experiments evaluate OLA across four MLLM backbones and four ordinal benchmarks, testing benchmark performance, component contributions, prompt dependence, and qualitative output structure. OLA consistently improves over native prompting and key baselines, with online target-restricted alignment providing the main ablation gain and residual errors generally preserving ordinal proximity.

4.1 Main Benchmark: OLA improves over Naive Prompt in every model-dataset setting, with the largest gains on DR and HCI.On DR, ACC rises from 0.156 to 0.920 for Qwen2.5-VL, from 0.137 to 0.922 for Gemma-4, and from 0.148 to 0.924 for LLaVA-NeXT; on HCI, it rises from 0.328 to 0.757 for Qwen3-VL and from 0.245 to 0.733 for Gemma-4.
4.1 Main Benchmark: OLA is higher than CAA in every model-dataset setting, supporting direct output-side correction over hidden-state intervention through the native language-model head.CAA injects a single residual-stream direction and relies on the native LM head, whereas OLA uses WS-anchored lenses and directly corrects digit-token logits.
4.1 Main Benchmark: OLA exceeds the LoRA-tuned OrderChain baseline in 14 of 16 model-dataset settings while keeping the vision encoder, language-model backbone, and unembedding matrix frozen.The exceptions are Qwen2.5-VL and Qwen3-VL on Aesthetic, where OrderChain is higher by 0.9 and 1.2 ACC points; OLA also exceeds Ord2Seq in all 16 cells and NumCLIP in 15 of 16 cells.
4.2 Component Ablation: OLA-Online improves over OLA-Offline in all 16 settings by an average of +1.80 ACC, while improving over Capacity Probe by +2.45 and Probe Only by +3.48 on average.OLA-Offline is higher than Capacity Probe in 12 of 16 settings with an average gain of 0.66 ACC points, so offline classification is useful but not the main source of the final improvement.
4.2 Component Ablation: Applying the OLA pipeline trained with the Designed Prompt under the Naive Prompt still substantially exceeds the Naive Prompt baseline across all 16 settings.For example, Qwen3-VL on DR reaches 0.794 versus 0.688, while LLaVA-NeXT on DR reaches 0.684 versus 0.148, without retuning the lens, fusion, or λ⋆.
4.3 Robustness and Qualitative Analyses: K = 6 fused mid-to-deep decoder layers is a stable default rather than a universal optimum, while broader K = 10 windows are less stable.Smaller windows are competitive in isolated cases, but K = 6 is best or close to best in most backbone-dataset curves and is fixed without test-set selection.
4.3 Robustness and Qualitative Analyses: OLA confusion matrices concentrate residual errors among adjacent ordinal classes, whereas Aesthetic remains less diagonal and reflects noisier subjective ratings.Adience shows a strong near-diagonal pattern, and DR and HCI also concentrate errors among nearby classes, consistent with MAE reductions.
4.3 Robustness and Qualitative Analyses: Layer-wise PCA shows class mixing in early layers but smoother class-ordered structure from mid-decoder layers onward.This visualization is consistent with ordinal labels becoming more recoverable from intermediate and deeper hidden states.

5 Conclusion

Across four ordinal benchmarks and four MLLM backbones, latent ordinal evidence is recoverable from frozen hidden states, but digit-token outputs expose less than 1.15% of this energy. OLA bridges this gap through WS-anchored lenses and target-restricted online correction without modifying the frozen MLLM.

Less than 1.15% of latent ordinal energy is exposed by the digit-token row space across four benchmarks and four MLLM backbones.
OLA uses WS-anchored lenses and target-restricted online correction to connect hidden ordinal evidence to digit-token logits.
OLA matches or exceeds LoRA-tuned baselines without modifying the frozen MLLM.

Limitations

The evaluation is scoped to four open-source MLLM backbones and four ordinal benchmarks, with additional constraints from single-digit labels and per-backbone-dataset training. Closed-source models, multi-digit outputs, and cross-dataset lens transfer are not covered.

The experiments cover four open-source backbones and four ordinal benchmarks, excluding closed-source MLLMs because OLA requires hidden-state and digit-token row access.
All benchmarks use single-digit ordinal labels with C ≤8; multi-digit outputs require tokenizer redesign or sequence-level discrepancy correction.
OLA trains lightweight lens and fusion parameters separately for each backbone-dataset pair, so the method is not fully training-free.
Lens transfer across datasets within the same backbone is left to future work.
The appendix describes four benchmarks with dataset-specific ordinal class structures and split protocols.

A.5 Cross-dataset summary

The cross-dataset analysis consolidates dataset protocols and diagnostic procedures, using designed prompts and frozen-state probes to assess ordinal structure across benchmarks. Designed prompts specify ordinal scales, answer formats, and coarse-to-fine reasoning, while probes report layerwise SRCC.

Table 8 summarizes each dataset’s ordinal class count, corpus size, test size, fold count, and split-protocol family.
Diagnostic probes extract hidden states from visual placeholders or assistant-start positions under neutral and task-designed prompts.
Designed Prompt principles: Designed prompts combine an AI role, domain background, explicit task statement, and coarse-to-fine reasoning.
Designed Prompt principles: Coarse-to-fine reasoning first selects a contiguous ordinal superclass and then a fine class within it, using a fixed dataset-specific hierarchy without label leakage.
Prompt comparison: Replacing the Designed Prompt with a bare class-mapping question substantially reduces last-token ordinal SRCC in deep layers.
Diagnostic probe: The per-layer linear probe uses frozen hidden states and reports maximum test SRCC across decoder layers.

C.2 Ordinal lens training

OLA training keeps the MLLM, unembedding matrix, and digit-token submatrix frozen while learning per-layer ordinal lenses and a softmax fusion over cached mid-decoder hidden states.

The MLLM, unembedding WU, and digit-token submatrix WS remain strictly frozen throughout both training stages.
Stage A: per-layer ordinal lens: Stage A independently trains a low-rank per-layer ordinal lens using WS-projected standardized hidden states and cross-entropy.
Stage A: per-layer ordinal lens: Each lens uses Aℓ = I + UℓV⊤ with rank r=64 and identity initialization.
Stage B: softmax fusion: Stage B learns softmax fusion weights over the K frozen lenses by optimizing cross-entropy of their combined ordinal distribution.
Stage B: softmax fusion: The implementation caches OLA ordinal distributions after fusion for training, validation, and test splits.

D Extended Experimental Results

This section presents per-layer SRCC diagnostics across three MLLM backbones, alongside main benchmark and ablation tables with five-fold standard deviations.

Per-layer SRCC is used as the diagnostic across three MLLM backbones.
The section includes the full main benchmark tables.
The ablation tables report five-fold standard deviations.

D.1 Per-layer SRCC across backbones and datasets

Ordinal structure is strongest in upper-middle decoder layers across backbones and datasets, with layer-wise PCA visualizations showing a corresponding ordinal arc in middle-to-late layers.

SRCC peaks in the upper-middle decoder, with Qwen3-VL peaks on Adience and Aesthetic inside OLA’s L = {16, 18, 20, 22, 24, 26} window.
Gemma-4 peaks later than Qwen3-VL, reaching L40 on Adience and Aesthetic and L26 on HCI, while DR plateaus at ρ ≈0.756.
[0.756, 0.938] is the range of best-layer SRCC across all 12 backbone–dataset cells.The best layer is consistently in the upper-middle decoder and never the final layer.
LLaVA-NeXT already shows ρ ≥0.66 at L2 on three datasets and peaks around L16–L30.
Layer-wise 3D PCA projects last-token hidden states onto three principal components and colors points by ordinal class.The resulting figures show an ordinal arc emerging in middle-to-late decoder layers, consistent with per-layer SRCC peaks.

D.3 Main results with five-fold standard deviations

The main benchmark reports results on four ordinal benchmarks under four MLLM backbones, with uncertainty summarized by fold-based standard deviations where applicable.

Mean ± std is reported over five folds for Adience, DR, and HCI.
Aesthetic is evaluated using one class-balanced fold.
Four ordinal benchmarks are evaluated under four MLLM backbones.

D.4 Ablation studies with five-fold standard deviations

The ablation studies compare eight methods across four backbones and four datasets, while the OLA algorithms define frozen-backbone lens learning, layer fusion, and inference-time digit-logit correction.

Ablation design: Eight methods are compared in each backbone–dataset cell, including OLA-Offline and OLA-Online.The comparison also includes Capacity Probe, Probe Only, Probe Steering, Single-Layer Logit, Designed Prompt, and OL + Naive.
Lens learning: OLA learns per-layer ordinal lenses anchored on the frozen digit-row matrix W_S, then fuses their logits with learned softmax weights.
Inference-time alignment: The inference phase selects λ⋆ on validation data and corrects only candidate digit-token logits during deterministic generation.Correction is applied at the answer slot before greedy decoding continues to EOS.
Relation to prior methods: OLA differs from LoRA-based adaptation by leaving the MLLM backbone frozen and training only output-side components.
Diagnostic framing: The diagnostic evaluates how much ordinal information frozen hidden representations linearly carry and examines how the native output head exposes it.