PaperlayerView exact revision on arXiv

Source-linked AI summary

FastJEV: Understanding Redundancy for Compact JEV Inference

Jie Ma, Jie Gao, Yihang Liu, Zhike Qiu, Junle Li, Chongyi Zhuang, Jiayi Ji, Xiaoshuai Sun

arXiv:2610.11379v1cs.CV

TL;DR

JEV directly scores candidate choices, but candidate paths can still repeat histories, duplicate inference states, and execute the full backbone. FastJEV combines context anchoring, prefix sharing, and decision guided pruning to organize compact candidate evaluation. At selected pruning budgets, it reduces candidate depth while retaining most original task scores on average across six evaluation sets.

  • Problem

    Although JEV encodes the common context once, candidate paths can still repeat token histories, duplicate inference states, and execute the full backbone.

    Source: §1 · p. 1

  • Method

    FastJEV shares recurrent context states and matching candidate prefixes, then prunes layers ranked by their effect on relative candidate scores using a small unlabeled calibration set.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    At selected pruning budgets, the complete method reduces candidate depth by 43.75% to 45.83% and retains 93.66% to 97.52% of original task scores on average across six evaluation sets.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    The execution benefit of history sharing depends on candidate overlap and branching structure, and shared context anchoring with pruning improves efficiency when candidate suffixes are evaluated in independent batches.

    Source: §5 · p. 13

    Sharing is valid when visual and token histories, positions, and masks are identical, while finite precision can introduce differences when batch shapes change.

    Source: §3.2 · p. 4, §3.3 · p. 5

Abstract

from arXiv · show

JEV models make multimodal decisions by directly scoring candidates. Although the common context is encoded once, candidate evaluation can still repeat matching token histories, duplicate inference states, and execute the full backbone. In this paper, we study these sources of redundancy and present FastJEV for compact candidate evaluation. We jointly organize history reuse and state storage, since sharing computation requires preserving states for later branches. We first introduce shared context anchoring to reuse recurrent initial states and omit unused final recurrent caches. We extend this reuse through candidate prefix sharing, retaining the intermediate states needed by subsequent branches. To further reduce the depth of these paths, we apply decision guided pruning based on relative score changes measured on a small unlabeled set. Our method retains full context encoding and all candidates without additional training. We evaluate FastJEV across three OmniJev model sizes on five public benchmarks and reconstructed LIBERO-10 offline questions. At the selected pruning budgets, the complete method reduces candidate depth by 43.75% to 45.83%, while retaining 93.66% to 97.52% of the original task scores on average across the six evaluation sets. Through controlled experiments, we show how candidate overlap and branching structure affect the execution cost of history reuse. In our implementation, candidate prefix sharing can reduce repeated computation while increasing latency. These findings motivate designing sharing granularity and execution schedules together for efficient JEV inference.

1 Introduction

JEV directly scores candidate choices, but candidate paths can still repeat histories, duplicate states, and execute the full backbone. FastJEV organizes history reuse, state storage, and pruning to make candidate evaluation more compact while preserving the context and candidates.

1 Introduction

  • Candidate paths can repeat matching histories, duplicate inference states, and execute the full backbone even when the common context is encoded once.Reusing histories removes repeated computation but requires retaining states for later branches; pruning changes computation on shared paths and their continuations.Source: §1 · pp. 1–2
  • At selected pruning budgets, FastJEV reduces candidate depth by 43.75% to 45.83% and retains 93.66% to 97.52% of original task scores on average across six evaluation sets.The evaluation covers three OmniJev model sizes, five public benchmarks, and reconstructed LIBERO-10 offline questions.Source: §1 · p. 2
  • Controlled experiments show that execution cost depends on candidate overlap and branching structure.The paper frames compact evaluation as organizing shared computation and selective reduction within direct candidate scoring.Source: §1 · pp. 1–2
  • FastJEV combines shared context anchoring, candidate prefix sharing, and decision guided pruning while retaining full context encoding and all candidates without additional training.The study connects repeated token histories, inference state storage, and execution depth.Source: §1 · p. 2

2 Related Work

Related work covers efficient multimodal inference, prefix and state sharing, and structured pruning. FastJEV studies candidate evaluation after context encoding, joining history reuse and state management with pruning guided by relative candidate scores.

2.1 Efficient Multimodal Inference

  • FastJEV focuses on computation for comparing candidates after a common context is encoded, retaining full context processing and all candidates while organizing remaining computation through history sharing, state reuse, and depth reduction.The cited literature also considers visual-token pruning and selective visual-attention skipping in generative vision-language models.Source: §2.1 · p. 2

2.2 Prefix and State Sharing

  • FastJEV extends prefix reuse to candidate scoring by sharing a recurrent context anchor and a trie of matching suffix histories, retaining continuation states while omitting unused final recurrent caches.Prior work includes block-based shared cache storage, tree-based prefix indexing, and recurrent-state management.Source: §2.2 · pp. 2–3

2.3 Structured Pruning

  • FastJEV ranks layers using relative candidate scores on a small unlabeled calibration set, then bypasses selected complete blocks during candidate evaluation without additional training.The same selected layers apply to shared histories and their continuations.Source: §2.3 · p. 3

3 Method

FastJEV organizes candidate evaluation around a fully encoded context, combining state reuse and matching-history reuse with selective reduction of candidate execution depth. Its design preserves the original candidate readout while retaining the states needed for later branches.

3 Method

  • FastJEV combines shared context anchoring, candidate prefix sharing, and decision guided pruning to reduce redundant work while retaining full context encoding.The three mechanisms target state storage, repeated matching histories, and candidate execution depth, respectively.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Preliminaries on JEV

  • The shared prefix is encoded once and candidate suffixes are evaluated from its cache, but branches can still duplicate recurrent states, matching histories, and full-depth computation.The decision head combines question and candidate representations with language-model features to produce scores.Source: §3.1 · pp. 3–4

3.2 Shared Context Anchoring

  • Candidate branches share recurrent initial states through a read-only buffer, while branches update separate working states and paths without continuations omit final recurrent-cache storage.Intermediate caches needed for continuation are retained, and separate storage remains for convolution states and attention KV caches.Source: §3.2 · p. 4

3.3 Candidate Prefix Sharing

  • A prefix tree evaluates matching token histories once, retaining fork caches and mapping merged nodes back to every candidate’s original readout entry.Merging respects token positions and preceding history, so equal tokens after different paths remain separate.Source: §3.3 · p. 5

3.4 Decision Guided Pruning

  • Figure 2 shows low-sensitivity later layers but the highest sensitivity in the final block at every model scale; pruning ranks layers by average relative-score perturbation on a fixed unlabeled calibration set.The selected mask applies to candidate branches, while the shared context continues to use all layers.Source: §3.4 · pp. 5–6, Figure 2 · p. 6

4 Experiments

The experiments evaluate FastJEV’s quality and execution costs across model scales, benchmarks, component combinations, state policies, and pruning budgets. They show that candidate-history sharing can reduce repeated work, but its latency and memory effects depend on workload structure and execution scheduling.

4.1 Experimental Setup

  • The setup uses three OmniJev v1.1 sizes, five public benchmarks, and reconstructed LIBERO questions; pruning masks come from sensitivity rankings calibrated on 32 OK-VQA inputs without target labels.Accuracy is reported on complete evaluation splits, and mean retention is the unweighted mean of six ratios to matched Original accuracy.Source: §4.1 · p. 7

4.2 Compact Candidate Evaluation across Tasks

  • At 45.83% candidate-layer pruning, mean retention is 97.52% for 0.8B and 94.92% for 2B; at 43.75% pruning, 4B retains 93.66%.These are the largest tested budgets meeting the mean-retention criterion for the respective scales.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Component Interaction and Efficient Execution

  • S+P is faster than Original in all 18 scale-by-dataset comparisons, while full T+S+P is faster only on POPE at the default budget.With T and P fixed, S lowers peak memory by 104.8 MiB on AI2D and 224.6 MiB on LIBERO without changing accuracy.Source: §4.3 · pp. 8–9, §4.2 · p. 8

4.4 State Storage within the Shared Structure

  • With T and six-layer P fixed, Share/Drop lowers peak live state storage in all four cases; on LIBERO A it falls from 544.1 MiB to 253.7 MiB.Across 16 runs, candidate choices and probabilities match the complete method; sharing initial states and omitting leaf caches reduce allocations.Source: §4.4 · pp. 9–10

4.5 History Reuse and Execution Structure

  • Counting logical candidate token occurrences before padding with the 4B model, complete-set counts fell 26.16%–34.88% on four datasets and 62.84% on LIBERO; POPE had no mergeable candidate histories.On LIBERO, history merging required an average of 18.73 candidate segment calls per frame; profiling three frames showed 17–19 candidate backbone calls for T+S+P, versus two for S+P.Source: §4.5 · p. 10
  • With 64 candidates and 75% shared content, T+S+P is 2.736× faster than S+P; with matched unique tokens, a two-level trie is slower than a flat trie in 22 of 24 pairs.Without shared content, peak memory still falls 49.62%, while only 1.54% of logical token occurrences merge, reflecting the combined trie and readout schedule.Source: §4.5 · p. 11

4.6 Reducing the Depth of Shared Paths

  • At fourteen skipped blocks, 4B mean retention is 93.66% for T+S+P and 93.87% for S+P, with speedups of 0.953× and 1.236× over Original; output sensitivity is competitive but task rankings differ.Pruning response varies by task and model scale; at ten skipped blocks, 4B retains at least 94.41% on every dataset, while 0.8B and 2B exceed 90% on all datasets at eleven blocks.Source: §4.6 · pp. 11–12

4.7 Discussion

  • Trie sharing can increase latency because segmentation, traversal, kernel launches, and state handling add costs, while regular batched suffix execution preserves batching opportunities.The discussion argues that useful sharing granularity depends on branching and remaining computation after pruning, motivating evaluation of prefix sharing together with execution schedules.Source: §4.7 · p. 12

5 Conclusion

FastJEV combines shared context anchoring and candidate prefix sharing to organize histories and state storage, then uses decision guided pruning to reduce candidate depth. Across six evaluation sets, the complete method removes about 44% to 46% of candidate layers while retaining over 93% of original task scores on average.

5 Conclusion

  • Shared context anchoring and candidate prefix sharing organize recurrent state storage with matching token histories.Decision guided pruning reduces candidate depth according to its effect on final scores.Source: §5 · p. 12
  • About 44% to 46% of candidate layers are removed while retaining over 93% of original task scores on average across six evaluation sets.The method retains full context encoding and all candidates without additional training.Source: §5 · p. 13
  • Controlled experiments show that the complete structure reduces repeated histories and state storage, while history-sharing benefits depend on candidate overlap and branching structure.Combining shared context anchoring with decision guided pruning improves efficiency when candidate suffixes are evaluated independently in batches.Source: §5 · p. 13

Supplementary Material

The supplement provides complete experimental results and extended pruning analyses for FastJEV, with experiments using OmniJev v1.1 and LIBERO results drawn from the reconstructed offline question set described in the main paper.

Supplementary Material

  • The supplement presents complete experimental results and extended pruning analyses, using OmniJev v1.1 and the main paper’s reconstructed LIBERO offline question set.All supplement experiments use OmniJev v1.1.Source: Supplementary Material · p. 16

A Additional Experimental Results

Across 18 comparisons, the complete method lowers total peak memory in six, is faster than Original on POPE at all three model scales, and is slower on the other five datasets. The reported comparison costs also combine history merging with execution scheduling choices that are not individually isolated.

A Additional Experimental Results

  • T+S+P versus S+P cost differences combine history merging, segment scheduling, and vocabulary projection scheduling, whose individual contributions are not isolated.Table 7 reports quality, latency, and peak GPU memory for the six-block evaluation across three model scales and six datasets.Source: A Additional Experimental Results · p. 16
  • 6 of 18 comparisons show lower total peak memory; the complete method is faster than Original on POPE at all three scales and slower on the other five datasets.POPE has no candidate histories mergeable under this protocol, so its speedup reflects other candidate-execution differences.Source: A Additional Experimental Results · p. 16
  • The complete method lowers total peak memory in six of 18 comparisons; on LIBERO, reductions of 15.70%, 8.43%, and 9.12% occur alongside higher latency.Source: A Additional Experimental Results · p. 16

A.1 Independent Timing Runs

Independent timing runs report consistent latency ordering across three processes for AI2D and LIBERO, alongside substantially more kernel launches for the trie path on LIBERO. Additional decision-quality measures report POPE F1 scores and episode-level bootstrap intervals for reconstructed LIBERO-10 QA.

A.1 Independent Timing Runs

  • On AI2D, latency is 144.98 ± 0.56 ms for Original, 124.42 ± 0.26 ms for S+P, and 161.56 ± 1.59 ms for T+S+P; on LIBERO, the values are 336.27 ± 2.45, 295.09 ± 2.16, and 723.27 ± 12.29 ms per frame.Three independent processes with fixed inputs and configurations preserve the same latency ordering on both datasets.Source: §A.1 · p. 16
  • Three 4B LIBERO frames have 4,963 to 5,027 candidate-computation kernel launches for S+P and 41,647 to 47,271 for T+S+P.These counts expose the trie path’s finer execution granularity.Source: §A.1 · p. 16

A.2 Additional Decision Quality Measures

  • On reconstructed LIBERO-10 QA, accuracy changes for 2B and 4B are −2.66 and −1.46 percentage points, with 95% confidence intervals of [−4.12, −1.27] and [−2.76, −0.07], respectively.The intervals use 2,000 paired bootstrap samples over 20 evaluation episodes, with episodes as the resampling unit.Source: §A.2 · p. 16
  • At the default six-block budget, POPE F1 is 86.05% and 88.29% for Original and T+S+P at 0.8B, 89.00% and 89.00% at 2B, and 90.52% and 90.50% at 4B.Source: §A.2 · p. 16

A.3 Understanding Compact Candidate Evaluation

FastJEV’s history sharing, state management, and pruning interact with execution granularity: shared work does not guarantee lower latency. State storage and execution scheduling also affect the measured cost.

Shared histories and execution granularity

  • In 22 of 24 matched-unique-token pairs, the two-level trie is slower than the flat trie, showing token reuse alone does not determine execution cost.Additional segments introduce more calls between shared paths and their continuations.Source: §A.3 · p. 16
  • T organizes matching histories into shared paths, S reduces the state copies needed for continuations, and P shortens path depth.S+P evaluates candidate suffixes directly in batches, providing a complementary view of execution efficiency.Source: §A.3 · p. 16

State storage and total memory

  • For LIBERO A, sharing initial states and omitting unused leaf caches lowers observed live state storage from 544.1 to 253.7 MiB without changing candidate probabilities.This isolates a storage benefit within the same candidate structure; model weights remain resident because context encoding retains the full backbone.Source: §A.3 · p. 17

Takeaway

  • The complete structure jointly reduces repeated histories, state copies, and executed candidate layers, while its latency benefit depends on shared work and execution granularity.Total GPU peak memory also includes weights, activations, attention caches, and readout buffers.Source: §A.3 · p. 17

A.4 Qualitative Comparisons

The qualitative examples show that reduced candidate depth can preserve correct choices even as probability distributions change, while calibration subsets yield stable pruning selections within the tested setup.

A.4 Qualitative Comparisons

  • With candidate depth reduced by 43.75%, diagram and science examples retain correct answers despite changed probabilities, while object-presence and counting examples change from incorrect to correct.In the counting example, the correct answer's probability decreases slightly but its rank rises because the competing answer decreases more.Source: §A.4 · p. 18
  • FastJEV selects the marked faucet handle in ERQA and correctly identifies an object held by the robot gripper in reconstructed LIBERO questions.The comparisons include diagrams, scene images, and robot observations with reduced candidate depth.Source: §A.4 · pp. 17–18

B.1 Stability of Layer Selection

  • For each OmniJev model, the selected pruning blocks remain unchanged across calibration sets of 8, 16, 32, and 64 inputs and three draws, matching the main-experiment masks.The models select different blocks from one another; context encoding retains all blocks.Source: §B.1 · p. 18, Table 8 · p. 18

B.2 Pruning Budgets and Depth Redundancy

The pruning sweeps show substantial candidate-depth reduction while retaining full-depth context encoding, with task-dependent retention and budget selection measured against matched Original references.

Budget selection and reference measurements

  • The initial 4B budget screen tests nine budgets on 128 fixed questions per public dataset and all 188 LIBERO frames, then evaluates the largest budget meeting 90% mean retention and its neighbors on complete splits.Latency and peak memory are measured against Original within the same process for each evaluated budget and dataset.Source: §B.2 · p. 18

Understanding depth redundancy

  • At 11 skipped blocks, all six datasets exceed 90% retention, with minima of 93.45% for 0.8B and 92.61% for 2B.The complete-split scan for these models evaluates 6, 8, 11, and 14 skipped blocks.Source: §B.2 · p. 18

Budget selection and reference measurements

  • Figures 5 and 6 show selected correct FastJEV answers across the evaluation sets, with probability bars not renormalized over the displayed candidates.Figure 6 includes both robot-camera views; the displayed LIBERO ground truth uses the frozen state-derived label.Source: Figure 5 · p. 19, Figure 6 · p. 20

Understanding depth redundancy

  • At roughly 44% to 46% candidate-depth reduction, context encoding remains at full depth; ERQA and LIBERO show larger retention losses than POPE and BLINK.At 14 skipped blocks with 4B, T+S+P and S+P mean retention differs by only 0.21 percentage points.Source: §B.2 · p. 21

C Limitations and Future Work

The paper identifies short-segment scheduling overhead and limits its analysis to OmniJev v1.1, while proposing investigation of token-pruning combinations as future work.

C Limitations and Future Work

  • The current implementation incurs scheduling overhead when shared paths are split into short segments, and the analysis is based on OmniJev v1.1.Further work is needed to reduce overhead and examine how findings extend to more JEV models.Source: C Limitations and Future Work · p. 21
  • Combining decision-sensitivity pruning of candidate depth with token pruning of sequence length remains future work, with effects on decision quality and execution cost to be evaluated.Source: C Limitations and Future Work · p. 21
Loading 2610.11379v1…