PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Amortization of Full-Budget Counterfactual Reasoning for Visual Token Communication

Qinglei Qi, Zhihe Liang, Fengzhan Jing, Shenao Zhu, Lei Zhang, Chenyang Zhang, Shuqing He, Jia Guo

arXiv:2609.30756v1cs.AI

TL;DR

In generative image communication, token value depends on the completed packet and receiver reconstruction, while exact counterfactual evaluation requires repeated reconstruction. ACV-Gate learns terminal candidate rankings and selectively evaluates bounded candidate sets. On CIFAR-10 at 0.20 bpp, its primary adaptive configuration improves PSNR over Local-MDL with substantially fewer candidate evaluations.

  • Problem

    A token’s value depends on the complete transmission and receiver reconstruction, but full-budget counterfactual evaluation requires repeated inference and reconstruction for each candidate.

    Source: §I · pp. 1–2

  • Method

    ACV-Gate learns terminal candidate rankings from full-budget evaluations and selectively assigns exact evaluation to bounded candidate sets under a budgeted state-allocation policy.

    Source: §I · p. 2

  • Results

    At 0.20 bpp on CIFAR-10, the primary full-proposal Adaptive configuration improves PSNR over Local-MDL by 0.636 ± 0.119 dB using 2.13 ± 0.53 evaluations per image, or 27.60% of the Exact-Full workload.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    Matched-candidate comparisons and evaluations on STL-10 and 384 × 384 systems support explicit operating points for allocating encoder computation, with the largest gains at low rates.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Generative image communication transmits compact semantic tokens under a limited packet budget, where token selection directly affects the final reconstruction quality after the complete packet is decoded. However, accurately estimating the terminal value of every candidate token requires repeated receiver-side reconstruction, resulting in substantial encoder-side computation. To address this problem, we propose ACV-Gate, an adaptive candidate evaluation framework that learns to approximate full-budget counterfactual evaluation and selectively assigns exact evaluations to the most informative candidates. Specifically, a set-aware student is trained using terminal advantages and regrets to predict candidate rankings directly, while a selective refinement mechanism evaluates only a bounded candidate set containing both Local-MDL and direct actions; cost-based thresholds further enable explicit control of the average evaluation workload. Experiments on CIFAR-10 show that ACV-Gate consistently improves reconstruction quality while substantially reducing candidate evaluations; at 0.20 bpp, the primary adaptive configuration improves PSNR over LocalMDL by 0.636 dB with only 2.13 candidate evaluations per image, corresponding to 27.60% of the calls required by the Exact-Full expert. Matched-candidate comparisons, synchronized GPU measurements, and evaluations on STL-10 and 384 *384 scale transfer further demonstrate consistent quality computation trade-offs, with particularly pronounced gains at low bit rates. These results show that combining terminal-value learning with selective candidate evaluation provides an effective and controllable mechanism for allocating encoder computation in packet-constrained generative image communication.

I. INTRODUCTION

Generative image communication selects visual tokens under a packet budget, but token value depends on the completed transmission and receiver reconstruction. ACV-Gate learns terminal candidate rankings and selectively allocates exact evaluations to control encoder computation.

I. INTRODUCTION

  • Under a fixed packet budget, transmitted tokens should improve the final reconstruction, but their value depends on prior and subsequent tokens and receiver completion.This terminal perspective matters especially at low bit rates, where each token occupies a substantial fraction of the budget.Source: §I · p. 1
  • Attention, saliency, diversity, and coverage estimate token importance, but may not capture how tokens interact with later transmission and receiver completion.For generative communication, selection benefits from a terminal-value criterion.Source: §I · pp. 1–2
  • Full-budget counterfactual evaluation measures candidate value after completing the remaining budget with the same continuation and receiver, but repeated inference and reconstruction impose substantial encoder computation.The transmitter must allocate both communication bits and computation used to decide how those bits are spent.Source: §I · p. 2
  • ACV-Gate learns terminal candidate rankings from full-budget advantages and regrets, enabling direct token selection and ranking alternatives for refinement.Its bounded screen retains the Local-MDL action, the direct student action, and highly ranked alternatives; cost-based state allocation controls when refinement is applied.Source: §I · p. 2
  • The formulation treats serialized packet bits, a per-image cap on exact evaluations, and an average candidate-query budget as coupled resources.Matched-budget comparisons, synchronized runtime measurements, and evaluations across image systems characterize the computation–quality trade-off.Source: §I · p. 2

A. Semantic communication and learned image transmission

For fixed visual representations and receivers, token selection remains a question of estimating each transmitted position’s effect on reconstruction. ACV-Gate applies terminal-value learning and selective computation to that decision while accounting for packet and encoder-computation budgets.

A. Semantic communication and learned image transmission

  • Progressive, content-adaptive, semantic-masking, and sparse-transmission methods address variable communication conditions, while token selection still requires estimating each transmitted position’s effect.For a fixed representation and receiver, ACV-Gate allocates computation used to make that decision.Source: A. Semantic communication and learned… · p. 2

B. Visual token selection and adaptive computation

  • Visual-token selection methods use attention, diversity, saliency, coverage, graph structure, or learned importance, whereas packetized reconstruction requires accounting for subsequent transmission and receiver completion.Adapted DivPrune, VisPruner, and SCOPE rules are evaluated with a common proposal and receiver, while ACV-Gate learns from completed-image quality.Source: B. Visual token selection and adaptive… · pp. 2–3

C. Selective prediction and decision distillation

  • ACV-Gate adds amortization and budgeting to the GCR-C terminal evaluator, learning candidate rankings and allocating exact evaluations to bounded candidate sets.The approach links token selection and encoder computation under a common packet budget and receiver.Source: C. Selective prediction and decision… · p. 3

III. SYSTEM MODEL AND PROBLEM FORMULATION

  • The constrained formulation distinguishes serialized communication bits, a hard per-image exact-evaluation cap, and an average workload budget.The controller decrements the remaining cap after expert calls; threshold calibration sets the average operating point, and experiments report realized evaluations and encoder runtime.Source: §III · p. 3

A. Fixed full-budget evaluator and compact proposal

The method uses full-budget evaluation of a compact candidate proposal to obtain terminal-quality supervision. A set-aware student learns candidate rankings, while state and candidate features support selective refinement.

A. Fixed full-budget evaluator and compact proposal

  • The Exact-Full reference evaluates each candidate in a compact proposal by appending it, completing the packet with Local-MDL, and measuring reconstructed-image PSNR.The proposal deduplicates Top-3 Local, Top-3 selected-set importance, and Top-2 coverage candidates, retaining Local; Exact-Full supplies the optimum for that proposal and continuation.Source: A. Fixed full-budget evaluator and… · p. 3

B. State and candidate representation

  • Candidate representations combine masked-prior state, codebook, position, budget, Local score, entropy, feasibility, proposal source, and relative features.Input statistics come from the current communication state, while terminal advantage and regret provide training targets.Source: B. State and candidate representation · p. 4

C. Set-aware student and learning objective

  • The student jointly processes candidate embeddings with a two-layer Transformer without positional embeddings, so permuting proposal rows permutes output scores while preserving set-level context.Regret-soft targets and a gap-dependent hard-target weight train candidate rankings; the objective combines H2, gain, and safety terms.Source: C. Set-aware student and learning… · p. 4
  • The full configuration trains through hard-CE/pair warm-up, regret-decision training, anchored adaptation, and a final allocation-score phase with the selector frozen.The allocation score estimates direct-action regret and lost positive gain.Source: C. Set-aware student and learning… · pp. 4–5

D. Budgeted selective expert allocation

Budgeted selective allocation uses a learned state score to decide whether to refine a state and a bounded screen to limit exact candidate evaluations. The screen preserves Local and direct actions while respecting the remaining per-image budget.

D. Budgeted selective expert allocation

  • The learned state score and rate-specific threshold determine whether the controller uses the direct action or a screened exact evaluation.Equal-budget experiments compare ACV allocation with student margin, Local uncertainty, and random allocation.Source: D. Budgeted selective expert allocation · p. 5
  • At selected states, the screen prioritizes Local, the direct student action, and remaining candidates by student rank, taking only as many unique entries as the candidate budget and proposal size allow.Refinement requires at least two available candidate slots.Source: D. Budgeted selective expert allocation · p. 5
  • Adaptive value over Local decomposes into headroom recovered by the direct action and additional recoverable value supplied by exact screening.The decomposition distinguishes student-recovered gain from the screen’s added value when the expert branch is evaluated.Source: D. Budgeted selective expert allocation · p. 5
  • Every called screen retains Local and the direct action, yielding a screened value at least as high as either action under the evaluator’s deterministic continuation.Each screened evaluation adds its screen size to the candidate count, preserving the per-image cap by construction.Source: D. Budgeted selective expert allocation · p. 5

E. Training and threshold calibration

Training separates direct-checkpoint selection from allocation-head fitting, while cost-based calibration selects an operating threshold from realized candidate expenditure on monitor images. The fixed threshold transfers to validation, with the per-image cap enforced during execution.

E. Training and threshold calibration

A. Datasets and comparison protocol

The study evaluates shared-tokenizer, receiver, and packet-setting comparisons on CIFAR-10, with separate scale-transfer experiments. It reports reconstruction, workload, and runtime measures using image-level splits and paired bootstrap intervals.

A. Datasets and comparison protocol

  • The CIFAR-10 study uses 200 development and 200 image-disjoint validation images across seven rates, with three selector seeds on the same validation images.Development supports fitting, checkpoint selection, allocation-head fitting, and threshold calibration; validation supplies reported performance.Source: A. Datasets and comparison protocol · p. 6
  • Methods within each image system share the tokenizer, packet syntax, CRC/FEC accounting, receiver, and Local-MDL continuation.Primary end-to-end evaluation uses the full proposal when the adaptive branch is selected; common-screen and allocation comparisons use Cmax = 4.Source: A. Datasets and comparison protocol · p. 6
  • The 384×384 scale-transfer study evaluates all 24 Kodak center crops at two calibrated packet-budget operating points.It uses a frozen LlamaGen VQ-16 tokenizer and decoder, a 24×24 token grid, and a locally trained masked prior.Source: A. Datasets and comparison protocol · p. 6

B. Metrics

  • Reported measures include PSNR, SSIM, actual bpp, candidate-query counts, expert invocations, and synchronized CUDA runtime.Runtime comparisons use synchronized CUDA boundaries; across-seed results are means±standard deviations, and paired image-bootstrap intervals resample image IDs while retaining associated states and rates.Source: B. Metrics · p. 6

VI. RESULTS

Full-budget terminal selection changes candidate rankings and can improve reconstruction while using far fewer exact evaluations than Exact-Full. The clearest gains and reduction in losses occur at low packet rates, where reconstruction headroom is greatest.

A. Terminal horizon changes candidate ranking

B. End-to-end reconstruction quality

  • At 0.20 bpp, Adaptive gains 0.636 ± 0.119 dB with 2.13 ± 0.53 candidate evaluations per image, recovering 46.99% of the Exact-Full expert gain with 27.60% of its candidate calls.This is the primary full-proposal adaptive configuration.Source: B. End-to-end reconstruction quality · p. 6
  • Adaptive recovers 18.22% and 12.18% of expert gain at 0.32 and 0.44 bpp using 16.05% and 8.14% of expert calls, while expert gain falls to 0.060 dB at 0.44 bpp.Adaptive-gain paired image intervals are positive at 0.20 bpp and cross zero at 0.32 and 0.44 bpp.Source: B. End-to-end reconstruction quality · p. 7
  • At 0.20 bpp, images with ΔPSNR below −0.1 dB fall to 23.0% for Adaptive versus 31.5% for Direct; the fractions are 18.0% versus 19.0% at 0.32 bpp and 3.5% versus 4.0% at 0.44 bpp.The largest reduction occurs at 0.20 bpp, alongside the larger mean benefit of refinement.Source: B. End-to-end reconstruction quality · p. 7

C. Common-screen terminal-value comparison

Common-screen tests separate candidate ranking from state allocation: learned rankings preserve more gain than Local-ranked-4, while practical allocation scores leave headroom for better state ranking. Runtime measurements quantify the quality–cost trade-off across methods.

C. Common-screen terminal-value comparison

  • At 0.20 bpp, learned four-call screens achieve 1.077–1.166 dB, compared with 0.271 dB for Local-ranked-4.At this budget, Full, A1 Set-context, Row-valueMSE, and DeepSets-regret-soft reach 1.119, 1.142, 1.077, and 1.166 dB, respectively.Source: C. Common-screen terminal-value… · p. 7
  • At 0.20 bpp, DeepSets-regret-soft reaches 0.874 dB at two calls per image, with its relative ranking versus Full changing across rates.The subsequent allocation experiments use Full, which combines the set-aware student and learned allocation head.Source: C. Common-screen terminal-value… · p. 7

D. Measured candidate expenditure and threshold transfer

  • With a one-call target, threshold transfer yields 0.71, 0.60, and 0.99 evaluations per image and gains of 0.482, 0.072, and 0.014 dB at 0.20, 0.32, and 0.44 bpp.The low-rate interval excludes zero, and every validation image stays within the four-call cap.Source: D. Measured candidate expenditure and… · p. 7

E. Fixed-screen allocation-score controls

  • At 0.20 bpp and one call per image, practical allocation scores yield 0.491–0.544 dB, while the same-screen V4 oracle reaches 1.048 dB.At 0.32 bpp, practical scores yield 0.074–0.118 dB versus 0.437 dB for the oracle; at 0.44 bpp, practical scores remain at or below 0.015 dB.Source: E. Fixed-screen allocation-score… · p. 7

F. Measured runtime–quality trade-off

  • In the unified run at 0.20 bpp, Full Adaptive-4 gains +0.459 dB at 365.1 ms with 0.66 evaluations per image, versus Full Direct’s +0.386 dB at 337.0 ms and zero evaluations.Source: F. Measured runtime–quality trade-off · p. 8
  • At 0.20 bpp, Exact-Full gains +1.353 dB at 830.3 ms, while Local-ranked-4 gains +0.271 dB at 416.1 ms using four calls per image.At routing probability 0.5, B1 reaches +0.680 dB at 482.8 ms with 3.94 calls, while B2 reaches +0.138 dB at 273.8 ms with 2.05 calls.Source: F. Measured runtime–quality trade-off · p. 9

G. Comparison with adapted token-selection methods

At 0.20 bpp, ACV-Gate Adaptive improves on Local with fewer exact candidate evaluations than Exact-Full, outperforming the adapted token-selection rules. A separately trained STL-10 system also shows gains over Local across three budget settings.

G. Comparison with adapted token-selection methods

  • At 0.32 bpp, all three structured adapters are at or below Local; at 0.44 bpp, zero-call adapters remain within 0.006 dB of Local.The reported 0.44 bpp proximity is consistent with small terminal headroom.Source: G. Comparison with adapted… · p. 9
  • At 0.20 bpp, ACV-Gate Adaptive gains +0.636±0.119 dB with 2.13 ± 0.53 calls, while Exact-Full gains 1.353 dB with 7.705 evaluations per image.ACV-Gate Direct gains +0.313±0.088 dB with zero exact candidate evaluations; the three adapted rules gain +0.128, +0.015, and −0.062 dB.Source: G. Comparison with adapted… · p. 9

H. Evaluation on STL-10

  • The separately trained STL-10 system gains +0.442±0.041, +0.140±0.046, and +0.110 ± 0.020 dB at r = 0.20, 0.32, and 0.44, with paired intervals excluding zero.Exact-Full achieves larger gains with 7.78 candidate evaluations per image.Source: H. Evaluation on STL-10 · p. 9

I. High-resolution terminal-value learning and selective computation

ACV-Gate combines terminal-value learning, bounded screening, and budgeted state allocation. The reported evaluations show reconstruction gains at 384×384 and on other datasets, with larger gains at low rates and workload and runtime as deployment considerations.

I. High-resolution terminal-value learning and selective computation

  • At 384×384, Exact-Full improves PSNR over Local by 1.033 dB at 0.0142 bpp and 0.847 dB at approximately 0.0257 bpp, using eight calls per image.Its selection runtime is 8.9 and 8.4 times Local, respectively.Source: §I · p. 9
  • A separately trained 256-source student gains 0.553 dB on Internal32 and 0.520 dB on Tecnick40 at low rates, using 0.90 and 0.98 evaluations per image.Both low-rate image-bootstrap intervals exclude zero; higher-rate gains decrease and vary by dataset.Source: §I · pp. 9–10

VII. DISCUSSION

  • Equation (13) separates gain recovered by student ranking from additional value obtained through candidate screening.The student ranks candidates with zero exact evaluations, while selective refinement evaluates a compact set of alternatives.Source: §VII · p. 10
  • The return on computation depends on transmission budget: gains concentrate at low rates, where receiver generation is greater and selected tokens can substantially affect completion.Larger budgets leave less Exact-Full headroom over Local; the separately trained STL-10 and 384×384 systems also concentrate gains at low rates.Source: §VII · p. 10
  • At common-screen operating points, A1 Set-context and DeepSets-regret-soft match or exceed Full, while practical state-allocation scores achieve 0.491–0.544 dB versus 1.048 dB for the same-screen oracle.The gap indicates potential headroom in learning when exact refinement is most valuable.Source: §VII · p. 10
  • Deployment decisions require both candidate-workload and runtime measurements, since runtime includes feature extraction, student inference, continuation, and reconstruction.The joint frontiers guide computational-budget choices.Source: §VII · p. 10

VIII. CONCLUSION

  • At 0.20 bpp on CIFAR-10, Adaptive improves PSNR over Local-MDL by 0.636 ± 0.119 dB with 2.13 ± 0.53 evaluations per image, or 27.60% of Exact-Full workload.The conclusion also reports that Direct gains 0.313 ± 0.088 dB with zero exact evaluations.Source: §VIII · p. 10

Related papers

Loading 2609.30756v1…