PaperlayerView exact revision on arXiv

Source-linked AI summary

Guess and Send: World-Model-Guided Semantic Packet Transmission for Robust Video Perception

Yan Gong, Zheng Chu, Wei Koong Chai, Hari Mohan Pandey, Jiankang Zhang, Jingrui Hou, Miao Zhang, Yuanjian Li, Pei Xiao

arXiv:2610.29587v1cs.MM

TL;DR

Video links must transmit information the receiver cannot already recover from its evolving scene state. WorldGT predicts and verifies that state, then prioritizes packets by receiver-relative task impact. Experiments report gains on KITTI and VisDrone, including an 11.3-point KITTI J&F improvement at 0 dB Rayleigh SNR over the strongest independent baseline at a 4.13% reference budget.

  • Problem

    For a video receiver with memory, task relevance alone does not determine whether new information needs transmission; packet utility depends on what the receiver already holds and can predict.

    Source: §I · p. 1

  • Method

    WorldGT synchronizes a receiver twin, predicts and verifies cached features, and uses counterfactual task-output differences to guide packet selection and unequal error protection.

    Source: §I · p. 2

  • Results

    WorldGT improves KITTI and VisDrone perception under challenging channels; on KITTI at a 4.13% reference budget, it improves J&F by 11.3 percentage points over the strongest independent baseline at 0 dB Rayleigh SNR.

    Source: B. VisDrone Results · p. 9, §VIII · p. 14

  • Takeaways & Limitations

    Receiver-state-aware predictive transmission is reported as effective for bandwidth-efficient, reliable visual semantic communication, with detection experiments demonstrating transfer beyond segmentation.

    Source: §VIII · p. 14, §VII · p. 14

Abstract

from arXiv · show

Reliable video perception over bandwidth-constrained wireless links requires transmitting information that the receiver cannot already recover from its evolving scene representation. This paper introduces WorldGT, a world-model-guided semantic packet transmission framework that prioritizes information according to its contribution to the receiver's perception. A shared feature-space world model predicts the receiver's scene state through affine motion compensation and learned residual displacement, while sender-side verification identifies reusable predictions. An acknowledgment-synchronized receiver twin estimates task-impact scores through two counterfactual evaluations, guiding packet selection and unequal error protection over LDPC-coded wireless channels. A compact 3-bit feature representation increases spatial coverage under stringent bandwidth constraints, while predictive fallback sustains perception during packet outages. Experiments on KITTI and VisDrone under Rayleigh fading and AWGN demonstrate substantial gains in low-SNR conditions. On KITTI at a 4.13% reference bandwidth budget, WorldGT achieves 47.5% J&F at 0 dB Rayleigh SNR, outperforming H.264 by 11.3 percentage points. Controlled ablations show that world-model prediction contributes up to 8.7 points under Rayleigh fading. Object-detection experiments further demonstrate the transferability of receiver-relative predictive transmission across visual perception tasks. These results establish world-model-guided, receiver-state-aware transmission as an effective approach to robust video perception under severe communication constraints.

I. INTRODUCTION

For stateful video perception, packet value depends on what the receiver can already infer from its accumulated scene state. WorldGT uses a receiver twin and verified prediction to prioritize updates that can improve the receiver’s output.

I. INTRODUCTION

  • Reliable visual perception matters for connected vehicles, roadside sensing, and UAVs, but the wireless link may be least reliable when timely perception is most valuable.For a receiver with memory, task relevance alone does not determine whether new information needs transmission.Source: §I · p. 1
  • WorldGT synchronizes a receiver twin from codeword acknowledgments, predicts and verifies cached features, and scores refreshes by comparing receiver outputs with all versus no packets refreshed.Those scores guide budget-constrained packet selection and unequal error protection with a skip option.Source: §I · p. 2

A. Semantic and Predictive Communication

  • Semantic communication and DeepJSCC address task-relevant or noisy-channel transmission, while predictive communication uses temporal models to avoid unnecessary updates.The paper identifies receiver-state-relative packet utility, verified prediction, and discrete channel protection as an uncombined challenge for dense video perception.Source: A. Semantic and Predictive Communication · p. 2, §I · p. 2

B. Collaborative Perception and Video Analytics

  • Collaborative-perception and video-analytics systems use spatial confidence, information scores, camera-side filtering, or server requests, complementing receiver-state tracking across time.The paper frames estimating receiver demand from accumulated state as a complementary perspective on spatial and temporal communication decisions.Source: B. Collaborative Perception and Video… · p. 2

C. World Models and Receiver Digital Twins

  • Prior predictive-communication work motivates shared state and motion-based feature prediction, but the paper identifies joint ACK-synchronized cache prediction, sender verification, receiver-relative scoring, and discrete LDPC protection as unaddressed for dense video perception.Existing motion-compensation and resource-allocation mechanisms also remain to be integrated in this receiver-aware framework.Source: C. World Models and Receiver Digital… · p. 2

D. Error Protection and Resource Allocation

WorldGT combines receiver-aware prediction and packet prioritization with discrete channel protection under a fixed budget. Its receiver maintains a feature cache and task memory, while codeword acknowledgments synchronize the sender’s twin.

D. Error Protection and Resource Allocation

  • Receiver-relative task values can be incorporated into unequal-error-protection and discrete budget-allocation mechanisms, while feature-cache prediction error and task-output changes offer distinct criteria for verification and priority.The section situates these choices alongside Lagrangian allocation, multiple-choice knapsack, and video-coding skip decisions.Source: D. Error Protection and Resource… · p. 3

A. Overview of WorldGT Framework

  • WorldGT selectively sends information expected to improve current perception, jointly choosing packets and unequal protection using receiver-twin scores, channel budget, and estimated channel reliability.Acknowledgments let the sender reproduce receiver-state updates; missing packets use prediction or predefined fallback rules.Source: A. Overview of WorldGT Framework · p. 3

B. Feature Packets and Receiver State

  • Each packet contains a 2 × 2 feature tile at 3-bit precision, and the receiver state includes its feature cache, cell ages, task memory, and prediction history.Successfully delivered features update the cache; the task head processes the resulting features to produce perception outputs.Source: B. Feature Packets and Receiver State · p. 3

C. Codeword Channel and Budget

  • The channel model uses QPSK and 5G NR LDPC codewords with rates 1/3, 1/2, 2/3, and 3/4; no retransmission occurs within the 100-ms frame interval.The full reference budget is 9P units, with evaluated normalized budgets β = 0.0413 and 0.125; the sender uses average-SNR error rates without instantaneous channel knowledge.Source: C. Codeword Channel and Budget · p. 3

D. Receiver-Relative Transmission Objective

WorldGT frames transmission as maximizing expected task quality subject to a per-frame budget. Its receiver twin estimates packet impact by comparing task outputs under refreshed and unrefreshed counterfactuals.

D. Receiver-Relative Transmission Objective

  • The transmission objective seeks expected task quality over frames while enforcing an average per-frame budget.The expectation is over channel outcomes, and decisions use an immediate task-output proxy without current task labels.Source: D. Receiver-Relative Transmission… · p. 4

A. Receiver Twin and Task-Impact Scores

  • An acknowledgment-synchronized twin evaluates current-frame counterfactual outputs using the same task memory, with all cells refreshed versus none refreshed.The prediction-only counterfactual assumes current verification information is available; complete frame outages are handled separately.Source: A. Receiver Twin and Task-Impact Scores · p. 4
  • The all-versus-none output discrepancy is spatially attributed to packets, assigning low priority to regions already reproduced by the predicted receiver state.This can include predictable background motion and already tracked objects despite large feature changes or high sender confidence.Source: A. Receiver Twin and Task-Impact Scores · p. 4

B. World-Model Architecture

WorldGT advances cached features with sender-estimated affine motion, then uses embedded receiver history and factorized spatiotemporal attention to predict the cache. The architecture combines global motion compensation with temporal and spatial context.

1) Affine motion compensation:

  • Affine motion compensation aligns the cached scene to current global motion using a field estimated by the sender and reconstructed by the receiver from shared quantized coefficients.The field is fit from local feature correspondences using confidence-weighted least squares.Source: 1) Affine motion compensation · p. 4

2) History embedding and spatiotemporal backbone:

  • The residual model embeds eight history slots with cached features, age, freshness, validity, position, and affine-motion information.At clip start, the earliest available history is repeated to fill the window, with validity supplied as an input flag.Source: 2) History embedding and… · pp. 4–5
  • Four factorized transformer blocks apply spatial attention within each slot, causal temporal attention at each cell, and a feed-forward network.Factorization reduces attention cost to HN^2+NH^2 rather than (HN)^2.Source: 2) History embedding and… · p. 5

3) Displacement and gate heads:

The world model predicts residual displacement and chooses whether to warp or hold cached features. It is trained on simulated, unlabeled receiver histories, while sender verification connects prediction quality to receiver-relative task impact.

3) Displacement and gate heads:

  • Separate heads predict residual displacement and a gate that chooses between residual-warped features and the unwarped cache.The newest-slot features feed both heads.Source: 3) Displacement and gate heads · p. 5

C. Label-Free Training of the World Model

  • Training uses unlabeled video and simulated receiver histories, with packet loss and stale observations included in most samples.In 87.5% of training samples, history ends 2–8 frames before the target; the loss includes prediction, displacement supervision, and rollout terms.Source: C. Label-Free Training of the World… · pp. 5–6

D. Sender Verification and Side Information

  • Sender verification guarantees no larger tile feature error than holding when its side information arrives, linking feature-level verification to task-level packet selection.The twin evaluates task-output changes relative to the verified prediction.Source: D. Sender Verification and Side… · p. 7
  • The verification mask and affine parameters accompany every codeword, so their availability is coupled to successful codeword decoding.The evaluation reserves a budget share for the mask; affine coefficients, packet identifiers, and ACK traffic are outside the reported forward budget.Source: D. Sender Verification and Side… · p. 7

E. Packet Selection, UEP, and Packing

Packet selection assigns each packet a skip or coding-rate option under a budget, then approximates the discrete allocation and packs packets into whole codewords.

E. Packet Selection, UEP, and Packing

  • Each packet chooses skip or one of four coding rates, with cost scaled by payload precision and a zero-cost, certain-failure option for skipping.For a 3-bit payload, each packet occupies half a nominal codeword relative to the 6-bit reference.Source: E. Packet Selection, UEP, and Packing · p. 7
  • The allocation minimizes an additive expected-loss proxy using packet scores and rate-dependent costs.Lagrangian relaxation selects packet rates; bisection finds a feasible multiplier and greedy upgrades spend remaining units.Source: E. Packet Selection, UEP, and Packing · p. 7
  • Whole-codeword packing enforces the physical constraint by removing low-value packets and repacking when rounding exceeds it.At 3 bits, two packets share each codeword and its erasure outcome; linearity of expectation preserves the additive proxy.Source: E. Packet Selection, UEP, and Packing · p. 7

F. Operation During a Frame Outage

During frame outages, the receiver follows deterministic fallback rules to maintain a continuous state trajectory. The evaluation covers payload precision, video datasets, task metrics, and multiple transmission baselines.

F. Operation During a Frame Outage

  • When no codeword arrives, the receiver reuses its last delivered affine field and suppresses unverified residual correction, preserving a continuous state trajectory across frame losses.If no affine field has ever arrived, it initially holds the cache; from frame five onward it uses the residual model with a zero affine condition and learned hard gate.Source: F. Operation During a Frame Outage · p. 7

A. Data, Tasks, and Evaluation Protocol

  • KITTI evaluation uses 80 clips from 9 validation sequences, while VisDrone evaluation uses 68 clips from 17 test-dev sequences and filled-box proxy masks.Both datasets use 32-step clips and track up to ten first-frame objects.Source: A. Data, Tasks, and Evaluation Protocol · p. 7

B. Metrics and Comparators

  • Segmentation quality is clip-averaged J&F, averaging region intersection-over-union and boundary F-score across tracked objects and clips.The first and last frames are excluded from scoring.Source: B. Metrics and Comparators · p. 8
  • H.264/H.265 plus LDPC, DeepJSCC, feature-change and sender-confidence priorities, frame-triggered transmission, and cyclic, periodic, and random refresh are included as comparators.Codec configurations are selected at each operating point from those within 105% of the nominal budget.Source: B. Metrics and Comparators · p. 8
  • All packet-selection baselines use the same 3-bit payload as WorldGT, while EEP at 100% sends every 3-bit packet at rate 1/3.This EEP setting matches the fixed 6-bit/rate-2/3 reference budget.Source: B. Metrics and Comparators · p. 8

G. Payload Precision

  • 55.68% J&F for 6-bit features and 54.44% J&F for 3-bit features, a 1.24-percentage-point reduction; 2-bit features achieve 52.97%.The 3-bit operating point is presented as a favorable trade-off between feature precision and spatial coverage.Source: B. Metrics and Comparators · p. 8, Table II · p. 8

VI. RESULTS AND ANALYSIS

WorldGT outperforms the evaluated baselines across measured Rayleigh SNRs at both budgets, with its largest advantage at low SNR and competitive performance as channel reliability improves.

1) Rayleigh Fading:

  • At −5 dB, the rate-1/3 code loses approximately 90% of codewords, and at 0 dB it loses 52%; predictive receiver state sustains segmentation between successful updates.Receiver-relative selection directs payload to features that improve task output despite frequent codeword losses.Source: 1) Rayleigh Fading · p. 8
  • At 20 dB, WorldGT reaches 51.6% and 52.4% J&F at the two budgets, within 2.9 and 2.6 points of the best baseline.At 4.13% budget, 51.6% is within 1.1 points of the full-budget 3-bit EEP reference at 52.7%.Source: 1) Rayleigh Fading · p. 8
  • At 4.13% budget, WorldGT achieves 47.5% J&F at 0 dB Rayleigh SNR, 11.3 points above H.264.It also outperforms every evaluated baseline at each measured Rayleigh SNR from −5 through 10 dB at both budgets.Source: 1) Rayleigh Fading · p. 8, Figure 4 · p. 9
  • At 0 dB and 4.13% budget, WorldGT reaches 47.5% J&F, versus 33.9% for frame triggering, 28.2% for feature change, and 30.3% for sender confidence.Cyclic, periodic, and random refresh also remain below the task-adaptive approaches at this operating point.Source: 1) Rayleigh Fading · pp. 8–9

2) AWGN:

Across AWGN and aerial-video evaluations, WorldGT performs best near difficult decoding conditions; ablations attribute gains to prediction and receiver-relative selection, with affine conditioning also contributing.

2) AWGN:

  • At −1.25 dB AWGN, WorldGT achieves 47.4% and 50.9% J&F at the two budgets, exceeding H.264’s 40.4% and 44.3%.At −2 dB, when all four calibrated LDPC options fail, predictive fallback reaches 32.2% J&F versus 22.8% for matched hold-last.Source: 2) AWGN · p. 9

B. VisDrone Results

  • At −5 dB Rayleigh, VisDrone WorldGT achieves 35.9% and 37.6% J&F, exceeding H.264 by 8.2 and 7.0 points at the two budgets.At 0 dB and 4.13%, WorldGT scores 39.8% versus 22.4% for feature change and 23.5% for sender confidence; Fig. 5 compares the aerial-video methods and budgets.Source: B. VisDrone Results · pp. 9–10, Figure 5 · p. 10

C. Controlled Ablations and Mechanism Analysis

  • At −5 dB Rayleigh, replacing the world model with hold-last features reduces J&F by 8.7 points; the reduction is 6.5 points at 0 dB.Prediction contributes both to receiver-cache updates and to the reference used for packet selection.Source: C. Controlled Ablations and Mechanism… · p. 10
  • Removing affine conditioning costs 5.0 J&F points at −5 dB Rayleigh, combining effects of alignment, residual-model conditioning, and global-motion persistence during outages.The gain is smaller at higher Rayleigh SNRs.Source: C. Controlled Ablations and Mechanism… · p. 10
  • At 0 dB Rayleigh, twin-based selection without prediction reaches 41.0%, versus 33.9% for world-model frame triggering; the complete method reaches 47.5%.A development experiment found at most a 0.8-point reduction when the best single rate per frame replaced per-packet UEP; payload coarsening increases spatial updates per codeword.Source: C. Controlled Ablations and Mechanism… · pp. 10–11

D. Joint Bandwidth-SNR Trade-off

WorldGT’s advantage is strongest when bandwidth or channel reliability is scarce, while its qualitative predictions remain useful during unreliable delivery. The profiled allocator is faster than the exact knapsack reference, but the reported timing is not end-to-end.

D. Joint Bandwidth-SNR Trade-off

  • At 5 dB, WorldGT reaches 47.9% J&F at 2.07% budget, exceeding H.264’s 47.5% at 8%.At 10 dB it remains ahead through 12.5% budget and is within 0.6 points of H.264 at 25%; at 15 dB it is nearly equal at 4.13%.Source: D. Joint Bandwidth-SNR Trade-off · p. 11
  • At −5 and 0 dB, WorldGT exceeds H.264 and DeepJSCC across the full bandwidth range; at 1.03% budget it retains 33.1% and 40.2% J&F, respectively.At 0 dB, 40.2% is close to H.264’s 40.4% at 12.5% budget, using approximately twelve times less nominal bandwidth.Source: D. Joint Bandwidth-SNR Trade-off · p. 11, Figure 7 · p. 12

E. Qualitative Comparison of Received Perception

  • On the displayed KITTI clip at 0 dB and 4.13% budget, WorldGT maintains a localized leading-vehicle mask; H.264 shows severe later-frame corruption.The packet row distinguishes delivered, lost, and unsent tiles; without the world model, additional foreground regions appear around vehicles, particularly in the final frame.Source: E. Qualitative Comparison of Received… · p. 11, Figure 8 · p. 12

F. Computational Cost

  • The Lagrangian allocator takes 30.9 ms versus 544.8 ms for exact knapsack, approximately 17.6× faster; profiled component medians total 93.2 ms.Measurements exclude feature extraction, affine warp, channel coding and decoding, transport, and feedback, so they do not represent end-to-end latency.Source: F. Computational Cost · pp. 11–12

G. Task Extension: Object Detection

WorldGT’s receiver-relative predictive transmission extends from segmentation to object detection, including analog feature transmission. The paper reports gains across challenging channel conditions and identifies how prediction and feature detail matter at different reliability levels.

G. Task Extension: Object Detection

  • At 12.5% budget, WorldGT outperforms H.264 and all-tile DeepJSCC from −2.5 to 15 dB; at 0 dB, it achieves 48.4% mAP.At 0 dB, this is 1.5 points above DeepJSCC and 29.2 points above H.264; at 4.13% budget, DeepJSCC retains a small advantage.Source: G. Task Extension: Object Detection · p. 13, Figure 9 · p. 13
  • At 4.13% budget, WorldGT + DeepJSCC achieves 29.5% mAP at −5 dB and 45.1% at 0 dB, improving over DeepJSCC by 11.8 and 8.2 points.At 12.5% budget and 0 dB, it achieves 50.0% mAP versus 46.9% for DeepJSCC.Source: G. Task Extension: Object Detection · p. 13

VII. DISCUSSION

  • Under severe fading, prediction maintains useful scene estimates between successful deliveries, while greater reliability raises the value of more detailed spatial information.The 3-bit operating point balances feature precision against spatial coverage.Source: §VII · pp. 13–14
  • WorldGT’s segmentation-oriented representation transfers to a separately trained detection head without retraining the world model or redesigning the transmission policy.The WorldGT + DeepJSCC results also show receiver-relative selection applied to analog feature transmission.Source: G. Task Extension: Object Detection · p. 13, §VII · p. 14

VIII. CONCLUSION

  • WorldGT’s synchronized receiver twin uses predicted scene state, sender-side verification, and task-impact scores to guide packet selection and unequal error protection.The detection experiment reuses segmentation-oriented packet choices rather than optimizing selection for detection.Source: G. Task Extension: Object Detection · p. 12, §VIII · p. 14
  • At 0 dB and 4.13% reference bandwidth, WorldGT improves KITTI J&F by 11.3 percentage points over the strongest independent baseline.At 20 dB, it remains within 2.9 points.Source: §VIII · p. 14

APPENDIX A DETAILED NUMERICAL RESULTS

The appendix tables provide detailed numerical results for the SNR and detection experiments discussed in the main text.

APPENDIX A DETAILED NUMERICAL RESULTS

  • Table V reports KITTI J&F results under Rayleigh fading.Its caption says shading identifies WorldGT.Source: Table V · p. 16
  • Tables VI, VIII, and VII report selected-point KITTI AWGN and VisDrone AWGN and Rayleigh-fading J&F results, respectively.The VisDrone tables cover 68 clips and use filled bounding boxes as proxy masks.Source: Table VI · p. 16, Table VIII · p. 17, Table VII · p. 17
  • Table IX reports detection-extension mAP@0.5 results under Rayleigh fading.Its caption marks the analog selection variant described in Section VI-G.Source: Table IX · p. 17

Related papers

Loading 2610.29587v1…