PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillComm: Skill-Driven Semantic Communication for Sequential Workflows via Incremental Token Transmission

Ziyang Meng, Lu Lu

arXiv:2607.11972v1cs.NIcs.ITmath.IT

TL;DR

Existing semantic communication methods generally treat workflow tasks as independent transmissions, despite later stages often needing only state-relevant updates. SkillComm synchronizes reusable skill states to prioritize transmitted tokens and reconstruct task-ready grids; on the MS COCO 2017 workflow, it reduces transmission cost by 51.2% while retaining 99.4% upper-bound-normalized AP at 15 dB Rayleigh fading.

  • Problem

    Existing SemCom methods mainly optimize each transmission independently and do not explicitly exploit reusable execution states across ordered workflows, where later stages often need state-relevant updates rather than full-token transmissions.

    Source: §I · p. 1

  • Method

    SkillComm synchronizes executable skills through a shared Skill-Book and uses cross-step memory to prioritize state-active tokens and reassemble task-ready token grids.

    Source: §I · p. 1

  • Results

    On the MS COCO 2017 workflow, SkillComm reduces token transmission cost by 51.2% and retains 99.4% upper-bound-normalized AP at 15 dB Rayleigh fading.

    Source: §I · p. 1

  • Takeaways & Limitations

    The results support reusable skill states as context for selective semantic updates in workflow-level SemCom.

    Source: §V · p. 5

Abstract

from arXiv · show

As wireless visual intelligence evolves from isolated task inference to ordered skill workflows, the communication bottleneck shifts from transmitting a single semantic representation to coordinating reusable skill states under channel constraints. Existing DeepJSCC and prompt-guided visual transmitters usually treat each task as an independent full-token transmission, with limited reuse of execution memory across semantic workflows. This is inefficient for workflows such as Detect, Segment, and Keypoint, where later stages often require only state-relevant semantic updates. To this end, we propose SkillComm, a skill-driven semantic communication framework that uses reusable skill states as shared context for workflow-aware token prioritization and memory-assisted token-grid reconstruction. A shared Skill-Book maps a high-level visual intent into a synchronized executable skill sequence at the transmitter and receiver. Conditioned on this workflow, adaptive token selection exploits cross-step memory to transmit only state-active tokens through joint source-channel coding, while the receiver reconstructs a task-ready token grid by combining decoded tokens with local historical memory. Experiments on the MS COCO 2017 validation set for the Detect-Segment-Keypoint workflow show that SkillComm reduces token transmission cost by 51.2% while retaining 99.4% upper-bound-normalized average precision at high SNR. These results demonstrate that reusable skill states enable selective semantic update delivery for future agentic and embodied visual intelligence.

I. INTRODUCTION

Existing semantic communication methods largely optimize transmissions independently, leaving reusable execution states underused in ordered visual workflows. SkillComm synchronizes skill execution and uses workflow context to prioritize tokens and reassemble task-ready representations.

I. INTRODUCTION

  • Existing SemCom methods mainly optimize each transmission instance independently, without explicitly exploiting reusable execution states across ordered visual workflows.Later stages may inherit bounding boxes, masks, or intermediate token memory and need state-relevant updates rather than another full-token transmission.Source: §I · p. 1

II. SYSTEM MODEL AND PROBLEM FORMULATION

  • A shared Skill-Compiler and Skill-Book decompose high-level intent into executable skills and coordinate workflow execution across the wireless link.The receiver is expected to execute an ordered visual workflow such as Detect→Segment→Keypoint under wireless channel constraints.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • A memory-driven selector transmits active semantic innovations, while the receiver merges decoded features with cached context to reconstruct task-ready token grids.This targets redundant background and context already available from earlier workflow stages.Source: §I · p. 1
  • On the MS COCO 2017 workflow, SkillComm achieves a 51.2% reduction in token transmission cost and retains 99.4% upper-bound-normalized AP under 15 dB Rayleigh fading.Source: §I · p. 1

A. Workflow Decomposition using Skill Compiler

A shared Skill-Book lets the transmitter and receiver execute the same ordered skill workflow from a high-level visual instruction. At each stage, skill-conditioned tokens are selected and sent through JSCC rather than transmitting the full token grid.

A. Workflow Decomposition using Skill Compiler

  • A skill compiler maps a high-level instruction to an ordered workflow of skill indices using the shared Skill-Book.The workflow is synchronized across transmitter and receiver.Source: A. Workflow Decomposition using Skill… · p. 2
  • The synchronized Skill-Book lets the receiver activate the same scheduled skill at each workflow stage without repeatedly receiving natural-language prompts.Each reusable skill includes a textual description, compact embedding, and associated task head.Source: A. Workflow Decomposition using Skill… · p. 2

B. Token Selection Criteria and JSCC Encoding

  • The system selects state-active tokens from a skill-conditioned token canvas, gathers them for JSCC encoding, and transmits their spatial indices over a reliable control channel.The transmitted tokens are conditioned on the scheduled skill embedding.Source: B. Token Selection Criteria and JSCC… · p. 2

C. JSCC Decoding and Token Reassembly Objective

The receiver decodes transmitted active tokens and scatters them back onto the original grid, filling unselected positions from synchronized historical memory. The reconstructed grid is passed to the active task head, within an objective that balances workflow distortion and communication cost.

C. JSCC Decoding and Token Reassembly Objective

  • Decoded active tokens are scattered to their original grid coordinates, while unselected slots are replenished from synchronized receiver memory.The reassembly operator produces a full token canvas.Source: C. JSCC Decoding and Token Reassembly… · p. 2
  • The reconstructed token canvas preserves the original grid shape so the activated task head can execute inference directly.The receiver updates its memory after reassembly.Source: C. JSCC Decoding and Token Reassembly… · p. 3
  • SkillComm minimizes cumulative workflow distortion subject to a total communication-cost constraint, with task priorities and an accuracy-rate tradeoff parameterized in the objective.The cumulative rate covers the full skill sequence.Source: C. JSCC Decoding and Token Reassembly… · p. 3

III. SKILL-DRIVEN SEMANTIC COMMUNICATION

SkillComm implements workflow-aware token prioritization through memory-aware scoring, adaptive Top-K JSCC transmission, and memory-assisted token-grid reassembly.

III. SKILL-DRIVEN SEMANTIC COMMUNICATION

  • Memory-aware token scoring prioritizes state-active semantic innovations across sequential skill stages.This is part of the workflow-aware token prioritization policy.Source: §III · p. 3
  • Adaptive Top-K selection sends prioritized tokens through JSCC across sequential skill stages.The selection policy is paired with reassembly at the receiver.Source: §III · p. 3
  • Memory-assisted reassembly reconstructs token grids from the delivered state-active semantic innovations.The reconstructed grids support sequential skill stages.Source: §III · p. 3

A. Token Scoring at Transmitter

Transmitter memory retains prior-stage features to support cross-task reuse, while token scoring combines current features, memory, skill relevance, and temporal differences. A rate network uses pooled features and channel SNR to set the retained-token fraction.

A. Token Scoring at Transmitter

  • Transmitter memory stores features generated at the previous stage, while receiver memory stores the corresponding previously reconstructed features.These cached states support cross-task information reuse without redundant transmission.Source: A. Token Scoring at Transmitter · p. 3
  • Cosine similarity alone cannot distinguish task-irrelevant background from task-critical novel features, motivating a delta-aware token gate.The gate combines a routing logit with explicit skill relevance and temporal-difference priors.Source: A. Token Scoring at Transmitter · p. 3
  • The rate MLP uses pooled current, memory, and absolute-difference features together with the skill embedding and normalized SNR to predict the retained-token fraction.The fraction is bounded by ρ_min=0.2 and ρ_max=0.8; hard Top-K is trained with a straight-through mask and soft sigmoid relaxation.Source: A. Token Scoring at Transmitter · p. 3

B. Memory-Assisted Token Reassembly at Receiver

After transmitting the active token subset, the receiver rebuilds the feature grid by overwriting selected positions and retaining unselected positions from local historical memory. This supports lower cumulative token use than transmitting all N tokens at every workflow stage.

B. Memory-Assisted Token Reassembly at Receiver

  • The receiver overwrites selected positions with newly received tokens and inherits unselected positions from local historical memory.The receiver lacks access to untransmitted portions of the transmitter’s current state, so it cannot fuse using that state.Source: B. Memory-Assisted Token Reassembly at… · p. 3
  • The mapping π_t(i) preserves the original global token index while locating that token in the compact transmitted sequence.The reassembly mechanism is described as a pure memory overwrite without supplementary communication overhead.Source: B. Memory-Assisted Token Reassembly at… · p. 3
  • For T stages, SkillComm incurs full communication cost only at the initial stage, unlike a full-token benchmark requiring TN tokens.The rate controller dynamically modulates K_t to reduce repeated transmission of redundant semantic tokens.Source: B. Memory-Assisted Token Reassembly at… · pp. 3–4

IV. EXPERIMENTAL RESULTS

The evaluation uses Detect→Segment→Keypoint as a representative compound-intent workflow. It assesses stage-specific intent conditioning, workflow-level token efficiency, and atomic-task AP.

IV. EXPERIMENTAL RESULTS

  • The representative workflow captures a person’s body mechanics through localization, body delineation, and joint estimation.The workflow is instantiated as Detect→Segment→Keypoint.Source: §IV · p. 4
  • The authors use this workflow to validate sequential skill execution, while stating that the proposed mechanism is not tied to this task chain.Source: §IV · p. 4
  • The reported evaluation dimensions are stage-specific intent conditioning, workflow-level token efficiency, and atomic-task APbox, APmask, and APkp.Source: §IV · p. 4

A. Experimental Setup

The experiments use 100 MS COCO 2017 validation images containing people and a three-stage workflow. Comparisons include reference systems and SkillComm variants, with evaluation under Rayleigh fading.

A. Experimental Setup

  • 100 person-containing MS COCO 2017 validation images are used, with 448 × 448 inputs producing 196 spatial tokens per stage.A prompt-conditioned Swin Transformer produces a 14 × 14 × 256 token canvas, and pre-trained Mask R-CNN and Keypoint R-CNN heads process the reassembled canvas.Source: A. Experimental Setup · p. 4
  • The conventional full-token Detect→Segment→Keypoint workflow requires transmitting 588 tokens.This is 3N tokens for N = 196 at each of the three stages.Source: A. Experimental Setup · p. 4
  • Training samples SNR continuously from 0 to 15 dB, and evaluation tests Rayleigh fading at six SNRs from 0 to 15 dB.The rate controller is trained with the backbone and task heads frozen, followed by Rayleigh-calibration iterations.Source: A. Experimental Setup · p. 4
  • References include a no-channel upper bound, TASC-f, and PGMT-SC, with PGMT-SC excluded from per-task AP curves.TASC-f is included only for applicable comparisons; it lacks automatic workflow compilation and workflow-level token reuse.Source: A. Experimental Setup · p. 4
  • SkillComm-Full transmits all 196 tokens at each stage with manually supplied prompts, while SkillComm-Fixed uses cosine Top-K selection at a fixed 50% downstream token ratio.SkillComm-Adaptive combines automatic skill compilation, cross-step memory, learned delta-aware gating, and adaptive SNR-aware rate control.Source: A. Experimental Setup · p. 4
  • Table II is scoped to stage-conditioning alignment and workflow-level token efficiency.Source: Table II · p. 4

B. Stage-Conditioning Alignment and Token Utilization

SkillComm compiles the compound intent into the required stage-specific routing sequence and reduces token use. The section evaluates routing alignment and describes per-task AP across Rayleigh fading channels.

B. Stage-Conditioning Alignment and Token Utilization

  • SCA is a normalized Brier-style score comparing effective routing weights with the target one-hot skill vector at each stage.Its normalization maps the score to [0, 1], with one indicating correct atomic conditioning at every stage.Source: B. Stage-Conditioning Alignment and… · p. 4
  • 100% SCA accompanies SkillComm’s zero routing error, and its learned gate uses 287.1 rather than 588 tokens, saving 51.2%.TASC-f obtains 33.3% SCA and PGMT-SC obtains 66.3% SCA for the same compound intent.Source: B. Stage-Conditioning Alignment and… · p. 4
  • TASC-f repeats detection routing at all stages, while PGMT-SC repeats its soft mixture; SkillComm compiles the correct one-hot sequence.The target routing vectors correspond to detection, segmentation, and keypoint in sequence.Source: B. Stage-Conditioning Alignment and… · p. 4
  • Fig. 3 reports per-task COCO AP across Rayleigh fading channels for the Detect→Segment→Keypoint workflow.Source: Figure 3 · p. 5

C. Per-Task AP Evaluation over Rayleigh Fading

Across the Detect-Segment-Keypoint workflow, SkillComm preserves task performance while reducing transmission cost, and its skill compiler maintains stage-specific execution in later tasks. The conclusion frames reusable skill states as a basis for transmitting semantic updates instead of repeated full representations.

C. Per-Task AP Evaluation over Rayleigh Fading

  • SkillComm maintains stage-specific execution in segmentation and keypoint, while TASC-f degrades significantly by continuing to use the detection condition.At the first stage, all SkillComm variants transmit the full token grid and achieve nearly identical detection performance.Source: C. Per-Task AP Evaluation over… · p. 5
  • SkillComm-Fixed is more robust at low SNR, whereas SkillComm-Adaptive surpasses it at high SNR by prioritizing tokens using channel and memory information.SkillComm-Full provides the all-token upper performance reference; Fixed and Adaptive reduce token use through selective transmission.Source: C. Per-Task AP Evaluation over… · p. 5

V. CONCLUSIONS

  • 51.2% lower token transmission cost accompanies 99.4% upper-bound-normalized AP at 15 dB Rayleigh fading; SkillComm-Adaptive retains 94.3% average AP relative to NC-UB across 0–15 dB and all three tasks.The 51.2% cost reduction and 15-dB AP result are reported for the Detect-Segment-Keypoint workflow.Source: §V · p. 5, C. Per-Task AP Evaluation over… · p. 5
  • The conclusion presents reusable skill states as a basis for transmitting semantic innovations rather than repeated full representations.Source: §V · p. 5
  • Future work will develop feedback-loop skills that adapt the next skill, token budget, and refinement transmission using receiver confidence, channel state, and memory-reassembly errors.Source: §V · p. 5
Loading 2607.11972v1…