PaperlayerView exact revision on arXiv

Source-linked AI summary

Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Xiangyu Liu, Yijie Shen, Yue Yin, Zhitong Yin, Yongqi Pan, Wentao Fan

arXiv:2610.34939v1cs.CV

TL;DR

RGB-X tracking benefits from auxiliary sensors, but centralized training requires collecting multimodal videos, while heterogeneous clients complicate which parameters and contributors should be shared. Fed-MAPTrack uses prompt-only federated training with global RGB sharing and modality-specific banks. It outperforms FedAvg on the three benchmarks under the evaluated partition, and the ablations underscore the joint role of sharing domains and contributor eligibility.

  • Problem

    Centralized RGB-X training assumes aligned multimodal videos can be collected together, while heterogeneous sensor availability leaves sharing domains and eligible contributors unspecified for tracker parameter groups.

    Source: §1 · pp. 2–3

  • Method

    Fed-MAPTrack trains prompt-related parameters over a frozen tracker, sharing RGB parameters globally and using modality-specific banks with selective contributor eligibility.

    Source: §1 · p. 3

  • Results

    Fed-MAPTrack achieves higher mean scores than FedAvg on LasHeR, VisEvent, and DepthTrack under the evaluated partition.

    Source: §6 · p. 16

  • Takeaways & Limitations

    Controlled comparisons show that extending bucketing without compatible participation rules can reduce performance, supporting joint consideration of sharing domains and contributor eligibility.

    Source: §5 · p. 16

    The evaluation uses twelve simulated clients, three repetitions, and one fixed partition; modality, dataset, and RGB-only data proportion vary together, limiting generality across partitions.

    Source: §5 · p. 16

Abstract

from arXiv · show

RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.

1. Introduction

RGB-X tracking uses thermal, depth, or event observations to complement RGB, but centralized training assumes aligned multimodal videos can be collected together. Fed-MAPTrack addresses heterogeneous sensor availability by combining prompt-only federated training with modality-aware aggregation and selective participation.

1. Introduction

  • Sensor-aware sharing requires defining both parameter-group sharing domains and eligible contributors, because RGB-only clients lack auxiliary measurements despite having useful RGB data.Fusion and normalization parameters interact with the auxiliary pathway even when they are not explicitly auxiliary prompts.Source: §1 · pp. 2–3
  • Existing approaches do not specify sensor-aware sharing and participation rules for RGB-X tracker parameter groups, while modality-separated federation loses a shared RGB bank.The introduction also notes that exchanging the full tracker is unnecessary when adaptation can be confined to prompts.Source: §1 · p. 3
  • Fed-MAPTrack uses a frozen pretrained tracker, globally aggregates RGB-shared parameters, and maintains thermal, event, and depth banks for other groups.RGB-only clients contribute to the common RGB bank, while their fusion and normalization updates are excluded from modality-specific averages.Source: §1 · p. 3
  • The study distinguishes auxiliary-sensor availability from dataset origin and jointly designs modality-aware bucketing with selective participation in a prompt-only framework.The backbone and prediction head are frozen, restricting local optimization and round-wise communication to prompt-related parameters.Source: §1 · p. 3
  • The framework is evaluated on thermal, event, and depth benchmarks with matched repetitions and controlled ablations, reporting improvements over FedAvg under the evaluated partition.The introduction also notes final-parameter analyses of the aggregation design.Source: §1 · p. 3

2. Related Work

Related work covers RGB-X tracking, federated visual and multimodal learning, and parameter-efficient prompt tuning. The paper focuses on sensor-aware sharing and contributor eligibility for tracking parameters across heterogeneous clients.

2. Related Work

  • The review frames its comparison around how prior areas handle sensor heterogeneity and parameter sharing in distributed RGB-X tracking.It spans four areas: RGB-X tracking, federated visual learning, multimodal federated learning, and prompt tuning.Source: §2 · p. 3

2.1. RGB-X Tracking

  • RGB-X tracking adds thermal, depth, or event cues to RGB for complementary information, but existing trackers typically assume centralized access to aligned multimodal videos.Thermal supports low-light tracking, depth provides geometric structure, and events offer high temporal resolution and motion sensitivity.Source: §2.1 · pp. 3–4

2.2. Federated Visual Learning

  • Conventional federated methods address optimization or personalization, but do not specify sensor-dependent sharing domains for RGB-X tracking parameters.Tracking also involves target localization and template-search interaction rather than only image-level recognition.Source: §2.2 · p. 4

2.3. Multimodal Federated Learning

  • Multimodal federated methods address modality heterogeneity through alignment, distillation, completion, or personalized aggregation, but these alone do not set participation rules for functional tracker groups.The paper positions its group-specific aggregation as tailored to tracking rather than a replacement for alignment or completion.Source: §2.3 · p. 4

2.4. Parameter-Efficient Prompt Tuning

  • Prompt tuning adapts pretrained models with few trainable parameters, and Fed-MAPTrack applies it to combine global RGB sharing with modality-specific adaptation and eligibility.The related work contrasts this focus with client-specific prompt generation in pFedPG.Source: §2.4 · p. 4

3. Method

Fed-MAPTrack couples parameter sharing domains with contributor eligibility for federated RGB-X prompt tuning. Its framework keeps videos local and separates common RGB sharing from sensor-specific adaptation.

3. Method

  • Fed-MAPTrack jointly defines where each parameter group is shared and which clients may contribute, combining modality-aware bucketing with selective participation.The local model is identical across aggregation variants.Source: §3 · p. 4, §3.1 · p. 5

3.1. Overall Framework

  • Clients adapt prompt-related parameter groups on a frozen pretrained tracker, keep videos local, and return prompt tensors with validity metadata.The groups cover RGB-shared, auxiliary-specific, fusion, and normalization parameters; RGB-only clients freeze the auxiliary-specific group but can optimize the other groups using masked auxiliary inputs.Source: §3.1 · pp. 4–6

3.2. Modality-Aware Aggregation

  • RGB-shared parameters use one global bank, while auxiliary-specific, fusion, and normalization groups use separate thermal, depth, and event banks.The sensor-dependent banks separate adaptation for different auxiliary modalities without changing the local model.Source: §3.2 · p. 6

3.3. Selective Participation

  • RGB-only clients retain valid RGB contributions, while their fusion and normalization updates are excluded from modality-specific server averages.Their auxiliary-specific contributions are already rejected by the validity mask; the controlled FNA comparison isolates fusion and normalization eligibility while holding bank assignment and local training fixed.Source: §3.3 · p. 7

4. Experiments

The experiments evaluate Fed-MAPTrack across the three tracking datasets and examine how bucketing scope, contributor eligibility, and modality-bank separation relate to performance and parameter patterns. They also report limited optimization and qualitative tracking evidence, alongside simulated prompt-state communication payload estimates.

4.1. Experimental Setup

  • Core comparisons use the same tracking objective, preprocessing, optimization budget, initialization, and fixed client partition across three matched runs.All methods are evaluated at round 120; reported standard deviations describe training variability under that partition, not variability across independently generated partitions. LasHeR AUC, VisEvent SR, and DepthTrack F-score are reported separately as percentages.Source: §4.1 · pp. 7–8

4.2. Main Results

  • Fed-MAPTrack gains 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack over FedAvg.Among nine methods, it has the highest mean thermal AUC and depth F-score; Independent FL's mean event SR is 0.07 points higher. Results are means across three matched runs under one fixed client partition.Source: §4.2 · pp. 9–10, Table 1 · p. 9

4.3. Ablation Study

  • Auxiliary-only bucketing improves FedAvg by 0.92 thermal AUC, 3.53 event SR, and 2.17 depth F-score points, but extending bucketing without S-A reduces scores.With the extended scope fixed, S-A improves FNA by 5.20 thermal AUC, 2.17 event SR, and 0.65 depth F-score points; depth gains are not positive in every run. Fed-MAPTrack exceeds auxiliary-only bucketing in thermal AUC by 1.43 points, while event and depth means are lower by 0.08 and 0.06 points.Source: §4.3 · pp. 10–11, Table 2 · p. 11, Figure 5 · p. 12

4.4. Selective Participation Analysis

  • LasHeR, VisEvent, and DepthTrack assign 48.0%, 84.9%, and 3.0% of training samples, respectively, to their one RGB-only client.FNA and Fed-MAPTrack retain those clients and their RGB-shared updates; S-A changes only whether their fusion and normalization updates enter modality-specific averages. The largest RGB-only share is on VisEvent, but the largest FNA-to-Fed-MAPTrack gain is on LasHeR, so the ratio alone does not explain the gain.Source: §4.4 · pp. 11–12, Figure 6 · p. 12
  • FNA and Fed-MAPTrack have largely overlapping RGB-shared and auxiliary weight distributions, with more visible differences in fusion and normalization.FNA normalization scales cluster near one, whereas Fed-MAPTrack's are broader and centered below one; these final-state histograms do not establish better adaptation or identical tensors.Source: §4.4 · pp. 12–13, Figure 7 · p. 13

4.5. Optimization and Tracking Analysis

  • The complete method records lower mean local training losses than FedAvg and FNA over much of training, while all three show an overall declining trajectory.Losses average twelve clients equally and are not held-out losses for a common server model; benchmark evaluations are for round 120 only, so the curves do not establish test-performance convergence or improved generalization.Source: §4.5 · p. 13, Figure 9 · p. 14

4.4. Selective Participation Analysis

  • Fusion dispersion ranges from 18.0% to 21.4% for the complete method versus 5.2% to 6.3% for FNA; normalization ranges from 1.24% to 2.80% versus 0.39% to 0.95%.Auxiliary-specific dispersion does not increase uniformly and is lower for the complete method in most blocks. The observed separation is concentrated in fusion and normalization; separation itself is not a quality metric or proof that it causes tracking gains.Source: §4.4 · p. 13, Figure 8 · p. 14

4.5. Optimization and Tracking Analysis

  • In the event example, Fed-MAPTrack stays near the annotated target while other methods localize another person; thermal and depth examples also show deviations and later drift.The sequences are illustrative examples from one fixed repetition, showing potential benefit and failure modes rather than benchmark-wide recovery or guaranteed robustness.Source: §4.5 · pp. 13–14, Figure 10 · p. 15
  • Figure 11's middle basketboy example shows a difference in estimated target extent between FNA and Fed-MAPTrack, while other examples have closer predictions.The localization maps are analytically derived from saved predicted boxes using the same Gaussian encoding; they visualize box geometry, not attention, activation, or confidence.Source: §4.5 · pp. 14–15, Figure 11 · p. 16

4.6. Communication Analysis

  • 80.60 MB per round and 9.673 GB over 120 rounds are estimated for full prompt-state exchange; mask-aware uploads reduce estimates to 72.63 MB and 8.716 GB.The mask-aware figures assume invalid RGB-only auxiliary tensors are removed before upload while full prompts are still downloaded; neither estimate is measured traffic, and both exclude metadata and protocol overhead. Prompt tuning reduces payload relative to full-model exchange, but does not establish latency, energy, or deployment improvements.Source: §4.6 · pp. 15–16

5. Discussion and Limitations

The discussion finds that effective aggregation depends jointly on parameter-sharing domains and contributor eligibility, particularly for fusion and normalization. Its conclusions remain bounded by the evaluated partition and client setting, and do not establish that bank separation causes accuracy gains.

Joint design of bucketing and selective participation

  • Adding selective participation to broader bucketing improves FNA and yields higher mean scores than FedAvg on all three evaluated benchmarks.The complete method also has higher means than Local, FedProx, FedPer, Ditto, FedMSplit-P, and FED-PRIME on each benchmark, while Independent FL has a slightly higher event mean.Source: §5 · p. 16
  • Bank separation and accuracy are not causally linked by the final-parameter analyses.The analyses show group-dependent differences but do not establish that separation causes the tracking gain.Source: §5 · p. 16

Evaluation scope

  • The evaluation uses twelve simulated clients and three repetitions under one fixed partition, so it does not establish generality across partitions.Modality, dataset, and RGB-only data proportion vary together.Source: §5 · p. 16

Dynamic sensor conditions

  • S-A uses client-level sensor availability rather than estimating auxiliary observation quality, leaving noisy, delayed, or persistently failing sensors as cases for finer eligibility decisions.The paper identifies observation-quality estimation and dynamic sensor availability as directions for extending the rule.Source: §5 · p. 16

6. Conclusion and Future Work

Fed-MAPTrack combines prompt tuning on a frozen tracker with global RGB sharing and modality-specific banks and eligibility rules. On the evaluated benchmarks, it outperforms FedAvg in mean scores, while future work targets broader evaluation and finer participation rules.

6. Conclusion and Future Work

  • Fed-MAPTrack achieves higher mean scores than FedAvg, Local, FedProx, FedPer, Ditto, FedMSplit-P, and FED-PRIME on each evaluated benchmark.It attains the highest mean thermal AUC and depth F-score among the evaluated methods, while its event SR is 0.07 points below Independent FL.Source: §6 · p. 16
  • The framework adapts prompt-related parameters on a frozen backbone and prediction head, sharing RGB parameters globally and using modality-specific auxiliary, fusion, and normalization banks.RGB-only clients contribute to the shared RGB bank, while their fusion and normalization updates are excluded from modality-specific server averages.Source: §6 · p. 16

Future Work

  • Future work will examine observation-quality-aware participation, dynamic sensor availability, finer sharing rules, additional partitions and data proportions, and measured edge-device communication and runtime.Source: §6 · p. 16

Related papers

Loading 2610.34939v1…