Deduplication-while-Training: A Resilient Paradigm for Privacy-Preserving Cross-Client Deduplication in Federated Learning

• Cross-client duplicates are difficult to detect without exposing data, while duplicates are linked to memorization and privacy risks and waste training resources; DbT serializes deduplication before training.Source: §1 · pp. 1–2
• DwT makes deduplication and training concurrent at the individual-data-item level, letting clients train on items they claim without waiting for global deduplication or other participants.Source: §1 · p. 2
• DwT-FL uses protected tags, a CAS-based state claim, and hot-cold queues so unique trainers proceed while duplicate copies remain available for takeover after disconnections.The state table records each item's tag, state, and owner; the system can transfer training rights to another client holding the same data.Source: §1 · p. 2
• DwT turns cross-client deduplication into an online service that reduces idle waiting, supports state recovery and task takeover, and handles new clients incrementally without repeating full deduplication.Source: §1 · p. 2
• Up to 93.04% reduction in failure recovery time, and up to 94.18%/78.87% lower dynamic-join time overhead in deduplication/training, respectively, versus the state-of-the-art scheme.Source: §1 · p. 3

2.1 Federated Learning

• Federated learning keeps data on clients, which upload model updates for server aggregation rather than sharing raw data.Source: §2.1 · p. 3
• Because raw records stay local, duplicate records are difficult to detect across clients, and repeated training adds computation and communication overhead while increasing overfitting and security risks.Source: §2.1 · p. 3

2.2 Existing Privacy-Preserving Cross-Client Deduplication Schemes for FL

• Existing schemes use multi-party cryptographic interactions; Abadi et al. use a TEE-assisted binary tree and PSI, while Ye et al. account for duplicate frequency through training weights and multiple PSI rounds.Source: §2.2 · p. 3
• These schemes follow DbT: disconnections can require rerunning the complete protocol, and dynamic client joining is unsupported, limiting scalability.Source: §2.2 · p. 3

3.1 Architecture

• DwT-FL's architecture assigns protected tag generation to the key server and deduplication, state management, and model aggregation to the aggregation server, while clients submit tags for their data.Source: §3.1 · p. 3, Figure 1 · p. 4
• After duplicate detection, a unique trainer uses the hot queue while other copies remain in cold queues for recovery; clients can replace a disconnected trainer, and new clients are handled incrementally.Source: §3.1 · p. 3
• DwT-FL implements exact deduplication, but the DwT paradigm can also use fuzzy methods by replacing the exact method with existing schemes.Source: §3.1 · p. 3

3.2 Threat Model

• The threat model assumes honest-but-curious aggregation and key servers that do not collude, honest clients, and disconnections or delays caused by network conditions.Source: §3.2 · p. 4

4.1 Design Goals

• DwT-FL targets private cross-client deduplication, resilience to client dropouts, and asynchronous client joining without interrupting online clients or rerunning global deduplication.These are the system's three stated design goals.Source: §4.1 · p. 4

4.2 Construction of DwT-FL

• CAS lets only one concurrent request claim an EMPTY data entry by changing it to PENDING and binding the trainer; data-entry creation is kept separate from rights allocation.Deferring allocation leaves entries EMPTY if a client disconnects during deduplication, so another client can claim the data.Source: §4.2 · pp. 5–6
• Clients place successfully claimed data in a hot queue for immediate training and retain unsuccessfully claimed duplicates in a cold queue for possible takeover.Clients retain cold-queue data locally rather than deleting it.Source: §4.2 · p. 6
• Across rounds, the system resets processing states but retains prior trainer records, reusing training rights to avoid repeating duplicate detection and CAS contention.In stable settings, the previous round's trainer receives TRAIN and other owners receive DEDUP.Source: §4.2 · p. 6
• After a training-phase disconnection, the server resets the disconnected trainer's PENDING tags and can direct another online owner or a new client to take over duplicate data.The server detects disconnection using heartbeat timeouts; unique data remains available for the original client to reconnect.Source: §4.2 · p. 7
• New clients append their tags to the index and receive DEDUP for already-claimed data or a CAS-based training right for data still marked EMPTY.Joining does not interrupt existing training or require rerunning global deduplication.Source: §4.2 · p. 7

4.3 Analysis

• Under the stated assumptions, OPRF tags and non-colluding servers protect input data during duplicate detection, while CAS enforces at most one active trainer per duplicate item.The threat model assumes honest clients and network instability as the sole cause of disconnections; malicious-client defenses and long-term key management are out of scope.Source: §4.3 · p. 8, §4.2 · p. 8
• Heartbeat-based state recovery lets other owners resume unfinished duplicate-data training without global rollback or loss of training data.The authors frame this fault-tolerance claim for client disconnections caused by network instability or device failure.Source: §4.3 · pp. 8–9

5 Implementation

• DwT-FL implements communication, heartbeats, and training in Python, while C++ handles the bidirectional index and concurrent state claims.The split avoids Python Global Interpreter Lock synchronization overhead and supports atomic state updates.Source: §5 · p. 9
• The server's bidirectional index uses a pre-allocated memory pool and open-addressing hash table, with full tag comparisons during probing to handle collisions.The tags are compared across 512 hexadecimal characters.Source: §5 · p. 9
• Model parameters are persisted and transferred in resumable chunks with SHA-256 integrity checks to prevent incomplete or corrupted files from entering aggregation.Local training uses PyTorch and EleutherAI/pythia-14m, with mixed precision and fused AdamW.Source: §5 · p. 9
• Aggregation runs only when no data remains PENDING and uses each client's COMMITTED records—its actually trained samples—to set weighted FedAvg contributions.The implementation counts COMMITTED records from the state table to determine aggregation weights.Source: §5 · p. 9

6.1 Experimental Setup

• Experiments compare DwT-FL with a reimplementation of NDSS’25 on a single workstation, using the Haiku corpus and the EleutherAI/pythia-14m model.The corpus has 15,281 processed records, and each client receives a randomized 0–20-second online delay to simulate asynchronous participation.Source: §6 · p. 9, §6.1 · p. 9, §6.2 · p. 9

6.2 Performance

• As local records increase from 256 to 1,536 per client, DwT-FL end-to-end time rises from 27.44 s to 35.35 s, versus NDSS’25’s 32.39 s to 43.05 s.Across 2 to 10 clients, both schemes’ end-to-end times grow approximately linearly, with DwT-FL’s advantage becoming more pronounced; at 10 clients, deduplication takes 2.95 s of 41.80 s end-to-end.Source: Figure 6 · p. 10, §6.2 · p. 10, Figure 4 · p. 10
• With 10 clients and 10,240 fixed input records, DwT-FL’s end-to-end time fluctuates between 41.21 s and 42.34 s as the duplicate ratio rises from 0.0 to 0.9.DwT-FL remains faster than NDSS’25 in this experiment; the tested total input count stays fixed as the duplicate ratio changes.Source: Figure 5 · p. 10, §6.2 · p. 10
• Up to 93.04% lower failure recovery time and 94.18% and 78.87% lower deduplication and training stabilization times, respectively, are reported for DwT-FL versus NDSS’25.Figure 7 reports the failure-recovery comparison, while Figure 8 reports dynamic-joining stabilization times.Source: Figure 7 · p. 11, Figure 8 · p. 11, §6.2 · p. 11
• CAS training-right claiming takes 0.121 ms versus 0.408 ms with a global Mutex, while bidirectional-index recovery locates and releases rights in 1.992 ms versus 12.964 ms for a full scan.CAS throughput is 85.27 Mops/s versus 25.09 Mops/s with a Mutex; full failure-recovery times are 75.092 ms with the bidirectional index and 85.951 ms with a full scan.Source: Figure 9 · p. 11, §6.2 · p. 12
• As clients increase from 2 to 10, metadata communication overhead rises from 5.818 MB to 38.593 MB, while its share of model transmission grows from 4.06% to 5.39%.At 10 clients, metadata storage is 17.989 MB, or 3.20% of the raw model payload; metadata overhead does not change with duplicate ratio.Source: Figure 10 · p. 12, §6.2 · p. 12

• DwT-FL enables parallel privacy-preserving cross-client deduplication and client-local training using a CAS-based concurrent state-claim mechanism and hot-cold dual-queue scheduling.Source: §7 · p. 13
• DwT turns cross-client deduplication from one-time, globally synchronous preprocessing into a continuous online service with state management, concurrent claiming, and failure recovery.Source: §7 · p. 13
• Evaluations show that DwT-FL significantly reduces the overhead of failure recovery and dynamic client joining.Source: §7 · p. 13