PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling

Nurettin Safak, Osman Tokluoglu, Enver Cavus

arXiv:2609.07762v1eess.SPcs.LG

TL;DR

FTN offers spectral-efficiency gains at the cost of deliberate ISI, and the paper tests whether CNN-inspired multi-window processing improves Bi-LSTM detection. It finds no robust advantage from those branches and instead combines pre-whitening with BCJR-posterior distillation, reaching 1.05× MAP BER at τ = 0.8 and 1.89× at τ = 0.7 with N = 6.

Abstract

from arXiv · show

Recurrent detectors such as bidirectional long short-term memory (Bi-LSTM) networks are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling. Motivated by convolutional detectors that build the intersymbol interference (ISI) structure into their architecture, we ask whether processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM. Across roughly 260 controlled trainings it does not: at a matched parameter budget and a matched readout, the multi-window architecture never significantly beats a plain Bi-LSTM. Nested windowing is an invertible rearrangement that adds no information, extra branches only add bottlenecks, and a distillation diagnostic shows the network is already near optimal for its window. The limitation is therefore the observation model, not the architecture. Keeping the architecture fixed, we pre-whiten the input, restoring the conditional independence that colored matched-filter noise violates, and distill the BCJR soft posterior into the network. With 3.4% more parameters this reaches 1.05 times the BCJR BER at a compression factor of 0.8 and 1.89 times at 0.7, improving to 1.47 times when the whitened window is widened. The 23.7% BER reduction at 0.8 requires an ill-conditioned ISI matrix but is not monotone in the conditioning, and it holds across five independent noise realizations and a symbol-level McNemar test.

I. INTRODUCTION

The paper asks whether transferring CNN-inspired, ISI-aware multi-window processing to a Bi-LSTM improves FTN detection. It reports no robust architectural advantage and proposes addressing the observation model with pre-whitening and BCJR-posterior distillation.

I. INTRODUCTION

  • FTN increases spectral efficiency by transmitting faster than Nyquist, at the cost of deliberate ISI, while BCJR complexity grows exponentially with channel memory.That complexity motivates learned detectors that trade some BER for lower complexity.Source: §I · p. 1
  • The question follows a CNN result that used ISI-aware kernels to approach BCJR performance for τ ≥ 0.7.The paper asks whether transferring that multi-window structure helps a recurrent detector.Source: §I · p. 1
  • Across roughly 260 trainings, matched readouts and parameter budgets leave multi-window recurrent branches without a robust BER advantage over a plain Bi-LSTM.The study reports this as a fair, readout-matched comparison.Source: §I · p. 1
  • A distillation diagnostic finds the plain network already near window-optimal, locating the remaining gap in the observation model.The proposed detector keeps the architecture unchanged while combining pre-whitening and BCJR soft-posterior distillation.Source: §I · p. 1

II. SYSTEM MODEL

The system model describes matched-filtered FTN observations as ISI-weighted BPSK symbols plus colored noise. Learned detectors use a finite observation window to estimate its center symbol, with BER normalized to BCJR MAP performance.

II. SYSTEM MODEL

  • The matched-filter output follows the Ungerboeck model: ISI-weighted BPSK symbols plus noise whose correlation follows the ISI sequence.The symbols are BPSK, and the ISI coefficients come from the pulse autocorrelation.Source: §II · p. 2
  • The exact log-likelihood is a sufficient statistic only over the whole sequence, and BCJR evaluates the corresponding MAP posterior.The likelihood is proportional to 2a⊤y − a⊤Xa, with X the Toeplitz ISI matrix.Source: §II · p. 2
  • Learned detectors observe a length-2N+1 window around y_k and estimate the center symbol a_k; BER is normalized by BCJR MAP BER, where 1.00× is optimal.N is the one-sided ISI length.Source: §II · p. 2

III. OBSERVATION-MODEL-AWARE BI-LSTM DETECTION

The section compares plain and multi-window Bi-LSTM detectors, including nested and distance-isolated branches. With readout held fixed, the multi-window design shows no robust BER advantage across the tested compression factors.

A. Multi-window architectures

  • The plain baseline feeds y_k into a single Bi-LSTM.The comparison includes a matched-parameter 2 × 2 architecture-by-readout experiment with identical protocols.Source: A. Multi-window architectures · p. 2, Table I · p. 2
  • At 8 dB with readout fixed, multi-window minus plain Bi-LSTM BER is favorable only at τ = 0.8 and unfavorable at τ = 0.7, 0.6, and 0.5.The figure caption characterizes the result as no robust multi-window advantage.Source: Figure 2 · p. 2
  • The multi-window detector uses separate Bi-LSTM branches for nested windows, concatenating their outputs into a shared dense head.A distance-isolated variant instead gives each branch only the symmetric triplet [y_{k−i}, y_k, y_{k+i}].Source: A. Multi-window architectures · p. 2

B. Readout is a confound, and it dominates

Readout position changes BER more than architecture: matched comparisons show a significant 22–33% readout effect, while multi-window branches do not consistently beat the plain Bi-LSTM. Nested windows add no information, and branch compression creates additional bottlenecks.

B. Readout is a confound, and it dominates

C. Why the architecture cannot help

  • Nested windowing is information-preserving because the largest window is the original input, so the rearrangement adds no information about the target.Any potential gain must therefore come from inductive bias alone.Source: C. Why the architecture cannot help · p. 2
  • Each recurrent branch irreversibly compresses its window before fusion, creating multiple bottlenecks rather than one.Distance-isolated branches also cannot separate their own distance contribution from the others; at τ = 0.7 they reach 8.96× MAP versus 3.61× for the plain baseline, or 2.48× worse.Source: C. Why the architecture cannot help · p. 3

D. Distillation diagnostic: already window-optimal

Distilling the full-sequence BCJR posterior into the plain network yields no significant gain, indicating that the network is already near-optimal for its input window. The remaining gap lies in the observation model.

D. Distillation diagnostic: already window-optimal

  • Distilling the full-BCJR posterior into the plain input-window network yields no significant gain, indicating that it is already near window-optimal.Because soft-cross-entropy distillation targets the window-conditional posterior, it helps only if a learning gap remains.Source: D. Distillation diagnostic: already… · p. 3
  • The proposed detector keeps the same Bi-LSTM and trains it against the optimal BCJR detector’s soft posterior.The teacher is used only during training.Source: D. Distillation diagnostic: already… · p. 3

E. Pre-whitening

Pre-whitening addresses the colored-noise observation model while leaving the Bi-LSTM architecture unchanged. BCJR soft-posterior distillation complements whitening, with their joint benefit confined to regimes where both mechanisms apply.

E. Pre-whitening

  • Pre-whitening factors the ISI matrix and transforms the input to reduce colored-noise dependence; at τ = 0.7, residual autocorrelation falls from 0.351 to 0.038.The transform uses an adaptive ridge when the ISI matrix loses numerical positive definiteness.Source: E. Pre-whitening · p. 3
  • Because the whitening filter is IIR, the whitened window carries information from outside the raw window, raising window-MAP BER performance at τ = 0.7, 8 dB, N = 6 from 1.68 × 10^-3 to 6.98 × 10^-4.The network receives both raw and whitened channels, with the architecture unchanged.Source: E. Pre-whitening · p. 3, Figure 4 · p. 3

F. BCJR soft-posterior distillation

  • At BER ∼10^-3, the network is regressed onto the full BCJR detector’s soft posterior because hard-label gradients are informative only at rare errors.The teacher is used only during training; at τ = 0.8, 0.7, and 0.6, whitening and distillation are weak alone but strong together, whereas at τ = 0.9 or 0.5 one component alone is best.Source: F. BCJR soft-posterior distillation · pp. 3–4

IV. SIMULATION RESULTS

In the reported 8 dB comparisons, whitening plus distillation reduces baseline BER by 23.7% at τ = 0.8 and 19.2% at τ = 0.7. Across the five-seed, N = 6 evaluation, the method reaches 1.05× and 1.89× MAP at those compression factors, with a widened whitened window improving the τ = 0.7 result to 1.47× MAP.

A. Experimental setup

  • The evaluation uses BPSK over AWGN with RRC β = 0.35, windows N = 6 (N = 8 in the whitened domain), and pooled training at 7–10 dB.Models are evaluated from 0–10 dB, with reported ratios at 8 dB; the plain and whitened detectors have 5809 and 6009 parameters, respectively.Source: A. Experimental setup · p. 4

B. Main results

  • 23.7% baseline BER reduction at τ = 0.8 and 19.2% at τ = 0.7; the method reaches 1.05× and 1.89× MAP at N = 6, respectively.Increasing only the whitened window to N = 8 reaches 1.07× and 1.47× MAP at τ = 0.8 and τ = 0.7, respectively.Source: B. Main results · p. 4
  • Across five seeds, paired t-tests give p = 2.9 × 10^-5 at τ = 0.8 and p = 2.6 × 10^-3 at τ = 0.7.The authors regard the block-wise and symbol-level tests as primary because they do not rely on seeds as the unit of analysis.Source: B. Main results · p. 4
  • The components are individually weak at τ = 0.8 (−13.2% for distillation and −10.8% for whitening) but jointly yield −23.7%.The same ordering holds at τ = 0.6, while at τ = 0.5 the inexact ridge-regularized whitening means adding the teacher no longer helps.Source: B. Main results · p. 4

C. Mechanism: the gain tracks conditioning

The gain depends on conditioning but does not increase monotonically with it: it peaks at τ = 0.8, while the well-conditioned τ = 0.9 case shows no room for improvement.

C. Mechanism: the gain tracks conditioning

  • At τ = 0.9 (κ = 1.8), all methods coincide with BCJR, leaving no room to improve.The methods reach 1.04× →0.99× MAP at 8 dB.Source: Figure 5 · p. 4
  • At τ = 0.8, the proposed detector approaches BCJR from 1.37× to 1.05× MAP at 8 dB.With N = 8, the result is 1.07× MAP.Source: Figure 6 · p. 4
  • 23.7% improvement peaks at τ = 0.8 (κ = 6.3), then declines as κ grows further.Ill-conditioning is necessary but does not predict gain size monotonically.Source: C. Mechanism: the gain tracks… · p. 4

D. Statistical robustness

At τ = 0.8 and 0.7, the proposed method wins across five independent noise blocks and passes symbol-level McNemar tests. Robustness tests did not cover τ = 0.6 or 0.5.

D. Statistical robustness

  • At τ = 0.8 and 0.7, the method wins in 5/5 independent blocks and the symbol-level McNemar tests are significant.The reported p-values are 6.5 × 10−6 and 2.3 × 10−5, respectively.Source: D. Statistical robustness · p. 5
  • Robustness tests cover only τ = 0.9, 0.8, and 0.7; results at τ = 0.6 and 0.5 rely on seed-level tests alone.The authors identify robustness testing at lower compression factors as a next step.Source: D. Statistical robustness · p. 5
  • Ill-conditioning is necessary but insufficient for a gain: it is absent at τ = 0.9, peaks at τ = 0.8, then declines as κ grows.Fig. 10 plots this non-monotonic relationship.Source: Figure 10 · p. 5

V. CONCLUSION

For FTN detection, the study finds that adding multi-window branches does not improve Bi-LSTM BER, while pre-whitening and BCJR-posterior distillation improve results with the architecture unchanged. Gains are supported at τ = 0.8 and 0.7 across independent noise realizations and symbol-level tests.

V. CONCLUSION

  • At τ = 0.8 and 0.7, whitening plus distillation moves the detector from 1.37× to 1.05× MAP and from 2.34× to 1.89× MAP, respectively.At τ = 0.7, widening only the whitened window to N = 8 reaches 1.47× MAP.Source: §V · p. 5
  • Transferring CNN-style multi-window structure to a Bi-LSTM does not improve BER; nested rearrangement adds no information and branches add bottlenecks.A distillation diagnostic indicates the network is already near window-optimal.Source: §V · p. 5
  • The proposed detector stays below the plain baseline at τ = 0.8 and 0.7, with positive gains in all five noise blocks.Symbol-level McNemar tests are significant at both compression factors.Source: Figure 11 · p. 6, Figure 12 · p. 6, Figure 13 · p. 6
Loading 2609.07762v1…