Self-Attention Transformer-Based Detector for Faster-than-Nyquist Signaling

Nurettin Safak, Osman Tokluoglu, Enver Cavus

In this study, a novel encoder-only Transformer-based receiver architecture is presented for BPSK signals transmitted over Faster-than-Nyquist (FTN) signaling channels that introduce intentional inter-symbol interference (ISI) with a compression factor of $τ=0.8$. A complete end-to-end communication chain encompassing BPSK modulation, RRC pulse shaping, and the ISI coefficients arising from matched filtering was constructed and evaluated. The proposed Transformer receiver was benchmarked against the optimal BCJR detector over an $E_b/N_0$ range of 0-8 dB. To systematically close the BER gap to the BCJR, a two-stage training strategy combining multi-SNR pretraining and per-SNR curriculum fine-tuning was developed. The computational complexity and inference latency of the Transformer receiver were analyzed in comparison with a GRU based receiver. Attention map visualizations revealed that the Transformer autonomously identifies the FTN-induced ISI memory structure without requiring any prior channel knowledge; as the SNR increases, the attention weights become significantly concentrated around the center token and its nearest neighbors.