The challenge stems from the very technique that makes optical wireless so spectrally efficient: high-order quadrature amplitude modulation, or QAM. Schemes such as 256-QAM and 512-QAM pack enormous amounts of data into each symbol by using extremely dense constellations of amplitude and phase combinations. But as the constellation density grows, the decision boundaries between neighboring symbols shrink to razor-thin margins, making the signal exquisitely sensitive to noise, phase fluctuations, and the nonlinear distortions introduced by optical components such as light-emitting diodes and laser diodes. In intensity-modulated direct-detection systems, where light intensity cannot go negative, engineers must also contend with a high peak-to-average power ratio, which further degrades signal quality. Traditional receivers that rely on analytical models and linear approximations simply cannot capture these nonlinearities, and their performance collapses precisely where 6G needs it most.

The research team built their evaluation around a two-user optical NOMA system, in which both users share the same time and frequency resources through power-domain superposition coding. The user with the weaker channel is allocated 80 percent of the transmit power while the stronger user receives 20 percent, and at the receiver a successive interference cancellation procedure separates the overlapping signals. The channel model incorporated a realistic battery of impairments: distance-dependent path loss with an exponent of 2.2, Rayleigh fading, additive white Gaussian noise, optical nonlinear distortion from the light source, and hardware impairments including synchronization errors. This deliberately harsh environment was designed to emulate the conditions a real 6G optical link would face, with independent channel realizations generated for each user to reflect heterogeneous propagation conditions.

Against this backdrop, the authors evaluated six detectors within a single unified framework. The minimum mean square error (MMSE) detector offers a favorable complexity-performance trade-off but remains fundamentally linear. The QR-decomposition with M-algorithm maximum likelihood detector (QRM-MLD) approaches optimal performance through tree search but becomes computationally prohibitive at high modulation orders. Convolutional neural network (CNN) detectors learn spatial signal patterns efficiently but cannot model temporal dependencies, while recurrent neural network (RNN) detectors capture channel memory and time correlations at the cost of slower convergence. Two hybrids, CNN-MMSE and the newly proposed RNN-MMSE, combine statistical estimation with deep learning refinement. Prior studies had examined these techniques only in isolation; the unified comparison is itself a significant contribution.

The proposed RNN-MMSE detector operates in three stages. First, an RNN learns the channel characteristics directly from pilot symbols, replacing conventional pilot-based estimation and capturing temporal variations in the process. Second, the MMSE stage applies a statistically optimal linear estimate that suppresses noise and linear interference, producing a structured initial symbol estimate. Third, a second RNN refines that estimate by learning only the residual nonlinear distortions and temporal dependencies that the linear stage could not remove. The authors provide a theoretical justification rooted in the orthogonality principle of linear estimation: because the MMSE output extracts all linearly available information, the residual presented to the recurrent network has a substantially smaller variance than the raw signal, which smooths the optimization landscape, lowers gradient variance, and accelerates training convergence.

The simulation results are striking. At 64-QAM, conventional optical NOMA detection required approximately 13.5 dB of signal-to-noise ratio to achieve a bit error rate of 10 to the minus 3, while the proposed RNN-MMSE detector reached the same target at just 6.5 to 7 dB. As the modulation order climbed, the gap widened. At 128-QAM the hybrid detector delivered a 7 to 8 dB improvement, at 256-QAM an 8 to 9 dB gain, and at 512-QAM, the most demanding scheme tested, it achieved a 9 to 10 dB advantage, reaching the target error rate at roughly 12.5 to 13 dB where the conventional baseline needed 21 to 22 dB. Standalone CNN and RNN detectors and the CNN-MMSE hybrid improved on the classical methods but consistently fell short of the proposed architecture.

Spectral behavior told a similar story. Using Welch power spectral density estimation with Hamming windowing, the team measured out-of-band leakage across all detectors. Conventional optical NOMA exhibited in-band power spectral density around minus 40 dB/Hz with poor suppression near minus 60 dB/Hz, while the RNN-MMSE detector pushed spectral leakage below minus 130 dB/Hz at 512-QAM, indicating excellent sidelobe suppression and minimal adjacent-channel interference. Training convergence favored the hybrid as well: over 20 epochs the RNN-MMSE model began at roughly 85 percent accuracy and converged to about 90 percent, outperforming standalone models by 18 to 32 percentage points. Statistical validation across 20 independent Monte Carlo runs confirmed the results were reproducible, with a standard deviation of only 0.22 dB on the key SNR metric and a 95 percent confidence interval of plus or minus 0.10 dB.

Practicality was a central concern. The entire framework was implemented in MATLAB R2024a on a workstation with an Intel Core i7 processor, 32 GB of RAM, and an NVIDIA RTX 3060 GPU. Training on 100,000 generated signal samples took only 14 to 16 minutes, and inference required approximately 3.8 milliseconds per OFDM frame, fast enough for near-real-time deployment. Memory consumption stayed below 800 MB during inference. The authors acknowledge that the hybrid architecture carries more computational overhead than a standalone linear detector, with complexity of the order of N cubed for the MMSE stage plus N squared terms for the recurrent stage, but they argue the performance gains justify the cost, particularly since the heavy training can be performed offline once while deployment requires only lightweight forward inference that GPUs, FPGAs, or edge accelerators can handle.

The study is candid about its limits. All results derive from Monte Carlo simulation rather than a physical testbed, and the authors identify experimental validation on an LED- or laser-based intensity-modulation link with software-defined-radio baseband processing as an immediate priority. Their comparison with Transformer-based detectors draws on literature benchmarks obtained under different channel models rather than a controlled head-to-head experiment, and their robustness analysis of imperfect channel estimation, imperfect interference cancellation, and synchronization errors is mechanistic rather than simulated. The two-user configuration, while modular and scalable in principle, will need extension to denser multi-user networks with adaptive grouping and dynamic power allocation. Even so, the findings make a compelling case that hybrid intelligent detection is not an incremental refinement but a necessary evolution for 6G optical networks, and the roadmap toward testbed validation, attention-based comparisons, and large-scale multi-user trials is already clearly drawn.

Subject of Research: Neural network-based signal detection for 6G optical non-orthogonal multiple access systems with high-order QAM modulation

Article Title: Comprehensive analysis of 6G optical NOMA waveforms using neural network-based detection for high-order QAM modulation

Article References: Kumar, A., Revathi, V., Gaur, N., & Nanthaamornphong, A. (2026). Comprehensive analysis of 6G optical NOMA waveforms using neural network-based detection for high-order QAM modulation. Results in Optics, 25, Article 101159. https://doi.org/10.1016/j.rio.2026.101159

Image Credits: AI Generated

DOI: 10.1016/j.rio.2026.101159

Keywords: 6G, optical wireless communication, NOMA, QAM, recurrent neural network, MMSE detection, deep learning, bit error rate, power spectral density, IM/DD, OFDM, signal detection