Joint Tomlinson-Harashima source and linear relay precoder design in amplify-and-forward multiple-input multiple-output two-way relay systems

Existing minimum-mean-squared-error (MMSE) transceiver designs in amplified-and-forward (AF) multiple-input multiple-output (MIMO) two-way relay systems all assume a linear precoder at the sources. Nonlinear source precoders in such a system have not been considered yet. In this paper, we study the joint design of source Tomlinson-Harashima precoders (THPs), relay linear precoder and MMSE receivers in two-way relay systems. This joint design problem is a highly nonconvex optimization problem. By dividing the original problem into three sub-problems, we propose an iterative algorithm to optimize precoders and receivers. The convergence of the algorithm is ensured since the updated solution is optimal to each sub-problem. Numerical simulation results show that the proposed iterative algorithm outperforms other algorithms in the high signal-to-noise ratio (SNR) region.