Deep Reinforcement Learning for RIS-Aided Multiuser MISO System with Hardware Impairments

Abstract

In this paper, we study a reconfigurable intelligent surface (RIS)-aided multiuser MISO system with imperfect hardware, where the transceiver design is based on the statistical channel state information (CSI). Considering the transceiver hardware impairments (HWI), we aim to maximize the minimum average user data rate, where the precoding matrices at the base station (BS) and the reflecting phase shifts at the RIS are jointly optimized. Since the problem is nonconvex and the objective function cannot be derived in closed form, we adopt the deep deterministic policy gradient (DDPG) algorithm to deal with this challenging optimization problem, where we generate a set of CSI vectors in an offline way, and then these data sets are used to train the neural networks. The simulation results demonstrate the rapid convergence speed of the adopted DDPG algorithm and also emphasize that it is crucial to consider the HWI when optimizing the transceiver.

1. Introduction

Thanks to its attractive properties of low power consumption and hardware cost, reconfigurable intelligent surface (RIS) is recognized as one of the most promising techniques in future sixth-generation (6G) wireless systems [1,2,3,4,5,6]. RIS consists of an array of passive and low-cost reflecting elements whose phase shifts can be tuned. The authors of [7,8] studied the RIS-aided multicell and RIS-aided simultaneous wireless information and power transfer, respectively. Low-complexity algorithms were developed to jointly optimize the precoding matrices at the base station (BS) and the reflecting phase shifts at the RIS. However, the above contributions in [7,8] were based on the ideal assumption of perfect hardware, which is difficult to hold in practice. In practical communication systems, there are inevitable transceiver hardware impairments (HWI), which would cause signal distortions and cannot be ignored in the transceiver design.

The authors of [9] derived the closed-form date rate expression for RIS-aided communication systems, and then the impact of HWI on the RIS-aided systems was analyzed. A RIS-aided single-user communication system with HWI was studied in [10], where the phase shifts of the RIS were optimized by the majorization-minimization (MM) algorithm. Recently, the joint beamforming and phase shift design was studied in a RIS-aided physical layer security system in [11]. Besides the transceiver hardware impairment, the authors of [12] further considered the impact of the phase noise at the RIS and derived the closed-form data rate expression, based on which the genetic algorithm was adopted to solve the phase shift optimization problem. In [13], the RIS-aided communication system for serving a mobile user was studied, and the authors proposed an interesting algorithm to predict the positions of the user under HWI. In [14], the authors analyzed the outage performance for RIS-aided non-orthogonal multiple access systems with HWI, where both near-field and far-field users were considered. Most recently, robust transceiver design for RIS-aided communication systems was studied in [15], where both imperfect CSI and HWI were taken into account. The semidefinite programming was proposed to solve the robust problem.

However, all the above papers were based on the assumption that the BS can acquire the instantaneous CSI, which is challenging in practice due to the limited channel coherence time. Recently, the researchers have focused on the phase shift design based on the statistical CSI such as location/angle information or channel distribution information such as channel covariance matrices, which varies in a much lower time scale than the instantaneous CSI. There are several advantages to using statistical CSI for transceiver design [16]. Firstly, the channel estimation overhead can be reduced as only statistical CSI is needed, which changes very slowly. Secondly, the computational complexity is significantly reduced as the phase shifts at the RIS are only needed to be recomputed when the statistical CSI has changed. Thirdly, the feedback overhead is decreased since the phase shift values of the RIS are only fed back to the RIS controller only when its values are updated, which changes with statistical CSI. Due to the above appealing advantages, the transceiver design based on statistical CSI for RIS-aided systems has attracted extensive research attention [17,18]. Specifically, the authors of [17] derived the closed-form date rate expression for a RIS-aided multiuser system. Then, a genetic algorithm was first proposed to optimize the phase shifts, which only depend on the statistical CSI. As a step further, the authors extended the work in [17] to the practical case when there are imperfect hardware, and a robust transmission design was proposed to optimize the phase shift by considering HWI.

However, the contributions in [17,18] considered the two-time scale design, where the BS designed its precoding matrices based on the instantaneous effective CSI, while only the phase shifts were designed based on statistical CSI. This means that the BS still needs to estimate the instantaneous effective CSI, which will incur sizable channel estimation overhead for highly-mobile scenarios. Against the above background, the authors of [19] studied the transceiver design for RIS-aided communication systems based on fully statistical CSI, where both the precoding matrices at the BS and the reflecting phase shifts at the RIS were designed based on statistical CSI. However, this work was based on the ideal assumption of perfect hardware, which is difficult to hold in practice. As a result, the contributions of this work are summarized as follows:

• We consider optimizing the precoding matrices at the BS and the reflecting phase shifts at the RIS based on statistical CSI to maximize the minimum user data rate to ensure fairness among the users, where the imperfect hardware is taken into account.
• Due to the expectation operator along with the hardware impairment, it is challenging to derive the closed-form data rate expression. Furthermore, the objective function in terms of the max-min format is discontinuous and non-differentiable. As a result, the existing algorithms based on mathematical derivations are not applicable. Instead, we resort to the powerful deep deterministic policy gradient (DDPG) algorithm to solve this challenging optimization problem.
• Note that the convergence speed is quite fast as it can converge within 600–900 iterations and the overall computational complexity are mainly from the calculation of rewards, which are only simple mathematical calculations. In addition, the calculated parameters can be used in subsequent steps and only need to be recalculated when the statistical CSI changes. Once the neural network is trained, it can be directly applied in real-time applications with only simple mathematical calculations. The neural networks only need to be retrained once the statistical CSI changes. Hence, the computational complexity is not high.

2. System Model

We consider a RIS-aided downlink multi-user system where the base station (BS) is equipped with M antennas and the user has a single antenna. The system architecture is shown in Figure 1. In this system, we assume that the RIS has N reflecting elements. Considering the hardware impairment, the transmit signal at the BS can be expressed as

$$\mathit{x}={\sum }_{k=1}^K\left({\mathbf{w}}_k{s}_k+{\mathit{\eta }}_s\right),$$

(1)

where ${\mathbf{w}}_k\in {\mathbb{C}}^{M\times 1}$ represents the beamforming vector from the BS to the k-th user and $s_k\sim \mathcal{CN}(0,1)$ represents the data signal symbol transmitted to the k-th user which satisfies $\mathbb{E}\left\{{\left|s_k\right|}^2\right\}=1$. Furthermore, ${\mathit{\eta }}_s$ denotes the independent Gaussian distortion noise, which satisfies the Gaussian distribution of zero mean, and its distortion noise power is proportional to the transmit power of the antenna. Then, ${\mathit{\eta }}_s$ can be represented as ${\mathit{\eta }}_s\sim \mathcal{CN}(\mathbf{0},k_s\mathrm{diag}\left\{{\sum }_{k=1}^K{\mathbf{w}}_k{\mathbf{w}}_k^H\right\})$, where $k_s\in \left(0,1\right)$ denotes the normalized variance of the emission distortion noise.

For K users, the beamforming matrix at the BS can be expressed as

$$\mathbf{W}=\left. [{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_K\right].$$

(2)

The beamforming matrix $\mathbf{W}$ has to satisfy the power constraints, which can be formulated as

$$\mathrm{tr}\left\{\mathbf{W}{\mathbf{W}}^H\right\}\le P_{\max }.$$

(3)

Furthermore, the channel between the base station and the RIS is denoted as ${\mathbf{H}}_{SI}\in {\mathbb{C}}^{N\times M}$, the channel between the k-th user and the BS is denoted as ${\mathbf{h}}_{SD,k}\in {\mathbb{C}}^{M\times 1}$ and the channel matrix from RIS to the k-th user is denoted as ${\mathbf{h}}_{ID,k}\in {\mathbb{C}}^{N\times 1}$. In this paper, we consider the Rician fading model, the channel ${\mathbf{H}}_{SI}$, ${\mathbf{h}}_{SD,k}$, and ${\mathbf{h}}_{ID,k}$ can be formulated as

$$\begin{array}{cc}\hfill {\mathbf{H}}_{SI}& =\sqrt{\beta }\left(\sqrt{\frac{\delta }{\delta +1}}{\overline{\mathbf{H}}}_{SI}+\sqrt{\frac{1}{\delta +1}}{\tilde{H}}_{SI}\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\mathbf{h}}_{SD,k}& =\sqrt{{\gamma }_k}\left(\sqrt{\frac{{\rho }_k}{{\rho }_k+1}}{\overline{\mathbf{h}}}_{SD,k}+\sqrt{\frac{1}{{\rho }_k+1}}{\tilde{\mathbf{h}}}_{SD,k}\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{h}}_{ID,k}& =\sqrt{{\alpha }_k}\left(\sqrt{\frac{{\epsilon }_k}{{\epsilon }_k+1}}{\overline{\mathbf{h}}}_{ID,k}+\sqrt{\frac{1}{{\epsilon }_k+1}}{\tilde{\mathbf{h}}}_{ID,k}\right),\hfill \end{array}$$

(6)

where $\beta ,{\gamma }_k$ and ${\alpha }_k$ are the large-scale path loss coefficients; $\delta ,{\rho }_k$ and ${\epsilon }_k$ are the Rician factors; ${\overline{\mathbf{H}}}_{SI}$, ${\overline{\mathbf{h}}}_{SD,k}$ and ${\overline{\mathbf{h}}}_{ID,k}$ represent the line-of-sight components, which are statistical CSI and remain unchanged over long time. When using the uniform area array model, the line-of-sight components ${\overline{\mathbf{H}}}_{SI}$, ${\overline{\mathbf{h}}}_{SD,k}$ and ${\overline{\mathbf{h}}}_{ID,k}$ can be formulated as

$${\overline{\mathbf{H}}}_{SI}={\mathbf{a}}_N({\theta }_A^a,{\theta }_A^e){\mathbf{a}}_M^H({\phi }_D^a,{\phi }_D^e),$$

(7)

$${\overline{\mathbf{h}}}_{ID,k}={\mathbf{a}}_M({\theta }_{D,k}^a,{\theta }_{D,k}^e),$$

(8)

$${\overline{\mathbf{h}}}_{SD,k}={\mathbf{a}}_N({\phi }_{D,k}^a,{\phi }_{D,k}^e),$$

(9)

where ${\theta }_A^a$ and ${\theta }_A^e$ deonte the azimuth and elevation angles of arrival at the RIS from the BS, respectively; ${\phi }_D^a$ and ${\phi }_D^e$ denote the azimuth and elevation angles of departure from the BS to the RIS, respectively; ${\theta }_{D,k}^a$ and ${\theta }_{D,k}^e({\phi }_{D,k}^a,{\phi }_{D,k}^e)$ are the the azimuth and elevation angles of departure from the RIS to the k-th user (from the BS to the k-th user). These angles are randomly generated. In addition, the array response vector is defined as

$$\begin{array}{c}\hfill {\mathbf{a}}_X\left({\vartheta }^a,{\vartheta }^e\right)=[1,\cdots ,e^{j2\pi \frac{d}{\lambda }\left(x sin{\vartheta }^a sin{\vartheta }^e+y cos{\vartheta }^e\right)},\cdots ,\\ \hfill e^{j2\pi \frac{d}{\lambda }\left((\sqrt{X}-1) sin{\vartheta }^a sin{\vartheta }^e+(\sqrt{X}-1) cos{\vartheta }^e\right)}{]}^T,\end{array}$$

(10)

where X can be substituted as M, N or K and ${\theta }_A^a$ and ${\theta }_A^e$ deonte the azimuth and elevation angles, respectively; d and $\lambda$ represent the antenna spacing at the BS and the wavelength, respectively.

Moreover, ${\tilde{\mathbf{H}}}_{SI}\sim \mathcal{CN}(0,{\mathbf{R}}_{HR}\otimes {\mathbf{R}}_{HB})$, ${\tilde{\mathbf{h}}}_{SD,k}\sim \mathcal{CN}(0,{\mathbf{R}}_{hB,k})$ and ${\tilde{\mathbf{h}}}_{ID,k}\sim \mathcal{CN}(0,{\mathbf{R}}_{hR,k})$ represent the non-line-of-sight component with ${\mathbf{R}}_{HR}$, ${\mathbf{R}}_{HB}$, ${\mathbf{R}}_{hB,k}$ and ${\mathbf{R}}_{hR,k}$ being the corresponding spatial covariance matrices, which are given by ${\left. [{\mathbf{R}}_{HB}\right]}_{i,j}={\rho }^{\left|i-j\right|}$, ${\left. [{\mathbf{R}}_{HR}\right]}_{i,j}={\rho }^{\left|i-j\right|}$, ${\left. [{\mathbf{R}}_{hB}\right]}_{i,j}={\rho }^{\left|i-j\right|}$ and ${\left. [{\mathbf{R}}_{hR}\right]}_{i,j}={\rho }^{\left|i-j\right|}$ and $\rho$ represents the correlation coefficient.

$$\begin{array}{c}{\left. [{\mathbf{a}}_X\left({\theta }^a,{\theta }^e\right)\right]}_x=exp\left\{2\pi j\frac{d}{\lambda }\left. [\frac{x-1}{\sqrt{X}}sin{\theta }^{a}sin{\theta }^e+\left(\left(x-1\right)\mathrm{mod}\sqrt{X}\right)cos{\theta }^e\right]\right\}.\end{array}$$

(11)

$$\begin{array}{c}{\mathrm{SIN}\mathrm{R}}_k=\frac{|({\mathbf{h}}_{ID,k}^H\mathbf{\Phi }{\mathbf{H}}_{SI}+{\mathbf{h}}_{SD,k}^H){\mathbf{w}}_k{|}^2}{{\sum }_{i=1,i\ne k}^K\left(1+k_B\right){\left|({\mathbf{h}}_{ID,k}^H\mathbf{\Phi }{\mathbf{H}}_{SI}+{\mathbf{h}}_{SD,k}^H){\mathbf{w}}_i\right|}^2+\left(1+k_B\right){\sigma }_k^2+{\Gamma }_k\left(\mathbf{w},\mathbf{\Phi }\right)}.\end{array}$$

(12)

$$\begin{array}{c}{\Gamma }_k\left(\mathbf{w},\mathbf{\Phi }\right)={\mathbf{h}}_{ID,k}^H\mathbf{\Phi }{\mathbf{H}}_{SI}+{\mathbf{h}}_{SD,k}^H\left. [k_B{\mathbf{w}}_k{\mathbf{w}}_k^H+(1+k_B)k_{s}diag\left\{\sum _{i=1}^K{\mathbf{w}}_i{\mathbf{w}}_i^H\right\}\right]\left({\mathbf{h}}_{ID,k}{\mathbf{\Phi }}^H{\mathbf{H}}_{SI}^H+{\mathbf{h}}_{SD,k}\right).\end{array}$$

(13)

Thus, the signal received at the k-th user can be written as

$$\begin{array}{cc}\hfill & y_k=\underset{\mathrm{reflected}\mathrm{link}}{\underbrace{{\mathbf{h}}_{ID,k}^H\mathbf{\Phi }{\mathbf{H}}_{SI}\left({\mathbf{w}}_k{s}_k+{\mathit{\eta }}_s\right)}}+\underset{\mathrm{direct}\mathrm{link}}{\underbrace{{\mathbf{h}}_{SD,k}^H\left({\mathbf{w}}_k{s}_k+{\mathit{\eta }}_s\right)}}\hfill \\ \hfill & +\underset{\mathrm{multiuser}\mathrm{interference}}{\underbrace{\sum _{i=1,i\ne k}^K{\mathbf{h}}_{ID,k}^H\mathbf{\Phi }{\mathbf{H}}_{SI}{\mathbf{w}}_i{s}_i}}+\underset{\mathrm{receiver}\mathrm{HWI}}{\underbrace{{\mathit{\eta }}_B}}+\underset{\mathrm{noise}}{\underbrace{{\mathit{\eta }}_k}}\hfill \\ \hfill & =\widetilde{y_k}+{\mathit{\eta }}_B.\hfill \end{array}$$

(14)

where $\mathbf{\Phi }=error\left(e^{j{\theta }_1},e^{j{\theta }_2},\cdots ,e^{j{\theta }_N}\right)$ denote the phase shift matrix at the RIS and ${\theta }_i$ is the phase shift of the i-th reflecting element; ${\mathit{\eta }}_B\sim \mathcal{CN}(0,k_{B}E\left\{{\left|\widetilde{y_k}\right|}^2\right\})$ deontes the user’s additional distortion noise, which satisfies the Gaussian distribution with zero mean and $k_B\in \left(0,1\right)$ denotes the normalized variance of the received distortion noise; ${\mathit{\eta }}_k\sim \mathcal{CN}(0,{\sigma }_k^2)$ denotes the additive Gaussian white noise by the k-th user.

Therefore, the k-th user’s instantaneous signal-to-interference-plus-noise ratio (SINR) is given by (12) and (13) on the next page. Based on (12) and (13), the instantaneous data rate of the k-th user can be expressed as

$$R_k={log}_2\left(1+{\mathrm{SINR}}_k\right).$$

(15)

Therefore, the optimization problem in this paper can be written as

$$\begin{array}{cc}\hfill & \underset{\mathbf{W},\mathbf{\Phi }}{max}\underset{k}{min}\mathbb{E}\left[R_k\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathrm{C}1:\mathrm{tr}\left\{\mathbf{W}{\mathbf{W}}^H\right\}\le P_{max},\hfill \\ \hfill & \mathrm{C}2:|{\theta }_i|=1,\forall i=1,2,\dots ,N,\hfill \end{array}$$

(16)

where the expectation in the objective function is taken over the nonline-of-sight components in the CSI. In the above optimization problem, C1 represents the power constraint at the BS, while C2 means the unit modulus constraints of the phase shifts of the RIS. Unfortunately, it is challenging to derive the closed-form expression of the objective function since the average data rate contains the expectation operation over numerous random small-scale channel gains. In addition, this work studied the impact of hardware impairment on both the BS and the users. The average data rate expressions would be much more complicated. Hence, there are no existing mathematical algorithms that can solve these kinds of optimization problems.

3. Proposed Algorithm

In this section, we propose a statistical CSI-based transmission scheme where the DDGP algorithm is adopted to solve the optimization problem.

3.1. Transmission Scheme

For the existing transmission schemes for RIS-assisted communication systems, the instantaneous CSI is adopted to adjust the beamforming and RIS phases shift, which requires channel estimation in each channel coherence time interval, as shown in Figure 2. However, this method has some drawbacks, as summarized as follows. For the instantaneous CSI-based scheme, the beamforming matrix and phase shift matrix need to be calculated in channel coherence interval, which increases the computational complexity. Furthermore, phase shifts of the RIS need to be updated frequently and sent back to the RIS controller, which incurs significant feedback overhead.

To address this issue, in this paper, we consider the design of the transmission scheme based on statistical CSI. As shown in Figure 2, for the statistical CSI-based scheme, the BS only needs to estimate the statistical CSI at the start of the transmission, and the rest of several channel coherence time intervals will be fully used to transmit the information, which significantly reduces the computational complexity and feedback overhead. Once the network is trained, it can be directly applied in real-time with only simple mathematical calculations. The neural networks only need to be retrained once the statistical CSI changes.

3.2. DDPG Algorithm

As the objective function in (16) does not have a closed-form expression, it is difficult to solve this problem using conventional optimization algorithms. Therefore, in this paper, we adopt a deep reinforcement learning algorithm, which can efficiently process complex environmental parameters and a large amount of state information by utilizing techniques such as stochastic gradient optimization and inverse parameter transfer in deep neural networks. Specifically, the DDGP algorithm is employed to solve the optimization problem in this paper, which is one of the deep reinforcement learning algorithms. The DDPG algorithm can be used to solve this challenging optimization problem with continuous variables.

The DDPG algorithm adopts the Actor-Critic architecture, which uses the policy network to output deterministic actions directly, and the functions of its four networks are introduced as follows:

(1)

$ActorCurrentnetwork$

The role of the actor current network is to iteratively update the policy network parameters $\theta$ and select the current action according to the state $S^{\left(t\right)}$ at time step t, which is composed of three parts: the beamforming matrix ${\mathbf{W}}^{\left(t\right)}\in {\mathbb{C}}^{M\times K}$, the phase shift matrix ${\mathbf{\Phi }}^{\left(t\right)}\in {\mathbb{C}}^{N\times N}$ and the channel matrices, i.e., ${\mathbf{H}}_{SI}^{\left(t\right)}\in {\mathbb{C}}^{N\times M}$, ${\mathbf{h}}_{SD,k}^{\left(t\right)}\in {\mathbb{C}}^{M\times 1}$, ${\mathbf{h}}_{ID,k}^{\left(t\right)}\in {\mathbb{C}}^{N\times 1}$. In addition, the actor current network also interacts with the environment to generate $S^{(t+1)}$ and reward $R^{\left(t\right)}$, which can be defined as $R^{\left(t\right)}={\displaystyle \underset{\mathbf{W},\mathbf{\Phi }}{max}}{\displaystyle \underset{k}{min}}\mathbb{E}\left[R_k^{\left(t\right)}\right]$ and $R_k^{\left(t\right)}$ is defined in (15). The expression of loss function $J^{\left(t\right)}\left(\theta \right)$ can be expressed as

$$J^{\left(t\right)}\left(\theta \right)=-\frac{1}{m}{\sum }_{j=1}^m{Q}^{\left(t\right)}\left(s_i,a_i,w\right).$$

(17)

(2)

$ActorTargetnetwork$

The actor target network serves to select the optimal action ${\mathbf{a}}^{(t+1)}$ based on the state $S^{(t+1)}$ at time $t+1$ sampled in the empirical playback pool. The action ${\mathbf{a}}^{\left(t\right)}\in {\mathbb{R}}^{2MK+N}$ is composed of two parts: the first N elements corresponding to the phase shifts of RIS reflecting elements and the remaining $2MK$ elements corresponding to the real part and imaginary part of the beamforming matrix, respectively. We take action ${\mathbf{a}}^{\left(t\right)}$ to optimize the beamforming matrix ${\mathbf{W}}^{\left(t\right)}$ and the phase shift matrix ${\mathbf{\Phi }}^{\left(t\right)}$, and the optimized results can be described as

$${\mathbf{W}}^{(t+1)}=\frac{\sqrt{P_{\max }}{\mathbf{W}}^{\left(t\right)}}{{∥{\mathbf{W}}^{\left(t\right)}∥}_F}.$$

$${\varphi }_n^{a(t+1)}={\varphi }_n^{a\left(t\right)}+a_j^{\left(t\right)}\pi$$

where ${\varphi }_n^{\left(t\right)}=cos\left({\varphi }_n^{a\left(t\right)}\right)+jsin\left({\varphi }_n^{a\left(t\right)}\right)$ is the n-th phase shift in ${\mathbf{\Phi }}^{\left(t\right)}$ and $a_j^{\left(t\right)}$ is the j-th action value in ${\mathbf{a}}^{\left(t\right)}$, $\forall n=j=1,2,\dots ,N$.

The target network parameter ${\theta }^{{\left(t\right)}^{\prime }}$ is periodically copied from the current network parameter ${\theta }^{\left(t\right)}$, which uses the soft update method, and the soft update factor is $\tau$.

$${\theta }^{(t+1)}=\tau {\theta }^{\left(t\right)}+\left(1-\tau \right){\theta }^{(t+1)}.$$

(3)

$CriticCurrentnetwork$

The critic current network is used to iteratively update the value network parameter $w^{\left(t\right)}$ and calculate the current value of $Q\left(S^{\left(t\right)},a^{\left(t\right)},w^{\left(t\right)}\right)$. The target value of $Q^{{\left(t\right)}^{\prime }}$ is given by

$$\begin{array}{c}\hfill y_i=R^{\left(t\right)}+\gamma Q^{\prime }\left(S^{\left(t\right)},a^{\left(t\right)},w^{{\left(t\right)}^{\prime }}\right).\end{array}$$

(18)

The loss function is given by

$$\begin{array}{c}\hfill J\left(w^{\left(t\right)}\right)=\frac{1}{m}{\sum }_{j=1}^m{\left(y_j-Q\left(\varphi \left(S_j^{\left(t\right)}\right),a_j^{\left(t\right)},w^{\left(t\right)}\right)\right)}^2.\end{array}$$

(19)

(4)

$CriticTargetnetwork$

The critic target network aims to calculate the $Q^{\prime }\left(S^{\left(t\right)},a^{\left(t\right)},w^{{\left(t\right)}^{\prime }}\right)$ portion of the target value Q. The network parameter $w^{{\left(t\right)}^{\prime }}$ is periodically copied from $w^{\left(t\right)}$, which uses the soft update method, and the soft update factor is $\tau$:

$$w^{(t+1)}=\tau w^{\left(t\right)}+\left(1-\tau \right)w^{(t+1)}.$$

At the same time, to increase some randomness and increase the coverage of learning in the learning process, the DDPG algorithm adds some noise $\mathcal{N}$ to the selected action A. That is, the expression of the final and interactive action A of the environment is

$$\begin{array}{c}\hfill a^{\left(t\right)}={\pi }_{\theta }\left(S^{\left(t\right)}\right)+\mathcal{N}.\end{array}$$

(20)

The structures of actor and critic networks are shown in

Table 1. Network Structures.

Parameter Networks	Number of Neurons	Activation Function
Actor	128	ReLU
	64	ReLU
	$N+M\ast K\ast 2$	$tanh(·)$
Critic	64	ReLU
	32	ReLU
	1	None

and both of them have three layers of neural networks.

The overall algorithm is summarized in Algorithm 1.

Algorithm 1 The Proposed DDPG Algorithm.

1: Randomly initialize ${\theta }^{\left(t\right)}$, $w^{\left(t\right)}$, $w^{(t+1)}=w^{\left(t\right)}$, ${\theta }^{(t+1)}={\theta }^{\left(t\right)}$. Empty the collection of experience playback D. 2: for I = 1,2,…, T do 3:    Initialize $S^{\left(t\right)}$ as the first state of the current state sequence, and get its eigenvector $\varphi \left(S^{\left(t\right)}\right)$. 4:    Get the action $A^{\left(t\right)}={\pi }_{\theta }\left(S\right)+\mathcal{N}$ in Actor’s current network based on state S. 5:    Perform the action $A^{\left(t\right)}$, get a new state $S^{(t+1)}$, reward $R^{\left(t\right)}$, and determine whether arrive the termination status `end’. 6:    Stores the array $\left\{\varphi \left(S^{\left(t\right)}\right),A^{\left(t\right)},R^{\left(t\right)},\varphi \left(S^{(t+1)}\right),end\right\}$ into the empirical playback set D. 7:    $S^{(t+1)}=S^{\left(t\right)}$ 8:    Get m samples $\left\{\varphi \left(S_j^{\left(t\right)}\right),A_j^{\left(t\right)},R_j^{\left(t\right)},\varphi \left(S_j^{(t+1)}\right),en{d}_j\right\}$, $\forall j=1,2,\dots ,m,$ from empirical playback sets, and calculate the current target $Q^{\prime }s$ value $y_j$: $$y_j=\left\{\begin{array}{c}{R}_j^{\left(t\right)}\hfill \\ R_j^{\left(t\right)}+\gamma Q^{(t+1)}\left(\varphi \left(S_j^{(t+1)}\right),{\pi }_{{\theta }^{\prime }}\left(\varphi \left(S_j^{(t+1)}\right)\right)\right).\hfill \end{array}\right.$$ (21) 9:    Use the mean-variance loss function $$\frac{1}{m}{\sum }_{j=1}^m{\left(y_j-Q^{\left(t\right)}\left(\varphi \left(S_j^{\left(t\right)}\right),A_j^{\left(t\right)},w\right)\right)}^2,$$ to update the Critic’s current network parameter w through the gradient backpropagation of neural networks. 10:    Use $$J\left(\theta \right)=-\frac{1}{m}{\sum }_{j=1}^m{Q}^{\left(t\right)}\left(s_i,a_i,w\right).$$ to update the Actor’s current network parameter $\theta$ through the gradient backpropagation of neural networks. 11:    If $t\%C=1$, update Critic’s target network and parameters of Actor’s target network: $$w^{(t+1)}=\tau w^{(t+1)}+\left(1-\tau \right)w^{(t+1)}.$$ $${\theta }^{(t+1)}=\tau {\theta }^{\left(t\right)}+\left(1-\tau \right){\theta }^{(t+1)}.$$ 12:    If $S^{(t+1)}$ is at termination status, end the current time step’s iteration; otherwise go to Step b. 13: end for

4. Simulation Results

In this section, the performance of the DDPG algorithm-based scheme is evaluated. Firstly, The locations of A and B are set at (0, 0, 30 m) and (100 m, 20 m, 10 m), respectively. Besides, the users are limited to a circle centered at (150 m, 0, 1.5 m) with a radius of 20 m. Other parameters are shown in

Table 2. Simulation Parameters.

Parameter Name	Sign	Parameter Value
Noise power density	${\rho }_n$	−174 dBm/Hz
Channel bandwidth	B	1 MHz
Reference path loss	$P{L}_0$	0–30 dB
Reference distance	$d_0$	1 m
Path loss coeffificients	$\beta$	$2.2$
	${\alpha }_k$	$2.2$
	${\gamma }_k$	$3.75$
Rician factors	$\delta$	3
	${\epsilon }_k$	3
	${\rho }_k$	3
Correlation coefficients	$\rho$	$0.1$
Normalized variance	$k_B$	$0.01$
Normalized variance	$k_S$	$0.01$
Numbers of antennas	M	8
Numbers of users	K	4
Numbers of reflecting elements	N	20–50

.

Based on the table above, we adopt the DDPG algorithm based on statistical CSI. The number of reflecting elements is set to $N=20$, 30, 40, and 50, respectively. For each time step, we use the beamforming matrix $\mathbf{W}$ and the phase shift matrix $\mathbf{\Phi }$ at time step t as the input of the DDPG neural network, and the output will be $\mathbf{W}$ and $\mathbf{\Phi }$ at time $t+1$. In Figure 3, Figure 4, Figure 5 and Figure 6, we illustrate the minimum average user date versus the time steps for different N.

As shown in Figure 3, Figure 4, Figure 5 and Figure 6, when the number of RIS reflecting elements is set to $N=20$, the MAUR (minimum average user date) converges to $0.75$ when the number is set to $N=30$, MAUR converges to about $0.8$. When the number of RIS reflecting elements increases to 40, MAUR increases to $0.9$, and when N is 50, the rate is about $1.1$. Hence, the MAUR increases with the number of reflecting elements. Meanwhile, the simulation results show that, under different conditions, the increase in the number of reflecting elements does not affect the converging speed of the proposed DDPG algorithm.

In Figure 7, Figure 8, Figure 9 and Figure 10, we respectively set $k_B=k_s$ as 0.01–0.15 to explore the convergence of MUAR under different conditions. It indicates that the MAUR decreases with $k_B$ and $k_s$. This conclusion is also consistent with the SINR formula in (13). Furthermore, when $k_B,k_s$ are set to 0, the result is identical to the situation when hardware impairments are not considered.

In addition, considering the influence of wind and rain in nature, we have added a random variable on the channel angles (angle of departure and angle of arrival), where the random variable is assumed to follow the uniform distribution. In general, the uniform distribution can be regarded as the worst case since the variables are uniformly distributed rather than peaking at one point for the Gaussian distribution. Then, we use the trained solution obtained from our DDPG networks for the realistic channels with angle variance to demonstrate the effectiveness of our algorithm. As observed from Figure 11, we can find that the performance degradation due to the channel variations is not too much, which confirms the robustness of our proposed algorithm.

Finally, in Figure 12, we compare the performance of the proposed algorithm with the existing non-optimized algorithm to evaluate the effectiveness of the optimization operations. In specific, for the non-optimized algorithm, the beamforming vector at the BS is randomly generated and the phase shift matrix is set to a unit matrix. It is observed from Figure 12 that the proposed algorithm significantly outperforms the existing non-optimized algorithm.

5. Conclusions

In this paper, we studied the downlink IRS-aided multiuser MISO system with imperfect hardware, which is based on statistical CSI design. The DDPG algorithm was applied to optimize the beamforming matrix at the BS and the phase shifts matrix at the RIS jointly. Furthermore, the transceiver hardware impairment was also considered to solve the problem of inevitable hardware loss in practical systems. The simulation results demonstrated that it is necessary to take into HWI, and the DDGP algorithm can achieve excellent performance.