Online Trajectory Planning Method for Midcourse Guidance Phase Based on Deep Reinforcement Learning

Abstract

Concerned with the problem of interceptor midcourse guidance trajectory online planning satisfying multiple constraints, an online midcourse guidance trajectory planning method based on deep reinforcement learning (DRL) is proposed. The Markov decision process (MDP) corresponding to the background of a trajectory planning problem is designed, and the key reward function is composed of the final reward and the negative step feedback reward, which lays the foundation for the interceptor training trajectory planning method in the interactive data of a simulation environment; at the same time, concerned with the problems of unstable learning and training efficiency, a trajectory planning training strategy combined with course learning (CL) and deep deterministic policy gradient (DDPG) is proposed to realize the progressive progression of trajectory planning learning and training from satisfying simple objectives to complex objectives, and improve the convergence of the algorithm. The simulation results show that our method can not only generate the optimal trajectory with good results, but its trajectory generation speed is also more than 10 times faster than the hp pseudo spectral convex method (PSC), and can also resist the error influence mainly caused by random wind interference, which has certain application value and good research prospects.

1. Introduction

The midcourse guidance phase of the interceptor takes the longest proportion of time in the composite guidance process, which directly determines the range and distance of the interceptor and plays an important role in determining whether the interceptor can have a good interception situation at the initial stage of the terminal guidance phase [1]. However, due to the influence of target maneuver, especially regarding the high-altitude, the high-speed and strong maneuvering targets, midcourse guidance trajectory needs online real-time planning to cope with real-time changing terminal constraints.

At present, the research of trajectory online planning mainly faces the following difficulties: (1) The real-time performance of the online planning method is difficult to guarantee; (2) In order to ensure the efficiency of online planning, it is difficult to generate the real optimal trajectory with high accuracy; (3) Dynamic pressure, heat flow density, overload and other constraints restrict the efficiency of trajectory planning, which brings great difficulties to engineering implementation. In order to solve the difficulties encountered in online trajectory planning, a large number of researchers have conducted in-deep research, References [2,3,4,5,6] introduced convex optimization theory into the online reentry trajectory planning of aircrafts, making it easy to solve the trajectory and providing a new idea for online trajectory planning. However, this method requires a specific simulated flight trajectory as the initial value, which leads to strong restrictions. The trajectory planning problem can also be regarded as an optimal control problem, and the pseudo spectral method is very popular in the numerical solution of optimal control problems [7,8]. Typical methods include the Gauss pseudo spectral method [9], Radau pseudo spectral method [10], etc. References [11,12,13] have applied various improved pseudo spectral methods in optimal trajectory generation design and achieved a good optimization performance. However, the pseudo spectral method cannot be directly applied to engineering practice due to an uncertain solution time [14]. References [15,16] proposed a new hybrid guidance method for the guidance problem of reusable launch vehicles during the descent and landing stages, combining convex optimization theory and the improved transcription of the generalized hp pseudo spectral method. It not only satisfies the constraints acting on the system and reduces optimization time, but also is more accurate than standard convex optimization, providing the possibility for the real-time online resolution of guidance problems. References [17,18,19,20] used an inverse dynamics in the virtual domain (IDVD) method, which transforms the trajectory planning problem from the time domain to the virtual domain by virtual factors and then optimizes the trajectory using the dynamic inversion method. It not only has good real-time and robust performance, but can also meet the process constraints. However, the accuracy problem that restricts the effectiveness of the method has not been well solved.

The foothold of the traditional trajectory online generation algorithm is based on the offline trajectory or discrete waypoints, which strictly limits the effective improvement of the real-time performance of the algorithm and increases the degree of nonlinear violation of the trajectory online planning problem. With the development of telemetry technology and simulation technology, more and more data can be observed and applied in the field of aircraft guidance and control, which provides a new design mode for trajectory online planning methods, that is, trajectory planning methods based on data-driven approaches, which are expected to solve problems that cannot be solved by traditional methods. In recent years, artificial intelligence (AI) has flourished in various fields of society, and is regarded as the main innovative technology to solve high-dimensional complex systems [21], which is the best way to achieve such methods. In particular, the technology of deep reinforcement learning (DRL), which combines reinforcement learning (RL) and deep learning (DL), has attracted extensive attention in various fields and it has been applied in the research of auto driving [22,23,24], UAV path planning [25,26,27] and robot motion control [28,29]. With the development of telemetry technology and simulation technology, more and more data can be observed and applied in the field of aircraft guidance and control, providing a new design mode for trajectory planning methods, namely that of trajectory planning methods based on a data-driven approach, and the DRL is the best way to achieve such methods. As a classical DRL algorithm, the deep Q network (DQN) algorithm [30] first introduced the deep neural network into RL as a function approximator, which enhanced the ability to fit functions, mastered the method of learning control strategies directly from high-dimensional inputs and successfully solved the continuous state space problem, but it could not be applied to the continuous action control and the training effect was unstable. In order to solve the problem of continuous action space, reference [31] proposed a deterministic policy gradient (DPG) algorithm, which used deterministic behavior strategies instead of random strategies, reducing the amount of sampled data and improving the computational efficiency. The deep determination strategy gradient algorithm adds the DQN algorithm idea to DPG algorithm [32] and better solves the problem of the actor–critic network convergence, making the training process more stable. Moreover, it solves the problem that the actor–critic network is difficult to converge, which also increases the stability of the training process. Reference [33] proposed the proximal policy optimization (PPO) algorithm, which mainly aims to improve the efficiency of the algorithm by updating the policy. The advantage of this method is that it can utilize the sampled samples multiple times, to some extent solving the problem of low sample utilization. References [34,35,36] applied the PPO algorithm to planetary power descent and landing phase guidance, missile terminal guidance and hypersonic strike weapon terminal guidance tasks, providing a reference for the application of DRL technology in the field of trajectory planning generation.

The algorithm design of trajectory online planning generation can be used as a control decision problem. The speed, position and part of the target information can be obtained through sensors, which can better describe the state of the dynamic environment. The control quantity acting on the friendly missile can be regarded as an action. Thus, the online trajectory planning problem can be studied under the reinforcement learning framework, and it is expected to overcome some difficulties of traditional algorithms. In addition, applying DRL to solve this problem can jump out of the limitations of traditional models and find potential performance and more excellent online trajectory planning methods. This can also provide a new algorithm model reference for experts in this field.

In conclusion, for the complex and high-dimensional midcourse guidance trajectory planning model, this paper selects the DDPG algorithm to train the interceptor for trajectory planning, and obtains the midcourse guidance online trajectory planning method according to the set training strategy. The main contributions are summarized as follows:

(1)

The Markov decision process (MDP) is designed according to the characteristics of the midcourse guidance trajectory planning model. The reward function is designed using the negative feedback reward contrary to the traditional positive reward, which makes the agent learning more consistent with the interceptor guidance mechanism;

(2)

The trajectory planning task training strategy is set based on the idea of course learning (CL) to deal with the low convergence of the direct training of the DDPG algorithm. The algorithm convergence is significantly improved by replacing the training task from easy to difficult, which is verified by simulation experiments;

(3)

The trajectory planning of midcourse guidance is investigated using the DDPG algorithm. The optimality of the trajectory it solves is similar to the traditional optimization algorithm, which has higher efficiency and is more suitable for online trajectory planning, as verified by simulation experiments.

The work in this paper is arranged as follows. This section describes the research status of the online trajectory generation algorithm and expounds the innovation of this paper. The second section describes the problem of midcourse guidance trajectory online planning. The third section gives the MDP design method. The fourth section gives the DDPG algorithm model. The fifth section describes the online planning method of the medium-guided trajectory. The sixth section performs the simulation verification for the research content. The last section summarizes the contents of this paper and looks forward to the next research direction.

2. Online Trajectory Planning Problem Formulation

2.1. Model Build

The context of this paper includes the ground-based interceptor and high-altitude high-speed strong maneuvering target, as shown in Figure 1.

The whole trajectory of the interceptor is mainly divided into four phases: the boosting phase, midcourse guidance–high throw reentry phase, midcourse guidance–gliding phase and terminal guidance phase. In the boosting phase, the interceptor will be launched vertically, the rocket booster will send the interceptor to the designated altitude quickly and give it a fast flight speed. After the rocket booster completes the task, the booster stage of the interceptor falls off and turns into the midcourse guidance–high throw reentry phase. The interceptor continues to fly up to the high throw point relying on inertia and reenters the adjacent space by adjusting its attitude. In the midcourse guidance–gliding phase, the interceptor glides freely with the help of the aerodynamic rudder of the correction stage and aerodynamic force. After flying to the shift handover window of the midcourse and terminal guidance, the correction stage of the interceptor falls off and enters the terminal guidance phase. The time of the terminal guidance phase is generally short. When the interceptor flies to the predicted hit area, the seeker starts to capture the target and the interceptor intercepts the target under the guidance of the terminal guidance law through the direct/air composite control. The research of midcourse guidance trajectory planning in this paper is mainly focused on midcourse guidance–gliding trajectory.

The simplified midcourse guidance motion model of the interceptor [37] is as follows:

$$\left\{\begin{array}{l}dx/dt=v\cos \theta \cos {\psi }_v\hfill \\ dh/dt=v\sin \theta \hfill \\ dz/dt=v\cos \theta \sin {\psi }_v\hfill \\ dv/dt=-D_a-g\sin \theta \hfill \\ d\theta /dt=\frac{L_a\cos \sigma }{v}-\frac{g\cos \theta }{v}+\frac{v\cos \theta }{h+r_e}\hfill \\ d{\psi }_v/dt=\frac{L_a\sin \sigma }{v\cos \theta }\hfill \end{array}\right.$$

(1)

where $x$, $h$ and $z$ are the coordinates of the interceptor’s center of mass under the northeast sky coordinate system, respectively, $v$ is interceptor speed, $\theta$ is the trajectory inclination angle, ${\psi }_v$ is the trajectory deflection angle, $D_a$ is the drag acceleration, $L_a$ is the lift acceleration, $g$ is the gravity acceleration, $\sigma$ is the angle of bank, $r_e$ is the Earth’s radius with a value of 6371.2 km.

The calculation of $D_a$, $L_a$ and $g$ is as follows,

$$\left\{\begin{array}{l}{D}_{\alpha }=C_{D}pS/m\\ L_{\alpha }=C_{L}pS/m\\ g=g_0{\left(r_e/(r_e+h)\right)}^2\end{array}\right.$$

(2)

where $C_D$ is the drag coefficient, $C_L$ is the lift coefficient, $p$ is dynamic pressure, $S$ is the characteristic area of the interceptor, $m$ is the interceptor mass, $g_0$ is the gravity acceleration at sea level and its value is 9.81 m/s².

The calculation of $C_D$ and $C_L$ is the function formula containing the angle of attack $\alpha$,

$$\left\{\begin{array}{l}{C}_D=0.012-0.01\alpha +0.6{\alpha }^2\\ C_L=-0.04+0.8\alpha \end{array}\right.$$

(3)

The calculation of $p$ is as follows,

$$p=0.5\rho v^2$$

(4)

where $\rho$ is the air density and its calculation formula is as follows,

$$\rho ={\rho }_0{e}^{-h/H}$$

(5)

where ${\rho }_0$ = 1.226 km/s³ is the air density at sea level and H = 7254.24 m is the reference height.

For the convenience of calculation, all the above variables can undergo dimensionless treatment by the normalization method, as follows:

$$\widehat{x}=x/R_e;\widehat{h}=h/R_e;\widehat{z}=z/R_e;\widehat{v}=v/\sqrt{g_0{R}_e};\widehat{t}=t/\sqrt{R_e/g_0};{\widehat{L}}_{\alpha }=L_{\alpha }/g_0;{\widehat{D}}_{\alpha }=D_{\alpha }/g_0/;\widehat{g}=g/g_0$$

(6)

where $\widehat{x}$, $\widehat{h}$ and $\widehat{z}$ represent dimensionless coordinate positions; $\widehat{v}$ represents the dimensionless speed; $\widehat{t}$ represents dimensionless time; ${\widehat{L}}_{\alpha }$, ${\widehat{D}}_{\alpha }$ and $\widehat{g}$ represent dimensionless lift, drag and gravity.

2.2. Problem Formulation

According to the model analysis in Section 2.1, the state equation of the interceptor can be expressed as

$$\dot{s}=f(s(t),u(t),t)t\in [t_0,t_f]$$

(7)

where $s=[x,h,z,v,\theta ,{\psi }_v]$ represents the state variable of the interceptor and $u=[\alpha ,\sigma ]$ represents the control variable of the interceptor.

In order to meet the requirements of fast interception, the performance index is selected as the shortest time to reduce the escape probability of the target and relieve the pressure of long-term target trajectory prediction.

$$J=\min (t_f)$$

(8)

In the midcourse guidance phase, the interceptor flies at high altitude and high speed for a long time, which will inevitably be restricted by the constraints of heat flow density $q$, dynamic pressure $p$, overload $n$ and control input $u$. The optimized trajectory should satisfy the following constraints:

$$\left\{\begin{array}{l}q=C{\rho }^{0.5}{v}^{3.05}\le q_{\max }\\ p=0.5\rho v^2\le p_{\max }\\ n=\sqrt{L_{\alpha }^2+D_{\alpha }^2}/g\le n_{\max }\\ \left|\alpha \right|\le {\alpha }_{\max }\\ \left|\sigma \right|\le {\sigma }_{\max }\end{array}\right.$$

(9)

where C = 1.291 × 10⁻⁴ is the calculation constant of $q$ and ${\square }_{\max }$ represents the maximum limit of the constraints.

In addition, the boundary conditions of the midcourse guidance phase are composed of the high throw reentry point of the trajectory and the position of the predicted handover point. The trajectory planning needs to meet the following boundary conditions

$$\left\{\begin{array}{l}[x(t_0)-x_0,h(t_0)-h_0,z(t_0)-z_0]=0\\ [x(t_f)-x_f,h(t_f)-h_f,z(t_f)-z_f]=0\end{array}\right.$$

(10)

where ${\square }_0$ and ${\square }_f$ represent the initial value and terminal value of the variable, respectively.

The online trajectory planning problem in the midcourse guidance phase can be expressed as follows: under the conditions of Equations (7), (9) and (10), the online trajectory planning with the requirements can be solved, and the value of Equation (8) should be as optimal as possible.

3. MDP Design

The interaction between the agent and environment in DRL follows the MDP, which mainly includes the state set S, the action set A, the reward function R, the discount coefficient γ and the transition probability. This paper uses the modeless DRL method, so the transfer probability is not considered. To facilitate the training and learning of the RL algorithm, the corresponding <S, A, R, γ> is designed based on the background of the midcourse guidance trajectory planning problem.

3.1. State Set

When designing the state set, the principle of taking information that is helpful to solve problems as much as possible and discarding information that may interfere with decisions is followed. According to the guidance mechanism of the interceptor, based on the original interceptor state variable, this paper defines the remaining distance of interceptor $s_{togo}$, the longitudinal plane component ${\eta }_{xh}$ and the transverse plane component ${\eta }_{xz}$ of the velocity leading angle of interceptor; their expressions are

$$s_{togo}=\sqrt{{(x_f-x)}^2+{(h_f-h)}^2+{(z_f-z)}^2}$$

(11)

$$\left\{\begin{array}{l}{\eta }_{xh}={\phi }_{xh}-\theta \\ {\eta }_{xz}={\phi }_{xz}-{\psi }_v\end{array}\right.$$

(12)

$$\left\{\begin{array}{l}{\phi }_{xh}=\arcsin \frac{h_f-h}{\sqrt{{(h_f-h)}^2+{(x_f-x)}^2}}\\ {\phi }_{xz}=\arcsin \frac{z_f-z}{\sqrt{{(z_f-z)}^2+{(x_f-x)}^2}}\end{array}\right.$$

(13)

where ${\phi }_{xh}$ and ${\phi }_{xz}$ are the components of the line-of-sight angle of the interceptor target in the longitudinal and transverse planes, respectively. Therefore, the state set of interceptors is defined as S = $[x,h,z,v,\theta ,{\psi }_v,s_{togo},{\eta }_{xh},{\eta }_{xz}]$.

3.2. Action Set

As the input of the interceptor guidance control, the action set is composed of control variables $\alpha$ and $\sigma$. According to the interceptor control constraint, the control set of the interceptor is defined as A = $[\alpha \in (-{\alpha }_{\max },{\alpha }_{\max }),\sigma \in (-{\sigma }_{\max },{\sigma }_{\max })]$.

3.3. Reward Function

The design of the reward function is the key to affect the learning convergence of the algorithm. In order to make the interceptor find the target point better, this paper not only gives the final reward, but also designs the feedback reward of the corresponding simulation step according to the prior knowledge, that is, ${\eta }_{xh}$ and ${\eta }_{xz}$ should be as small as possible, so that the interceptor can always fly towards the direction near the target. The reward functions are given by

$$\left\{\begin{array}{l}{R}_f=1000-1.1{\displaystyle \sum \Delta t}if{s}_{togo}<\epsilon \\ R_{\Delta t}=-{\eta }_{xh}^{\Delta t}\Delta t-{\eta }_{xz}^{\Delta t}\Delta tifdone=False\end{array}\right.$$

(14)

where $\epsilon$ is constant, $\Delta t$ is the simulation step, ${\eta }_{xh}^{\Delta t}$ and ${\eta }_{xz}^{\Delta t}$ are, respectively, the velocity leading angle component of the corresponding $\Delta t$. $R_f$ represents the final reward obtained by the interceptor when it meets the condition $s_{togo}<\epsilon$, and the final reward is inversely proportional to the sum of the simulation steps, that is, the interceptor is encouraged to complete the task in the shortest time. $R_{\Delta t}$ represents the feedback reward of corresponding $\Delta t$, and its negative value is to encourage the interceptor to keep the speed direction always towards the target so as to optimize the shortest flight time; $done$ represents the termination condition. In combination with the background of interceptor head-on interception mode in practice, $done$ is set as

$$done=\left|{\eta }_{xh}\right|>\pi /2or{s}_{togo}<\epsilon$$

(15)

Combining Equations (14) and (15), the reward function is the sum of the final reward and feedback reward, that is, $R=R_f+R_{\Delta t}$.

3.4. Discount Coefficient

The discount coefficient γ is between 0 and 1, which represents the degree of impact of future rewards on current rewards. γ = 0 refers to the current reward only relying on the reward from the next moment, γ = 1 refers to the future rewards that are equally important for the current state, regardless of how far they are from the present. In practical applications, γ generally does not choose 1 for the following reasons: (1) It is difficult to mathematically ensure the convergence of the formula; and (2) Future behavior is uncertain. Usually, the farther the future behavior is from the present, the less impact it should have on the present. To enable the obtained trajectory to comprehensively consider the overall performance, based on experience, the value of γ is set as 0.99.

4. DDPG Algorithm Model

The DDPG algorithm adopts an actor–critic network architecture, and it creates a mutually independent actor target network (hereinafter referred to as ${\mu }^{\prime }$ network) and the critical target network (hereinafter referred to as the $Q^{\prime }$ network) as copies of the actor online network (hereinafter referred to as the $\mu$ network) and the critic online network (hereinafter referred to as a Q network), respectively, as shown in Figure 2.

The Q network updates the parameter ${\omega }_Q$ using the temporal difference error method of the DQN algorithm, and the loss function L is represented by the minimum mean square error

$$L=\frac{1}{N}{\displaystyle \sum _t{(y_t-Q(S_t,A_t\left|{\omega }_Q\right.))}^2}$$

(16)

where N is the number of samples, $y_t$ is an accurate label of the $Q^{\prime }$ network, which is obtained using the Bellman optimal equation. In order to make the learning process of parameter ${\omega }_Q$ more stable and improve the convergence performance, the ${\mu }^{\prime }$ network is also used in its calculation

$$y_t=R_t+Q^{\prime }(S_{t+1},{\mu }^{\prime }(S_{t+1}\left|{\omega }_{{\mu }^{\prime }}\right.)\left|{\omega }_{Q^{\prime }}\right.)$$

(17)

Based on the standard back propagation method, the parameter ${\omega }_Q$ is optimized and updated according to the gradient ${\nabla }_{{\omega }_Q}L$,

$${\nabla }_{{\omega }_Q}L{\left|s\right.}_t=\frac{1}{N}{\displaystyle \sum _{t}2(Q(S_t,A_t\left|{\omega }_Q\right.)-y_t){\nabla }_{{\omega }_Q}Q(S_t,A_t\left|{\omega }_Q\right.)}$$

(18)

$${\omega }_Q={\omega }_Q+{\lambda }_Q{\nabla }_{{\omega }_Q}L$$

(19)

where ${\lambda }_Q$ is the learning rate of the Q network.

The update of parameter ${\omega }_{\mu }$ in the $\mu$ network follows deterministic strategy,

$${\nabla }_{{\omega }_{\mu }}\mu {\left|s\right.}_t=\frac{1}{N}{\displaystyle \sum _t{\nabla }_A}Q(S_t,A_t\left|{\omega }_Q)\right.{\nabla }_{{\omega }_{\mu }}\mu (S_t\left|{\omega }_{\mu }\right.)$$

(20)

$${\omega }_{\mu }={\omega }_{\mu }+{\lambda }_{\mu }{\nabla }_{{\omega }_{\mu }}\mu$$

(21)

where ${\lambda }_{\mu }$ is the learning rate of the $\mu$ network.

The parameters of the ${\mu }^{\prime }$ network and $Q^{\prime }$ network are updated in the way of moving average

$$\left\{\begin{array}{l}{\omega }_{{\mu }^{\prime }}\leftarrow \zeta {\omega }_{\mu }+(1-\zeta ){\omega }_{{\mu }^{\prime }}\\ {\omega }_{Q^{\prime }}\leftarrow \zeta {\omega }_Q+(1-\zeta ){\omega }_{Q^{\prime }}\end{array}\right.$$

(22)

where $\zeta$ is the update parameter and the value of $\zeta$ is 0.01. This makes the target network parameters update slowly and improves the stability of learning.

To ensure the effective exploration of agents in the continuous space learning process, the DDPG algorithm introduces the Ornstein–Uhlenbeck stochastic process as the random noise $\mho$

$$A_t=\mu (S_t\left|{\omega }_{\mu }\right.)+{\mho }_t$$

(23)

To deal with different dimensional problems, the DDPG algorithm also uses batch standardized DL technology, which can normalize each dimension in small batch samples to obtain the unit mean and variance so as to improve the learning efficiency of the algorithm.

In addition, the DDPG algorithm uses the experience playback mechanism of the DQN algorithm for reference. When generating sample data, the data obtained from the environment will be stored in the memory with a state transition sequence as the unit (S_t, A_t, R_t, S_t+1) to deal with the Markov problem of RL data.

5. Online Trajectory Planning Method for Midcourse Guidance

In this section, based on the midcourse guidance motion model, the MDP design and DDPG algorithm model, the interceptor is trained and the midcourse guidance trajectory planning method is obtained.

5.1. Training Strategy Based on CL

The CL refers to the method of gradually increasing the difficulty of tasks in machine learning to increase the learning speed. Agents are more likely to obtain the final reward in simple tasks, and the idea of CL is to transfer the strategy of completing simple tasks to complex tasks to gradually achieve the desired learning effect. Its advantage is that it can effectively reduce the exploration difficulty of complex tasks and improve the convergence speed of the algorithm.

The midcourse guidance trajectory planning problem is a complex nonlinear optimization problem, which not only needs to meet various constraints, but also needs to ensure the optimal performance as far as possible. If the above problem is not handled and the DDPG algorithm is used directly to learn and train the problem, it is likely to cause the interceptor to be difficult to reach the termination condition and the algorithm cannot converge. Therefore, the CL method is extended to the midcourse guidance phase trajectory training strategy.

Regarding the steps of the training task setting of the strategy, we first arbitrarily give the position of the handover point at the end of midcourse guidance phase; with the termination condition as ${\epsilon }_1$ = 5 km and does not consider process constraints of the trajectory. Furthermore, we consider the process constraints and set the termination condition as ${\epsilon }_2$ = 0.5 km. Note that arbitrarily giving the position of the handover point at the end of midcourse guidance phase is mainly reflected in the changes of $h_f$ and $z_f$ caused by the target maneuver. Process constraints are expressed in the form of termination conditions in the algorithm, as follows

$$\begin{array}{l}ifq>q_{\max }orp>p_{\max }orn>n_{\max }\\ done=True\end{array}$$

(24)

In the training process, the learning task from easy to difficult is set by adding the process constraints and improving the convergence criteria, and the midcourse guidance trajectory planning ability of the interceptor is trained step by step, which has a good effect upon improving the generalization of the model.

Therefore, a training strategy combining the CL idea and the DDPG algorithm is proposed for the midcourse guidance trajectory planning training task. The training network is shown in Figure 3, and the specific training strategy process is as follows:

The low-precision training network without process constraints and the high-precision training network with process constraints are designed;

The interceptor is trained in low-precision network to learn to complete simple tasks and obtain network parameters;

The low-precision network parameters are transferred to the high-precision network, and the interceptor continues to be trained in the high-precision network to learn to complete complex tasks.

5.2. Online Trajectory Planning Method

The main limitation of online trajectory planning is the real-time problem of the algorithm, especially when the interceptor and target speed are very fast, many optimization algorithms do not meet the constraint requirements of the problem. Therefore, this paper applies the above offline training strategy to train the interceptor trajectory planning capability, and proposes an online trajectory planning method based on this. The specific process is as follows:

(1)

Transform the interceptor midcourse guidance trajectory planning model into the corresponding MDP of the DDPG algorithm;

(2)

Based on the DDPG algorithm model, refer to Section 3 for details, and set the offline algorithm training process. The pseudocode is detailed in Algorithm 1.

Algorithm 1. Offline DDPG algorithm training.

Initialize critic network $Q(S,A\left|{\omega }_Q)\right.$ and actor $\mu (S\left|{\omega }_{\mu }\right.)$ with weights ${\omega }_Q$$\mathrm{and}{\omega }_{\mu }$ Initialize target network $Q^{\prime }(S,A\left|{\omega }_{Q^{\prime }})\right.$$\mathrm{and}{\mu }^{\prime }(S\left|{\omega }_{{\mu }^{\prime }}\right.)$ with weights ${\omega }_{Q^{\prime }}\leftarrow {\omega }_Q$$,{\omega }_{{\mu }^{\prime }}\leftarrow {\omega }_{\mu }$ Initialize the memory capacity for episode = 1, M do     Initialize the random process $\mho$ (as exploration noise)     Set the initial state of interceptor $S_0$     Randomly select the target point position $x_f,h_f,z_f$        for t = 1, T do        Select the action of the interceptor $A_t=f_{clip}(\mu (S_t\left|{\omega }_{\mu }\right.)+{\mho }_t)$ according to the current strategy and exploration noise        Execute At, calculate the current reward Rt, integrate the next state St+1          Store the state transition sequence (St, At, Rt, St+1) in the memory        Randomly sample small batch training data of N sequences (St, At, Rt, St+1) form memory          Update ${\omega }_{\mu },{\omega }_Q,{\omega }_{{\mu }^{\prime }},{\omega }_{Q^{\prime }}$ according to the DDPG network model (Equations (16)–(22))        end for end for

(3)

Set the low-precision training network without considering constraints, substitute the interceptor into the offline training process of step (2) and obtain the optimal parameters of the network;

(4)

Set high-precision training network considering constraints as well as substitute the interceptor into the offline training process of step (2), note that the optimal parameters trained in step (3) are taken as the initial parameters of the network and obtain the optimal parameters of the network;

(5)

Based on the offline trained network parameters, the midcourse guidance planning trajectory is quickly generated online. The pseudocode is detailed in Algorithm 2.

Algorithm 2. Online trajectory planning.

Input the initial state of interceptor $S_0$ and the target point position $x_f,h_f,z_f$ Import optimal network parameters ${\omega }_{\mu }^{\ast },{\omega }_Q^{\ast }$ to the network Select the action of the interceptor $A_t^{\ast }=f_{clip}({\mu }^{\ast }(S_t\left|{\omega }_{\mu }^{\ast }\right.))$ according to the optimal strategy ${\mu }^{\ast }$ Integral calculation of the state sequence of the interceptor Output interceptor midcourse guidance planning trajectory

6. Simulation Verification

6.1. Experimental Design

Assuming that the interceptor has completed the trajectory planning of the boost phase and the middle guidance high throw phase before the middle guidance gliding phase, the mass of the interceptor is 900 kg and the characteristic area of the interceptor is 0.7 m². The initial state parameters and process constraint parameters are set as shown in

Table 1. Initial state parameters.

x0/(km)	h0/(km)	z0/(km)	v0/(m·s−1)	θ0/(°)	ψv0/(°)
0	70	0	3000	−5	0

and

Table 2. Process constraint parameters.

nmax/(g)	qmax/(W·m−2)	pmax/(Pa)	|α|max/(°)	|σ|max/(°)
10	1 × 106	1 × 105	30	85

.

The change range of the forecast handover point needs to fully consider the impact of the target maneuver, mainly reflected in the change of h_f and z_f, and the terminal position parameters are set as shown in

Table 3. Terminal position parameters.

xf/(km)	hf/(km)	zf/(km)
338	[20, 40]	[−60, 60]

.

This paper uses Python language for simulation the experiment programming, MATLAB software for simulation drawing and Pytorch module to support DRL. The input layer of the $\mu$ network and ${\mu }^{\prime }$ network has 9 nodes, and the output layer has 2 nodes, the input layer of Q network and $Q^{\prime }$ network has 11 nodes and the output layer has 1 node, all networks contain a hidden layer with 300 nodes, and the learning rate of all networks is set to 0.0001. The Adam Optimizer is used as the optimizer of the network parameters in the simulation. The memory capacity is set to 1 × 10⁶, the number of samples for random N is set to 2000, and the maximum number of workouts is set to 1 × 10⁵. The value of the simulation step in this paper is 1.

In addition, except for the algorithm anti-interference verification in Section 6.5, all other simulation experiments in this article do not add disturbances.

6.2. Algorithm Convergence Verification

Define two training strategies, one of which is the training strategy based on CL, and the other is the direct training strategy. The average reward index (that is, the average of the total rewards of 50 rounds, when the round is less than 50 rounds, use the average of the total rewards of total rounds as the average reward) is used to observe the convergence of the algorithm. The simulation data are shown in Figure 4.

In the simulation, when the average reward reaches 800, the training achieves the desired effect, and the algorithm converges. It can be seen from the simulation results in Figure 4 that when the training strategy based on CL is adopted for training, the DDPG algorithm achieves the training effect when the number of training iterations in the step 1 task training is 4536,and the training effect when the number of training iterations in step 2 task training is 20,368. When the training strategy based on direct training is adopted, the DDPG algorithm is trained to the maximum number of iterations (1 × 10⁵), but the expected effect was not achieved. By comparing the training effects of the two training strategies, the convergence based on the CL training strategy is more prominent. In addition, the training strategy of the direct training significantly declines in the training effect during the late training period, as shown in Figure 4c; this shows that the method of direct training from scratch is not suitable for the complex high-dimensional interceptor midcourse guidance trajectory planning task, and it is easy to lead to the interceptor, which cannot find a feasible solution to the problem.

6.3. Algorithm Effectiveness Verification

In order to verify whether the trajectory generated by the algorithm meets the constraint requirements, the Monte Carlo experiment method is used to conduct simulation experiments on the networks trained in two strategies, the terminal position parameters are randomly generated and 10,000 Monte Carlo tests are conducted, respectively; the simulation data are shown in

Table 4. Test results of two training strategies.

Training Strategy	Success Rate/(%)	Average Reward (Maximum, Minimum)	Standard Deviation
CL training	90.16	852.67 (872.90, −52.77)	214.73
Direct training	40.96	394.84 (854.65, −132.59)	536.65

.

According to the test results in Table 4, the success rate and average reward of the training strategy based on CL designed in this paper are far greater than direct training, and its standard deviation is also less than direct training, which shows that the DDPG algorithm combined with the CL idea has more effective training effect than direct training. In addition, this paper only divides the trajectory planning task into two steps of training, whilst the stage training task can be further refined in the future to improve the success rate and convergence of the training strategy.

6.4. Algorithm Optimization Performance Verification

In order to verify the optimization performance of the method in this paper, the trajectory online planning method in this paper (CLDDPG) is used to optimize the trajectory online, and the terminal position parameters are random. At the same time, under the same simulation conditions, the hp pseudo spectral convex method (PSC) in the literature [15] is used to compare with the method in this paper, and the PSC method selects 100 nodes. The simulation data are shown in Figure 5 and

Table 5. Comparison of simulation results of two methods.

Method	Flight Time of Simulated Trajectory/(s)	Time Required for Trajectory Simulation/(s)
CLDDPG	118	0.15
PSC	119.09	1.62

.

It can be seen from Figure 5 that the two optimized trajectories can meet the guidance requirements (see Figure 5a). The final speed of the two optimized trajectories are almost the same (see Figure 5b). Compared to the trajectory inclination angle (see Figure 5c), the difference in their trajectory deflection angle is greater (see Figure 5d), because their angle of attack changes have the same trend (see Figure 5e), the bank angle of the PSC trajectory has always been positive, while the bank angle of the CLPPDG trajectory is negative for a period of time in the later stage (see Figure 5f), which directly leads to a much larger increase in the trajectory deviation angle of PSC compared to CLDDPG. They can all meet the constraints of heat flux density (see Figure 5g), overload (see Figure 5h), and dynamic pressure (see Figure 5i), but the heat flux density of the PSC trajectory is relatively high, which is caused by the lower height of the PSC trajectory compared to the CLPPDG trajectory. It can be seen from Table 5 that the flight time of the CLPPDG trajectory is almost the same as that of the GPM optimized trajectory, but CLPPDG is far superior to PSC in the time required for optimizing the trajectory, which may benefit from CLPPDG’s extensive offline training experience. The simulation results show that the online planning method based on deep reinforcement learning proposed in this paper can not only generate a trajectory with good optimization performance online, but also have excellent real-time performance.

6.5. Algorithm Anti-Interference Verification

In the course of an interceptor midcourse guidance flight, it will be affected by many interference factors, such as atmospheric random interference, measurement noise, etc. To verify the anti-interference of the method in this paper, random wind interference is added to the experiment in Section 6.4. The interference model is shown in Equation (25).

$$\tilde{A}=A+\Gamma (2rand(1)-1)$$

(25)

where $\tilde{A}$ is the control variable after interference, $rand(1)$ represents a function that generates random numbers between 0 and 1, $\Gamma$ is the interference intensity and the $\Gamma$ value selected in this paper is 2°. The simulation results are shown in Figure 6.

It can be seen from Figure 6 that the interceptor can still meet the guidance requirements with the addition of random wind interference (see Figure 6a). The change in speed (see Figure 6b) shows slight fluctuations, and there is a slight deviation in the final values of the trajectory inclination angle (see Figure 6c) and trajectory deviation angle (see Figure 6d), all of which are within a controllable range and almost do not affect the trajectory. Due to the direct effect of random wind interference on the control variables, the fluctuations in the angle of attack (see Figure 6e) and angle of bank (see Figure 6f) are more pronounced. At the same time, it can be clearly seen that the trajectory with increased random wind interference can still meet the constraints of heat flux density (see Figure 6g), overload (see Figure 6h) and dynamic pressure (see Figure 6i). The simulation results show that the online trajectory planning method based on deep reinforcement learning proposed in this paper has strong anti-interference characteristics.

7. Conclusions

This paper aims to solve the online planning problem of the interceptor midcourse guidance trajectory, and the main conclusions of this paper are as follows:

(1)

An MDP designed according to the characteristics of the midcourse guidance trajectory planning model can enable the interceptor to use the DDPG algorithm for offline learning training;

(2)

The training strategy based on CL is proposed, which greatly improves the training convergence of the DDPG algorithm;

(3)

The simulation results show that the online trajectory planning method based on the deep reinforcement learning proposed method has a good optimization performance, strong anti-interference and an outstanding real-time performance.

In addition, with the deepening of learning and training, adjusting the training strategy becomes very important to improve the convergence of the algorithm. For future research, the training strategy should be further investigated to improve the rate of convergence of the algorithm.