Maneuver Decision-Making through Automatic Curriculum Reinforcement Learning without Handcrafted Reward Functions

Abstract

Maneuver decision-making is essential for autonomous air combat. However, previous methods usually make decisions to aim at the target instead of hitting the target and use discrete action spaces instead of continuous action spaces. While these simplifications make maneuver decision-making easier, they also make maneuver decision-making more unrealistic. Meanwhile, previous studies usually rely on handcrafted reward functions, which are troublesome to design. Therefore, to solve these problems, we propose an automatic curriculum reinforcement learning method that enables agents to maneuver effectively in air combat from scratch. On the basis of curriculum reinforcement learning, maneuver decision-making is divided into a series of sub-tasks from easy to difficult. Thus, agents can gradually learn how to complete a series of sub-tasks, from easy to difficult without handcrafted reward functions. The ablation studies show that automatic curriculum learning is essential for reinforcement learning; namely, agents cannot make effective decisions without curriculum learning. Simulations show that, after training, agents are able to make effective decisions given different states, including tracking, attacking, and escaping, which are both rational and interpretable.

1. Introduction

Autonomous maneuver decision-making is a sequential decision-making problem. In air combat, the goal of agents is to maneuver according to different states and launch missiles to defeat opponents. Usually, decisions are made by a human pilot. However, with the development of artificial intelligence, there have been many studies regarding programs or algorithms [1,2] as virtual pilots to make decisions in simulation environments, which aims to realize autonomous decision-making for unmanned combat aerial vehicles in the future.

For example, Hu [3] proposed an improved deep Q-network [4] for air combat maneuver decision-making, verifying the feasibility of deep reinforcement learning (RL) for air combat maneuver decision-making. Eloy et al. [5] applied game theory to the confrontation process of air combat and proposed a differential game method combined with the missile attack area [6] in order to attack the static high-value targets. Dantas et al. [7] compared different methods of evaluating the most effective moment for launching missiles during air combat. They found that supervised learning on the basis of simulated data can promote the flight quality in beyond-visual-range air combat and increase the likelihood of hitting the desired targets. Fan et al. [8] used asynchronous advantage actor critic algorithm [9] to address the problem of air combat maneuver decision-making. They proposed a two-layer reward mechanism, including internal rewards and sparse rewards. The simulation results indicated that the method could reduce the correlation between samples through asynchronous training. Wang et al. [10] adopted deep deterministic policy gradient for beyond-visual-range air combat and validated the effectiveness of the method in simulations. Huang et al. [11] applied Bayesian inference and moving horizon optimization to air combat maneuver decision-making. The method adjusted the weights of maneuver decision factors by Bayesian inference theory, and then computed the control quantities by moving horizon optimization. Ide et al. [12] proposed a hierarchical maximum-entropy reinforcement learning with reward shaping on the basis of expert knowledge. This approach achieved the second place among eight competitors in the final DARPA’ s AlphaDogfight Trials event.

Although these studies are valuable and lay foundation for replacing human pilots with computers in real environments, there are still some shortcomings. First, in order to simplify the complexity of decision-making, many studies use discrete action spaces, whereas real action spaces are continuous. Second, in these studies, hitting the target by a missile is usually simplified as aiming at the target. However, in reality, even if the target has been locked, it may not be hit. Finally, these studies use handcrafted reward functions; namely, the agent obtains a non-zero reward at each time step. The disadvantage of handcrafted reward functions is that they may not be necessary, and the design of these functions may cost lots of effort and time.

On the other hand, Graves et al. [13] introduced a method of automatically selecting curricula based on the increase rate of prediction accuracy and network complexity in order to improve the learning efficiency. Matiisen et al. [14] introduced teacher–student curriculum learning, in which the former selects sub-tasks for the latter and the latter attempts to solve these sub-tasks. Automatic curriculum learning methods also produced great performance in addition of decimal numbers [15] and Minecraft [16]. A goal proposal module was introduced in [17]. This method prefers goals that can decrease the certainty of the Q-function. The authors investigated the certainty through thirteen robotic tasks and five navigation tasks and illustrated great performance of the method.

Castells et al. [18] created a new method which automatically prioritizes samples with a little loss to efficiently fulfill the core mechanism of curriculum learning without changing the training procedure. Experimental results on several different computer vision tasks indicate great performance. Self-paced curriculum learning approach is introduced in [19], which takes into account both the prior knowledge known before training and the learning process. Stretcu et al. [20] proposed a new curriculum learning method that decomposes challenging tasks into intermediate target sequences for pre-training the mode. The results showed that the classification accuracy of the method on the data set improved by 7%. Sukhbaatar et al. [21] proposed an automatic curriculum learning method based on asymmetric self-play. The method assigns two different minds to an agent: Alice and Bob. Alice aims to provide a task, and Bob aims to fulfill the task. The core is that Bob can comprehend the environment and fulfill the final task faster by self-play.

Rane used curriculum learning to solve tasks with sparse rewards [22]. The experimental results showed that curriculum learning can improve the performance of agents. Pascal et al. [23] interpreted the curricula as a task distribution sequence interpolated between the auxiliary task distribution and the target task distribution and framed the generation of a curriculum as a constrained optimal transport problem between task distributions. Wu et al. [24] proposed bootstrapped opportunistic adversarial curriculum learning, which opportunistically skips forward in the curriculum if the model of the current phase is already robust. Huang et al. [25] proposed GRADIENT, which formulates curriculum reinforcement learning as an optimal transport problem with a tailored distance metric between tasks.

Therefore, we propose an air combat maneuver decision-making method called automatic curriculum reinforcement learning (ACRL), and the main contributions are: 1. ACRL can make the missile hit the target instead of just aiming at the target. Since hitting the target by a missile is usually simplified as aiming at the target in most previous research. However, even if the target has been locked, it may not be hit. 2. ACRL does not require any human data or handcrafted reward functions, but only uses the results of air combat as the reward function. Because reward design is nonexistent, ACRL can save plenty of time and effort. 3. ACRL uses continuous action spaces in maneuvering decision-making instead of discrete action spaces, which are not realistic. 4. ACRL is verified by ablation studies, and the ability of decision-making of the trained agent is demonstrated by simulations.

2. Method

2.1. Aircraft Model and Missile Model

The aircraft model is listed as follows [26]:

$$\begin{array}{l}\left\{\begin{array}{l}\dot{x}=v\cos \gamma \cos \psi \\ \dot{y}=v\cos \gamma \sin \psi \\ \dot{z}=v\sin \gamma \\ \dot{v}=g(n_x-\sin \gamma )\\ \dot{\gamma }=\frac{g}{v}(n_z\cos \mu -\cos \gamma )\\ \dot{\psi }=\frac{g}{v\cos \gamma }{n}_z\sin \mu \end{array}\right.\\ n_x\in [-0.5,1.5]\\ n_z\in [-3,9]\\ \mu \in [-\pi ,\pi ]\end{array}$$

(1)

where x, y, and z are three-dimensional coordinates of the aircraft. $\gamma$ and $\psi$ are pitch angle and yaw angle, respectively. v represents the aircraft speed. g is the gravitational acceleration. $\mu$, $n_x$, and $n_z$ are control signals. The missile model is [27]:

$$\left\{\begin{array}{l}{\dot{x}}_m=v_m\cos {\gamma }_m\cos {\psi }_m\\ {\dot{y}}_m=v_m\cos {\gamma }_m\sin {\psi }_m\\ {\dot{z}}_m=v_m\sin {\gamma }_m\end{array}\right.$$

(2)

$$\left\{\begin{array}{l}{\dot{v}}_m=\frac{(P_m - Q_m)g}{G_m}-g\sin {\gamma }_m\\ {\dot{\psi }}_m=\frac{n_{mc}g}{v_m\cos {\gamma }_m}\\ {\dot{\gamma }}_m=\frac{n_{mh}g}{v_m}-\frac{g\cos {\gamma }_m}{v_m}\hspace{0.17em}\hspace{0.33em}\hspace{0.33em}\end{array}\right.$$

(3)

where x_m, y_m, and z_m are three-dimensional coordinates of the missile. v_m represents the missile speed. ${\gamma }_m$ and ${\psi }_m$ are pitch angle and yaw angle, respectively. n_mc and n_mh are control signals. P_m, Q_m, and G_m are thrust, resistance, and mass, respectively:

$$P_m=\left\{\begin{array}{l}\begin{array}{ll}{P}_0& t\le t_w\end{array}\\ \begin{array}{ll} 0 & t>t_w\end{array}\end{array}\right.$$

(4)

$$Q_m=\frac{1}{2}\rho v_m^2{S}_m{C}_{Dm}$$

(5)

$$G_m=\left\{\begin{array}{c}\begin{array}{cc}{G}_0-G_{t}t& t\le t_w\end{array}\\ \begin{array}{cc}{G}_0-G_t{t}_w& t>t_w\end{array}\end{array}\right.$$

(6)

where t_w = 12.0 s, $\rho$ = 0.607, S_m = 0.0324, and C_Dm = 0.9. P₀ is the average thrust, G₀ is the initial mass, and G_t is the rate of flow of fuel. K is the guidance coefficient of proportional guidance law.

$$\left\{\begin{array}{l}{n}_{mc}=K\cdot \frac{v_m\cos {\gamma }_t}{g}[\dot{\beta }+\tan \epsilon \tan (\epsilon +\beta )\dot{\epsilon }]\\ n_{mh}=\frac{v_m}{g}\frac{K}{\cos (\epsilon + \beta )}\dot{\epsilon }\end{array}\right.$$

(7)

$$\left\{\begin{array}{l}\beta =\arctan (r_y/r_x)\\ \epsilon =\arctan (r_z/\sqrt{r_x^2+r_y^2})\end{array}\right.$$

(8)

$$\left\{\begin{array}{l}\dot{\beta }=({\dot{r}}_y{r}_x-r_y{\dot{r}}_x)/(r_x^2+r_y^2)\hspace{1em}\hspace{1em}\\ \dot{\epsilon }=\frac{(r_x^2 + r_y^2){\dot{r}}_z - r_z({\dot{r}}_x{r}_x + {\dot{r}}_y{r}_y)}{R^2\sqrt{r_x^2 + r_y^2}}\end{array}\right.$$

(9)

where n_mc and n_mh are control commands of the missile. $\beta$ and $\epsilon$ are yaw angle and pitch angle of line-of-sight, respectively. The line-of-sight vector is the distance vector r, where $r_x=x_t-x_m, r_y=y_t-y_m, r_z=z_t-z_m$, and $R=‖r‖=\sqrt{r_x^2+r_y^2+r_z^2}$. If the missile has not hit the target after 27 s, the target is regarded as missed. If the azimuth angle of the target relative to the aircraft exceeds 60° (off-axis angle) when at the launch time, the target is regarded as missed. The conditions for the end of simulations are: 1. One of both sides in air combat is hit by the missile (the missing distance is 30 m); 2. The missiles of the both sides miss their targets; 3. The simulation time has reached 100 s. Meanwhile, we do not use any handcrafted reward functions; namely, the reward obtained by the agent is 1 if it defeats the opponent, −1 if it is defeated by the opponent, and 0 in other cases.

2.2. Proximal Policy Optimization

Policy gradient method tries to calculate an estimate of policy gradient and uses it for stochastic gradient ascent [28]. Both trust region policy optimization (TRPO) [29] and proximal policy optimization (PPO) [30] are policy gradient methods. PPO is built on TRPO and is more effective. The objective function of TRPO is constrained by the policy update. After linear approximation of the objective and quadratic approximation of the constraint, the problem can be approximately solved by the conjugate gradient method. However, TRPO is complex and incompatible with noise and parameter sharing.

PPO is a better method that achieves the data efficiency and reliable performance of TRPO, and it only uses first-order optimization. Schulman proposed a new target with a clipped probability ratio to generate an estimate of the lower bound of policy performance. The standard policy gradient method performs an update on the sampled data while PPO alternates between sampling data from the policy and performing several times of optimization on the sampled data. Schulman compared different surrogate targets, including not clipped target, clipped target, and KL penalties (with fixed or variable coefficients).

Specifically, PPO modifies the objective function to penalize policy changes that cause the probability ratio to deviate from 1. The objective function of PPO is:

$$\begin{array}{l}{L}^{CLIP}(\theta )={\rm E}\left[\min (r_t(\theta )A_t,\mathrm{clip}(r_t(\theta ),1-\epsilon ,1+\epsilon )A_t)\right]\\ P_r(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{old}}(a_t|s_t)}.\end{array}$$

(10)

By clipping the probability ratio and modifying the surrogate target, PPO can prevent excessive updates and make the optimization process more concise and robust than TRPO.

PPO is used in self-play to train air combat agents. Figure 1 shows the self-play architecture of training agents. An action or maneuver is output by the policy and applied to the air combat environment. Then, a transition consisting of states, actions, and rewards is obtained and stored in the experience pool. After several episodes, transitions are sampled from the experience pool and sent to the learner, where the policy is trained on the transitions by PPO. After training, new policy is used for self-play in the air combat environment to obtain better transitions than previous policies, and these better transitions can be used for training in order to obtain better policies. Therefore, the self-play architecture can gradually improve the decision-making ability.

2.3. Automatic Curriculum Reinforcement Learning

The goal of RL is to explore the environment to maximize returns [31,32,33]. Usually, the agent at the beginning makes decisions randomly because it has not yet been trained. The agent is trained by the RL algorithm with the samples it acquires, which enables it to learn how to obtain more returns. Then, the trained agent can acquire samples with more returns by means of actions it generates, and by repeating this process continuously, the agent can generate actions with more returns. On the other hand, human data can be used to pre-train agents to enable them to make rational decisions at the beginning [34,35] rather than completely random decisions. After that, RL algorithms are used to train agents for better decisions.

Without pre-training, the training process is more concise (collecting effective expert data requires plenty of time and effort, and getting a well-trained agent requires plenty of time and effort as well). Due to the randomness, agents have a certain exploratory nature; that is, they may be able to find more rewards in the environment by random behaviors. On the other hand, the randomness may be disadvantageous. For example, in an environment with sparse rewards, it is difficult for agents to obtain rewards by random behaviors, so it is difficult for the agent to complete the task.

The maneuvering decision problem in air combat can be regarded as a task with sparse rewards. The schematic diagram of air combat maneuver decision-making is shown in Figure 2, which shows three possible results of decisions. The green aircraft and the blue aircraft represent the two sides of air combat. The solid lines represent the flight trajectories generated by the green aircraft, and the gray triangles represent the off-axis angle of the missile. As shown in the two red trajectories of Figure 2, due to the fact that the azimuth angle of the target is larger than the off-axis angle, the missile will miss the target if it is launched. Only if the target azimuth angle is less than the off-axis angle can the missile possibly hit the target, as shown in the green trajectory of Figure 2.

Therefore, it is difficult for agents to hit the target by random decisions. Meanwhile, in the process of training, we found that these agents cannot make effective decisions. To solve this problem, we propose ACRL to train agents. ACRL decomposes the original task into a task sequence from easy to difficult, and then uses PPO to train the agent to complete all tasks in the task sequence, ultimately enabling the agent to complete the original task. In air combat, the initial state is random, including the initial distance, initial velocity, and initial angle. When the initial azimuth angle of the opponent is larger, it is more difficult for the agent to overcome the opponent. Therefore, we propose a method to automatically generate a task sequence to improve the training efficiency.

Concretely, perform several simulations with random initial states at first, where the initial angles and distances are selected within [−1, 1] (the original intervals are [−180°, 180°] and [4000 m, 16,000 m], respectively. For simplicity, these two intervals are normalized to [−1, 1]), and record the initial angles and distances of the simulations in which missiles hit the targets. The maximum and minimum values of these initial angles and distances are a_max, a_min, b_max, and b_min, respectively. The four values form two intervals [a_min, a_max] and [b_min, b_max], which are proper subsets of [−1, 1]. Therefore, the initial angle and initial distance are first selected within these two intervals, which forms the first sub-task easier than the original task; that is, the agent is trained in the two intervals. If the agent is able to make effective decisions in the two intervals, the interval length will be increased, which corresponds to the second sub-task, [a_min − $\delta$, a_max + $\delta$] and [b_min − $\delta$, b_max + $\delta$]. It is easier than the original task but more difficult than the first sub-task. After completing the second sub-task, the length of the interval is changed to [a_min − 2$\delta$, a_max + 2$\delta$] and [b_min − 2$\delta$, b_max + 2$\delta$] until the interval becomes [−1, 1]. $\delta$ is set to 0.1, and if the number of wins is greater than 20 and the number of losses, the interval length will be increased. Otherwise, the interval will not be changed.

2.4. Air Combat State

As shown in

Table 1. Air combat state.

State	Symbol	Formula
yaw angle	$\psi$	$\psi ={\psi }_0+{\displaystyle \int \frac{g}{v\cos \gamma }{n}_z\sin \mu }\mathrm{dt}$
pitch angle	$\gamma$	$\gamma ={\gamma }_0+{\displaystyle \int \frac{g}{v}(n_z\cos \mu -\cos \gamma )}\mathrm{dt}$
velocity	v	$v=v_0+{\displaystyle \int g(n_x-\sin \gamma )} \mathrm{dt}$
altitude	z	$z=z_0+{\displaystyle \int v\sin \gamma }\mathrm{dt}$
distance between the two sides	d	$d=‖r_1-r_2‖$
launch missile	f1	0 or 1
yaw angle of the missile	${\psi }_1$	${\psi }_m={\psi }_{m0}+{\displaystyle \int \frac{n_{mc}g}{v_m\cos {\gamma }_m}}\mathrm{dt}$
pitch angle of the missile	${\gamma }_1$	${\gamma }_m={\gamma }_{m0}+{\displaystyle \int \frac{n_{mh}g}{v_m}-\frac{g\cos {\gamma }_m}{v_m}}\hspace{0.17em}\hspace{0.33em}\mathrm{dt}\hspace{0.33em}$
distance between the missile and the other side	d1	$d_1=‖r_{m1}-r_2‖$
heading crossing angle	$\beta$	$\beta =\arccos (\frac{v_1\cdot v_2}{‖v_1‖‖v_2‖})$
launch missile from the other side	f2	0 or 1

, the input of the neural network is a one-dimensional vector with 11 elements: $\psi , \gamma , v, z, d, f_1, {\psi }_1, {\gamma }_1, d_1, \beta , f_2$. Min–max normalization is used, and hyperbolic tangent function is the activation.

3. Experiments

The effectiveness of the proposed method is verified by ablation studies and simulations in this section. The ablation studies compare the training process of ACRL with the original RL algorithm. Simulations verify the decision-making ability of agents trained by ACRL. For each method, five independent experiments are conducted, and the results are recorded, which include forty iterations. A total of 36 past agents are selected randomly in the test to fight against the current agent. The initial distance and initial angle are randomly selected within the corresponding intervals. Hyperparameters are shown in

Table 2. Hyperparameters.

Hyperparameter	Value
velocity	(250 m/s, 400 m/s)
batch size	1024
optimizer	Adam
actor learning rate	0.0002
critic learning rate	0.001
actor architecture	(256, 256, 4)
critic architecture	(256, 256, 1)
activate function	tanh
epoch	6
$\gamma$	0.99

. We simulate this work in Python. The aircraft model and missile model are written in Python 3.9.7, and the neural networks are trained by Pytorch 1.11.0. Neural networks are fast and light; therefore, we can implement neural networks for practical projects.

3.1. Ablation Studies

Figure 3 shows the changes in the number of wins, losses, and draws in the training process of ACRL, PPO without curriculum (PPO-WC), PPO with reversed curriculum (PPO-RC, from difficult to easy), and SAC [12] with curriculum (SAC-C). The solid line represents the mean of the number of wins, losses, or draws of the corresponding method, and the shaded part represents the standard deviation of the number of wins, losses, or draws.

As shown in Figure 3a, in the training process, the number of wins of ACRL is always greater than that of PPO-WC. The reason why the number of wins of ACRL first decreases and then increases is that the early curricula is easier; thus, the missile is more likely to hit the target, resulting in more wins. As the difficulty of the curricula gradually increases, it becomes difficult for the missile to hit the target, so the number of wins gradually decreases. Therefore, the number of wins of PPO-RC does not increase at the beginning. Since it is difficult to win in reversed curriculum, it is difficult for agents to acquire rewards and improve their abilities, and the number of wins is less than that of ACRL. On the other hand, the agent gradually learns how to make effective decisions, resulting in a gradual increase of the number of wins of ACRL, as shown in Figure 3a. At the same time, more losses also represent better training because both wins and losses are the results of the agent’s self-play.

However, the numbers of wins of PPO-WC, PPO-RC, and SAC-C are much less than that of ACRL at the end of the training, which means that: 1. ACRL is more effective. 2. Both curriculum and PPO are essential. On the other hand, the number of draws for ACRL first increases and then decreases, which is consistent with the results in Figure 3a,b. This is because in early curricula, missiles are more likely to hit the target, resulting in fewer draws. As the difficulty of the curricula increases, it becomes more difficult to hit the target, resulting in more draws. The agent gradually learns effective decisions during training, which results in a decrease of the number of draws.

3.2. Simulation Results

In this section, the maneuvering decision-making ability of the proposed method is verified. We categorize simulations as four different scenarios to emphasize the effectiveness of the proposed method: 1. Two untrained agents. 2. The fortieth agent and the first agent. 3. The fortieth agent and the fifteenth agent. 4. The fortieth agent and the thirtieth agent.

The results of untrained agents, i.e., the results of random decisions, are shown in Figure 4, which shows the three-dimensional trajectory and its top view. The initial distance range is (4000 m, 16,000 m), the initial angle range is (−180°, 180°), and the initial velocity range is (250 m/s, 400 m/s). In Figure 4, the solid line represents the flight trajectory of the aircraft, and the dashed line represents the flight trajectory of the missile. The blue and yellow dots represent the starting points of the two agents, respectively.

In simulation 1, both agents adopt random decisions. The initial yaw angle of Agent 1 is −138.0°, and the initial yaw angle of Agent 2 is −74.4°. From Figure 4b, it can be seen that due to the untrained nature of the agents, they do not make any effective decisions. Agent 1 launches the missile at the beginning of the simulation, while Agent 2 launches the missile after flying forward for some time. However, both missiles miss the target; thus, the simulation ends.

In simulation 2, the initial yaw angle of Agent 1 is 67.6°, and the initial yaw angle of Agent 2 is −60.0°. From Figure 4d, it can be seen that both Agent 1 and Agent 2 choose to launch missiles at the beginning of the simulation, and Missile 2 misses the target. Finally, Missile 1’s flight time exceeds 27 s; thus, the simulation ends. In simulation 3, the initial yaw angle of Agent 1 is −3.3°, and the initial yaw angle of Agent 2 is 36.1°. From Figure 4f, it can be seen that Missile 2 misses the target because of the off-axis angle, while Missile 1 misses the target because of the maximum flight time. As shown in the above scenario, without training, agents maneuver randomly; therefore, they cannot make any rational and effective decisions to hit the target.

Figure 5 shows the results of the fortieth agent and the first agent. Agent 1 represents the fortieth agent, and Agent 2 represents the first agent. In simulation 1, the initial yaw angle of Agent 1 is 56.5°, and the initial yaw angle of Agent 2 is −66.4°. Agent 1 launches the missile after flying for some time. Then, Agent 1 quickly moves away from Agent 2, as this decision can reduce the probability of being hit by Agent 2’s missile. After some time, Agent 2 launches the missile. Finally, Agent 2 is hit by Agent 1’s missile; thus, the simulation ends.

In simulation 2, the initial yaw angle of Agent 1 is 19.2°, and the initial yaw angle of Agent 2 is −50.2°. Agent 1 adopts similar decisions as in simulation 1, which involves first approaching and aiming at the target, then launching the missile and moving away from the target. Meanwhile, Agent 2 is unable to make effective decisions. It launches the missile at the beginning of the simulation, resulting in missing the target. In simulation 3, the initial yaw angle of Agent 1 is −92.0°, and the initial yaw angle of Agent 2 is 101.1°. Agent 1 first turns back and then tracks Agent 2. Due to the long distance between the two agents, Agent 1 does not launch the missile, but Agent 2 keeps flying forward instead of making any effective decisions, and launches the missile without aiming at the target; thus, it misses the target. As shown in the above scenario, the decisions of the fortieth agent are obviously better than that of the first agent, which means that the proposed method is effective.

Figure 6 shows the results of the fortieth agent and the fifteenth agent. Agent 2 represents the fifteenth agent. In simulation 1, the initial yaw angle of Agent 1 is −99.1°, and the initial yaw angle of Agent 2 is −10.4°. Agent 2 does not launch the missile at the beginning, indicating that the agent gradually learns when to launch the missile during the training. Agent 1 first approaches Agent 2 from the side and then launches the missile. Finally, the missile of Agent 1 hits Agent 2.

In simulation 2, the initial yaw angle of Agent 1 is 123.3°, and the initial yaw angle of Agent 2 is 125.0°. Agent 1 launches the missile after targeting Agent 2, but Agent 2 fails to target Agent 1 when launching the missile, resulting in missing the target. Finally, due to Missile 1’s flight time exceeding 27 s, Missile 1 misses the target. In simulation 3, the initial yaw angle of Agent 1 is 100.3°, and the initial yaw angle of Agent 2 is −80.7°. Agent 1 quickly flies away from Agent 2 to avoid being hit by the missile launched by Agent 2. After launching the missile, Agent 2 hardly changes its flight direction and only flies upwards to increase its altitude, so it is hit by Missile 1. As shown in the above scenario, the decisions of the fifteenth agent are better than that of the first agent and still worse than that of the fortieth agent, which means that the agent’s ability gradually improved during training.

Figure 7 shows the results of the fortieth agent and the thirtieth agent. Agent 2 represents the thirtieth agent. In simulation 1, the initial yaw angle of Agent 1 is −155.5°, and the initial yaw angle of Agent 2 is 8.6°. Agent 1 and Agent 2 are facing away from each other at the beginning. Both agents try to maneuver to make the other’s azimuth angle less than the off-axis angle before launching missiles. Agent 1 changes its flight direction after launching the missile to avoid being hit by Agent 2’s missile. Finally, Agent 2 is hit by the missile launched by Agent 1, so the simulation ends.

In simulation 2, the initial yaw angle of Agent 1 is 2.5°, and the initial yaw angle of Agent 2 is 66.1°. Agent 2 is in a status of being pursued by Agent 1. Agent 2 flies forward and launches the missile after some time, but the missile misses the target. Agent 1 first tracks Agent 2 and flies away after launching the missile. Finally, Agent 2 is hit by the missile of Agent 1. In simulation 3, the initial yaw angle of Agent 1 is −63.7°, and the initial yaw angle of Agent 2 is 96.9°. Both agents turn to each other at first, launch missiles, and then stay away from each other after launching. Finally, the missile of Agent 2 hits Agent 1. As shown in the above scenario, the decisions of the thirtieth agent are much better than that of the first and fifteenth agents, and its behavior is very similar to the fortieth agent.

4. Discussion

In this article, agents use continuous action space, starting from completely random decisions, gradually learn how to make effective maneuver decisions by ACRL, and ultimately are able to cope with targets of different situations. Different from previous research, ACRL uses continuous action spaces in maneuver decision-making and regards miss distance as the criterion for air combat rather than whether the target has entered the missile attack zone. Therefore, this study is more in line with reality.

Designing reward functions is a time-consuming job, and unreasonable rewards can prevent agents from effective learning. Different from previous research, ACRL does not require any handcrafted reward function and only uses the results of air combat as reward signals. Meanwhile, according to Figure 3, ACRL can increase the number of wins during training, which indicates that ACRL is a concise and efficient method, and automatic curriculum learning proposed in this article is vital.

At the end of the training, the averages of wins of ACRL, PPO-WC, PPO-RC, and SAC-C are 28.5, 12.0, 21.2, and 15.4, respectively. The averages of draws of ACRL, PPO-WC, PPO-RC, and SAC-C are 63.3, 86.7, 80.0, and 78.9, respectively. These statistics indicate that ACRL is more effective than the other three methods.

As shown in Figure 5b, Figure 6b and Figure 7d,f, agents can make decisions to attack and then move away, which reflects that they have learned to use missiles after training. This ability is acquired through explorations, and there are no handcrafted reward functions throughout the training process to guide the agent in making such decisions, which indicates the effectiveness of ACRL. Meanwhile, these behaviors are interpretable: If one continues to approach the other after launching its missile, it may be hit by the missile launched by the other. Therefore, during the training process, agents learn to stay away from the others after launching missiles in order to reduce the probability of being defeated.

There are also some limitations: 1. The proposed method can only be used in one-on-one air combat. However, real air combat usually contains many planes. Therefore, the method is not suitable for multi-agent systems. 2. The aircraft model is a three-degree model; therefore, it is a simplified and unrealistic model. Meanwhile, more complex and realistic models are more expensive. 3. This paper does not include airplane guns. However, both missiles and guns are common weapons in air combat.

5. Conclusions

In this article, we propose ACRL to solve maneuver decision-making problems, which aims to use missiles to hit targets in different situations or avoid being hit by targets. ACRL has several advantages that previous methods do not have: Its action spaces are continuous rather than discrete, which makes it more realistic. It can make the missile hit the target instead of just aiming at the target. It does not need any handcrafted reward functions, which makes it more concise and efficient. It can make the agent cope with targets in different situations, which indicates its effectiveness, because targets with larger azimuth angles are more difficult to defeat. The decisions made by agents are rational and interpretable. For example, agents learn to attack first and then move away to reduce the probability of being defeated by opponents without handcrafted reward functions.

However, we only investigate one-on-one air combat in this article. In future, we may devote to developing algorithms for multi-agent systems. The multi-agent systems may contain different kinds of models, and these models are more complex and more realistic than the three-degree model.