Risk Analysis of Airplane Upsets in Flight: An Integrated System Framework and Analysis Methodology

Abstract

Generally, airplane upsets in flight are considered a precursor to loss of control in flight (LOC-I) accidents, and unfortunately LOC-I is classified as the leading cause of fatal accidents. To further explore the risk factors, causal relationships, and coupling mechanism of airplane upsets, this study proposed a risk analysis model integrating the Interpretative Structural Modeling (ISM) and Bayesian Network (BN). Seventeen key risk factors leading to airplane upsets were identified through the analysis of typical accident cases and the literature. The ISM approach was used to construct the multi-level interpretative structural model of airplane upsets, which could reveal the causal relationship among various risk factors and risk propagation paths. Then, taking 286 accident/incident investigation data as training samples, a data-driven BN model was established using machine learning for dependency intensity assessment and inference analysis. The results reveal that the interaction among risk factors of fatal accidents caused by airplane upsets is more significant than that of non-fatal accidents/incidents. Risk factors such as pilot-induced oscillations/airplane-pilot coupling and non-adherence to Standard Operating Procedures (SOPs)/neglect of cross-validation have a significant effect on airplane upsets in flight among seventeen risk factors. Moreover, this study also identifies the most likely set of risk factors that lead to fatal accidents caused by airplane upsets. The research results have an important theoretical significance and application value for preventing airplane upsets risk.

1. Introduction

An airplane upset was previously defined as a flight condition where the pitch of the airplane exceeded +25/−10 degrees or a bank angle exceeded ±45 degrees or the airplane flew within the aforementioned parameters but at inappropriate airspeeds [1]. However, the civil aviation industry considers this definition flawed because an aircraft’s controllability is related to its altitude. For example, an airplane may be controllable at low altitudes even with a bank angle of 30 degrees but may cause an accident at high altitudes [2]. As a result, the International Civil Aviation Organization (ICAO) has revised the definition of the airplane upset as follows: an unexpected airplane state with the characteristics of unintentional divergences from parameters normally experienced during operations, which may involve pitch and/or bank angle divergences as well as inappropriate airspeeds for the conditions [3]. An upset can even be considered to exist whenever an airplane is diverging from the pilot’s expectations [4]. In case of an airplane upset, the airplane may return to its normal flight state through natural airplane reaction to accelerations, automatic flight system response, or pilot intervention. If protective measures are not implemented positively and correctly, deviations from the normal flight state will become larger and may lead to LOC-I, causing a catastrophic accident [5]. ICAO and International Air Transport Association (IATA) also believe that an airplane upset will gradually evolve into three outcomes: recovery to a normal state, causing new errors, or leading to a LOC-I accident [3,6].

As a serious consequence of airplane upsets, LOC-I has been identified as one of the three high-risk accident categories in global commercial aviation and is even considered the most important cause of fatal accidents [7,8]. According to statistics from Boeing, there were 36 fatal flight accidents in commercial aviation worldwide between 2012 and 2021, resulting in 1738 deaths. Of these accidents, nine were caused by LOC-I and resulted in 757 deaths, accounting for approximately 43.6% of the total mortality rate [9]. IATA has statistically analyzed 777 flight accidents and selected the top 6 accident types based on severity, as shown in Figure 1 [8]. The results show that although LOC-I accidents account for only about 8% of commercial aircraft accidents; they have the highest fatality rate with 94% involving the deaths of passengers or crew members. This makes LOC-I a typical “low probability-high risk” accident type [10]. ICAO believes that although LOC-I accidents are uncommon, they often cause catastrophic consequences, with few survivors. Therefore, ICAO lists the prevention of LOC-I as one of its top safety priorities [1]. In fact, the vast majority of LOC-I accidents are caused by airplane upsets [11]. Therefore, it is very important to study the risk of airplane upsets and explore the risk evolution mechanism for improving flight safety.

The primary objectives of this study are as follows. Firstly, the key risk factors leading to airplane upsets are identified through the analysis of typical accidents and the literature. Secondly, a multi-level interpretative structural model of risk factors leading to airplane upsets is constructed using ISM theory to explore the interactions and causal relationships between various risk factors. Finally, a data-driven BN model is established using 286 commercial transportation aviation and general aviation accident/incident reports as sample data. BN parameters are obtained through machine learning to comprehensively analyze the risk evolution mechanism and coupling effect of risk factors leading to airplane upsets.

2. Literature Review

The civil aviation safety system is complex with risk factors involving employees, all kinds of equipment, the working environment, and the organization [12,13]. The civil aviation industry has reached a consensus that civil aviation accidents are often not caused by a single factor but by defects at various levels of the system in most cases [14,15]. Similarly, the risk factors leading to airplane upsets are complex and changeable. These risk factors interact with each other to form a nonlinear risk dynamic evolution process. Risks may spread simultaneously at multiple nodes of the system and if not intervened promptly may lead to an increasingly large deviation from the desired airplane state over time.

A study of historical accidents has found a variety of causes of airplane upsets, involving mechanical failures, icing, wakes, improper responses, spatial disorientation, and stalls [16]. Hyunseong Lee et al. [5] identify three main factors leading to airplane upsets: airborne equipment failures or design defects (33.3%), icing (21.4%), and inappropriate crew response (14.3%). A statistical report from the National Aeronautics and Space Administration (NASA) shows that pilots losing airplane state awareness (ASA) is closely related to auto flight confusion, invalid source data/loss of air data, distraction/attention allocation, inappropriate control actions, lack of external visual references, and ineffective crew resource management (CRM) [17]. ICAO believes that there are numerous reasons why upsets occur and summarizes them into three categories: environmentally induced, systems-induced, and pilot-induced [3]. The above studies have focused on active risk factors that cause airplane upsets but rarely involve the latent risk factors of management.

Since most LOC-I accidents result from airplane upsets, we have also analyzed the risk factors leading to LOC-I. IATA analyzed data on 64 LOC-I accidents from 2009 to 2018, which resulted in 2462 fatalities. The research results showed that in many cases the triggering factors of LOC-I accidents were the external environment, predominantly meteorological factors. However, pilots’ performance deficiencies often worsen the situation, making it difficult to recover until it was too late. In addition, insufficient training, inadequate safety management systems, and aircraft design defects were also considered potential threats that could lead to LOC-I accidents [10]. Jack Smith et al. [18] analyzed 46 general aviation LOC-I accidents in the UK from 2018 to 2019 based on the precursor sequence analysis model. The results showed that most non-fatal accidents were caused by inappropriate crew response after upsets, insufficient energy management, or improper procedures; fatal accidents were more likely to be irreparable due to dangerous mental or physical conditions or incorrect understanding of the situation, and more than half of the fatal accidents involved decision errors and skill errors.

Due to the severity of the consequences, some organizations and aircraft manufacturers had developed a series of measures and training strategies for airplane upset prevention and recovery [1,3,4,6], aimed at helping flight crews identify and avoid airplane upsets, taking appropriate and timely measures to prevent further divergence, and using airplane maneuvering techniques to recover from airplane upsets.

In summary, the existing research on airplane upsets mainly focused on two aspects: the identification of risk factors and preventive measures. Most studies conducted static and qualitative analysis through statistical historical accidents, but few studies involved the coupling mechanism of causal factors and the dynamic evolution of risks, which might lead to a lack of targeted and prioritized preventive measures. To explore the inherent correlation and coupling effects of airplane upset risk factors, this paper proposed a hybrid model integrating ISM and BN to achieve static analysis and dynamic prediction, which could provide theoretical guidance for airplane upset risk management.

3. Methodology

3.1. Framework of Methodology

To further clarify the evolution mechanism of airplane upsets, the key risk factors are identified based on the analysis of typical accident cases and the relevant literature. Then, the ISM and BN models are combined to construct a comprehensive risk assessment framework. The ISM model is mainly used to determine the interaction between risk factors and the path of risk transmission. The BN model can quantitatively analyze the interaction intensity of various risk factors and conduct a comprehensive risk assessment. The process diagram of the methodology is illustrated in Figure 2.

3.2. Method Introduction

3.2.1. Interpretative Structural Modeling (ISM)

Interpretative structural modeling (ISM) is a structured modeling technology for system analysis proposed by Professor Warfield in 1974 [19]. Its basic idea is to decompose a complex system into several subsystems (elements), explore the causal relationship and correlation between various factors with the help of experts’ practical experience and knowledge, and finally, transform the complex system into a multi-level hierarchical relationship model through Boolean logic operations. The biggest advantage of ISM is to transform fuzzy concepts and views into hierarchical topology with good structural relations without losing system functions. The causal hierarchy and correlation of complex system factors can be clearly understood through the hierarchy diagram. Therefore, ISM is more suitable for system analysis with many factors, complex relationships, and fuzzy structure and has become a widely used analysis method in modern system engineering [20,21].

The calculation steps of ISM are as follows [22,23]:

Step 1: Identify the set of factors that affect the system. Usually, these variables can be identified from existing literature or through exploratory research.

Step 2: Determine the adjacency matrix A. For complex systems, there are usually two kinds of relationships between variables: direct influence or indirect influence. The knowledge and experience of experts are usually used to judge whether there is a direct correlation between variables. For a_ij, if variable S_i has an impact on variable S_j, a_ij = 1; if not, a_ij = 0. When i = j, a_ij = a_ji = 0. Then, the adjacency matrix A can express the relationship between n variables of the system by using 1 and 0, as shown in Equation (1).

$$A={\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ a_{21}& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{n1}& a_{n1}& \cdots & a_{nn}\end{array}\right]}_{n\times n}$$

(1)

Step 3: Determine the reachability matrix M. The reachability matrix M refers to the degree to which different variables in a digraph can reach each other through a certain length of the path. The adjacency matrix A and the identity matrix E are carried out matrix power operation until Equation (2) is satisfied; then, the reachable matrix M can be calculated, as shown in Equation (3). In the reachability matrix M, the variable with the value “1” indicates that there is a path from one variable to another; that is, one variable has a direct impact on another variable. The variable with the value “0” indicates that there is no path from that variable to another, and there is no direct relationship between them. The identity matrix E can be presented in Equation (4).

$${(A+E)}^1\ne {(A+E)}^2\ne \cdots \ne {(A+E)}^{K-1}={(A+E)}^K=M$$

(2)

$$M_{ij}={\left[\begin{array}{cccc}{m}_{11}& m_{12}& \cdots & m_{1n}\\ m_{21}& m_{22}& \dots & m_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ m_{n1}& m_{n1}& \cdots & m_{nn}\end{array}\right]}_{n\times n}$$

(3)

$$E={\left[\begin{array}{cccc}1& 0& \dots & 0\\ 0& 1& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & 1\end{array}\right]}_{n\times n}$$

(4)

Step 4: Divide the hierarchy of variables.

The hierarchical division of variables is the basis for establishing a multi-level structure model [24]. Based on the reachability matrix M, the reachability set R(S_i) (the set of all variables that can be reached from S_i), the antecedent set A(S_i) (the set of all variables that can reach S_i), and the common set C(S_i) can be obtained, which can be expressed as Equations (5)–(7).

$$R(S_i)=\left\{S_j|S_j\in A,S_{ij}=1\right\},i,j=1,2,\cdots ,n$$

(5)

$$A\left(S_i\right)=\left\{S_j|S_j\in A,S_{ji}=1\right\},i,j=1,2,\cdots ,n$$

(6)

$$C\left(S_i\right)=R\left(S_i\right)\cap A\left(S_i\right)$$

(7)

Then, the same variables in the reachability set R(S_i) and the common set C(S_i) are found and used as the first level of the ISM. After deleting the rows and columns of all variables of the first level from the reachability matrix M, a new reachability matrix M’ is formed. By repeating the above steps, all variables at each level can be found.

Step 5: Establish a multi-level hierarchical structure model. A multi-level hierarchical structure model can be obtained by layering the variables according to the hierarchical order and then connecting the interaction relationship between the variables with the directed vector line [25].

3.2.2. Bayesian Network (BN)

A Bayesian Network, also known as belief network, is a directed acyclic graph (DAG) model that combines probability theory and graph theory, which is considered a powerful method in the field of uncertainty reasoning. The distinct advantage of a Bayesian Network is its strong ability in uncertainty reasoning, representing the correlation of nodes and updating probabilities [26,27,28,29], so it has been widely used in the field of risk assessment [30,31,32].

A Bayesian Network is composed of nodes, the directed edges connecting these nodes, and the strength of dependencies between nodes, where the nodes represent variables, the directed edges represent the direct causal relationship between link nodes, and the Conditional Probability Tables (CPTs) assigned to nodes specifies the degree of interaction between link nodes [33,34]. A typical Bayesian Network structure is shown in Figure 3, where node A and node B are root nodes and also the parent nodes of node C, node E is the child node of node C, and node D is the leaf node.

Each BN can be represented by N = {(V, E), P}, where “V” is a set of nodes (X₁, X₂, X₃, …, X_n), “E” represents a set of the directed edges, and “P” represents the joint probability of the network. “P” can be expressed in Equation (8).

$$P\left(X_1,X_2,\cdots ,X_n\right)=\underset{i=1}{\overset{n}{{\displaystyle \mathsf{\Pi }}}}P(X_i|parent(X_i))$$

(8)

After obtaining new information (also known as evidence) through investigation, statistical analysis, or other methods, the BN needs to update and revise the prior probabilities of nodes to obtain more realistic probabilities [35].

$$P(A_i|B)=\frac{P(B|A_i)P\left(A_i\right)}{P\left(B\right)}$$

(9)

where P(A_i) and P(B) represent the prior probabilities of occurring for A_i and B, respectively; P(A_i|B) represents the probability of A_i given B; P(B|A_i) is the probability of B given A_i.

3.2.3. Converting ISM Model to BN Structure

To convert an ISM model into a BN structure, the following rules need to be met [36]: (1) nodes transformation. The corresponding variables in ISM model are converted into BN nodes. (2) Structure transformation: the hierarchical structure of each variable in ISM model is converted into BN structure. (3) Directional edges transformation: the causal relationship between variables in ISM model is converted into BN directional edges.

4. Application of the Methodology

4.1. Identifying Key Risk Factors

Airplane upsets are the result of the nonlinear and dynamic evolution of risk under the interaction of various risk factors [14]. To ensure the reliability and accuracy of the study, we mainly adopt two methods to extract the main risk factors that lead to airplane upsets: (1) by sorting and analyzing the research literature in Section 2, the risk factors were preliminarily extracted; (2) retrospective analysis and classification analysis were carried out on the 72 investigation reports of typical accidents or incidents in 2000–2020 (40 commercial transport aviation investigation reports and 32 general aviation investigation reports) caused by airplane upsets. Finally, seventeen key risk factors were identified, involving flight crew, aircraft equipment, flight environment, and organization management, as shown in

Table 1. Key risk factors leading to airplane upsets.

Category	Risk Factors and Code	Brief Description
Flight crew risk factors	Non-adherence to SOPs/neglect of cross-validation (H1)	Standard Operating Procedures (SOPs) define strict operation specifications for flight crew, including checklists and cross-check requirements, which can help the crew effectively manage external threats and possible human errors [25].
	Improper attitude and thrust adjustment (H2)	The pilot must correctly adjust pitch, roll, and thrust to control the aircraft. If the pilot’s control inputs are passive, unplanned, or inappropriate, the aircraft reaction may be completely unexpected [3].
	Loss of situational awareness (H3)	Situational awareness refers to the ability of pilots to correctly perceive and understand external environmental factors and predict future development trends in a specific time and space [25].
	Blind confidence or fluke (H4)	It refers to blind confidence in one’s flight ability or experience, proof of flight ability through risky operation, or self-assumption that everything is normal and insufficient attention to a safety risk or even intentional violation [14].
	Spatial disorientation (H5)	It refers to the phenomenon that the crew members are unable to correctly judge the aircraft’s position or status due to sensory illusion [37].
	Misuse of airplane automation (H6)	Some crew members rely on automation excessively. Once there is a malfunction or uncertainty in the aircraft automation system, they are often helpless or dare not question [38].
	Pilot-induced oscillations/airplane-pilot coupling (H7)	When the pilot’s control input is uncoordinated with the aircraft movement, unpredictable aircraft attitude may be generated, accompanied by pitch or roll oscillation, causing greater deviation [3].
	Pilot incapacitation (H8)	It refers to the crew of mental or physical dysfunction, loss of consciousness, or even sudden death. Although not common, it is worth vigilance [39].
Equipment risk factors	Airborne equipment failure (M1)	Airborne equipment here refers to flight instruments, auto flight systems, flight control, other systems, etc.
	Aircraft design defects (M2)	Aircraft design defects, including design errors, functional errors, or logic errors, will lead to airplane upsets or loss of control of aircraft in a specific situation [40].
Meteorological risk factors	Turbulence (E1)	Turbulence may cause large deviations in airspeed, altitude, or attitude, and severe turbulence may even lead to airplane upsets and/or structural damage.
	Thunderstorm (E2)	Thunderstorms can be accompanied by severe weather, making it difficult for pilots to control the aircraft.
	Icing (E3)	Icing will lead to aircraft surface roughness and airfoil pollution, resulting in a large degradation in airplane performance and unexpected airplane handling [6].
	Wake turbulence (E4)	An encounter with wake turbulence usually results in a rapid but short rolling or pitch moment.
Organizational risk factors	Inadequate training (O1)	Adequate and effective training can increase the skills, situational awareness, and knowledge of the flight crew and help the pilots to correctly manage the aircraft status [15].
	Improper operation plan (O2)	For example, the operating pressure and workload are too high; the crew is not matched properly; the flight is executed with hidden aircraft, or the operation plan is unreasonable [41].
	Absent or deficient safety management (O3)	Airlines and regulatory authorities have deficiencies in the construction of safety culture, standards and regulations, and supervision.

.

4.2. Construction of ISM

The interaction among various risk factors that lead to airplane upsets is relatively complex, and the direct correlation among the factors is different. To accurately judge the interaction between various risk factors in Table 1, 10 experts in the field of civil aviation safety management were invited to score according to Equation (1), including 4 senior flight captains, 4 civil aviation safety management experts, and 2 scholars in the field of aviation human factors engineering. If experts believed that variable S_i has a direct impact on variable S_j, the score would be 1; otherwise, it would be 0. Based on the scoring results of 10 experts, if 8 or more experts scored 1, the final result was 1; otherwise, it was 0. Finally, the adjacency matrix A could be obtained, as shown in

Table 2. The adjacency matrix A.

	H1	H2	H3	H4	H5	H6	H7	H8	M1	M2	E1	E2	E3	E4	O1	O2	O3
H1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H2	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
H3	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
H4	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H5	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H6	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H7	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H8	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
M1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
M2	1	1	0	0	0	0	1	0	1	0	0	0	0	0	0	0	0
E1	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0
E2	0	0	1	0	1	0	0	0	1	0	0	0	0	0	0	0	0
E3	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0
E4	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
O1	0	1	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0
O2	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0
O3	0	0	0	1	0	0	0	0	0	1	0	0	0	0	1	1	0

.

The reachability matrix M could be obtained by matrix power operation of the adjacency matrix A and the identity matrix E according to Equation (2), as shown in

Table 3. The reachability matrix M.

	H1	H2	H3	H4	H5	H6	H7	H8	M1	M2	E1	E2	E3	E4	O1	O2	O3
H1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
H2	0	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
H3	1	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0
H4	1	1	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0
H5	1	0	1	0	1	0	1	0	0	0	0	0	0	0	0	0	0
H6	1	1	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0
H7	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
H8	1	0	1	0	0	0	1	1	0	0	0	0	0	0	0	0	0
M1	0	1	0	0	0	0	1	0	1	0	0	0	0	0	0	0	0
M2	1	1	0	0	0	0	1	0	1	1	0	0	0	0	0	0	0
E1	1	1	1	0	0	0	1	0	1	0	1	0	0	0	0	0	0
E2	1	1	1	0	1	0	1	0	1	0	0	1	0	0	0	0	0
E3	1	1	1	0	0	0	1	0	1	0	0	0	1	0	0	0	0
E4	1	0	1	0	0	0	1	0	0	0	0	0	0	1	0	0	0
O1	1	1	1	0	0	1	1	0	0	0	0	0	0	0	1	0	0
O2	1	1	1	0	0	0	1	0	1	0	0	0	0	0	0	1	0
O3	1	1	1	1	0	1	1	0	1	1	0	0	0	0	1	1	1

.

According to step 4 in Section 3.2.1, all risk factors were selected and classified in turn, and the results are shown in

Table 4. Risk factor hierarchy results.

Levels	R(Si) = C(Si) = R(Si)∩A(Si)
L1	{H1, H7}
L2	{H2, H3}
L3	{H4, H5, H6, H8, M1, E4}
L4	{M2, E1, E2, E3, O1, O2}
L5	{O3}

.

According to Table 3 and Table 4, a multi-level interpretative structural model of risk factors leading to airplane upsets could be drawn, as shown in Figure 4.

4.3. Construction of BN

4.3.1. Determination of BN Topology and Node States

The multi-level interpretive structure model in Figure 4 was mapped into BN structure by Netica 6.0 (NORSYS software corp., Vancouver, BC, Canada), as shown in Figure 5. Node T represented the event of an airplane upset.

For simplification, the states of node T were set as “non-fatal” and “fatal”, and the states of other nodes were set as yes/no based on the characteristic of the nodes.

4.3.2. Determination of BN Parameters

The determination of Bayesian Network parameters is the basis for quantifying the causal relationship and dependency relationship between things, including the prior probability of nodes and the conditional probability table of non-root nodes [42]. There are usually two methods to determine the parameters of Bayesian Network: (1) BN parameters are estimated by experts in the field based on knowledge and experience, and the results are subjective [43]; (2) in the case of obvious conditional dependence of causality, BN parameters can be obtained from training sample data by machine learning, which has the advantage of small deviations [44]. Because we had obtained dependencies among risk factors leading to airplane upsets by ISM model, a second method was used to determine the BN parameters to ensure the accuracy of the results.

Obtaining BN parameters based on the machine learning method has strict requirements for training sample data. We selected 286 accidents/incidents caused by aircraft upsets from the accident database from 1995 to 2023 as training samples, involving 169 commercial transport aviation accidents/incidents and 117 general aviation accidents/incidents. Similarly, these accident reports were extracted from the aviation accident database of the National Transportation Safety Board (NTSB), the Aviation Safety Network, SKYbrary, and the Aviation Safety Information System of China. Among them, 240 accidents/incidents were used as training samples for BN model parameter learning. Another 46 accidents/incidents were used as validation samples for the prediction accuracy of the model. For some accident reports with missing information, we used the EM (Expectation-Maximization) algorithm to quantify BN parameters, which could effectively deal with missing data and hidden variables in the training samples. The training sample data were structured and imported into Netica software to obtain the learning results of BN parameters, as shown in Figure 6.

At the same time, the CPTs of non-root nodes could also be obtained by learning the training sample data. An example of CPT of node T was shown in Figure 7.

4.3.3. Model Verification

To further verify the accuracy of the training model, the data of 46 accidents/incidents were input into the Netica software as validation samples. Then, the output results were compared with the actual rate of the 46 accidents/incidents. The verification results were mainly judged according to the error rate of the model. The error rate of the model was calculated by the following formula [45]:

$$Errorrate=\frac{{\displaystyle \sum Cas{e}_f}}{{\displaystyle \sum Cas{e}_f}+{\displaystyle \sum Cas{e}_t}}$$

(10)

where ${\displaystyle \sum Cas{e}_f}$ refers to the total number of samples whose predicted values are inconsistent with the actual accidents/incidents; ${\displaystyle \sum Cas{e}_t}$ refers to the total number of samples whose predicted values are consistent with the actual accidents/incidents.

The output results were shown in

Table 5. The results of model verification.

Predicted Value		Actual Situation	Error Rate
Non-Fatal	Fatal
27	3	Non-fatal	8.7%
1	15	Fatal

. The calculation results showed that the error rate was 8.7%, indicating that the prediction results of the BN model had certain accuracy, which was basically consistent with the actual situation.

5. Results and Discussion

5.1. Analysis of Multi-Level Interpretative Structural Model

The multi-level interpretative structural model of airplane upsets provides a visualization of the interaction and interdependence among various risk factors, as shown in Figure 4. The structural model is divided into five levels and can guide management priorities and preventive measures. The risk factors in L₁ are considered to be the direct factors leading to airplane upsets, including non-adherence to SOPs/neglect of cross-validation (H₁) and pilot-induced oscillations/airplane-pilot coupling (H₇). These two factors are also recognized as the most direct and perceptible risk factors that lead to airplane upsets. L₂, L_3, and L₄ are in the middle position, including 14 risk factors, which are indirect factors. Such factors are mainly reflected in the high coupling effect of various risk factors of “human-machine-environment-organization”. They play a connecting role in the whole process and are the necessary conditions for airplane upsets and need to be paid attention to. Absent or deficient safety management (O₃) is divided into L₅, which is the deep factor leading to airplane upsets. In other words, absent or deficient safety management (O₃) ultimately leads to airplane upsets through influencing indirect and direct factors, so it is often considered the deepest and most fundamental (but often overlooked) risk factor and should also be given the highest management priority. In addition, some exogenous variables, such as turbulence (E₁), thunderstorms (E₂), icing (E₃), and wake turbulence (E₄), are not pointed to by other risk factors according to Figure 4, indicating that they are less affected by other risk factors. However, it does not mean that these risk factors are not important. In fact, meteorological risks will lead to aircraft failure, wing pollution, and large deviation in airspeed, altitude, or attitude, which will seriously affect aircraft performance and the flight crew’s situational awareness. Since the flight crew cannot change the environment, meteorological risks should be closely monitored, and effective response measures should be taken promptly.

The multi-level interpretative structural model can also reflect the main risk transmission paths (accident chains) of the interaction among the risk factors that lead to airplane upsets, such as O₃ → H₄ → H₁ → airplane upsets. This transmission path can be explained as follows: due to insufficient safety supervision of airlines, some violations of the flight crew cannot be corrected in time. Over time, these violations are rationalized by some pilots, leading to their weak safety awareness [41]. When these pilots face some emergencies, they tend to have a bad psychological state (such as blind confidence and flawed mindset) and will not strictly follow the SOP procedures and cross-check the instrument status of the aircraft [15,46], which may eventually lead to an airplane upset. The diversity of risk transmission paths reveals the complexity of the coupling among risk factors. Managers need to take targeted precautions against these transmission paths, weaken the positive role of each path, and prevent the further evolution of risks.

5.2. Analysis of BN Model

5.2.1. Inference Analysis

The inference analysis of BN includes causal reasoning (the result is derived from the causes) and diagnostic reasoning (when the result is known, find out the possibility of various causes) [45]. This study mainly uses BN’s bottom-up diagnostic reasoning ability. To this end, we set the values of the two states of node T (non-fatal and fatal) to 100%, respectively, to obtain the posterior probability of each node in the two scenarios, as shown in Figure 8.

It can be seen from Figure 7 that the posterior probability of each node T with the “fatal” state is basically greater than that of node T with the “non-fatal” state, which shows that the interaction of various risk factors resulting in the fatal accidents caused by airplane upsets is more significant. At the same time, we can also infer the probability of failure of each node based on the posterior probability.

5.2.2. Sensitivity Analysis

The sensitivity analysis of the Bayesian Network is used to quantify the change of output probability of the target node due to local parameter changes and to identify sensitive nodes in the model [47]. The Netica software measures the sensitivity of nodes by calculating the mutual information, the percent, and the variance of belief as indicators. The larger the value, the more sensitive the node is. To identify the most sensitive risk factors of node T, the T node is set as the target node in the Netica software, and the calculation results are shown in

Table 6. Results of the sensitivity analysis.

Node	Mutual Info (MI)	Percent	Variance of Belief	Node	Mutual Info (MI)	Percent	Variance of Belief
T	0.98818	100	0.2459148	O3	0.00013	0.0129	0.0000434
H7	0.05433	5.5	0.0183792	M1	0.00004	0.00401	0.0000135
H2	0.02428	2.46	0.0082863	H8	0.00002	0.00180	0.0000061
H1	0.01397	1.41	0.0047446	E2	0.00002	0.00153	0.0000052
H3	0.00420	0.425	0.0014359	E1	0.00001	0.000934	0.0000031
O1	0.00162	0.164	0.0005528	O2	0.00001	0.000861	0.0000029
H6	0.00125	0.127	0.0004305	E4	0.00000	0.000492	0.0000017
M2	0.00072	0.0726	0.0002454	H5	0.00000	0.000221	0.0000007
E3	0.00017	0.0175	0.0000590	H4	0.00000	0.000136	0.0000005

.

According to Table 6, it can be seen that pilot-induced oscillations/airplane-pilot coupling (H₇), improper attitude and thrust adjustment (H₂), non-adherence to SOPs/neglect of cross-validation (H₁), loss of situational awareness (H₃), and inadequate training (O₁) are the top five sensitive risk factors for node T. When the values of these five nodes are set to “Yes = 100%” in the Netica software, the result shows that the probability value of node T with the “fatal” state has increased from 43.6% to 70%, as shown in Figure 9. The result of sensitivity analysis shows that strengthening the training of flight crew’s technical skills (ability to control the aircraft) and non-technical skills (situational awareness) and improving the level of flight crew’s safety awareness (adherence to SOPs/cross-validation) have significant effects on preventing airplane upsets.

5.2.3. Most Probable Explanation

A Bayesian Network can be used as the most probable explanation (MPE) function in the Netica software to identify the risk factor set that is most likely to lead to a specific result [48,49]. To identify the risk factor set that is most likely to lead to airplane upset fatalities, the “fatal” state of node T was set to 100%, and then the “Most Probable Explanation” function in Netica software was used to obtain the results. The results show that the most likely risk factor set is {O₃, O₁, M₂, E₃, H₁, H₂, H₃, H₇} when an airplane upset fatality occurs, as shown in Figure 10. This conclusion is highly consistent with the current scientific understanding of airplane upset accidents. Therefore, BN’s reverse reasoning ability can provide a scientific basis for the risk management of airplane upsets.

5.3. Managerial Implications

With the findings, this paper identifies the causal relationships and interaction mechanisms that cause airplane upsets, and the effects of various risk factors are complex and nonlinear, which can serve as a useful reference. To implement targeted training and prevent airplane upsets, the following suggestions are provided.

(1) Compared to meteorological and equipment factors, human factors have a greater impact on the evolution of airplane upsets. Therefore, the flight crew must constantly strengthen their safety awareness and implement active and effective monitoring during flight. As can be seen from Figure 4 and Figure 8, non-adherence to SOPs/neglect of cross-validation is an important cause of airplane upsets. The blind confidence and fluke mentality of the flight crew can easily lead to operational behaviors that deviate from regulations and procedures, and it is necessary to eliminate these negative psychological states. Active and effective monitoring refers to monitoring the external environment, the airplane’s current energy state, the flight performance of other crew members, and flight path trajectory. This can enable crew members to improve situational awareness, predict future aircraft states, promptly detect deviations, and take proactive corrective measures. In short, the flight crew should establish a belief that everyone will make mistakes, even pilots with considerable experience and training, and should maintain vigilance at all times during the flight.

(2) Airlines should develop a comprehensive and standardized evidence-based training (EBT) program for airplane upsets so that the crew members can master upset recovery technologies, including technical skills (energy management of airplanes), non-technical skills (situational awareness and threat and error management), and emergency management capabilities. Once an airplane upset occurs, the flight crew can correctly assess the airplane’s energy state, arrest any flight path deviation, and recover to a stabilized flight path before an extreme upset ever occurs. Using the certified FSTD (Flight Simulation Training Device) for airplane upsets training is an effective method. Various high-fidelity scenarios can be simulated on the simulator devices, such as adverse weather, spiral dive, stall, mechanical failure, etc., to improve the flight crew’s capabilities of upset prevention and recovery. Airlines should make full use of pilots’ daily flight and training data, establish a data-driven early warning mechanism for airplane upsets, find out the ability defects of pilots in time, and provide relevant references for flight training.

(3) Organizational deficiencies are often considered the greatest potential threat to civil aviation safety. Although the risk factors that lead to airplane upsets involve the flight crew, meteorological environment, and equipment, these risk factors are more or less related to deficiencies in organizational management. Airlines should establish the concept of system safety, strengthen supervision, improve the construction of the supervision system, and cultivate an excellent safety culture.

(4) Airborne equipment failure and aircraft design defects are also important risk factors that cannot be ignored, especially aircraft design defects, which can make it difficult for the flight crew to understand and handle emergencies. Therefore, in addition to strengthening regulation, future technology-based solutions are needed, such as automatic energy management systems and improved airplane health monitoring systems.

6. Conclusions

Numerous risk factors lead to airplane upsets, including active risk factors (equipment failure, operation errors, or meteorological factors), as well as latent risk factors (management defects). This study proposes an integrated model based on ISM-BN to identify the key risk factors that lead to airplane upsets and explore the causal relationship and interaction mechanism. This method cannot only systematically and comprehensively analyze the mechanism of airplane upsets but also provide targeted and prioritized preventive measures for airplane upsets risk management. The primary conclusions and contributions are as follows.

(1) Based on the analysis of typical accident cases and the literature, seventeen key risk factors leading to airplane upsets have been identified, involving flight crew, equipment, meteorology, and organization.

(2) The multi-level interpretative structural model of airplane upsets is constructed by using ISM model, which can provide a visualization of the interaction and interdependence among various risk factors. Non-adherence to SOPs/neglect of cross-validation (H₁) and pilot-induced oscillations/airplane-pilot coupling (H₇) is recognized as one of the most direct and perceptible risk factors, and absent or deficient safety management (O₃) is considered the deepest and most fundamental (but often overlooked) risk factor. In addition, the multi-level interpretative structural model can also reflect the multiple risk transmission paths (accident chains), revealing the complexity of the airplane upsets mechanism.

(3) Taking 286 commercial transport aviation and general aviation accident/incident reports as sample data, BN parameters are obtained by machine learning, and a data-driven BN model is established. BN’s powerful reasoning ability is used to conduct an in-depth analysis of the mechanism of airplane upsets, including inference analysis, sensitivity analysis, and MPE analysis. The results show that the interaction among risk factors of fatal accidents caused by airplane upsets is more significant than that of non-fatal accidents/incidents. Among them, pilot-induced oscillations/airplane-pilot coupling (H₇), improper attitude and thrust adjustment (H₂), non-adherence to SOPs/neglect of cross-validation (H₁), loss of situational awareness (H₃), and inadequate training (O₁) are listed as the top five most sensitive risk factors. The MPE analysis results show that the most likely risk factor set leading to an airplane upset fatality is absent or deficient safety management (O₃), inadequate training (O₁), aircraft design defects (M₂), icing (E₃), non-adherence to SOPs/neglect of cross-validation (H₁), improper attitude and thrust adjustment (H₂), loss of situational awareness (H₃), and pilot-induced oscillations/airplane-pilot coupling (H₇).

The ISM-BN method is feasible and effective in analyzing the mechanism of airplane upsets, and the research can be used as an important theoretical basis for formulating the risk management strategy for airplane upsets. However, this study still has some limitations. For the accidents/incidents caused by airplane upsets, we only roughly divided the node states into fatal and non-fatal and did not strictly divide them according to the severity level. This is because BN is a probabilistic model, which requires more extensive basic data to improve the reliability for a more detailed classification of node states. Similarly, the machine learning algorithm also needs a larger sample size to obtain high-precision BN parameters. Although our sample data range has been expanded to 1995–2023, the total number of accidents reported is still limited. Therefore, further research is needed in the following areas: (1) more accident investigation reports should be collected as a supplement to improve the accuracy of BN model; (2) the accidents/incidents caused by aircraft upsets will be divided into commercial transport aviation and general aviation, and more detailed statistical analysis will be carried out according to the severity; (3) new training protocols will be developed based on the above research to prevent airplane upsets and LOC-I.