Development of an IoT-Based Solution Incorporating Biofeedback and Fuzzy Logic Control for Elbow Rehabilitation

Abstract

The last few years have seen significant advances in neuromotor rehabilitation technologies, such as robotics and virtual reality. Rehabilitation robotics primarily focuses on devices, control strategies, scenarios and protocols aimed at recovering sensory, motor and cognitive impairments often experienced by stroke victims. Remote rehabilitation can be adopted to relieve stress in healthcare facilities by limiting the movement of patients to clinics, mainly in the current COVID-19 pandemic. In this context, we have developed a remote controlled intelligent robot for elbow rehabilitation. The proposed system offers real-time monitoring and ultimately provides an electronic health record (EHR). Rehabilitation is an area of medical practice that treats patients with pain. However, this pain can prevent a person from positively interacting with therapy. To cope with this matter, the proposed solution incorporates a cascading fuzzy decision system to estimate patient pain. Indeed, as a safety measure, when the pain exceeds a certain threshold, the robot must stop the action even if the desired angle has not yet been reached. A fusion of sensors incorporating an electromyography (EMG) signal, feedback from the current sensor and feedback from the position encoder provides the fuzzy controller with the data needed to estimate pain. This measured pain is fed back into the control loop and processed to generate safe robot actions. The main contribution was to integrate vision-based gesture control, a cascade fuzzy logic-based decision system and IoT (Internet of Things) to help therapists remotely take care of patients efficiently and reliably. Tests carried out on three different subjects showed encouraging results.

1. Introduction

Physical medicine adopts treatment protocols based on repetitive exercises to improve the functioning of the joints. However, these types of repetitive exercises consume a lot of energy and time for the medical staff. Integrating technologies such as robotics and virtual reality into the rehabilitation process can make this strenuous work more affordable. Robotics mainly focuses on the development of systems and control strategies to facilitate the recovery of lost motor skills. Systems dedicated to automated rehabilitation have benefited from numerous technological advances in robotics, such as sensors, actuators and control approaches [1,2,3,4,5,6,7]. In this context, we aimed to take advantage of the progress made in remote control systems, gestural control, IoT-based system design and fuzzy logic-based decision-support systems to design remote elbow rehabilitation solution. Much attention has been given to the development of robotics for teleoperation purposes [8,9,10]. Several human–robot gesture recognition interfaces have been developed [11,12]. The Kinect camera shows excellent vision capabilities. Thus, it has been used in many applications, such as people assistive systems [13,14,15], rehabilitation and stimulation [16,17] and game-based learning systems [18]. The Kinect device is able to recognize human movement. The recognized human patterns are used in several computer environments, such as LabVIEW [11] and MATLAB [19]. In [20], a Kinect gesture recognition approach was proposed to improve the gesture implementation process. Additionally, an eigenspace-based approach was introduced in [21] to identify and recognize human gestures using 3D Kinect. A Kinect-based tracking approach was presented in [22]. This approach is mainly for pick and place applications. Thus, the Kinect sensor offers excellent capabilities to be used for remote handling purposes.

In addition to the vision-based solutions, electromyography (EMG) has been widely used in robotic applications [14,23,24,25,26,27]. A real time telemanipulation technique is presented in [23] using EMG-based arm tracking and markless images. Based on EMG signals a myoelectrically controlled robotic system was proposed for elbow training assistance [14]. In [24], EMG was used to control in real time a robotic arm system. In [25], the authors designed a rehabilitation training system based on virtual reality and EMG feedback. A novel concept of robot therapy using EMG thresholds was presented in [26]. A review on the EMG-based control of the human upper limb motion is provided in [27].

Additionally, Fuzzy control has been extensively used in many robotic applications [28,29,30] because of its power in modeling uncertainty and vague data. Many researches have been focused on fuzzy logic-based control schemes for rehabilitation robots. An adaptive admittance controller based on fuzzy logic has been developed for 2-DOF actuated parallel ankle rehabilitation robot [31]. A PID controller based on a fuzzy inference system for rehabilitation of shoulder flexion/extension has been presented in [32]. A fuzzy-impedance control law was developed to estimate the human–robot interaction force [33]. A robotic system providing fuzzy-based adaptive assistance was presented in [34]. In contrast to the studies cited above, by adopting fuzzy logic as a control law for trajectory tracking [31,32] or for assistance [33,34], the proposed solution integrates the concept of fuzzy logic into the control architecture to build a safety system that takes into account pain that could be felt by the patient. In this study, a fuzzy logic controller was employed with the proposed rehabilitation architecture in order to model patient pain during rehabilitation therapy. Incorporating the fuzzy controller into the developed rehabilitation system gives the system the ability to adapt efficiently and reliably to changes in the rehabilitation conditions, including patients’ conditions.

By using IoT concepts, a system’s parts/objects can be accessed anytime anywhere. Remote rehabilitation can be adopted to relieve stress in healthcare facilities by limiting patient numbers in clinics [35]. Indeed, remote rehabilitation via the Internet allows therapists to control and monitor their patients at home. We note that in the current situation where humanity is fighting against COVID-19, the most used keywords are “stay at home” and “do not touch.” The proposed solution includes these two goals: gesture control, which means no direct contact, and Internet-based control, which means rehabilitation while staying at home.

The main contribution is to integrate vision-based gesture control, a cascade fuzzy logic-based decision system and IoT to help therapists remotely take care of patients efficiently and reliably. In fact, the proposed solution not only allows remote, in-home rehabilitation but also incorporates a fuzzy logic-based decision support system which takes into consideration the pain felt by the patient. The available rehabilitation systems are very large, complicated, immobile, unwieldy and very expensive. In contrast to the previously cited works [1,2,3,36,37], the proposed solution is light and portable since the objective was to design small robots with a well-defined target in order to be accessible for use at home. Compared to [14,23,24,25,26], the developed system is a vision-based telemanipulation solution that avoids the inconveniences of the EMG-based solutions, such as signal sensitivity. Many studies on robotic rehabilitation have not taken into account the pain felt by the patient in the robot control architecture; see [38,39,40,41,42], to name a few. However, this pain can prevent a person from interacting positively with therapy. Continued rehabilitation beyond pain could cause material and moral damage. In addition to the possibility of complicating the situation of the affected member, one could create a state of refusal of rehabilitation by the subject who suffered enormously. To cope with this problem, some studies suggest incorporating a current return, where the peak may reflect resistance exerted by the patient due to a feeling of pain [28]. However, this solution seems limited because the engine starting already presents a current peak which can mislead the decision system. In other works, the authors rely on the detection of muscle contraction to estimate fatigue and therefore pain [43,44]. On the other hand, one cannot avoid certain pain in the advanced steps of rehabilitation. Therefore, the decision-making system must allow the robot to go ahead even in the presence of a current peak. In this situation, we need a combination of inputs and a table of rules. In this developed control architecture, we propose a sensor fusion to estimate the degree of pain. Indeed, the patient’s safety during the therapy is achieved by integrating a cascaded fuzzy logic controller that estimates patient’s pain. EMG signal, current sensor feedback and the position encoder feedback are used to provide the fuzzy controller with the data needed to estimate pain. This measured pain is fed back into the control loop and processed to generate safe robot actions. In most of the research presented in the literature [37,45,46,47], robotic rehabilitation was carried out via a control panel in the robot control station. Gesture control makes the control process easier and more natural. The developed solution applies a complete system that allows the physiotherapist to act by gesture from a distance to control the robot and get visual feedback. The architecture implemented based on the Message Queuing Telemetry Transport (MQTT) protocol solves the delay problem often encountered with remote control systems.

2. Control Design

The idea is to offer a remote assistance service for home rehabilitation. Since the physiotherapist and the patient are at different sites, the system must ensure that the physiotherapist controls and supervises the rehabilitation process and that the patient repeats the same actions performed by the physiotherapist. To this end, a human–machine interface (HMI) has been developed to allow the physiotherapist to remotely control the robot and supervise the rehabilitation process. As shown in Figure 1, based on the Microsoft Kinect Sensor V2 for Windows, the physiotherapist’s skeleton is detected and the elbow joint is calculated. The desired position is acquired by the HMI software in order to calculate the corresponding torque on the basis of an adaptive controller. Then, the torque generated is sent via an MQTT protocol to the robot’s on-board card. The range of motion (RoM) measured by a rotary encoder, the output of the current sensor and the raw EMG signal are sent to the HMI software. TD features are taken directly from the pre-processed EMG and classification is performed by a first fuzzy logic system (FLS) to estimate muscle contraction. A second FLS is implemented to estimate pain based on muscle contraction, current sensor feedback and RoM.

In the proposed control architecture, we used the Microsoft Kinect Sensor V2 for Windows as the generator of the desired range of motion. Kinect v2 is a 3D video camera that provides the distance between the camera and the objects in the scene. The camera has the ability to track, at the same time, up to six skeletons with 25 points for each. Kinect is an active stereoscopic 3D camera with an infrared light source and an infrared camera with a resolution of 512 × 424 pixels. In addition to the stereoscopic 3D vision component, the Kinect device also has a color camera with a resolution of 1920 × 1080 pixels. For each pixel of the image, it finally provides four pieces of information: the three color components and the depth. The sensor sends a stream composed of the distance between the camera plane and the nearest object found. Each pixel of the resulting image contains the given distance expressed in millimeters. Unlike the Kinect v1, the new Kinect employs a time-of-flight, or ToF, camera. ToF cameras function by measuring the time it takes for light emitted by a Kinect to reach an object and come back. Knowing the speed of light in a typical living room atmosphere, the Kinect can calculate the distance between itself and the object. The measurement of depth as a triangulation process is described by the inventors in [48]. To connect the Kinect sensor to LabVIEW, we used the Haro3D library that contains API VIs to access the following features of the Kinect v2:

• Bodies (people and joints tracking);
• Colored 3D cloud of points;
• Depth sensing;
• Color high-definition camera;
• Active infrared imaging;
• 3D volume reconstruction.

Feature extraction is the process of selecting useful information and deleting unwanted EMG parts. In order to correctly extract the information conveyed by the EMG signal, algorithms for signal processing and pattern matching were developed, which perform the extraction of characteristics and the classification of patterns. These algorithms depend directly on the characteristics used to represent the acquired signals. The main features in analysis of the EMG signal are: the time domain (TD), the frequency domain (FD) and the time-frequency or time-scale (TS) representation [49]. The TD features, and unlike the FD and TS features, are the easiest to calculate and implement since they do not require any other additional transformation and they are extracted directly from the raw EMG signal. Indeed, the TS and FD features require additional transformations which impose very complex calculations and consequently dedicated processors. In this paper, the TD features are used as inputs of the first block of the fuzzy logic controller to estimate the muscle contraction. Indeed, the EMG signal is used to extract muscle contraction which is used further to estimate patient pain during the rehabilitation session for safety purposes. EMG-RMS, EMG-SSI, EMG-vorder and EMG-Logdetect features are extracted within an N sample analysis time window, and for each, X_i represents the ith sample of EMG signal [49].

The root mean square (RMS) is modeled as the amplitude-modulated Gaussian random process where the RMS is related to the constant force, and the non-fatiguing contractions of the muscles.

$$RMS=\sqrt{\frac{1}{N}\sum _{i=1}^N{X}_i^2}$$

(1)

The simple square integral (SSI) expresses the energy of the EMG signal as a usable feature:

$$SSI=\sum _{i=1}^N\left|X_i^2\right|$$

(2)

EMG_v-order (V) V is a non-linear detector that computes implicitly the strength of a muscular contraction.

$$V={\left(\frac{1}{N}\sum _{i=1}^N{X}_i^v\right)}^{\frac{1}{v}}$$

(3)

where v is an optimum value has been reported to be 2 in [50] and used as $v=1$ in this work to be equivalent to the integral absolute value (IAV) [49].

The log-detector (logdetect) provides an estimate of the strength of muscle contraction.

$$log\mathit{detect}=e^{\frac{1}{N}{\sum }_{i=1}^{N}log\left(\left|X_i\right|\right)}$$

(4)

In this application, we are only interested in the flexion-extension movement. For this reason, we adopted a 1 DoF dynamic model, as presented by Equation (5).

$$m\ddot{q}+b\dot{q}+g\left(q\right)=\tau$$

(5)

where

• m is the link mass.
• q is the joint variable.
• $\dot{q}$, and $\ddot{q}$ are the joint velocity and acceleration.
• b is the dynamic coefficient of friction.
• g is the gravitational force.

Equation (5) can be written as:

$$m\dot{z}=y\vartheta -\tau -bz$$

(6)

where

$$z=\gamma e+\dot{e}$$

(7)

$e=q_d-q$ is the tracking error, $q_d$ represents the desired trajectory of the joint variable and $\gamma >0$.

$$y\vartheta =m\left(\ddot{q_d}+\gamma \dot{e}\right)+b\left(\dot{q_d}+\gamma e\right)+g\left(q\right)$$

(8)

where y represents the regression matrix.

$y=\left[y_{11}{y}_{12}\right]$

$\vartheta$ is $2x1$ vector of the unknown constant parameters

The following controller is proposed:

$$\tau =y\widehat{\vartheta }+k_{v}z$$

(9)

where $k_v$ is a scalar value representing the control gain.

With the adaptive update rule:

$$\dot{\widehat{\vartheta }}=-\dot{\tilde{\vartheta }}=\lambda y^{T}z$$

(10)

where

• $\widehat{\vartheta }$ is a vector representing the estimate of the unknown parameters;
• $\widehat{\vartheta }$ is $2x1$ matrix; ($\widehat{\vartheta }\in {\Re }^{2x1}$), $\tilde{\vartheta }=\vartheta -\widehat{\vartheta }$;
• $\lambda =\left(\begin{array}{c}{\lambda }_{1}0\\ 0{\lambda }_2\end{array}\right)$, and ${\lambda }_i>0$.

The stability of the proposed controller can be achieved using the following Lyapunov function V:

$$V=\frac{1}{2}m\left(q\right)z^2+\frac{1}{2}{\tilde{\vartheta }}^T{\lambda }^{-1}{\tilde{\vartheta }}^2$$

(11)

$$\dot{V}=z\dot{z}+{\tilde{\vartheta }}^T{\lambda }^{-1}\dot{\tilde{\vartheta }}$$

(12)

By substituting Equation (6) into Equation (12), we get:

$$\dot{V}=z\left(y\vartheta -\tau \right)-b{z}^2+{\tilde{\vartheta }}^T{\lambda }^{-1}\dot{\tilde{\vartheta }}$$

(13)

Substituting Equation (9) into Equation (13) yields:

$$\dot{V}=-k_v{z}^2+{\tilde{\vartheta }}^T\left({\lambda }^{-1}\dot{\tilde{\vartheta }}+y^{T}z\right)-b{z}^2$$

(14)

Substituting Equation (10) into Equation (14) yields:

$$\dot{V}=-k_v{z}^2-b{z}^2<0$$

(15)

V is lower bounded by zero, $\dot{V}$ is negative semidefinite and $\ddot{V}$ is bounded by Barbalat’s Lemma

$$\underset{t\to \infty }{lim}\dot{V}=0$$

(16)

By the Rayleigh–Ritz theorem, we can prove that

$$\underset{t\to \infty }{lim}z=0$$

(17)

which means that

$$\underset{t\to \infty }{lim}e=0$$

(18)

The novelty proposed in the developed solution is to deal with the pain that the patient feels during the rehabilitation process. Indeed, this pain can affect a person’s reactivity and therefore their susceptibility to a rehabilitation protocol. We assume that there is a limit to pain, beyond which the susceptibility in question is affected. Indeed, it is not possible to refer only to a peak of current often detected due to patient resistance to confirm a pain threshold. On the other hand, a current peak accompanied by muscle fatigue could give a good estimate of the pain. In fact, there is no rehabilitation without pain. However, pain experienced when performing a RoM already performed in the last exercise cannot be considered pain requiring a special intervention. For those advanced reasons, a fuzzy logic-based decision support system could be established as an appropriate solution. The proposed fuzzy logic controller consists of two blocks. The first block is for estimating muscle contraction based on four signals extracted from the EMG sensor, namely, EMG-RMS, EMG-SST, EMG-vorder and EMG-logdetect. The second block consists of estimating the level of pain as a function of the muscle contraction, the patient’s resistance and the last RoM reached. The fuzzy architecture is presented in Figure 2.

The control process begins with a simple gesture launched by the physiotherapist and taken up by the Kinect. Image processing and recognition algorithms are implemented in the HMI. After the required processing, the desired elbow angle is communicated to the adaptive controller block and to the 3D virtual environment block. The calculated torque is sent via the MQTT protocol to the elbow robot. The information collected from the current sensor, the EMG sensor and the encoder is sent back via the MQTT protocol to the control station. The raw EMG signal acquired goes through extraction algorithms to detect muscle contraction. The EMG features and the current sensor feedback are used as input to the fuzzy system to estimate the pain felt by the patient. Position feedback is used in the control motion block and in the fuzzy logic input to make the appropriate decision. The parameter settings and the performed results are stored in the database. Although HTTP (Hypertext Transfer Protocol) is the most widely used protocol, MQTT (Message Queue Telemetry Transfer) has grown rapidly in recent years. The most important characteristic of the MQTT protocol that really distinguishes it is the real-time aspect; a criterion that avoids the problems due to latency. Additionally, the MQTT protocol is easy to implement and enables high speed data transmission by means of publish/subscribe operations, which are essential in medical applications, and ideal for machine-to-machine communication. The use of the network must be minimized to avoid disturbances due to the high demand for data exchange. The MQTT protocols minimize the data packets, which alleviate the use of the network and consequently avoid disturbances caused by the high demand for data exchange. MQTT is a low-power protocol offering the possibility of using a battery as an alternative solution to power the portable robot. The MQTT central server (called broker) is a message dispatcher. IoT devices can subscribe or publish messages to this server. A “publish” operation occurs when a client intends to send data to the server. When a client intends to receive data from the broker, we call this a “subscribe” operation. Actually, the devices are subscribed and published to topics. The function of the broker is to manage the data transmission between devices. In the developed architecture, the physiotherapist plays the role of broker. The topic to publish in this situation is the computed torque. The elbow robot is subscribed in this topic which allows it to be notified of each update. Similarly, the physiotherapist’s station must be entered in the current sensor feedback, the EMG feedback and the encoder position feedback. Figure 3 illustrates the implemented architecture. The system operation is described by the flowchart presented in Figure 4.

3. Results

The robot used in the experimental tests was equipped with a Nema 34 stepper motor and a 1000 CPR (count per revolution) encoder. The actuator technical specifications are given in

Table 1. Technical specifications of the robot actuator.

Type	Bipolar Stepper
Step Angle	1.8 deg
Holding Torque	4.6 N.m
Rated Current	6 A
Encoder Driving Voltage	5 V
Encoder Output Current	55 mA
Encoder Resolution	1000 CPR

.

To measure the current, the ACS712 module based on the hall effect principle has been used. This module can measure both AC and DC current providing a perfect isolation from the load and easy integration with any conventional micro-controller. On the other hand, a Myoware muscle sensor was used to measure a filtered and rectified electrical activity of the muscle with a resolution of 12 bits. The EMG signal was acquired using the multi-channel surface electromyography (SEMG) signal during dynamic contraction. Myoelectric signals were detected by placing three electrodes—two of them for measurement with a distance equal to 3 cm, and the third serving as a reference electrode placed at the proximal end of the elbow. The selection of muscles and the placement of the pre-gelled electrodes are based on the related literature [51] for elbow flexion/extension. Based on [52], the maximum bandwidth of the EMG signal is approximately 500 Hz. Therefore, according to Shannon theory, the sample rate should be greater than twice the signal bandwidth. Thus, the sampling frequency of the acquisition system was fixed at 1000 Hz. The patient should receive specific training to learn how the rehabilitation system is used, including the placement of the electrodes and the installation of the system. The main features of the EMG sensor are given in

Table 2. Specifications of the Myoware EMG sensor.

Parameter	Value
Supply Voltage	+2.9 to 5.7 V
Polarity reversal protection	Yes
Output Modes	EMG envelope and EMG Raw
Adjustable Gain	Yes
Physical size	0.82${}^{″}$ × 2.06${}^{″}$

.

The developed user-friendly HMI containing three different control screens. The first screen shown in Figure 5 has the role of configuration and supervision. In this step, the interface provides an authentication service. This service improves the security aspect of the networked system. Since we are in the context of vision-based control using a 3D camera, the position of the user must respect specific parameters to be correctly recognized. To solve this problem, the developed software provides some indicators on the user 3D position. When the physiotherapist’s position in front of the 3D camera is inappropriate for control, the indicators are displayed in red, as shown in the first screen of the Figure 5a. In order to get the actual information, the Kinect camera requires a predefined position. If the user shows up outside this position, the system interface displays an invalid position alert (see Figure 5a). When the appropriate position is detected, the three indicators of the user’s position become colored green, as shown in Figure 5b.

Figure 6 shows that the system offers real time streaming. An integrated IP camera in the developed control architecture is used to send videos to the router. The physiotherapist’s computer connected to the router displays a real time video of the patient during the rehabilitation process. Such visual interaction could facilitate the process of rehabilitation. By controlling the robot in a gestural way, the video of the patient following the robot’s actions is not enough for the physiotherapist. The physiotherapist must know the real time angle achieved by the elbow in motion. To this end, the joint angle recorded during the exercise is displayed in real time on the physiotherapy interface, as shown in Figure 5. To ensure safety, an indicator is displayed on the physiotherapist’s station to offer information about the pain felt by the patient, as shown in Figure 5. Both videos of the Kinect and IP camera are transmitted to be displayed in the HMI as shown in Figure 6.

The membership functions for the inputs and for the output of the first fuzzy system are shown in Figure 7 and Figure 8, respectively. The membership functions for the inputs and for the output of the second fuzzy system are shown in Figure 9 and Figure 10, respectively.

Electronic health records have many benefits for the medical profession and for patients. Grouping and remote access to patient records are allowed. It is therefore easier to transfer a patient from one rehabilitation center to another and to follow the evolution of his medical file. In this context, the software developed, as shown in Figure 11, offers the possibility to record patients’ personal data and exercises performed, and reports on patient follow-up.

Experiments were performed on three subjects, all with an elbow fracture, just one week after removal of the cast. The subjects underwent a complete rehabilitation protocol to recover the flexion/extension movements for 10 days. The exercise was repeated for 3 sets of 12. The connected robot followed the physiotherapist’s instructions. The first tests carried out with the three subjects showed that the subjects had lost approximately $64\%$ of the RoM of a normal elbow. Subject 1 was received on 12 February 2020, as shown in Figure 12a. He underwent a 10-day work protocol punctuated by two days of rest (5 days of work, 2 days of rest, 5 days of work). He obtained a RoM of 96 degrees during the last day with rehabilitation progress factor (RPF) equal to $5.3$. This rate was calculated according to the maximum RoM that can be reached by a healthy subject: movable range = 145 degrees for flexion and −5 degrees for the extension. Likewise, subject 2 was received on 27 February 2020 and he underwent the same protocol. He completed his last session with a RoM of 114 degrees and a RPF of $6.3$, as shown in Figure 12b. Subject 3 began his rehabilitation protocol on 19 March 2020 and managed to reach a RoM of 142 degrees on 29 March 2020 with an RPF of $7.8$, as shown in Figure 12c.

Table 3. Table of clinical tests.

Subject	Initial RoM (degree)	Loss Percentage	Achieved RoM (degree)	Recovery Percentage	RPF
Subject 1	55	64	96	64	5.3
Subject 2	75	50	114	76	6.3
Subject 3	105	30	142	94	7.8

summarizes the obtained results.

4. Conclusions

Neuro-robotics is a multidisciplinary field that combines neuroscience and engineering. This field has seen major breakthroughs, particularly with the development of technology and control strategies. In this paper, we have presented a solution for remote elbow rehabilitation via the Internet. Indeed, due to the current circumstances of COVID-19’s spread, providing remote services is becoming a real challenge. A human–machine interface (HMI) has been developed to allow the physiotherapist to remotely control the robot and supervise the rehabilitation process. The control is based on the gestural behavior detected by the Kinect camera and sent via the MQTT protocol to the elbow rehabilitation robot. Information on resistance force, RoM and muscle contraction is fed back to the physiotherapist’s computer. An intelligent system based on fuzzy logic has been implemented to detect the presence of the pain threshold. Indeed, on the basis of a cascaded fuzzy logic controller, we first estimate the muscle contraction and then the pain felt by the subject during the rehabilitation exercise. The results obtained showed the efficiency and reliability of the solution developed.