HARVEST: High-Resolution Haptic Vest and Fingertip Sensing Glove That Transfers Tactile Sensation of Fingers to the Back

Featured Application

This study proposes a haptic vest and a sensing glove for transferring the tactile information from the fingers to the back. We developed a haptic vest installed with 80 voice-coil type vibrators and a sensing glove that can detect up to 100 points of fingertip pressure distribution with an interval of 2 mm. We conducted perceptual experiments to evaluate the device performance and user experience. It was found that when the glove and haptic vest were used, the shapes of figures could be recognized correctly when compared to the case where only the glove was worn.

Abstract

Human fingertips are densely populated with tactile receptors and are hence incredibly sensitive. However, wearing gloves on the fingers drastically reduces the tactile information available to the fingertips, such as the texture and shape of the object, and makes it difficult to perform dexterous work. As a solution, in this study, we developed a high-resolution haptic vest that transfers the tactile sensation of the fingertips to the back. The haptic vest contains 80 voice-coil type vibrators which are located at each of the two discrimination thresholds on the back and can be driven independently. The tactile sensation of the fingertips is transferred to the back using the developed haptic vest in combination with a sensing glove that can detect the pressure distribution on the finger skin at up to 100 points. Different experiments were conducted to validate the performance of the proposed haptic vest and sensing gloves. The use of the haptic vest and the sensing glove enabled the user to perceive the shape of a planar object more accurately when compared to the case where the user wore only the glove.

1. Introduction

Human tactile perception is surprisingly sensitive. In particular, the finger is a body part with the highest density of tactile receptors, and it can acquire information necessary for manual manipulation tasks, such as texture, shape, and weight, when it touches an object [1]. However, tactile information such as the texture and the shape of an object is drastically degraded for people who must wear gloves on their hands for work and may hinder their ability to work effectively.

One solution to these problems is to transfer the sensation of the fingertip to a body part. One reason for not presenting the tactile information on the fingertips is the unavailability of small actuators that do not interfere with the fingertip movement. Such devices are also large and complicated. If the fingertips are not used as a presentation target, a large-area presentation device can be placed, for example on the back. This can overcome the issue of the size of the actuator in the haptic device and allows the user to work without hindering the movement of the fingertips.

Tactile transfer is a common method in the study of sensory feedback-enabled prosthetic hands, and there have been many attempts to place vibrators on the arms and shoulders [2]. Furthermore, as an example of the application of tactile transfer techniques, a new tactile presentation method in a virtual reality (VR) environment has been proposed to transfer the tactile sensation of the fingertips to the arm and face [3,4,5]. However, few of these devices aimed to replace the ability of the fingertips with other parts of the body, and they only convey the sense of touch symbolically.

A two-fingered glove is developed as described in [6] that can sense the pressure distribution of a fingertip with a 2 mm interval, which is equivalent to or less than the discrimination threshold of two points on the fingertip [1]. By using the glove, the pressure distribution is transferred to the back, in combination with 144 eccentric vibration motors placed on the back and the abdomen [7]. They focused on the back because the number of discriminable points on the back is greater than the number of discriminable points on the fingertips. Additionally, it is not difficult to attach or detach the device from the back even if it is large and heavy, unlike the case with the arms and other parts of the body. Because the eccentric vibration motor has a low frequency response (150 ms) as described by [7], another prototype was also developed using 100 voice-coil type vibrators with a higher frequency response (1 ms) [8]. A prototype using a voice-coil type vibration motor showed that it is possible to support the tactile perception ability of the fingertips by transferring the fine tactile signal that is obtained from the fingertips to the back.

In this study, we developed a haptic vest installed with 80 voice-coil type vibrators, and a sensing glove that can detect up to 100 points of fingertip pressure distribution with an interval of 2 mm. We examined whether objects were correctly identified when the tactile signal from the fingers was mapped to the back (Figure 1). A new drive method using a shift-register is proposed and implemented to drive all the vibrators independently at different amplitudes and frequencies, although it is necessary to provide an appropriate AC signal to the voice coil type transducers independently. In this study, we first evaluated the hardware of the developed devices, and finally conducted two types of experiments to evaluate the performance of the device. We found that the objects were identified correctly in the gloved and haptic vest state when compared to the case without the haptic vest (without tactile transfer to the back).

2. Related Work

There are many studies and devices that have attempted to transfer the haptic sensation to the body to enhance the immersive experience. There are reports of wearable and installed type devices that are embedded with vibrators to provide tactile feedback [9,10,11]. There are also a few reports on haptics displays developed for use in VR/AR [12,13,14]. However, most of them are not based on tactile transfer.

Tactile transfer, in which tactile information from one part of the body is transferred to other parts of the body, is a common method in the study of prosthetic hands, and there have been many attempts to place motors and eccentric vibrators on the arms and shoulders [2,3,15]. The study aimed to expand human tactile perception by using a method known as assistive devices. However, these devices could transmit haptic sensation only symbolically using vibration or electrical stimulation and could not recognize the shape of objects or support human skills.

Similarly, there are some examples of applying tactile transfer technology such as VR technology. Okano et al. performed a large-area tactile presentation on the soles [16]. In this study, a mapping of palmar tactile sensations to the soles of the feet was demonstrated, which can be considered intuitive because of the similarity of the shapes of the palms and soles. The tactile information of the fingertips has also been transferred to the arm using a five-segment linking mechanism [3], to the face using an air-driven suction device [5], and to the wrist using a bracelet-type device [17,18,19]. However, most of these studies treated fingertips as a single contact point, and none used a dense wearable tactile sensor to transfer high density tactile information acquired by a single finger.

For extending the tactile ability of the human body, a fingertip-mounted microscope that magnifies a touched area haptically was proposed [20]. It could connect the micro and macroscopic worlds through intuitive fingertip movements. In addition, there are studies that provide feedback on the pressure distribution at the fingertip to the finger skin using electrical stimulation [21]. Although the extension of fingertip sensing capabilities was achieved, the haptics display itself inhibited the movement of the fingers.

There were several researches on tactile cue presentation on the abdomen and back [10,22], but as far as the authors know, they did not intend to transfer fingertip tactile information.

Here, we demonstrate tactile transfer based on the following idea. A large number (80) of tactile presentation elements are placed in a large area on the back to present the tactile information of the finger with sufficiently high resolution. The fingertip is fitted with a glove that can sense the pressure distribution at the fingertip with 2-mm resolution, which is equivalent to the two-point discrimination threshold of the fingertips. We aim to extend the human ability by reconstructing the tactile ability of the hands on the back. If we consider the tactile ability as the number of discriminable tactile points, the area divided by the resolution, we can increase the number of tactile points on the back, so that the tactile presentation on the back exceeds the tactile ability of the fingertips. There have been studies in which vibrators are placed on the back and vibration stimuli are applied as an output, such as a study that realized navigation of walking for visually impaired and healthy people [23,24]. In another study, 400 vibration actuators were mounted on a chair and images were converted and projected on the back as vibration sensations to realize image recognition [25]. However, these studies were not about the replacement of human tactile abilities. If the proposed approach can be used to extend the remaining tactile area, in a work environment where gloves are worn, the information necessary for the work can be presented on the back to support human skills.

3. System Design

In this section, we provide an overview of the HARVEST (Haptics AR Vest) system, which is a combination of the developed haptic vest and a sensing glove.

3.1. Haptic Vest

3.1.1. Design of Haptic Vest and Driving Method of Voice Coil Type Vibrators

Figure 2 shows the hardware of the prototype haptic vest. 80 haptic reactors (Alps Alpine), which are voice-coil type vibrators with high response, are used in the prototype, and they are spaced evenly at 50 mm intervals. The two-point discrimination threshold for the back was set at 50 mm, and the area where the vibration stimulus was presented was set at a distance equivalent to this two-point discrimination threshold [1].

The vibrators are connected to 10 cascaded 8-bit shift registers (TPIC6B595) through a microcontroller (ESP32) by serial connection, enabling the 80 vibrators to be driven independently (Figure 3). The data from the PC to the microcontroller include the output of the 80-stage shift register and the vibration intensity parameter. The data received in the microcontroller is projected onto the spatial arrangement of the vibrators built into the haptic vest. When the output of 10 bytes to all the shift registers is completed, the outputs of all shift registers are updated.

As a voice coil vibrator is used instead of an eccentric motor, it is necessary to generate an appropriate AC signal as a pulse. For example, when stimulating a vibration frequency of 400 Hz, it is necessary to generate a 400 Hz square wave, and it is necessary to switch the output of the shift register at least 800 times per second. At least a 64 kHz clock is required to switch the 80-stage shift register. Furthermore, in order to control the strength of vibration by means of pulse width modulation (PWM), it is considered necessary to transmit the data at approximately 2 MHz in order to fine-tune the pulse width with 32 gradations. The shift registers and the SPI communication output of the microcontroller used here can transmit and receive signals at a much faster rate. In the developed system, the voice coil type vibrator can express 32 gradations, and the response speed is 1 ms. The experiments with various users confirmed that when the 80 vibrators were driven independently, all the vibrators could be distinguished and perceived.

3.1.2. Frequency Characteristics Measurement of the Voice Coil Type Vibrators Were Measured

The frequency characteristics and vibration intensity of the voice coil type vibrators were measured. Here, we limited the measurement to the vibration presented on the back. For this purpose, a voice coil type vibrator was mounted on the back with an acceleration sensor attached its side surface. The acceleration sensor (BMX055, InvenSense) was used to measure the acceleration, and the sampling frequency was 1 kHz.

For this experiment, the author sat on a chair. The acceleration sensor was attached to the side surface of the voice coil type vibrator with double-sided tape, and the voice coil type vibrator was attached on top of the clothes on the back of the user by means of Velcro-tape. The frequency was incremented by 50 Hz in the range between 50 Hz and 400 Hz. The input to the voice coil type vibrator is the PWM signal output from the shift register, and the duty ratio of the PWM signal was incremented by 10% for each measurement. It should be noted that, since this system treats a square wave as an input signal, when driven at a low frequency, noise is generated inside the voice coil vibrator, which may be caused by collisions inside, so it was not investigated. The 3-axis acceleration was measured at each frequency for 5 s, and the square root of the average value of the amplitude for each PWM duty ratio was calculated.

From the acceleration measurement in Figure 4, it can be seen that the voice coil type vibrator has the largest acceleration amplitude at 400 Hz and 200 Hz with a 50% duty ratio (16 g). Based on the results of the experiment, we decided to conduct a perceptual experiment at 350 Hz, as for this case the vibration intensity changes approximately linearly with respect to the PWM duty ratio.

3.2. Sensing Glove

3.2.1. Pressure Sensor Specifications

Figure 5 shows the prototype of the sensing glove. The built-in pressure sensor used in the sensing glove was adopted from the pressure-sensitive resistance method developed by Yem et al. [6]. In their study the pressure sensors were installed at an interval of 2 mm, and the number of pressure detection points was 5 × 10 = 50 points. In this study, the number of sensors was expanded from 50 points to 10 × 10 = 100 points per finger. The sensing glove covers three fingers: the thumb, index finger, and middle finger. The glove can measure at approximately 300 Hz when transmitting with 8-bit resolution and can measure at approximately 900 Hz when transmitting information from only one finger. Because the two-point discrimination threshold of the distal phalanx of the finger is approximately 1.5 mm at the tip and approximately 2 to 3 mm at the fingertip [1], the resolution of the pressure sensor is approximately the same as the number of pressure points that the fingertip can discriminate. The thickness of the glove was approximately 1.2 mm. This thickness protects the sensor and acts as an appropriate spatial low-pass filter. The thickness of this glove is approximately the same as that of a general leather glove. As will be revealed later in experiment 2, the thickness of the glove considerably impairs the sense of touch and causes difficulty in understanding the shape of the object. As mentioned in the introduction, this is thought to simulate the decline in tactile sensation due to ageing and the decline in tactile sensation in the case where gloves are necessary.

3.2.2. Pressure Measurement

The output of the pressure sensor was measured. The pressure sensor was placed between two sponges, and a known weight was placed on top. The output results of the sensor are shown in Figure 6. A pressure of 25 hPa corresponds to a force of approximately 10 g on the fingertip, and 250 hPa corresponds to a force of 500 g. From this result, it was confirmed that the output of the sensor changes significantly with a small pressure, and the output saturates as the pressure increases. The magnitude G of each voice coil vibrator is given by following Equation (1), where the output Soutput of the each pressure sensor.

$$G=0.15Soutput-2$$

(1)

4. Experiment

To investigate whether the tactile information transferred to the back can be perceived correctly using the proposed system, two types of perceptual experiments were performed in this study.

In the first experiment, we investigated whether the subjects/users could correctly identify four straight lines at different angles presented on the back, under four different scenarios. The scenarios are as follows: (1) the subject touches the straight line object with their bare hands, without wearing a glove or haptic vest, (2) the subject wears the glove and touches the straight line object without wearing the haptic vest, (3) the subject wears both the glove and the haptic vest and touches the straight line object, and (4) the subject wears only the haptic vest and the object is presented on the back of the user.

In the next experiment, we investigated whether the subjects could correctly recognize four different objects presented on the back and performed the experiments under four different scenarios. The scenarios are as follows: (1) the subject touches the object with bare hands, without wearing the gloves or the haptic vest, (2) the subject wears the gloves and touches the object without wearing a haptic vest, (3) the user wears both the glove and the haptic vest, and the haptics stimulation is performed in real time on the back, and (4) the user wears the glove and the haptic vest, and the touched object is presented to the back by driving the vibrator one row at a time from the left.

In all the experiments, the distal phalanx of the index finger was targeted for stimulation, and mapping to the back was attempted. The pressure sensor can measure up to 10 × 10 = 100 points. However, in this experiment, 10 × 8 = 80 detection points were used to achieve a one-to-one correspondence with the maximum number of vibrators (80 voice coil type vibrators) as shown in Figure 7. As it is possible to measure at approximately 900 Hz with one finger, the experiments were conducted without including the thumb and the middle finger. The sensing glove and the haptic vest are each connected to the PC via a USB port, and the tactile stimulus was presented based on the information received from the PC side.

4.1. Experiment 1: Line-Direction Recognition

In the first experiment, a combination of the sensor glove and the haptic vest was used to examine whether the user could correctly recognize straight lines with different angles presented on the back. There are four types of slopes of the straight line: horizontal, vertical, 45° to the right, and 45° to the left (here, they are represented by the symbols “-”, “│”, “/”, and “\”, respectively). The straight line object is 1 mm in diameter, which is smaller than the pressure detection interval of the pressure sensor used in this experiment. The straight line object used in this experiment and the output of the pressure sensor are shown in Figure 8.

4.1.1. Subject

The experiments were performed by nine subjects with ages in the range of 21–42 (1 female, 8 males), all of whom were right-handed.

4.1.2. Experimental Environment and Procedure

The experiments were carried out according to the four scenarios described at the beginning of this section. In addition, the slopes of the four types of straight lines were randomly presented five times each, for a total of 20 times (the total number of trials was 80). In each trial, one straight line object was placed on the desk, and after the subject touched the object, the angle of the perceived straight line was determined. The subject touched the straight line object only once. The subjects had to determine the angle of the straight line object within 5 s, and the correct answer rate for each straight line object was recorded.

Prior to the experiment, a one-minute training was provided for each tactile condition, and the participants were allowed to freely touch the straight line object. Before the start of the experiment, the subjects were asked to wear headphones and made to hear white noise. All the subjects performed the experiments with their eyes closed (Figure 9).

4.1.3. Result for Experiment 1

The results of the experiment are shown in Figure 10 and Figure 11. The correct answer rate of the slope of the straight line under each scenario is tabulated. A high correct answer rate of 86% or more was recorded on average for all combinations of tactile conditions and objects. Even in the scenario when only the haptic vest is worn and a straight line is presented on the back, a high correct answer rate of 95.6% is recorded on average, assuring that the haptic vest works fine to present simple patterns. However, in this experiment, the correct answer rate was very high even under the condition where only the gloves were worn, and the effect of the haptic vest could not be observed.

4.2. Experiment 2: Shape Recognition

In the next experiment, we investigated whether the subjects could correctly recognize the shape of the objects presented on the back using the sensor glove and the haptic vest. There are four types of objects presented: a cross, a circle, a triangle, and a square (indicated by the symbols “×”, “○”, “△”, and “□”). Each object was presented under four different scenarios as described at the beginning of this section. The objects have a border thickness of 1 mm, and a size of 1.5 × 1.5 cm. As in experiment 1, the subject touched the object only once. Answers were recorded within 5 s of touching the object, and the percentage of correct answers for each object was recorded. The objects used in this experiment and the output of the pressure sensor are shown in Figure 12.

4.2.1. Subject

This experiment was performed by 10 subjects with ages in the range of 21–70 (1 female, 9 males), all of whom were right-handed. 9 of these subjects participated in experiment 1.

4.2.2. Experimental Procedure

The experiments were carried out according to the four scenarios described at the beginning of this section. Contrary to the first experiment, we added the fourth condition. In which the vibrators are driven one row at a time every 100 ms. This condition was introduced because it is well-known that stimulation of multiple points at the same time makes it difficult to recognize [26]. It has been reported that the longer the drive interval of the vibrator in each row, the higher the identification rate of the figure presented on the arm etc. [27]. However, the system proposed here aims to improve the tactile perception ability of the fingertips during work. Therefore, real-time performance is required. Hence, the vibration interval was set as 100 ms, which was found to be the shortest drive interval between each row that resulted in a high correct answer rate for the author (Figure 13).

4.2.3. Results of Experiment 2

The average identification rates of the tactile presentation under different scenarios are compared: (1) bare hands: 94%, (2) gloves only: 42%, (3) real-time transfer between gloves and haptic vest: 52%, and (4) transfer between gloves and haptic vest for each row, 60%. A higher discrimination rate was recorded in the scenarios where the subject wears the haptic vest than in the scenario where the subject wears only the glove. In addition, the object identification rate with gloves on average is 42%, which shows that the identification rate is significantly lower when compared to the scenario where the subject touches the object with bare hands (94%). As mentioned earlier, the sensing gloves are equivalent to the leather gloves used on a daily basis, and it can be seen that the tactile sensation is significantly impaired. This makes it possible to simulate the case of declining tactile sensation due to aging and work environment with gloves. More than 20% of the respondents answered “○” and “□” as “△”. The space in the center of each of these objects could have contributed to the decrease in the discrimination rate. Moreover, 44% answered “△” as “×” and 28% answered “×” as “△”. It can be inferred that the inclusion of straight lines that are tilted diagonally compared to other figures made the shape identification difficult in this case. In addition, the subjects appeared to be confused between “□” and “○”, which were similarly misidentified under all scenarios (Figure 12).

We attempted to understand if the difference in the identification rate had a direct dependence on the four scenarios of tactile presentation and the differences in the shape of the object. In this problem, the independent variables are the type of tactile presentation and the shape of the object, and the dependent variable is the identification rate. ANOVA and multiple comparisons were performed (Figure 14 and Figure 15). It was found that the shape of object was the dominant factor (p < 0.05). Focusing on scenarios other than the subject using bare hands, it can be seen that for each of the shapes “△”, “□”, and “○”, the scenario of transferring to each row when both the glove and the haptic vest are used has the highest correct identification rate. In addition, when comparing the results of the glove alone with the scenario of transfer of each row while using the glove and the haptic vest, a dominant difference was found between “○” and “□” (p < 0.05).

4.3. Discussion

Compared with experiment 1, experiment 2 had a lower discrimination rate under all tactile conditions. In the scenario using the haptic vest, the identification rate was higher than that in the case of using the glove, where the stimulation is performed by driving the vibrators one row at a time from the left. Compared to the experiment 2, the objects of different shapes have many combinations of vertices and straight lines. This led to the misidentification of simple shapes. In contrast, under the condition that vibration is performed row by row using a haptic vest, the information presented to the back at one time is reduced, and remapping is performed in the brain based on the spatiotemporal transition of the stimulation point. Hence, in this case, the identification rate of the shapes is higher than that in the case of transferring to the back in real time in most of the figures, “△”, “□”, and “○”. However, the longer the time it takes for the stimulus point to move, the longer it takes for the information to be transmitted, and the type of information that can be presented is restricted by the type of information presented and the situation.

From the above results, it was found that when the tactile sensation of the fingertip was transferred to the back, the method of transferring the tactile sensation to the back by each row had the highest figure identification rate. Many subjects also commented that “the conditions for transfer in each row were the easiest to understand”, and that “the vertices of diagonal straight lines and shapes were presented more clearly, and remapping in the brain was easier than the condition of mapping to the back in real time”.

To increase the identification rate of shapes, other methods such as providing a tactile sensation only to the apex and tactile drawing methods using multiple fingers can be used. By emphasizing the vertices, it becomes possible to grasp the position and number of corners, and it becomes easier to grasp the features for recognizing shapes. In addition, the tactile map of an object is often recognized by touching it with multiple fingers. In this experiment, the tactile signals were obtained only from the index finger. In the future, we will attempt tactile sensation transfer to the back from multiple fingers. Also, we will try to use a linear actuator to present haptic feedback to the back, since humans have very low spatial resolution for vibration sensation and also humans are not that sensitive to the direction of high-frequency vibrations due to the lack of directional sensitivity [28] that would lead to the low identification rate on each scenario. In addition, when confirming the shape and material of an object, humans evaluate the tactile sensation by relying on information on the shearing force of the fingertips and the magnitude of the applied force. In future work, we will investigate whether the identification rate of the shapes changes when a shear force in any direction of the finger pad is applied without restricting the movement of the finger.

5. Conclusions

In this study, we developed a haptic vest installed with 80 voice coil type vibrators. We also developed a sensing glove that can detect the pressure distribution from up to 100 points on the fingertips at intervals of 2 mm and investigated whether the tactile perception ability of the fingertips could be correctly reconstructed on the back in combination with the haptic vest. We evaluated the hardware of the developed device and system and conducted two types of perceptual experiments. It was found that when the glove and haptic vest were used, the shapes of figures could be recognized correctly compared to the state where only the glove was worn.

A unique feature of the proposed device is that the part that interacts with the real object and the part that presents the tactile sensation are separated. We have found that the proposed system shows promise in correctly identifying the inclination of straight lines and shapes. One possible practical implementation of this system is that it can strongly support the tactile perception ability of the fingertips which require delicate manipulation, such as threading a needle and thread through a small hole.

However, for the reconstruction of the tactile sensation of the fingertips of the tactile perception ability of the fingertips during work involving the wearing of gloves, further improvements need to be made to the proposed device. Specifically, as a future work, we will design an experiment on the premise of remapping the brain by training and investigate if the user experience of the proposed device can be improved by training the users. In addition, we will conduct a mapping survey of vibration stimuli corresponding to multiple fingers and a mapping survey of the back according to the spatial coordinates of the fingers. Through these evaluations, we will verify whether this research can improve human tactile perception ability.