Optimization of Wheelchair-Mounted Robotic Arms’ Base Placement by Fusing Occupied Grid Map and Inverse Reachability Map

Abstract

In a household setting, a wheelchair-mounted robotic arm (WMRA) can be useful for assisting elderly and disabled individuals. However, the current WMRA can only perform movement and grasping tasks through joystick remote control. This method results in low efficiency due to poor coordination between the mobile platform and the robotic arm as well as the numerous operational steps required. To improve the efficiency and success rate of the robot in task execution, this paper proposes a parking location optimization method that combines the occupied grid map (OGM) and the inverse reachability map (IRM). Firstly, the SLAM algorithm is used to collect environment information, which is then stored in the form of an occupied grid map. The robotic arm workspace is then gridded, and the inverse reachability map is calculated based on the grasping pose of the target object. Finally, the optimal position of the mobile platform is obtained by comparing the optimal location point in the inverse reachability map and the obstacle information in the occupied grid map. This process achieves base placement optimization based on the grasping pose. The experimental results demonstrate that this method reduces the user operation time by 97.31% and overall task completion time by 40.57% when executing household environment tasks compared with the joystick control, increasing the range of executable tasks compared with the algorithm of the EL-E robot and reducing task completion time by 23.48% for the same task. This paper presents a parking location optimization method that can improve the grasping efficiency of the robotic arm and achieve parking location position selection for the WMRA in a household environment.

1. Introduction

Currently, the global population is continuously increasing and care for elderly and disabled individuals is becoming an increasingly greater pressure. In this context, assistive technology for the elderly and disabled is constantly evolving, and various assistive devices are being innovated. Among them, the WMRA combines the electric wheelchair and robotic arm, allowing for indoor movement and various operations in the home environment, effectively reducing care pressure and improving the quality of life for elderly and disabled individuals.

However, the WMRA still faces many problems during use. The traditional control method for electric wheelchairs and robotic arms is through joystick remote control. When performing various household tasks, all degrees of freedom of the electric wheelchair and robotic arm need to be controlled. Therefore, it is challenging to control the WMRA through a joystick, especially for users with a certain degree of disability. Traditional WMRA operation involves many steps and has low human–machine interaction efficiency, which greatly hinders its application.

In recent years, many studies have focused on improving the coordination between the mobile platform and arm motion planning. By optimizing the base placement to assist the robotic arm in completing tasks, the coordination between the mobile platform and the arm can be improved. The optimal base placement can be selected by calculating the inverse reachability map of the robotic arm, which evaluates the impact of different base placements on the robotic arm task. However, this method is not applicable when obstacles are near the target task. When the target object is on a wall or tabletop, the optimal base placement may interfere with the obstacle, preventing the mobile platform from reaching the designated position. There are currently many SLAM methods available for detecting and collecting obstacles in the household environment. SLAM uses LiDAR or cameras to collect information and combines it with odometer information to construct an environmental map, finally storing the information on environmental obstacles in the form of OGM. This method allows mobile robots to avoid obstacles based on environmental information and reach the user-specified two-dimensional target point. However, the user-specified target point may not be suitable for subsequent robotic arm task completion, affecting the completion efficiency of the robotic arm and the coordination of the mobile robotic arm.

To address these issues, this paper proposes a novel approach to optimize the base placement of the WMRA equipped with a robotic arm, by integrating the occupied grid map and inverse reachability map. To evaluate the algorithm’s performance, a variety of household tasks were performed. Firstly, the system was constructed, and the human–machine interaction and visual systems were integrated. Secondly, the household environment was collected using the SLAM algorithm and stored using the occupied grid map. Then, the reachable capability of the robotic arm was calculated to generate an optimal distribution of base placements. These positions were then compared and filtered against the household environment information, resulting in the completion of the OGM-IRM optimization method of base placement.

2. Related Works

2.1. Research Status of WMRAs

Currently, there is a growing body of research on WMRAs that can perform simple planar grasping tasks and navigate through home environments. Sateesh [1] combined voice interaction with arm and head sensors to increase interaction options and improve task efficiency. Alexander et al. [2] segmented the door opening task into multiple stages and used visual information alone, without an environmental map, to complete the task by narrowing down the error through multiple positionings. Pasteau [3] proposed a visual navigation framework that guided the wheelchair through a corridor using visual servoing methods.

All of the above WMRAs perform navigation and grasping tasks separately. However, they do not consider the effect of the base placement on arm operations, leading to the fixed selection of the base placement, which cannot adapt to various grasping tasks, resulting in a low overall task completion efficiency and success rate.

To improve the navigation efficiency and obstacle avoidance ability of the WMRA, obtaining information about the surrounding environment is crucial. The most common method for collecting environmental information is using the SLAM algorithm to construct occupied grid maps. By matching sensor information and map information, self-positioning can be achieved, thus avoiding obstacles during navigation.

2.2. Avoiding Obstacle Methods Based on Occupancy Grid Maps

Luca et al. [4] utilized the RTAB-Map algorithm to collect information in a classroom and implemented mobile robot indoor ultraviolet disinfection. Zhen et al. [5] used the Cartographer algorithm to construct an environmental map and used OGM information and sensor information to achieve positioning and complete navigation in complex environments. Zhao et al. [6] extracted feature information from OGM to decompose navigation tasks into multiple subtasks and used real-time visual information matching to estimate self-orientation and complete the positioning task in the navigation task. Li et al. [7] used the Gmapping algorithm in combination with RGB-D cameras to construct maps and used the Adaptive Mentcarto Localization (AMCL) algorithm to estimate the position and direction of the WMRA, achieving autonomous navigation.

2.3. Optimization of Base Placement Based on Inverse Reachability Map

Mobile robotic arms require a suitable base placement for efficient operation and increased success rate while avoiding the need for repetitive parking location adjustments. Several researchers have studied the optimization of base placement for mobile manipulators, such as Abhijit et al. [8], who determined the placement of the robotic arm by constructing an accessibility map; Felix et al. [9], who presented an approach to select an optimal stance location for a humanoid robot’s feet relative to a desired grasp pose; and Snehal et al. [10], who proposed a reinforcement learning algorithm based on actor-critic that uses the workspace data of the robotic arm to train accessibility, thereby improving computational efficiency and achieving effective coordination of robot mobile operations.

These studies mainly focus on the use and generation of inverse reachability maps. The current research is centered on enhancing IRM usage and algorithm efficiency, but there are still problems with obstacle fusion in household environments.

Recent studies have shown improvements in indoor navigation methods for WMRA, with increased navigation and human–machine interaction efficiency. These robots recognize and collect environmental information using sensors for improved navigation safety and efficiency. Additionally, they can select the optimal base placement to enhance the success rate for different tasks. However, issues persist in combining the two during execution, and a well-established solution is yet to be developed for optimal base placement and environmental obstacle interference.

3. Motivation

In a household setting, a WMRA can be useful for assisting elderly and disabled individuals. In previous work, we used the beta process autoregressive hidden Markov model to pour water from the bottle to the cup [11], and a laser pointer was used to facilitate the interactions between humans and a WMRA in an indoor environment [12]. We focused on programming by the demonstration approach to improving the intelligence and adaptability of the WMRA [13] and proposed an object-affordance-based implicit interaction technology using a laser pointer [14]. The WMRA with laser interaction was designed to identify the desired intention faster and perform tasks with less limb involvement from the user, as shown in Figure 1.

Previous research on the WMRA focused on its stationary state or proximity to the target object. However, in real-life household environments, disabled users may need to remotely lock onto a target object using lasers. In such scenarios, the WMRA must travel a certain distance to reach the vicinity of the target object. This presents a new challenge: determining the optimal parking position for the WMRA that allows the robotic arm to capture the target object in a shorter time and with a higher success rate. There are multiple empty spaces near the target object where the WMRA can park, and identifying the best position becomes crucial.

In response to this challenge, the work of this paper is to obtain the optimal location for wheelchair parking by integrating OGM and IRM. Moreover, various experiments in this paper have also proven that it can effectively address the aforementioned challenges.

Currently, for executing household tasks, the WMRA often uses separate planning and execution methods for navigation and grasping. The base is mainly responsible for completing the navigation task, while the arm is responsible for completing the grasping task. However, achieving coordination between them is difficult, resulting in coordination being a major issue, as shown in Figure 2.

To solve the above problems, motion planning methods with process structure are often used, such as the EL-E assistive robot [15], which segments the navigation task into three parts, rough navigation, fine navigation, and final navigation, ultimately bringing the robot to a position 40 cm away from the task target. However, this method does not take into account the robotic arm’s capabilities when selecting the parking location for the mobile base. As a result, the chosen parking location may not be suitable for the subsequent robotic arm task. Figure 3a illustrates the specific process.

To improve coordination in mobile manipulation and help the robotic arm better accomplish tasks, the optimal parking location for the mobile base must be found. The commonly used method is the optimization of base placement. However, when obstacles are present near the target object or the target object is on the surface of an obstacle, the optimal base placement may interfere with the obstacle, which would affect the effectiveness of base optimization. Therefore, this article proposes a combination of the OGM and IRM methods to study the problem of optimal base placement and environmental obstacle interference for the WMRA, as shown in Figure 3b.

4. Optimization Methods of Base Placement

4.1. Wheelchair-Mounted Robotic Arm System

The WMRA in our study is comprised of a mobile wheelchair, a robotic arm, various sensors, and a host computer, as shown in Figure 4. The host computer, running Ubuntu 18.04, is an Nvidia Jetson TX2, and the communication among the visual, SLAM, navigation, and control modules is facilitated by a robot operation system (ROS). An Arduino Mega 2560 serves as the lower-level controller of the mobile wheelchair, and two drivers control the PWM speed of the two DC motors that enable the wheelchair to achieve differential steering and forward and backward movements. To obtain color images and depth information, an Intel RealSense D435i depth camera sensor is used. The experimental platform utilizes both the eye-to-hand and eye-in-hand configurations to capture local and global information, respectively. The eye-in-hand camera is mounted on the Kinova Jaco 6DOF robotic arm, while the eye-to-hand camera is equipped with a two-degree-of-freedom rotating cloud platform that expands the camera’s field of view and enables it to better obtain environmental information.

During operation, the user interface of the experimental platform displays the current image information from the global camera, the map of the home environment, and the mobile wheelchair’s current location. The user selects a target object on the touch screen and the WMRA returns its coordinate value relative to the camera’s position. Using a coordinate transformation, the platform computes the relative position between the target point and the wheelchair and sends it to the inverse reachability map as the target point for the robotic arm’s end effector to calculate the next step.

4.2. Generation of Occupied Grid Map and Inverse Reachability Map

4.2.1. Occupied Grid Map (OGM)

In this study, the Cartographer algorithm [16] was utilized to establish environmental information for a complex task. The algorithm consists of two main components: the front end and the back end. The front end is responsible for handling sensor data to generate initial pose estimates and construct the map, while the back end optimizes the pose estimates and map construction results to achieve greater accuracy. Specifically, the Intel RealSense D435i sensor was employed in the front end to provide point cloud and IMU information, which were fused to estimate the robot’s pose. The front-end point cloud data are used to obtain the initial position of the robot in the environment, followed by camera data to calibrate the robot’s rotational pose and IMU data for position calibration. This approach allowed Cartographer to achieve more accurate pose estimation than a single sensor. In the back end, a graph optimization method was utilized to optimize pose estimates and map construction results.

The Cartographer algorithm treated the pose estimates and map construction results of the robot’s movement as a graph, with each node representing the robot’s pose estimate and map construction result at a certain point in time, and each edge representing the robot’s motion relationship between two points in time. The graph optimization algorithm was then utilized to minimize errors and obtain more accurate pose estimates and map construction results. The final results demonstrated that the Cartographer algorithm was effective in establishing environmental information for a complex task, as shown in Figure 5.

To provide LiDAR information in the implementation of the Cartographer algorithm, the “depthimage_to_laserscan” ROS package was employed. This package is responsible for converting the 3D point cloud information obtained from the depth camera into 2D laser scan information. By storing the minimum value of each column of the depth image in a laser array, it provides the distance of the closest obstacle in the forward direction as well as the distance of each obstacle point to the camera in the lateral direction.

Once the sensor setup was completed, the transformation between the IMU and WMRA’s TF coordinate was configured. Running the Cartographer algorithm initiated the process of constructing the map of the household environment. The resulting household environment occupancy grid map is depicted in Figure 6.

4.2.2. Inverse Reachability Map (IRM)

The inverse reachability map is a useful tool for evaluating the grasping ability of a robotic arm in a specific location in space, based on various base placements. To obtain the IRM, the accessibility of the robotic arm must be analyzed. This involves creating a unified robot description format (URDF) model of the robotic arm, which contains information about the arm’s joints, links, inertial and geometric properties, and collision model. The workspace is then divided into grids, with each voxel grid being subdivided based on its orientation and joint span. This study uses a URDF model of the WMRA with seven links and six joints, and the workspace is divided into 144,000 grid units, with a grid interval of 0.05 m [17].

The joint space of the robotic arm is discretized within each joint’s restricted range, with a selected joint span of 22.5°. By combining the rotational range of each joint, the joint space is traversed and calculated. The forward kinematics of the robotic arm is computed for each set of joint angles to obtain the spatial pose of the end effector relative to the base placement. To efficiently calculate a large number of kinematic solutions, this study uses the CUDA-based robotic arm kinematic solution algorithm [18] developed by the University of Michigan.

After obtaining 104,658 forward kinematic results, the results are categorized based on the grid partition of the workspace. The number of forward kinematic results associated with each voxel grid unit is examined to determine the number of forward kinematic results per voxel grid unit [19]. The voxel grids that received forward kinematic results are classified and marked with different colors (red, yellow, green, or blue) to represent different numbers of results received. The specific results and classifications are presented in Figure 7.

Upon obtaining the description of the arm’s reachable space, we utilized a visual plugin offered by the Spatially Aware Manipulation [18] package to display the outcomes in RVIZ. The RVIZ visualization displays each sphere as the center of a unit grid, while the four colors demonstrate the reachability of the arm for that unit grid. Blue grids signify positions with the most forward kinematic solutions, indicating that the arm can reach that position in more poses. Green and yellow grids demonstrate lower reachability, while red grids indicate that the arm faces difficulties reaching that position or that it is a singularity point of the arm [20]. The RVIZ visualization results are presented in Figure 8a.

Subsequently, we constructed the inverse reachability map after obtaining the arm’s reachable space description. Initially, we determined the target point (x, y, z) of the end effector, which is obtained through user interaction by clicking on the location information of the target object.

Based on the reachability space description, we obtained the forward kinematics results of the arm for each unit grid at the current base placement. Here, (x, y, z) represents the position of the tool center point relative to the global coordinates, with the global coordinate system centered on the center point of the four wheels of the WMRA, representing the center position of the WMRA. As each pose in the reachability map is a transformation matrix (T) starting from the robot’s origin, we could obtain the grasping ability description of the arm for a certain point in space at different base placements by inverting each forward kinematic transformation matrix and storing it as the IRM. As the mobile base of the armchair robot can only move in SE(2) space, we projected the IRM onto a 2D plane to obtain the IRM facing the mobile base, as shown in Figure 8b.

$$T_{global }^{TCP}=\{x,y,z\}$$

(1)

The inverse reachability map is generated from the description of the reachability map, and each point indicates that the WMRA has some inverse kinematics solutions at that position to reach the tool center point. Different colors represent different numbers of inverse kinematics solutions. With more inverse solutions of kinematics, the robot can reach the target position in more positions, and the WMRA has higher reachability to the tool center point.

The IRM includes all of the base placements that can cover the tool center point (TCP) and can visually present the coverage ability of the TCP by different base placements. Thus, the IRM can be utilized to find suitable base placements for the WMRA. After constructing the occupancy grid map and the IRM, we analyzed and compared the data to remove unreachable base placements caused by home environment factors, selected the optimal point from the remaining base placements, and completed the base optimization.

4.3. Optimization Methods by Fusing OGM and IRM

Based on the IRM information obtained from the target end effector pose, we could obtain the distribution of grasping ability base placements, where the grasping ability is expressed by the spherical range covered by the end effector at the target point [10].

$$M=\underset{x,y,\gamma }{\arg \max }IRM(x,y,\gamma )$$

(2)

We extracted the base placements that could enable the TCP to complete more poses, i.e., the optimal base placements, and compared them to the environmental information in the OGM to check for interference. If interference occurred, we selected the next best base placement for comparison until we obtained a base placement that did not interfere with the environmental information, which became the optimal base placement and was published as the Goal_Point to the controller of the base through ROS. The specific algorithm principles are depicted in the table below, and the actual RVIZ visualization of the OGM-IRM map is presented in Algorithm 1.

Algorithm 1: Optimization of Base Placement

Input: Target Pose T $\in$ SE(3), Inverse Reachability Map, Occupied Grid Map Output: Robot base pose B ϵ SE(2)   1:  P ← IRM.setPose(T);   2:  O ← OGM.setObstacle;   3:  MaxP ← arg max(P);   4:  while (MaxP ! = 0) do   5:     if (max(P).isInCollision(O)) then   6:       remove MaxP from the list P;   7:       MaxP ← arg max(P);   8:     else   9:       Optimal P ← MaxP; 10:     end if 11:  end while 12:  return Optimal P

5. Experiment and Results

In this section, we present the experimental validation of the feasibility and effectiveness of the proposed OGM-IRM base placement optimization algorithm in a household environment. The user selects the task target through the touch screen on the WMRA, which automatically selects the optimal base position and moves it to that position. We tested the algorithm on three objectives, namely, grasping a globe located at the edge of the table, grasping a cup located above the table on the opposite side of the table’s edge, and grasping a book located above the table adjacent to the wall, as shown in Figure 9. The experiments were repeated ten times in the given environment, and experiments were completed on the WMRA using the proposed OGM-IRM algorithm, the handle remote control method, and the process-structured navigation method of the EL-E robot. The total time to complete the task was recorded and compared to evaluate the performance of the OGM-IRM algorithm.

The experiment at position 1 validated the feasibility of the algorithm and measured the level of efficiency enhancement when using the OGM-IRM algorithm on the WMRA. The experiment at position 2 tested the reliability of the IRM information and evaluated the results of the base optimization, while the experiment at position 3 tested the algorithm’s ability to consider the interference of environmental information while calculating the base optimization and maximizing its function, as shown in Figure 10.

The specific experimental results are presented in

Table 1. The experiment results of the desktop task.

Task	Joystick Total Time (s)	Algorithm of EL-E Robot Total Time (s)	OGM-IRM
			Total Time (s)	Interaction Time (s)
1	87.66	66.63	48.05	1.25
2	95.21	78.50	62.77	1.27
3	90.54	Fail	50.01	1.14

, which indicates that the OGM-IRM algorithm achieved superior results compared with the process-structured navigation method used by the EL-E robot. The EL-E robot’s selection of a target navigation point that was far from the task target made it impossible for it to complete the third task.

The second experiment involved operating in a vertical environment to test the algorithm’s performance when the optimal base placement overlaps with obstacle information, as shown in Figure 11 and Figure 12. Specifically, the experiment involved completing tasks on a table drawer and an elevator button, with obstacles added to the latter to test the algorithm’s obstacle recognition ability.

For the drawer task, the EL-E robot’s process-structured navigation method failed as a result of the TCP being too far from the task target, a problem that was successfully overcome using the OGM-IRM algorithm.

The second task involved obstacle avoidance in an elevator shaft, which demonstrated the algorithm’s ability to optimize the robot’s position and avoid obstacles, as shown in Figure 13. The experimental results in

Table 2. The experiment results of the vertical task.

Task	Joystick Total Time (s)	Algorithm of EL-E Robot Total Time (s)	OGM-IRM
			Total Time (s)	Interaction Time (s)
Drawer	78.12	Fail	49.15	1.55
Elevator	65.54	49.21	37.87	1.47

indicate that the proposed OGM-IRM algorithm significantly reduces the task completion time and human–machine interaction time, enhancing the efficiency of completing high-difficulty tasks and expanding the range of executable tasks for the WMRA.

The experimental results in

Table 3. The experiment results.

Classified Time	Joysticks Time (s)	Algorithm of EL-E Robot Time (s)	OGM-IRM Time (s)
Human operation time	83.41	3.34	1.33
Non-human operation time	83.41	61.44	48.24
Total time	83.41	64.78	49.57

indicate that the proposed OGM-IRM algorithm significantly reduces the task completion time and human–machine interaction time, enhancing the efficiency of completing high-difficulty tasks and expanding the range of executable tasks for the WMRA.

In the task for the book, the algorithm of the El-E robot selects a relatively fixed docking position to the edge of the table, but the placement of the book is on the other end of the table, resulting in the WMRA being too far from the task target to complete the task as shown in Figure 14a. Similarly, in the task for the drawer, the algorithm of the El-E robot selects a docking position that is too close to the drawer to complete the task, as shown in Figure 14b.

In conclusion, the OGM-IRM algorithm presented in this paper enhances the capabilities of robotic arms in completing desktop tasks by enabling precise evaluations of target object positions that may have been difficult to reach because of surrounding obstacles. The algorithm also improves the task completion speed, demonstrating the benefits of positional optimization for subsequent robotic arm tasks and enhancing the coordination of the WMRA, as shown in Figure 15.

6. Conclusions

In response to the issue of low efficiency and inadequate coordination of mobile robotic arms in home environments, we propose a novel method named OGM-IRM. This method leverages the Cartographer algorithm to gather environmental data and store them in the form of an occupancy grid map. Subsequently, the working space and joint angles of the robotic arm are discretized, and distribution maps of the robotic arm’s grasping ability at various positions in the working space are generated. These maps are converted into an inverse reachability map to determine the optimal base placement of the mobile chassis by considering obstacle information in the occupancy grid map, thereby achieving base optimization based on the grasping pose. In summary, this paper introduces an approach for selecting the optimal base placement of a wheelchair robot in a home environment based on task objectives.

The proposed algorithm significantly enhances the efficiency of the WMRA in performing a wide range of home tasks and reduces the interaction time between humans and machines by a significant margin. Experimental validation demonstrates that the OGM-IRM approach can lower human–machine interaction time by 97.31% and reduce task completion time by 40.57%, increasing the range of executable tasks compared with the traditional method and reducing task completion time by 23.48% for the same task, showcasing its practicality and promoting its value.

There are still some shortcomings in the proposed method; for example, the current OGM-IRM algorithm necessitates substantial kinematic solving, leading to a long execution time and an inability to produce the optimal base placement of the WMRA in real time.

In the future, the algorithm needs to be further optimized in terms of execution time and computing resource consumption, so that it can be better deployed in embedded systems. To address these limitations, future research could consider screening the workspace to reduce the number of voxels and improve the algorithm speed.