Usability evaluation for driving simulation with the mechanical and joystick manual controllers

For lower limb disabled driving a car, mechanical manual controllers are mainly used to control the brake and accelerator. However, the joystick is used chiefly to drive an airplane; is seldom used in driving a car. This study aimed to evaluate the usability of the joystick-style and mechanical manual controller for the lower-limb disabled driving vehicle. Twenty participants were divided into experimental and control groups (10 persons for each group). The experimental group was lower disabled the control group was the non-disabled. Each subject performed a driving simulator experiment with the two manual controllers. Driving performance, physiological load


INTRODUCTION
According to WHO and World Bank in the World Report on Disability (WORLD REPORT ON DISABILITY, 2011), more than 1 billion people have disabilities worldwide. They lack services in daily life and face many obstacles and difficulties, for example, poorer health outcomes, lower educational achievements, less economic participation, higher poverty rates, increased dependency and restricted participation, etc. According to statistics from the Ministry of Health and Welfare of the Republic of China (2020), there were 1,186,740 disabled people in Taiwan in 2019.
The percentage of disabled individuals in Taiwan is 5.03%. The most significant portion of which are physically disabled at 30.35% or 360,234. As the number of people with disabilities continues to grow, issues related to barrier-free environments have gradually caught public attention in recent years. However, solving the problem of physically disabled people, significantly how to drive a car effectively and safely, deserves further discussion. For drivers with disabilities, customized and modified vehicles are the most critical way to enable them to act independently. By driving a car, people with disabilities can improve their independence and quality of life (Peters and Östlund, 2005). Going by themselves is the goal of the acquired recovery of the disabled, and it is usually necessary to modify the vehicle to promote this result (Hutchinson et al., 2019). The main reason is that people with disabilities cannot live independently due to their inability to control their vehicles. Under certain conditions, they cannot successfully find employment without a vehicle. The research report of Di Stefano et al. (2015) points out that most of the participants in their survey will not be able to drive independently without modification. The most common health conditions that require vehicle modification are paralysis or spinal injury. The most modified items consist of manual controls and steering aids. If physically disabled people can drive a car, they can improve mobility and reduce dependence on family members. They will also increase social participation and opportunities for leisure and recreational activities. They even can go out for work and improve the family's financial situation.
People with lower limb disabilities generally use wheelchairs as transportation when performing short-range movements. After treatment, rehabilitation, and a doctor's assessment of suitability for driving, you can obtain a modified car driver's license for long-distance mobility. Manual control devices refer to driving a car using a particular control device that replaces the foot pedal function. It allows people with lower limb disabilities to drive the vehicle independently. Currently, the standard manual control devices on the market have two styles: linktype and joystick-type. The link type has been widely used in vehicle modification, while the joystick type is used primarily for flight controllers, game consoles, and electric wheelchairs. Peters (2001) studied 26 drivers with tetraplegia and compared their driving performance and workload when using two types of linkage manual controllers for acceleration and braking. It is found that when the driver uses separate levers, the standard deviation of the lateral lane position is more significant. And using a combined lever will make you more tired due to acceleration and braking. Compared with non-disabled drivers, quadriplegic drivers have a longer reaction time and must generate more load and spend more energy to reach the driving level. Compared with regular drivers, drivers with tetraplegia need longer reaction time, create more loading, and devote more energy to get the driving class. Tudor (2015) developed a drive-by-wire linkage manual controller, and human trials have shown that this controller can improve the driving performance of people with spinal cord injuries.
It contains two main control elements. One is a lever device that controls acceleration and braking, and the other is a small wheel device that controls steering. Peters (2004) has collected six different styles of linkage manual controller samples. The primary purpose of these modification cases is to help these drivers drive as safely as drivers without disabilities. Different types of link-type manual controllers have other operating methods. However, which manual controller is most suitable for people with spinal cord injury or lower limb disability? There is still no relevant standard. In other words, developing evaluation methods for modified vehicles to test the driving performance of disabled people and evaluate their usability is a fundamental issue.
Through the data analysis of joystick controls to measure the driving skills of wheelchair users, novice or professional users can be distinguished in various driving tasks. In more manageable tasks, the driving skills of the expert group are equivalent to those of the novice group. Still, in more challenging and space-constrained tasks, the expert group uses less joystick movement to complete the job, and the time required is about half of the novice group (Sorrento et al., 2011). Considering the opinions and expectations of patients with lower limb disabilities, a drive-by-wire joystick-type manual controller has been developed, and the vehicle's driving performance using only the upper limbs has been further studied. This device only needs to move the hands and arms to control the vehicle's steering, accelerator, brakes, and gears. Mrabet et al. (2018) introduce a method and process of implementing an intelligent control system on an electric wheelchair. The system adds an artificial intelligence algorithm to the typical joystick manual controller. This can correct all hand movements of the disabled. Compared with the traditional joystick-type manual controller, it has a better operation effect. Different from the link-type and joystick-type manual controllers, Murata and Yoshida (2013) have designed a gestureoperated car steering interface for the disabled. Experiments using a gyro sensor and a driving simulator prove that the interface is similar to the traditional steering wheel operation method. The results of many studies (Liu et al., 2013;Park et al., 2004;Zheng et al., 2017) have shown the potential of the car to steer through the joystick, which is comparable to the traditional steering wheel, but in the formal investment before production, more stability and safety tests are needed.
Virtual reality (VR) driving simulation helps reduce the fear of patients with spinal injury when driving, and it can safely evaluate and improve the driving ability of patients (Ku et al., 2002). Sung et al. (2012) incorporate virtual reality (virtual reality) into the rehabilitation plan of patients with spinal injuries, which can accelerate the recovery of the patient's driving ability. After simulated driver's training, participants can: park the vehicle more accurately at the stop line and significantly reduce traffic violations. Compared with actual road driving tests, VR driving simulation is expected to improve the immersion and cost-effectiveness of simulated driving, and it can provide a safe test environment and objective data to measure driving performance (Kim et al., 2019). Jung and Kim (2012) use intelligent devices (like iPod and iPhone) to develop modified vehicles suitable for people with physical disabilities and evaluate their performance in VR driving simulation experiments. They have found that people with disabilities, compared to those with regular driver's licenses, can improve their driving skills by driving the simulated training device and have the same stable driving performance. The above research results show that VR driving simulation has many advantages for the driving training process, such as objectivity, cost reduction, efficiency, and safety improvement. If you use VR driving simulation to evaluate the driving performance of vehicle modification, I believe it can also have the effect of learning and training.
Due to the previous research on refitting vehicles, most of the extended discussions and designs have been carried out on the steering wheel with the link-type manual controller. Still, the joystick-type manual controller is rarely studied. In addition, joysticks are commonly used in equipment such as electric wheelchairs, aircraft pilots, and physically disabled people. There is little research to explore its application in modified vehicles, and there is no relevant research to evaluate the comparison of link-type and joystick-type manual controllers of the difference in driving performance. Therefore, the primary purpose of this study is to explore which type of manual controller is more suitable for people with lower limb disabilities. In addition to comparing the driving performance of link-type and joystick-type manual controllers, it also explores how participants use the two controllers for driving to physiological loading.

Independent Variables
This experiment has two independent variables: the subject group and the manual controller. The description is as follows: (1) Subject group The groups of test subjects are divided into the experimental group and the control group. The subjects in the experimental group are participants with lower limb disabilities, aged between 18 and 60 years old, five have a car driver's license, and five have no car driver's license. On the other hand, the subjects in the control group are those with sound limbs and have no history of musculoskeletal disease in the past. Five had a car driver's license, and five did not have a car driver's license.
(2) Manual controller During the experiment, there are two types of manual controllers: mechanical (I. mechanical) and joystick operation (II. joystick). The two methods are different in controlling a virtual car, as mentioned in section 2.2.1.

Dependent Variables
The dependent variables to be collected during this experiment include three parts: driving performance, physiological load, and System Usability Questionnaire (SUS) score, which are described as follows: (1) Driving performance a. Total grade The total grade is 100 minus all of the deduction of points in the eight driving tests. Each of the eight driving tests had criteria for the deduction of points according to the Taiwan driver's license examination regulations. When a subject violated a rule, the scoreboard displayed deducted points immediately, a warning sound was heard, and the counter recorded the total number of line touches. In five driving tests, including curve advance and retreat, crossroads, up and down ramps (railway level crossings), zebra crossings, and straight acceleration, 32 points will be deducted if the subject violates one regulation. In the three items of reversing and warehousing, parking on the roadside, and up and down ramps, 16 points will be deducted if one rule is violated; no deduction will be counted when driving around the field. b. Number of violations Sensing pipelines are on both sides of the road or in front of the signs. If the tire presses on the sensing pipeline when the vehicle moves, it is considered a violation. The number of repeated pressure pipes is accumulated until the experiment is completed. Detouring only judges that the driving is stable and unstable, so the number of times is not counted. c. Completion time In the formal driving training experiment, the driving completion time can be used to judge the participant's proficiency in the driving training items, and the driving completion time of the participants in the eight training items is recorded to determine whether the experimental group's proficiency is better than the control group.
(2) Physiological load This experiment uses a Polar heart rate watch for sports. The heart rate is measured wirelessly during the investigation, and the subject's heartbeats per min (bpm) can be obtained. The participant's heart rate was recorded every min at the beginning of each experiment. During the experiment, the participant's heart rate was recorded every 5 s until the end of the experiment. After the experiment, we can obtain the average heart rate (AHR), the maximum heart rate (MHR), and the heart rate at rest (HR_rest). This is to obtain relative heart rate (RHR) as the following: (3) System Usability Questionnaire (SUS) score The quality of the controller design needs to be evaluated by suitable standards, and the System Usability Scale (SUS), after some modifications, is an appropriate evaluation tool. Brooke (1996) pointed out that SUS is reliable and correlated with other subjective measurements. SUS was first used to measure the usability of the software. The advantage of SUS is that it is simple, easy to use, and does not require too much time.

Subjects
In this study, 20 people were recruited as voluntary participants and divided into experimental and control groups, as shown in Table 1. Ten people with lower limb disabilities with an average age of 52.2 years were in the experimental group. Ten healthy adults with an average age of 24.3 years were the control group. Five participants in the experimental group had a car driver's license, and the other five did not have a car driver's license, as did the control group. And explain to the participants the experimental process and purpose. It is necessary to use two different manual controllers for the experiment. Before the investigation, to let the participants be thoroughly familiar with the manual controller before the formal experiment starts, let the participant practice operating the controller at each level of the scene. After 30 min of operation, it was confirmed that the participant had adapted and understood the operating mode of the manual controller and then formally started the experiment. All test subjects must sign the test consent form.

Experimental Equipment
The driving simulator system uses a projector (TOSHIBA) to project onto the screen (length 135 cm, width 105 cm) as a visual presentation device for the virtual situation. It allows the experimenter to observe the subject's driving behavior in the driving simulator system during the experiment. Fig. 1 shows the driving simulation system in the laboratory. The following indicates essential equipment in this study.

Two types of Manual Controllers
This study has two types of manual controllers for driving a car. One is the mechanical manual controller (I. mechanical), and the other is the manual joystick controller (II. joystick), as shown in Fig. 2 and Fig. 3. These two different driving operating methods are detailed in Table 2.  The virtual driving training system was established in the previous study (Chiu et al., 2020). The virtual field was constructed using Virtools Dev 4.0 according to the scale of an actual driving training class using 3D Max (see Fig. 4). The test items in this study referred to the existing road test items used in Taiwan. This experiment included eight simulated driving test items: (1) around the field, (2) Sshaped forward and backward movement, (3) reversing the car into a garage, (4) performing parallel roadside parking, (5) moving up and down a hill, (6) forked road intersection (traffic lights), (7) crosswalk (flashing yellow lamp), and (8) stability when changing gears (linear acceleration).

Heart Rate Watch
The Polar heart rate watch for sports (RS800CX) was used. It includes a training heart rate monitor used as a wireless heart rate receiver display. After the experiment, use the Polar infrared transmitter (IrDA USB) to transfer the heart rate data to the computer software Pro Trainer 5. According to different subjects, read the heart rate of each data and then use the Excel software to remove the extreme value of the heart rate.

Experimental Procedures
Before the formal experiment, the entire experiment process, purposes, and other relevant precautions were expressed to the participant. Suppose there was no doubt and agreement to this research experiment; in that case, the subject was asked to fill in the experimental consent form, the subject's basic information. To avoid the subject being unfamiliar with the control method of the manual controllers for driving a car, the different control modes were explained twice so that the test subject understood each controller's control mode, allowing the test subject to use these two controllers. In the virtual driving training class scene, adapt by familiarizing yourself with the operation of various controllers. Practice driving a few laps around the field, curving forward and backward, reversing into the garage, roadside parking, etc., first give the test subject half an h of practice. After the training, the experimenter would observe the test subject driving around the field for one lap and then determine whether the subject is proficient and can start a formal experiment. If the driving situation is unstable, give a period to adapt to the ordinary operation until the subject feels the practice is sufficient.
Eight standard driving tests will be carried out when the subject has adapted and understood exactly how the controller operates. Each experiment will be carried out randomly, and each will be carried out three times. After the subjects completed an experimental test, they were asked whether they were fatigued. If you need to rest, give the test subjects a 3-min rest, and proceed with the experiment until the eight experimental tests are completed. After the experiment, the subject is asked to fill out System Usability Scale (SUS).

Data Statistics and Analysis
After the experiment was completed, using SPSS 12 statistical software to analyze the experimental data and calculate the narrative statistics of each dependent variable, such as the average and standard deviation. Variance analysis was used in driving performance, physiological load, and SUS score to judge the differences between the subject groups and manual controllers. The significance Level is set to α = 0.05. Tables 3 and 4 show the driving performance results during the overall driving experiment. The overall driving experiment includes eight driving tests. There was a significant difference (p < 0.01) between the two manual controllers in the average number of violations (ex: pressing to the sideline). It is noteworthy that significantly fewer violations occurred with the joystick than with the mechanical manual controller. However, there was no significant difference between the mean total grades between the two manual controllers. The control group had significantly higher total grades, fewer violations, and shorter completion times than the experimental group.   Table 4 shows the completion time results for each driving test item. The experimental group took significantly more time to complete each driving test item than the control group. The top three driving test items that the experimental group took the most time to complete were items (2), (3), and (4), as shown in Table 5. These three items are all difficult items, and most subjects would have more violations (ex: pressing to the sideline). The following subsection will present the detailed number of violations and analysis of the physiological load results for the three difficult test items.

Number of Violations and the Physiological Load for the Three Difficult Test Items
Tables 6 and 7 show the number of violations and the physiological load results for those, as mentioned earlier, three difficult test items. It's worth noting that using a joystick would result in significantly fewer violations than using a mechanical manual controller in test items (2) and (3). Further, a joystick would result in significantly fewer relative heart rates than a mechanical manual controller in all three difficult test items. Considering the effect of the subject group, the experimental group had significantly more violations than the control group in test items (2) and (3). In addition, the relative heart rates of the experimental group were significantly less than those of the control group in all three difficult test items.

SUS Score
Use SUS to evaluate the suitability of the tested manual controllers. There was no significant difference in the SUS score between the two manual controllers. Still, there was a significant difference in SUS score between the two groups after the experiment (see Fig. 5). Further observation shows that the SUS score (51.0) of the mechanical manual controller for the control group is lower than that of the joystick type (60.5); the SUS score of the experimental group for using the mechanical manual controller (78.3) is higher than that of the joystick type (70.8).

The Manual Controller Effect
For the overall driving performance, it is worth noting that using the joystick controller caused significantly fewer mean violation numbers (6.5) than the mechanical one (13.1). It is because the joystick controller has better control of the driving direction and consequently causes fewer violation numbers, such as pressing the sideline. Furthermore, fewer violation numbers shortened the completion time because the participants had to reverse the vehicle to correct the driving direction every time they pressed the sideline. Consequently, the joystick controller spent significantly less mean completion time (482 s) than the mechanical one (649.2 s). Therefore, the joystick controller is superior to the mechanical one in the overall simulated driving tests.
When it comes to the most three difficult driving test items (see Tables 6 and 7), violations were still significantly less when using the joystick controller. In addition, according to the relative heart rate data, using the joystick controller also has a significantly lower physiological load compared to the mechanical one. Because using the joystick to control the car's movement requires only one hand and less effort. However, with the mechanical controller, you must use the left hand to hold the steering wheel and the right hand to control the acceleration and braking so that the effort will be more significant.
For the SUS (System Usability Scale) results, there was no statistically significant difference between the two manual controllers. So subjectively speaking, whether it is a mechanical or joystick controller, it is generally considered to have similar usability. If looking at individual groups of subjects, the SUS score of the mechanical manual controller (51.0) is lower than that of the joystick (60.5) in the control group; in contrast, the SUS score of the mechanical manual controller (78.3) is higher than that of the joystick (70.8) in the experimental group. Judging from the mean SUS score, the control group thinks the joystick type is more suitable for use. Still, the experimental group feels the mechanical type is more suitable for use. However, these differences were not statistically significant.

The Subject Group Effect
Based on the mean total grade, the experimental group obtained a significantly lower total grade (57.4) than the control group (68.5). This is because the experimental group's mean age is much older than the control group; therefore, the experimental group's action of operating the driving simulator is relatively inflexible. For the results of violation numbers, the mean violation number of the experimental group (13.7) was significantly more than that of the control group (5.9). This may be because participants with lower extremity disabilities had poorer handling stability in the driving simulator, so they could easily violate the rules by pressing on the sideline. Furthermore, the average completion time of lower limb disabled participants was more (716.4 s) than that of the healthy participants (414.7 s). This finding is similar to previous studies (Peters, 2001;Chiu et al., 2020), showing that the reaction time of lower limb disabled participants is longer than that of the healthy participants.
For the most difficult three test items, the experimental group spent nearly twice the completion time of the control group, as shown in Table 5. This means that when the driving test becomes more complex, people with lower extremity disabilities take more time to complete the driving test work than healthy people. In addition, in the two most challenging driving tests, it was found that people with lower extremity disabilities more than doubled the number of violations on the simulated driving test than healthy people.
Interestingly, the mean relative heart rates of the experimental group in the most difficult three test items were significantly lower than those of the control group. This may be due to that the mean age of the experimental group (52.2 years) is substantially older than that of the control group (24.3 years). In addition, the lower limb disabled participants have had experience in driving with the refitted manual controller and are more familiar with the operation of the tested manual controller. Hence, they were not as nervous as the control group. So, their physiological load is smaller than that of the inexperienced control group.
From the SUS scores, there was a significant difference between the two groups of participants. The experimental group gave higher usability ratings for these two controllers than the control group. This may be because the experimental group is usually used to driving refitted cars, so the evaluation of these two manual controllers is higher. On the other hand, the control group uses their feet to control the accelerator and brakes and is not used to manual controllers in driving a car. Hence, the evaluation of these manual controllers is lower.

Limitations
Due to cost and time factors, the numbers of the two genders in the recruited experimental group were inconsistent, and the number of subjects in the experimental group was primarily female. Besides, the age difference between the control and experimental groups in this study was significant. Gender and age may affect driving performance. Future research can expand the sample of experimental subjects to discuss the effects of gender and age.
In addition, because the experimental scene is set in a driving training class, it is designed to be a slower driving situation in terms of vehicle speed. But in actual road driving, there will be high-speed driving. Therefore, this study does not determine whether using the joystick controller to drive on high-speed roads is safe. Further research is required to confirm the appropriateness of using the joystick controller on high-speed roads.

CONCLUSION
The purpose of this study is to evaluate which of the joystick and mechanical hand controllers is more suitable for use. The experiment obtained the driving performance, physiological load, and SUS scores of the lower limb disabled and healthy people using the two manual controllers. These research findings can be summarized as follows: (1) When using the joystick controller, the subjects performed better than the mechanical controller in the number of violations and completion time; (2) When using the joystick controller, the physiological load of the subjects was significantly lower; (3) The subjects' subjective evaluation of the joystick controller is similar to that of the mechanical controller. However, further research must confirm the appropriateness of using the joystick controller on actual high-speed roads.