Recognition Characteristics of Interface Designed for Gesture Operations of 3D Objects

This study focuses on the development of an interface for gesture control and Boolean operation of 3D objects in 3D computer-aided design (CAD). Although mouse and keyboard have been mainly used as computer input devices, novel computer interfaces are becoming increasingly popular with rapid development of computer technologies. Among several computer interfaces, the gesture interface is well-known as an system operable through the natural human behavior. However, the application to 3D CAD has been scarcely studied so far. From the backgrounds, a simple gesture interface was developed to operate a 3D object in 3D CAD. Leap Motion Controller (LMC) was used as a motion sensor device. A dialog-based graphical user interface (GUI) was also created for the parameter setting. As a result, several operational functions could be successfully executed through various hand-finger motions. Moreover, the system enabled us to experience an intuitive interaction with 3D CAD. Then, the several demonstrations were performed to identify the recognition characteristics of the gesture interface. The differences in the recognition rate were revealed as the results. Especially, it confirmed that a combination of static and dynamic gestures could help to enhance the recognition rate in the gesture interface using LMC. In accordance with the findings, the functions for Boolean operation were also implemented for the manipulation of a 3D object, such as extrusion and hole making.


INTRODUCTION
Next-generation interfaces are one of the key technologies for computer usage in our society. The related technologies have achieved remarkable development in recent years [1][2][3][4]. Thus, screen touch, voice input, gesture, etc. have become more common as input operations for a computer. Those natural user interfaces are operable through a natural interaction between a human and a computer. Additionally, each interface has some advantages compared with the conventional devices such as a mouse and keyboard. The gesture interfaces have continuously attracted attention from the early days, and various forms for gesture interface have been designed to replace the conventional ones [5].
As a sensor for markerless motion capture, leap motion controller (LMC) has been widely used to recognize the hand gestures and finger positions [6]. Figure 1 shows a practical situation of the gesture interface with LMC. The device is a compact sensor incorporating two CCD cameras, and the object positions can be obtained from the stereo-vision images. The detection accuracy of the sensors is approximately 0.01 mm. With the convenient features, the basic characteristics of LMC have been studied from various viewpoints. Vinayak and Ramani designed and evaluated a geometric interaction system using LMC for bare-hand mid-air virtual pottery [7]. Bachmann et al. analyzed the user's performance in selection tasks with LMC [8]. Then, they pointed out the movement times twice as large as for a mouse device and an error rate of 7.8% for the LMC and 2.8% for the mouse device. Naidu and Ghotkar investigated the gesture recognition of LMC for Indian hand signs [9]. A recognition system of hand signs for Japanese language was also studied by Kobayashi and Takeda [10]. Shao reported significant characteristics obtained by detection using LMC [11]. Guerra-Segura et al. discussed an evaluation of different measurements provided by LMC for the characterization of air strokes and their stability [12]. Smeragliuolo et al. reported the LMC's characteristics capable of providing the data that are clinically meaningful for wrist flexion/extension [13].
Some studies also focused on the several kinds of applications with LMC. Katahira and Soga proposed a system using LMC to produce a realistic picture of augmented reality [14]. Kincaida et al. proposed a system using LMC to measure tremors in clinical settings [15]. A prototype of an anthropomorphic gripper with five fingers was made by Staretu and Moldovan [16]. LMC was used as the controllable interface of gripper. Tang and Webb designed a contactless gesture control system using LMC for industrial robot [17]; moreover, they evaluated the system through a unique assessment method. Zubrycki and Granosik also described a system with LMC for 3-finger dexterous gripper [18].
Meanwhile, LMC has been applied to 3D computer-aided design (CAD) in recent years. Xiao and Peng proposed a hand gesture-based design interface using LMC for the review of CAD models [19]. Pareek and Sharma reported a LMC based CAD interface using number, position, velocity and direction of fingers as the input data [20]. Song et al. created a gesture interface using LMC for 3D modeling with applicable level of performance [21]. Except for these studies mentioned above, the authors are unaware of any study concerning the gesture interface with LMC for 3D CAD. In addition, the practical findings for the interface design have been scarcely obtained so far. Although the conventional devices are still used widely as input devices, the gesture interface would provide intuitive usability and additional value to 3D CAD. In this study, LMC was used as the motion sensor device of the gesture interface for 3D CAD, and the interface was set up to recognize several hand-finger motions for program functions, taking the advantage of the gestures for 3D CAD.
The following sections are organized as follows. Section 2 describes a gesture interface with LMC; moreover, the development of the interface, including the setting dialog, was reported. Section 3 represents the implementation of program, and the discussion was made in this study. Section 4 gives conclusions based on several findings.

DEVELOPMENT OF GESTURE INTERFACE WITH LMC
The system configuration of 3D CAD with a gesture interface is shown in Figure 2. The four main processes are Detection, Verification, Execution, and Perception. In Detection, the interface captures the hand-finger gestures through a motion sensor. LMC was used as the motion sensor device. As briefly described in Sec. 1, the device consists of two CCD cameras and three infrared LEDs. These track infrared light with a wavelength of 850 nanometers, which is outside the visible light spectrum [22]. The objects' positions can be obtained from the stereo-vision images. An interaction can be detected in an inverse pyramid area up to 600 mm. The apex of the pyramid is placed on the device with a maximum field of view of 150° [23]. The maximum frame rate is 300 fps, and the detection accuracy in fingertip position is approximately 0.01 mm. The relative study [24] reported showed that the accuracy for static setups was below 0.2 mm and that for dynamic setups was 0.4 mm in realistic scenarios. After the information including a hand form and motion was obtained from the former process, the integrity was instantaneously verified with reference to the predefined gestures in Verification. Swipe, circle and screen tap were set as the three gestures recognizable effortlessly in LMC. These gesture images are shown in Figure 3. Figure 3 (a) represents a swipe motion dealing with the hand-finger motions in vertical or horizontal direction. With reference to Figure 3 (b), circle was also defined as a hand-finger motion like circle drawing. The rotational direction is distinguishable according to either clockwise or counterclockwise motion. Figure 3 (c) shows a screen tap motion like pushing a screen in the detection space over LMC. Here, the gestures mentioned above are typical examples with the forefinger. Unlike those gestures, a brief stay was considered and newly added as a recognizable gesture in LMC. It is a two-handed, static gesture as shown in Figure 4, and the recognition was performed when the palms of two hands were held over the sensor. In Execution, a program function can be executed according to the setting behaviors in 3D CAD after a hand-finger gesture is detected contactless and verified through the interface. Then, a model on 3D CAD acts in accordance with the program function. Visual Studio 2015 was used as an integrated development environment (IDE) to develop the interface, and several functions were set using the IDE. Furthermore, the programs were written in Visual C#. Finally, the behaviors of 3D model are visually displayed in 3D CAD. Then, a movement of 3D model as a result output by the execution can be noticed. This is the Perception process. Inventor Professional 2019 was used as a 3D CAD because of the application programming interface (API) offering various program functions useful for creating a gesture interface.
A setting dialog was created to promote the integration between LMC and 3D CAD. On the dialog, various setting parameters are located and connected to program functions. Figure 5 shows the framework of dialog. There are several groups based on the characteristics of the program  Fig. 3. Gestures predefined in LMC [6] functions. The settings for coordinate systems are placed in the top left of dialog. We can adjust a sensitivity and deadzone's region of LMC through this group. The group concerning the "gesture" in the top center of dialog includes the gesture-related functions. This group includes the swipe and circle check boxes. One or both items can be selected, and some messages are displayed according to the ticks. If the state of these check boxes is true, the settings can be changed; moreover, the history of parameter change and gesture recognition is displayed on the "Log" area at the bottom of the dialog.
There are the settings for swipe in the second one from the top center of dialog. Minimum length and velocity are necessary for detecting the hand-finger motions associated with swipe. If the length or velocity of a hand-finger motion is smaller than the set values, a preconfigured program function is not executed. The settings for circle are also set in the second one from the top left of dialog. Minimum angle and radius are necessary for detecting the hand-finger motions associated with circle. A preconfigured program function can be performed unless the angle or radius of a hand-finger motion is smaller than the set values. The right or left hand can be selected using this dialog. After the modification using the dialog, the button "apply" in the top left of dialog is clicked to apply all the settings to the gesture interface of 3D CAD. Then, the corrections and gesture operations are continuously displayed on the "Log" area.

DEMONSTRATIONS AND DISCUSSIONS
A detectable region over LMC is defined on the basis of an orthogonal coordinate system with the X, Y, and Z axes. In LMC, the X axis is set as a horizontal direction. The Y axis is also set as the vertical direction. The Z axis is placed as a right-handed coordinate system with the X and Y axes. The gesture operations along with the Z axis are used for a near or far movement of 3D model and so on. Figures 6 to 9 illustrate each one-handed operation implemented in this study, respectively. The interface provides translation, rotation, and scaling. Compared with the conventional operations, the gesture operation offers an intuitive interaction with 3D CAD. One finger's translational motion led to 3D model's translational movement in accordance with the finger motion. However, in mouse operation, the mouse movement is required after pressing the wheel button. The 3D model rotation started with detecting more than four fingers, while the mouse operation entails the movement after pressing the wheel button and holding down the shift key on the keyboard. With detecting more than five fingers, the scaling of 3D model was adjustable through a near or far behavior of hand-finger motion over LMC. On the other hand, the turn wheel on a mouse needs to be rotated. For scale-up, the wheel is rotated in back direction, whereas, for scale-down, the wheel is rotated in forward direction. A 45-degree rotation of 3D model was also implemented when the swipe motion was detected as a one-handed operation with two fingers. The function can be  As a two-handed operation, the interface can also give a 90-degree rotation of 3D model and return to a home viewpoint. Here, a home viewpoint means an initial perspective in 3D CAD. One of the two-handed operations is a combination of static and dynamic gestures. A 90-degree rotation of 3D model is available through detecting a palm of one hand and swipe motion parallel to the X or Y-axis. As shown in Figures 10 and 11, detecting two hands is indispensable in the execution. In addition, a brief stay was designed as a static gesture of two hands. As shown in Figure 4, returning to a home viewpoint is performed when palms of two hands are held over the sensor for several seconds. The demonstrations of each gesture operation enabled to experience an intuitive interaction with 3D CAD. In conventional operations, ViewCube and Home-button shown in Figure 12 are usually used for achieving the same operations. Using a mouse, the 3D model can be rotated through clicking a surface, edge, or vertex of ViewCube. Returning to a home viewpoint is also executed after the Home-button is clicked.
As explained above, the program functions added to 3D CAD could be executed through the gesture operations recognizable in LMC. Thus, it was verified that the gesture interface created in this study could provide each basic function designed with practical consideration for the use in 3D CAD.
The gesture interface was investigated in operational tests. A 45-degree rotation, 90-degree rotation, and brief stay were selected as the conditions. Each operation represents a dynamic gesture, a combination of static and dynamic gestures, and a static gesture. The experimental conditions were set as 200 mm for the center height of detectable region, 100 mm for the minimum distance of swipe motion, 1000 mm/s for the minimum speed of swipe motion, respectively. Each operation was repeated 30 times to identify the recognition rates in the gesture interface created in this study. In the case of brief stay, the predefined gesture was given to revert to an initial perspective in 3D CAD after a 3D model was moved through the other operation. Then, the recognition rate within 10 seconds was evaluated. After the tests, each recognition rate was calculated through counting the logs in "Log" area.  6. Translational motion of 3D model Fig. 7. Rotation of 3D model Fig. 8. Expansion of 3D model Fig. 9. Reduction of 3D model Figure 13 shows the recognition rates obtained in each operational test. In the figure, each label of column chart represents Dynamic for dynamic gesture by means of a 45-degree rotation with one hand, Combination for a combination of static and dynamic gestures by means of a 95-degree rotation with two hands, and Static for a static gesture by means of brief stay with two hands, respectively. From the results, the recognition rate of brief stay was the highest among the three conditions. The numerical value was 96.7%. In this test, almost all of the gesture operations were recognized except for once. As a cause of the failure, it was considered that one hand exceeded the detectable region. On the other hand, the recognition rates in 45-degree and 90-degree rotation were 71.7% and 88.3%, respectively. The comparison of results in 45-dgree and 90-degree rotations indicated a beneficial feature which a combination of static and dynamic gestures had; namely, a combination of both gestures can help to enhance the recognition rate in gesture interface using LMC. Two possibilities are offered by this comparison. It is highly possible to recognize a dynamic gesture of one hand readily if the detection state is preliminarily made by a static gesture of the other hand. The other possibility is that a static gesture of one hand becomes a visual mark of a detectable region when a dynamic gesture of the other hand is made over LMC. Accordingly, it was considered that the recognition rate was improved with benefit from the synergetic effect of them discussed above. Several Boolean functions in 3D CAD were customized based on the recognition characteristics. The functions were executed through a combination of the static and dynamic gestures. As shown in Figure 14, screen tap was used as a dynamic gesture. In the operations, the gesture interface using LMC enables us to execute Boolean functions in 3D CAD after a 2D reference shape was preliminarily given to create a 3D shape. Moreover, several operations executed using screen tap are selectable through changing the number of fingers. Figure 16 represents a few of the examples. Extrusion can be executed with one finger and hole making can be achieved with two fingers.

CONCLUSIONS
In this study, a gesture interface using LMC was designed to operate 3D objects in 3D CAD, and it confirmed that several functions could be successfully executed. One finger's translational motion led to the 3D model's translational movement, in accordance with the finger motion. The 3D model rotation started with detecting more than four fingers. In addition, the scaling of 3D model was adjustable through a near or far behavior of hand-finger motion over LMC. A 45-degree rotation of a 3D model was also implemented as a swipe motion using one hand. A 90-degree rotation of a 3D model was available using both hands making a combination of static and dynamic gestures. The interface also enables to return to a home viewpoint with detecting both hands. Furthermore, several Boolean functions in 3D CAD could be executed through a combination of static and dynamic gestures. They are selectable through changing the number of fingers in a dynamic gesture. Each gesture operation indicated that the developed gesture interface could offer an intuitive interaction with 3D CAD.
Several demonstrations were performed to identify the recognition characteristics of the gesture interface. The differences of the recognition rate were revealed as the results. Especially, the comparison of results in 45-degree and 90-degree rotations indicated a beneficial feature of a combination of static and dynamic gestures. In other  words, a combination of both gestures can help to enhance the recognition rate in the gesture interface using LMC. Thus, a static gesture should be simultaneously given with a dynamic gesture to design a gesture interface using LMC with high recognition rate.
In the future work, further manufacturing-related functions would be designed and implemented to realize intuitive operations in 3D modeling.