Glove-based Interaction in Virtual Surgical Training System

A virtual training system has been developed to train novice surgeons on using the new jig and fixture in a knee replacement surgery. In this study, we described how a virtual hand was employed in a Virtual Knee Replacement Surgery Training System. The virtual hand is equipped with HOMER-G, a selection and manipulation technique that combines the Arm-extension and Ray-casting technique for easy selection and manipulation of virtual object with the virtual grasping capability during selection task. The virtual jig and fixture are selected using Raycasting technique and the arm extension technique of HOMER-G will help to manipulate the object. The virtual hand’s movement is controlled by a data glove and Flock of Birds tracking device. HOMER-G has the grasping capability that makes the manipulation of object looks more natural.


INTRODUCTION
Virtual environment provided an intuitive way for human-computer interaction.Mine (1995) had identified three major categories of interaction technique in a virtual world.The three categories are direct user interaction, physical controls and virtual control.Direct user interaction includes the use of hand tracking, gesture recognition and pointing to specify the parameter of the interaction task.It is a natural intuitive human-computer interaction.This makes virtual hand an important avatar for user in direct user interaction technique and data glove as a tool to simulate user hand movement.Data glove that consists of numerous sensors is able to give feedback on real-time hand movement signal to the computer to trigger the virtual hand movements in VR application.Therefore the virtual hand can complete the interactive operations the same as real hand, for example, conveniently grasping and releasing objects.Many researchers had adopted this technique to provide an intuitive way for the user to interact with the application.Those applications include tasks like assembly (Sankar et al., 1999) folding of sheet metal plates (Pouliquen et al., 2007), midwifery training (Pan et al., 2007) and sketching (Kavakli and Jayarathna, 2005).
Virtual training system in the medical field is a combination between 3D image modeling and virtual reality technique as a clinical training for the doctors (Sourina et al., 2000).It is also an alternative to the conventional training before the doctors implement the real process in the operating room (Gallagher et al., 2005).There are many types of virtual training system that were developed for medical training purposes which focused on certain learning aspects such anatomical studies, surgical procedure and surgical instrument usage (Gibson et al., 1997;Pheng-Ann et al., 2004;Posneck et al., 2005;Sabri et al., 2010).
In this study, a virtual training system for newly designed jig and fixture usage in computer assisted knee replacement surgery was developed.The jig and fixture is a surgery tool used to align the cutter blade in the right position before the sawing begins.The research started when the surgeon having a problem to align the existing jig during the computer assisted knee replacement surgery.The process to position the jig at the right angle will take a lot of time.To overcome this problem, the jig and fixture was redesigned by a group of researcher.Training is required for the novice surgeons to practice on the jig and fixture usage before the knee replacement surgery.The right technique to apply the jig and fixture is important to reduce the time taken to get the optimum position of the jig during the surgery (Awang, 2009).Thus, a virtual training system was developed to assist the computer-assisted knee surgery where the user can interact with virtual environment to determine the optimum position of the jig at tibia.A virtual hand has also been created for selecting and manipulating the jig and fixture to provide natural interaction between the surgeon and the virtual training system.

LITERATURE REVIEW
A large number of interaction techniques have been developed.The classical approach to design the direct manipulation for virtual reality application is to provide the user with virtual hand.Virtual hand is a human hand shaped like 3D cursor.The visual and motion realism of virtual hand as a visual feedback in the immersion of the virtual environment is important and helpful to enhance the immersion of virtual environment.The virtual hand model developed by Bin and Shuling (2009) and Wan et al. (2009) are good enough to fulfill the quality of visual feedback requirement.
Selection and manipulation include choosing a virtual object and setting its position by controlling the virtual hand.Selection and manipulation technique can be divided into two categories which are arm-extension and ray-casting technique (Bowman, 1996).The armextension technique allows the user to extend their reach in a virtual environment by using non-linear mapping to the user's hand.It provides the ability for users to extend their arms in virtual environment much further than normal reach (Poupyrev et al., 1996).The Ray-casting technique will eliminate a ray from virtual hand to point at the object for selecting and manipulating objects.The object can be chosen and manipulated when the virtual ray intersects with the virtual object.Several types of ray-casting technique have been developed such as aperture (Forsberg et al., 1996), spotlight (Liang and Green, 1996) and laser ray technique (Jacoby et al., 1994).
A great deal of research effort has been directed toward developing virtual reality surgery training in recent years and various interaction techniques have been applied.Studies on how to enhance the quality of visualization and 3D imaging for medical purpose have been done rapidly for many types of surgery.Various techniques have been applied to produce a quality 3D image and real effect of visualization and simulation.Due to cost effective and more efficient method compared to traditional methods, a virtual-reality training system for knee arthroscopic surgery was developed (Pheng-Ann et al., 2004).The system simulates soft tissue deformation with topological change in real-time using finite-element analysis and offers realistic tactile feedback using tailor-made force feedback hardware.The system presented mesh generation, real-time soft tissue deformation and cutting and collision detection.A virtual reality simulation system of telerobotic spine surgery was proposed for training surgeons in developing basic skills needed in telerobotic spine surgery.This system consists of two Phantom Omni haptic devices to provide the force feedback to the user.Immersion of the system is also improved by the haptic device that provides haptic sensation during drilling (Xie et al., 2011).
The complexity of the procedure and high level of costs for surgical training (caused by traditional training methods like anatomical studies and surgical training on human cadavers) were the decisive reasons to develop a virtual training system in endoscopic sinus surgery (Slack and Bates, 1998).In this research, they generated the virtual model based on MRI dataset of real patient.The dataset was then segmented in VE-SUV (segmentation software for medical image dataset).A virtual training system for surgery makes the inexperienced surgeons better prepared for the real operation (Weghorst et al., 1998;Rudman et al., 1998).
Virtual reality technology is one of the novel tools with great potential in medical field.The application of this technology is not limited to the systems described above.Other applications such as virtual dental surgery (Pohlenz et al., 2010) used in dental school had recommended that virtual simulation can be an additional modality in dental education.In cataract surgery simulator (Choi et al., 2009) is a virtual reality system for trainees to practice phacoemulsification procedures with computer-generated models in a virtual environment.
The learning practice of Minimally Invasive Surgery (MIS) makes unique demands on surgical training programs and that inspired the researcher to develop a virtual reality simulation for operating room (Gallagher et al., 2005).The focus for this research is proficiency-based training as a paradigm shift in surgical skill training.The results show that VR is more likely to be successful if it is systematically integrated into a well-thought-out education and training program.Validated performance metrics should be relevant to the surgical task being trained in the VR training program.They also stressed that VR is only a training tool that must be thoughtfully introduced into a surgical training curriculum to improve surgical technical skills.Gibson et al. (1997) and McCarthy and Hollands (1998) applied real-time techniques in the knee arthroscopic surgery system.The focus mostly relies on the high-end workstation for real-time visualization.Imaging technologies have been used for orthopedic visualization and simulation (Peng Ann, 2007) for orthopedics training of the upper limb region.Tissue identification was performed on the CVH (Chinese Visible Human) volume data.Volume-rendered visualization algorithm is used to perform translation and rotational transformation on the volume data set.

Development of the virtual training system:
The processes involved in the development of the Virtual Knee Replacement Surgery Training System for Knee Replacement Surgery (Ramli et al., 2010) are shown in Fig. 1.
Many virtual objects appear in virtual environment are created through the process of modeling.Modeling is an important step to make the virtual reality system successful.The modeling process is better to be completed at the earliest stage of the development phase.The main 3D models created in the system are the hand, knee bone, jig and fixture.There is various modeling software available in the market, it is essential to choose suitable software to handle the modeling process.3ds Max offers an excellent development environment for creating high quality 3D models.It supports a wide range of modeling techniques from low polygon modeling to modeling with compound objects to mesh.When the virtual object has been constructed, 3ds Max can also allow the user to assign different textures on it to improve the realism of the virtual object.3ds Max is the main tool for modeling process in this study, the virtual hand is constructed by using this software while the knee bone, jig and fixture are modeled using Solidworks (2007) and CMM machine.
To permit the interaction between the virtual models, the visualization and animation part were developed using the Unity 3D software.Besides that, the collision detection algorithm was applied to trigger the detection between jig and tibia.This is to the jig from penetrating the tibia while the visualization is running.The technique utilizes the narrow phase approach in detecting the intersection between two virtual planes (Ramli et al., 2010).
3D modeling of jig and fixture: The jig and fixture used in the knee replacement surgery include the guiding block, base and arm.The models were produced in 3D by using the 3D software (Solidworks, 2007) as in Fig. 2.
In order to produce a virtual process on how to position the jig, other components like bone, pin and the bone saw have to be modeled.In this study, segmentation on a series of Magnetic Resonance Imaging (MRI) image was made and a volume in which the bone part was tagged was obtained.After that, the surface mesh is created from the series of 3-D contours using Catia V5 (Fig. 3) and will convert the surface mesh to solid.

Virtual hand modeling:
The complexity of the hand structure makes its shape modeling a complicated and tedious process.Many modeling techniques, from polygonal modeling to parametric surface modeling have been proposed for modeling geometry of the human hand.For virtual hand modeling, the built polygonal model should be accurate enough to reflect the human hand shape and on the other hand it should not be too complicated to hinder the real time simulation of hand motion (Wan et al., 2004).Hand shape model: In our study, we used polygon modeling to make the virtual hand as close to the real human hand as possible (Lam and Arshad step of the virtual hand modeling is to set up a reference plane so we can refer to the plane while virtual hand.The reference plane is the picture of a real human hand from different angle which are viewed from top, bottom and side.It makes the modeling process easier and accurate compared to modeling without any reference.After setup is finished, a box is created and then converted to editable poly.Ring, cut, In our study, we used polygon modeling to make the virtual hand as close to the real Arshad, 2011).First step of the virtual hand modeling is to set up a reference plane so we can refer to the plane while modeling the virtual hand.The reference plane is the picture of a real human hand from different angle which are viewed from top, bottom and side.It makes the modeling process easier and accurate compared to modeling finished, a box is created and then converted to editable poly.Ring, cut, bridge is then used to create more vertices on the box and start moving vertices to form the palm of the virtual hand.After that, the finger base shape is created by using an extrude function to achieve more accurate finger form.Finally, relax function is applied on the virtual hand to smooth out the messy geometry.The hand shape model is as illustrated in Fig. 4.

Skeleton model:
The movement of the virtual hand is depending on the skeleton model.This model is constructed based on the anatomic bridge is then used to create more vertices on the box and start moving vertices to form the palm of the virtual hand.After that, the finger base shape is created by ude function to achieve more accurate finger form.Finally, relax function is applied on the virtual hand to smooth out the messy geometry.The hand shape model is as illustrated in Fig. 4.
The movement of the virtual hand is depending on the skeleton model.This model is anatomic structure of the  and Lawrence, 1997).
Each joint has local coordinate system; the motion of one joint is affected by other joints.As shown in Fig. 6, this model was built using bone feature in 3ds Max.The bones system is a jointed, hierarchical linkage of bone objects that can be used to animate other objects.It is a suitable technique to be used for building the skeleton model to represent the anatomic structure of the human hand.

Skin layer:
The skin layer is for displaying purpose just like the skin of humans.A skin material has been created to make the virtual hand more realistic.Materials describe how an object reflects or transmits light.3ds Max provides an option for applying texture to object.We have used texture mapping to reach a good visual realism.The skin texture is used as the texture.Material Editor is the dialog that can be used for creating and altering materials and also applying and adjusting the mapping.After the properties of the material have been adjusted, it is ready to be applied to an object by dragging and dropping it from the material editor to the virtual hand.The skin layer of the virtual hand is as shown in Fig. 7. Now the skeleton model can be mounted into a virtual hand in order to animate the virtual hand to perform grasping and releasing movement.The skin modifier is applied to all bones and assigns the bone to the hand shape model.The bones are animated in 3ds Max by setting the animation key frames where the hand shape model will follow the movements of the bones.Key frames record the beginning and end of each transformation of bone.The grasping and releasing movement can be done by setting the appropriate rotation value for finger joint and recorded in key frames.Figure 8 shows the virtual hand grasping animation.

HOMER-G technique:
A one hand selection and manipulation technique has been designed and named HOMER-G.Our HOMER-G pro illustrated in Fig. 9. HOMER-G is an improvement of HOMER developed by Bowman and The difference between HOMER and HOMER HOMER-G has an additional virtual grasping capability in the selection task.This new ability incre nature of virtual hand movement and provides a better visual feedback to the user.In HOMER object is shot by the ray from virtual hand, the virtual hand will start moving towards the object.Virtual will stop in front of the virtual object and the user can start to fist their hand to select the virtual object.The object then can be manipulated directly by virtual hand.After manipulation is done, the object is released and virtual hand will return to its initial position.
HOMER-G technique is applied to the Virtual Knee Surgery Training System to structure of the human hand is shown in Fig. 5 (Kapit Each joint has local coordinate system; the motion one joint is affected by other joints.As shown in Fig. 6, this model was built using bone feature in 3ds Max.The bones system is a jointed, hierarchical s that can be used to animate other objects.It is a suitable technique to be used for building the skeleton model to represent the anatomic The skin layer is for displaying purpose skin material has been created to make the virtual hand more realistic.Materials describe how an object reflects or transmits light.3ds Max provides an option for applying texture to object.We have used texture mapping to reach a he skin texture is used as the texture.Material Editor is the dialog that can be used for creating and altering materials and also applying and adjusting the mapping.After the properties of the material have been adjusted, it is ready to be applied to object by dragging and dropping it from the material editor to the virtual hand.The skin layer of the virtual Now the skeleton model can be mounted into a virtual hand in order to animate the virtual hand to perform grasping and releasing movement.The skin modifier is applied to all bones and assigns the bone to the hand shape model.The bones are animated in 3ds Max by setting the animation key frames where the hand shape model will follow the movements of the bones.Key frames record the beginning and end of each transformation of bone.The grasping and releasing movement can be done by setting the otation value for finger joint and recorded in key frames.Figure 8 shows the virtual hand grasping

RESULTS AND DISCUSSION
A one hand selection and manipulation technique has been designed and named G process flow is as G is an improvement of HOMER developed by Bowman and Hodges (1997).The difference between HOMER and HOMER-G is G has an additional virtual grasping capability in the selection task.This new ability increased the nature of virtual hand movement and provides a better visual feedback to the user.In HOMER-G, when the object is shot by the ray from virtual hand, the virtual owards the object.Virtual hand ual object and the user can start to fist their hand to select the virtual object.The object then can be manipulated directly by virtual hand.After manipulation is done, the object is released and virtual hand will return to its initial position.
G technique is applied to the Virtual to select the jig and HOMER-G has the grasping capability that makes the manipulation of object looks more natural as shown in Fig. 10.

CONCLUSION
Virtual hand is an important avatar in direct user interaction technique.This study discussed the application of virtual hand in the selection and manipulation of jig and fixture in the virtual knee replacement training system.The Ray tracing and arm extension technique has been applied with the grasping capability to enhance the natural and intuitive user interaction in virtual environments.This study will be extended where two hand interaction will be developed and used to provide more control of the selection and manipulation of virtual object.

Fig. 1 :
Fig. 1: Development process of the training system