AUDIO EFFECTS ON HAPTICS PERCEPTION DURING DRILLING SIMULATION

Virtual reality has provided immersion and interactions through computer generated environments attempting to reproduce real life experiences through sensorial stimuli. Realism can be achieved through multimodal interactions which can enhance the user’s presence within the computer generated world. The most notorious advances in virtual reality can be seen in computer graphics visuals, where photorealism is the norm thriving to overcome the uncanny valley. Other advances have followed related to sound, haptics, and in a lesser manner smell and taste feedback. Currently, virtual reality systems (multimodal immersion and interactions through visual-haptic-sound) are being massively used in entertainment (e.g., cinema, video games, art), and in non-entertainment scenarios (e.g., social inclusion, educational, training, therapy, and tourism). Moreover, the cost reduction of virtual reality technologies has resulted in the availability at a consumer-level of various haptic, headsets, and motion tracking devices. Current consumer-level devices offer low-fidelity experiences due to the properties of the sensors, displays, and other electro-mechanical devices, that may not be suitable for high-precision or realistic experiences requiring dexterity. However, research has been conducted on how to overcome or compensate the lack of high fidelity to provide an engaging user experience using storytelling, multimodal Cómo citar este artículo: Y. Valbuena, A. Uribe-Quevedo, A. Velasco-Vivas, “Audio effects on haptics perception during drilling simulation,” Revista Ingeniería Investigación y Desarrollo, vol. 17 (2), pp. 6-15, Julio, 2017. ISSN Impreso: 1900-771X, ISSN: 2422-4324 (En Línea) Audio effects on haptics perception during drilling simulation Revista Ingeniería, Investigación y Desarrollo (I2+D) Vol.17 (2), pp. 6-15. Julio-Diciembre, 2017. Sogamoso-Boyacá, Colombia. 7 interactions and gaming elements. Our work focuses on analyzing the possible effects of auditory perception on haptic feedback within a drilling scenario. Drilling involves multimodal interactions and it is a task with multiple applications in medicine, crafting, and construction. We compare two drilling scenarios were two groups of participants had to drill through wood while listening to contextual and non-contextual audios. We gathered their perception using a survey after the task completion. From the results, we believe that sound does influence the haptic perception, but further experiments are required to better comprehend the implications and possible medical applications.


INTRODUCTION
Multimodal interactions play an important role in our daily activities and provide us with feedback that allows to complete a task [1].When developing skills, our senses (i.e., sight, hearing, taste, touch, and smell) are key to determine and fine-tune our responses [2].These interactions and the correlation of all sensory modalities to convey information on the environment have been widely addressed [3] particularly, on how we perceive haptics and its influence on other senses, which can alter the experience for better or for worse [4].
Currently, Virtual Reality (VR) is having a positive impact in society as it allows to provide immersive and engaging experiences in various contexts (education, training, social inclusion, and cultural heritage amongst others) [5].VR allows users to engage in safe, customizable scenarios not possible in real life due to hazards and life-risks.Some application can be found in medical, military, education, and entertainment scenarios.
VR was born in 1960 when Morton Heilig develop the Sensorama [6], a motorbike simulator with audio, wind, chair, smell, and stereoscopic feedback.Although not successful at the time, VR simulation became relevant when airplane manufacturers realized the potential of flight simulation as a tool to train pilots in a controlled environment and expose them to critical situations to develop skills and decision making [7,8].Early approaches focused on haptic feedback (i.e., mechanical, electromechanical and hydraulics actuators) given the lack of computers at the time.However, visual immersion gained momentum after the Sword of Damocles [9], where the information was traditionally presented through screen projectors (one for each eye), that later evolved through the advances of computer graphics and optics (convex lenses to map our field of view into an immersive experience), which provides realistic virtual generated environments.Alongside with the haptics and video, audio has also evolved through different forms of feedback to increase the immersion.Some examples of audio advances in VR are ambisonics, ambiophonics and binaural sound that provide further realism that stereo or surround sound [10].
In terms of interactions, one of the main challenges of VR is the lack of tactile perception, this is why haptics is extremely important in several tasks, as it provides an insight of objects characteristics and surfaces properties.Indeed, a very well-known application of haptic devices is in the field of medical applications [11].Haptic technologies allow manipulating virtual objects providing force and touch feedback, through mechanical devices [12].Several systems, e.g.Phantom, Falcon, among others are available for applications as robotics, simulation or medicine.In [13], authors present a survey of the state of the art on the role of haptics in medical training simulators.
On the other hand, sounds can also provide information for material perception [14].On this premise in [15], authors presented a system for material classification, which by means of features like sound, image, friction force and acceleration, allows distinguishing among different materials with good accuracy (i.e. they show how the interaction of any tool with different surfaces produce vibrations which lead to audible sound waves to classify the surfaces).Multimodal interactions play an important role in several activities, where sight, touch and hearing play an important role during the execution of the task.
For our work, we focus on drilling.This action requires visual and haptic cues resulting in multimodal interactions.The multimodal feedback can provide information that can help determining if the task is going as planned, if there is any mechanical flaw, or if the drill bit is working properly.In this paper, we aim to study the effects of sound perception over haptics in a wood drilling scenario using a low-fidelity haptics device.Through the haptics scenarios, we also aim Yair Valbuena Álvaro Uribe-Quevedo Alexandra Velasco-Vivas to obtain a better understanding of the potential of multimodal interactions as a form to compensate low fidelity haptics.We chose drilling because it involves multimodal interactions and it is used in medical applications for implants and surgeries [16,17,18,19] home and construction, and artistic sculpting.

BACKGROUND
Perception is the reception and interpretation of stimuli within a nervous organism.This involves sensory and cognitive mechanisms of humans.Our senses, i.e. sight, hearing, smell, touch, and taste are the physiological tools for perceiving the information from the environment.There are several factors that alter perception of people; for instance, with respect to the intensity and the physical dimensions of the stimulus, it is possible to use objects that vibrate with the same intensity and frequency but more features can be added by exploiting sounds and even the shape of the object [20].
According to the research presented in [12], vision and audition are said to be the dominant senses which convey most of the information about the environment.First, the visual sense is based on absorption and conversion of light by the eye into neural messages.Second, the human auditory system transmits sound waves through the outer, middle, and inner ears.In the latter, the sound wave is transformed into neural messages, and then it is transmitted to the auditory cortex for processing.Instead, the sense of touch is distributed all over the body and it is mainly associated with active tactile senses such as the hands.Humans sense and respond to the environment by means of tactile receptors in the body.In general, humans are sensitive to touch, but skin discriminates different sensitivities i.e. touch, cold, heat, and pain all over the body.In this way, two or more of these sensitivities characterize other sensations like roughness, wetness, or vibration.Humans use touch interface in everyday tasks.Haptics is a term introduced at the beginning of 1900's, which refers to the science of manual sensing and manipulation through touch.Haptics allow determining the level of realism that can be achieved by enabling touch interactions with virtual environments.Currently, it is widely used in different disciplines (e.g.psychology, engineering, and computer science, etc.) referring to the study of human touch and force feedback with the external environment [12].
In several applications, manipulators with sensors or actuators are used as haptic devices, e.g.tactile-based or force feedback devices, to support the process of perceiving the characteristics of objects through touch.Additionally, these devices can be interfaced with the computer through development platforms that allow creating virtual environments, in which a real scenario is simulated and adding tactile and force feedback, these virtual environments allow to recreate more realistic situations and tasks [12].
Simulation is acknowledged as an important training tool and multiples types of simulators have been and are being developed to help train, develop and maintain skills.However, high-end systems involving high fidelity multimodal training such as the DaVinci robot [21], require a large investment in infrastructure, facilities, training, maintenance, and curricula, which may result inaccessible to various medical educational institutions [22].This scenario is challenging regarding the wide use and availability of simulation.This problem has been addressed with low-end technology that in recent years has been catching up in the form of mobile computing and 3D printing.The literature shows an ongoing growing interest in designing and using consumer-level devices for simulation purposes as a form to overcome the difficulties [23,24].This trend has also expanded to other medical applications such as arthroscopy surgery [25], and needle insertion [26].However, there are concerns related to the fidelity of simulation (level of realism) and its effects on the simulation outcomes [27].In [28], relationship between fidelity and transfer of knowledge was studied, and it was concluded that there were no significant differences amongst low and high fidelity simulation in auscultation skills, surgical techniques, and complex management skills such as cardiac resuscitation.
It is possible to conclude from the literature review that multimodal training is very important and simulators have been the only appropriate solution.However, current technological trends are providing opportunities to explore the use of low-fidelity devices that have the potential to impact VR scenarios.

DEVELOPMENT
The development of the drilling scenario is comprised of three main stages, namely i) scenario characterization, ii) programming, and finally iii) preliminary testing.Since drilling is a multimodal activity where visual, audio and haptic feedback takes place, we begin by understanding how the drilling mechanics work and how can it best be modeled in a low-fidelity scenario.

Drilling Characterization
The drilling process is characterized for perforating a hole with a circular cross-section using a drill bit which has multiple cutting points.The cutting process happens when pressure is applied through the drill resulting in the insertion of the drill bit into the material, while revolving at various speeds depending on the material.Mechanical models for drilling have been proposed in different conditions as predictors of behavior to better address drilling challenges [29].The cutting process varies depending on the target material, the drill bit (e.g., diameter, material, drilling speed and mode), and the applied force from the user [30].
We chose the Novint Falcon haptics as the user interface that conveys drilling interactions, the main characteristic is its low-fidelity, which places the device on a consumerlevel, making it affordable in comparison to advanced haptics systems (Table 1).The Novint Falcon is described as an entertainment-oriented device with various applications in games, education, and simulation [31].From Table 1, it can be seen that Novint Falcon is a low fidelity haptics interface.It is composed of a parallel mechanism that allows providing force feedback interactions across the three coordinate axes upon contact with virtual objects [33].However, because of the mechanism, the Novint Falcon can only offer haptics feedback from translations over its three degrees of freedom (DOF), which limits the interactions of the user and its applications, since objects in the space have six DOF (three translations and three rotations).To address this limitation, the prior research examined how to increase the Novint Falcon's DOF, Shah et al., designed a five DOF system by coupling two Novint Falcon devices [32].However, this approach may not be cost-effective as it requires an additional Novint Falcon device and the fabrication of the linkage to join both haptic devices, thus increasing the overall system complexity, cost, and space required to use the two coupled devices.Another solution to the lack of the rotational limit is presented by Uribe et al. [34], where they address the addition of two DOF using the rotational inertial sensor information from a mobile phone attached to the gripper as a prototype to virtually train cardiac auscultation.
To program the drilling interactions, we simplified the mechanical model drilling interaction by using a springmass system to implement the haptics with the Novint Falcon as presented in Fig. 1.We did this to focus the interaction on the moment of drilling were the user encounters resistance from the drilled material.To calibrate the force feedback we used an Arduino Uno with a FLIXFORCE pressure sensor A201 to measure various forces applied while drilling through wood with different drill bit diameters.From this procedure and using the mechanical properties of wood, we configured the mass-spring model with a stiffness between 8,000 and 10,000 N/m with a static friction coefficient of 0.3, and a dynamic friction coefficient of 0.4.

Multimodal Implementation
With the device and haptics model defined, we started the drilling scenario development with Chai3D [35], an open source haptics framework that uses C++.We implemented the mass-spring model in Chai3D using 3D virtual models created using Autodesk Maya.The 3D objects were imported as OBJ and all import properties were verified to guarantee proper interactions during drilling.To adjust the visual layout of the scene, the drill local coordinates and pivots were adjusted to match the position of the Novint Falcon's gripper as presented in Fig. 2.After the textures, lights and cameras were properly configured, we programmed the interactions (i.e., drill activation and drilling) between the drill and the user, and the drill bit with the wood.Once the interaction between the drill bit and the wood occurs, the contextual/non-contextual sounds are reproduced.To provide realistic audio feedback, the drilling sound recording was made in an Eckel audiometric room to limit any external noise and reverberation of the generated sounds within the environment, at a sampling rate of 44.1 kHz.Noncontextual audios such as nature and a hall were also recorded at the same rate.
To analyze the effects of audio, the drilling sound must be activated in similar conditions to real scenarios.
To achieve this, we configure the Novint Falcon's main button to serve as the trigger.The virtual wood block will enable the reproduction of the sound upon collision with the drill bit.Once the drilling sound and the non-contextual sounds were included, we defined three different drilling scenarios for our participants to experience.
The first test scenario for the participants focused on showing the drilling process including only the drill sound without load (in the air) and drill sound in load (wood drilling).The second scenario, included contextual sounds, and a non-contextual sound that alludes to nature, trying to reproduce a calm and tranquility scenario.Finally, the third scenario is a variation of the second one, where ambient sound of people, city noise or metal music trying to reproduce a stressful scenario is added.Since we aimed to understand the possible effects of audio over the drilling haptics, we created a survey with a likert-5 scale to rate each scenario from a user experience point of view.

RESULTS
We collected data from two control groups.The first group tested only the first scenario, i.e. the one with contextual sounds, and the second group tested all the three scenarios, i.e. one with contextual and two with non-contextual sounds.
Each group is formed by 18 participants, 60% male 40% female, most of them from ages between 20 and 39, who may be potential users of a system like the one presented in this paper, i.e. 36 users tested the system.40% of the participants expressed using the drill from 3 to 10 times per month and the other 60%, expressed using the drill from 1 to 3 times per month.To maintain the trial unbiased regarding the perception of drilling, all the participants were required to have previous drilling experience and not having auditory problems.Even though previous use of haptic devices was not required, half of the participants reported having used one at least once before.In terms of the interactions, 95% of the participants expressed their interest in playing video Yair Valbuena Álvaro Uribe-Quevedo Alexandra Velasco-Vivas games with some frequency and 35% have used serious games and simulators before.
The experiment was designed so both groups drilled through the wooden object and rated the haptic perception in terms of the audio effects.Each scenario was tested five times randomly, and this was corroborated by using the Research Randomizer [34], so each participant was exposed the same amount of times to each scenario.In the same manner, we included the drilling depth variable which was also selected randomly for each drilling test and for each participant.The drilling depth selection was of 3,4 cm and 5 cm.After finishing the trials, the participants were asked to answer a questionnaire to gather their impressions using a likert-5 scale.
Once in front of the system the users were asked to drill on the piece of wood.After reaching the target depth participants were asked about their perception obtaining the following: ease of use (78%), understandability (96%), enjoyability (76%), friendliness (72%) and safety (70%), but fairly motivating (62%).
Moreover, as shown in figures 4a. and 5a. the users to rate their perception and the influence of sound on the haptics feedback.The first group, who only tested the scenario with contextual sounds considered that the sound has some influence the perception of haptic fidelity and that the sound during the drilling task is very important.The second group, after testing the three scenarios with contextual and non-contextual sounds agree with the high influence (84%) that sound has on perception of haptic fidelity, and half of this group, thinks that the sound played during the drilling task is extremely important (88%).
In general, regarding the influence of sound in the perception of haptic fidelity, and considering the answers of all the participants, the mean is 4.15 over a 5 scale, which shows the high importance with standard deviation of 0.54.In the same way, the importance of sound during the drilling task was rated by all participants with a mean of 4.21 and standard deviation of 0.68.These results confirm the hypothesis that sound influences the haptics perception [38].
Additionally, we compared our results with another haptics scenario developed with Unity [37] (Fig. 3), where participants conducted four trials with five different scenarios comprised of i) no sound, ii) drilling sound, iii) metal music sound, iv) Classical music sound, and v) white noise.The participants performed the same task and provided feedback through the same questionnaire applied in our case, using a likert-7 scale.The results for the Unity-based scenario are presented in Fig. 5, where we can see that for that test, most of the participants think that the scenario is very understandable, enjoyable, and easy to use, but they find it fairly motivating.Regarding the influence of sound in the haptics fidelity, participants considered that it does influence very much (73%) and that it has some influence (23%).Finally, we can see that the users agree with the importance of the sound played during the drilling task.

CONCLUSION
Here we have presented the preliminary results of our study motivated on an ongoing work to better understand sound as a tool to provide haptic fidelity.From the results, we identified opportunities to expand the scope of our scenario, and although greater work is required to obtain a statistical significance, our preliminary findings are encouraging.During the trials, the majority of the participants believed sound played an important role within the haptics scenario as they felt different haptics feedback on every trial, even though the haptics model never changed.Additionally, participants who conducted more trials provided greater scores.Overall, participants found the Chai3D and Unity scenarios enjoyable, easy to use, and varied opinions about how motivating it was.
With this study, we aimed to prove that independently from the scenario applied, the sound would influence haptic perception which is confirmed with the results presented.After analyzing data statistically, we found that to reject null hypothesis the p-value is p = 0.037<0.05,sound is important for people when per-forming a drilling task, and the null hypothesis is rejected with this analysis because results obtained from both groups are similar.
For most people sound is fundamental in the haptic perception of the devices, this indicates to us that a good sound system is able to alter the haptic perception of the people, many of them expressed to feel a close affinity with reality with the drill when they were exposed to the environment full of stressful sounds, because despite being an environment full of external noises outside the drilling process, this was very close to reality and therefore it was easier to perceive both the vibration of the device and the sound it emitted when it was drilled.
Multimodality plays an important role in virtual reality and simulation, from this preliminary study, we believe that sound can be used to compensate haptics feedback.To continue building upon this work, future endeavors will focus on improve and change visual feedback, expand our participant numbers, and add more realistic audio immersion to a task at hand.We will also work on determining the suitability of our findings in other drilling simulation contexts such as in medical applications.

Figura 2 .
Figura 2. Virtual drilling scenario with the wood block and the drill.

Figure 4 .Figure 5 .Figure 6 .
Figure 4. Questionnaire results for the Chai3D scenarios: Influence of sound in the perception of haptics fidelity.a. Scenarios with contextual sounds.b.Scenario with non-contextual sounds