Emotion recognition techniques using physiological signals and video games – Systematic review –

Emotion recognition systems from physiological signals are innovative techniques that allow studying the behavior and reaction of an individual when exposed to information that may evoke emotional reactions through multimedia tools, for example, video games. This type of approach is used to identify the behavior of an individual in different fields, such as medicine, education, psychology, etc., in order to assess the effect that the content has on the individual that is interacting with it. This article shows a systematic review of articles that report studies on emotion recognition with physiological signals and video games, between January 2010 and April 2016. We searched in eight databases, and found 15 articles that met the selection criteria. With this systematic review, we found that the use of video games as emotion stimulation tools has become an innovative field of study, due to their potential to involve stories and multimedia tools that can interact directly with the person in fields like rehabilitation. We detected clear examples where video games and physiological signal measurement became an important approach in rehabilitation processes, for example, in Posttraumatic Stress Disorder (PTSD) treatments.

Emotion recognition techniques using physiological signals and video games -Systematic review-

I. IntroductIon
The analysis and study of human emotions is a complex field of research that can be approached from different points of view depending on the mechanisms employed for the stimulation, measurement, and analysis of emotional reactions in the human body.Currently, video games are among the tools with the greatest potential to stimulate the emotions of an individual; their strength lies in immersing the player in a virtual world, where they can experience different emotional reactions with no need to move large distances, engaging their attention, concentration, and time, in order to achieve the goals established by the immersive story of the game.Video games use stories, challenges, objectives, characters, scenarios, music, animation, and other audio-visual multimedia tools to capture the players' attention, and generate emotions that allow the individual to feel identified with the game, and thus, promoting a more outstanding game experience for those who would like to take up the challenge and use their time for this kind of activities.Along with books, music, images, and movies, videogames constitute a group of audio-visual tools that generate emotional reactions in individuals, according to the level of empathy they generate in the person.However, in contrast with the previous examples, video games represent a greater source of information on reaction, given that they are essentially interactive [1].
Apart from being one of the largest industries of technological development nowadays, video games have become a great source of research in applications oriented toward entertainment, as well as in applications known as "Serious Games," which try to solve problems in fields such as medicine [2the new Medicine degrees of the European Higher Education Area (EHEA, 3], the military [4], education [5,6], business [7], politics, citizenship [8], etc.Throughout the last decade, the convenience and importance of using video games or "Serious Games" have been demonstrated by analyzing the response of the players and potentiating some of their skills, simulating different virtual environments, and objectives that involve the direct participation of the people with the digital world.This study was carried out starting from the premise that a videogame can generate empathy in the player, producing different emotional reactions during gameplay, and that these reactions can be captured, measured, and analyzed through emotion recognition techniques and affective computing [9], such as face recognition [10], speech recognition [11], and physiological signals [12,13].The objective was to identify evidence and current research that integrate techniques of emotion recognition, involving physiological signals, such as electrocardiography (ECG), Galvanic skin response (GSR), and electromyography (EMG), with the use of video games for stimulating emotions; for this, we conducted a systematic review between 2010 and 2016, in which we identified a growth in the number of articles focused on video games and emotion recognition systems, demonstrating the work that is developed in the field.

A. Selection criteria
The articles were selected from eight different bibliographic and academic databases related to engineering and medicine: ScienceDirect, IEEEXplorer, SCOPUS, EngineeringVillage, ISI Web of knowledge, ACM, EMBASE, and PubMed.The articles corresponded to studies focused on emotion recognition from the measurement of physiological signals (ECG, GSR, or EMG), and the use of video games as tools for emotion stimulation, published between 2010 and 2016, due to the growth of these approaches employing these techniques.

B. Search strategy
The search and selection of articles that deal with developments in the mentioned field was based on two searches using different terms that refer to the field of study.
In the first search, the selected works contained in their title, abstract, or key words, the terms "emotion recognition" and "physiological signals", and were published between January 2010 and April 2016.The search was carried out in the databases previously mentioned and, as a result, the following number of papers was obtained: ScienceDirect (14), IEEEXplorer (162), SCOPUS (144), EngineeringVillage (125), ISI Web of knowledge (291), ACM (7), EMBASE (16), and PubMed (13).Given that we were looking for studies that used video games as emotion stimulation tools, the previously search was complemented Emotion recognition techniques using physiological signals and video games -Systematic reviewincluding the word "games" in any section of the article.The new search was conducted in the same eight academic databases and produced the following results: ScienceDirect (4), IEEEXplorer (7), SCOPUS (19), EngineeringVillage (8), ISI Web of knowledge (7), ACM (4), EMBASE (0), and PubMed (0), obtaining 49 articles in total.
A second search was carried out to complement the results obtained from the first search.This time, the objective was to identify works published between January 2010 and April 2016 that contained in their title, abstract, or key words the term "affective gaming."This term refers to video games that interact with the player through commands or actions using specialized tools, and respond or modify according to the emotions experienced by the player [14].The consulted databases produced the following results: ScienceDirect (121), IEEEXplorer (196), SCOPUS (27), EngineeringVillage (27), ISI Web of knowledge (16), ACM (451), EMBASE (1), and PubMed (15).Then, along with the previous search, the term "physiological signals" was included to be found in any section of the article; this search produced the following results: ScienceDirect (23), IEEEXplorer (14), SCOPUS (10), EngineeringVillage (5), ISI Web of knowledge (3), ACM (7), EMBASE (1), and PubMed (1), obtaining 64 articles in total.

C. Selection study
In total, 113 articles were found in the consulted databases (49 in the first search and 64 in the second search) containing the stated selection criteria.After excluding the duplicated papers, the number of articles was reduced to 74.Out of them, 46 were available in full.Finally, the accepted articles were classified according to the type of physiological signals measured, and the tool used to stimulate the individuals' emotions.15 articles reported to have measured physiological signals such as ECG, EMG, or GSR, along with video games as a stimulation tool for emotion recognition, and described the results of this process.

D. Data analysis
The search was conducted to determine state-ofthe-art techniques of emotion recognition based on physiological signs and video games.Of the 46 articles, 50 % (23) used video games as emotion stimulation tools, 10.87 % (5) used videos or movies as emotion stimulation tools, 4.35 % (2) used official images from the International Affective Picture System (IAPS), and 4.35 % (2) used some kind of mechanical device to stimulate emotions (vibrations [15]yet it is largely unmeasured, modeled or char-acterized.A better understanding of user perception will aid the design of tactile behavior that engages touch, with an experience that satisfies rather than intrudes.We measured 30 subjects\u2019 affec-tive response to vibrations varying in rhythm and frequency, then examined how differences in demographic, everyday use of touch, and tactile processing abilities contribute to variations in affective response.To this end, we developed five affective and sensory rat-ing scales and two tactile performance tasks, and also employed a published \ u2018Need for Touch\u2019 (NFT, LEGOS [16]).The remaining 30.43 % ( 14) corresponded to studies in the field of emotion simulation [17][18][19][20], the development and adaptation of tools for measuring emotions [21], literature revisions and editorials [6,14,[22][23][24], and the interaction of the individual with their environment and other individuals [25][26][27][28]2009. ACII 2009: IEEE.

III. results
Out of the 46 publications found in this work, 14 reported developments using physiological signals different from ECG, GSR, and EMG (Table 1).Of the other 32, seven articles reported having worked with emotion stimulation tools other than video games (Table 2).Finally, of the last 25, 10 corresponded to articles on developments in emotion simulation, revision of literature regarding machine learning algorithm techniques and serious games, and development of tools such as toolkits and editorials for technical events (Table 3).
Mauro Callejas-Cuervo -Laura Alejandra Martínez-Tejada -Andrea Catherine Alarcón-Aldana The 15 articles selected in the study are shown in Table 4, which briefly summarizes the most important aspects taken from each study.We can identify that these studies, mostly, carried out multimodal emotion recognition, in which they integrated signals from different sensors to measure the physiological responses of the individual interacting with the video game, and thus, identified the emotion that was being experienced during the game.GSR, along with heart rate (HR) from ECG or blood volume pulse (BVP), are the most used signals to identify emotions, due to the high efficiency that can be achieved in their detection and measurement, as well as the ease of access to the measurement equipment.We can also observe that 17.4 % of the referenced articles involved the measurement of signals through electroencephalograms (EEG) as the main measured signal, or as a complement, given that the information extracted from the behavior of the brain waves is relevant for detecting emotions.Including sensors to measure EMG signals allow to identify movements in the corrugator and zygomaticus major muscles, which are in charge of controlling the movement of the eyebrows and cheeks in the face, and therefore are directly related with the recognition of the facial expression.
It is important to mention that a large amount of the literature reported in academic databases does not clearly identify the key words or the indexed terms; therefore, the search protocol and the subsequent meta-analysis did not include all the studies that report innovative and important developments in the field of study.
Emotion recognition techniques using physiological signals and video games -Systematic review-

IV. conclusIons
From the systematic revision, we can conclude that video games have become an innovative approach as emotion stimulation tools, due to their potential to involve stories and multimedia tools that can interact directly with the person.However, video games still lack a classification that allows them to be linked with the emotions they may evoke.When using video games in studies of affective recognition, their selection is carried out according to the situation that will be stimulated, in order to obtain the same reaction from the player to the virtual environment and, as we suppose, to a real one.The use of commercial video games, paid or free (the most popular ones are riddles, jigsaw puzzles, simulators, and survival/horror, among others), is a common denominator among the studies referenced in this article.However, a selection methodology that allows for an initial classification of different video games that might be used in the same study is not reported in the different works, nor which game is more suitable to the requirements of the research being carried out.
Signals like GSR and ECG are widely used in this type of studies for detecting and measuring parameters related to emotions, such as HR, SCL and SCR, due to their high efficiency showing the physiological change connected to the emotions experienced by the individual.The EMG signals are used to recognize facial expressions and their changes; however, when considering the use of virtual reality tools, such as immersive glasses, this type of measurement is compromised due to the necessity of putting sensors on the face.For this reason, it is necessary to identify areas of the anatomy where it is possible to put sensors to measure the muscular activity when some kind of emotion is experienced, mainly stress, fear, and anxiety.
Finally, the field of affective video games is growing, with different possible action fronts; from the inclusion of avatars, which can demonstrate emotion through biofeedback, to the inclusion of emotion detection from players to improve the video game development process, as well as other fields of social, educational, and health.To obtain an accurate and robust technological system for the health and treatment field using videogames, it is necessary to develop new hardware and software tools that allow to measure and analyze signals in real time, without interfering with the normal behavior of the individual when is interacting with the video game.For example, biofeedback is a novel tool developed in the field of game design, with the potential to not only affect the players' game experience, but also create new branches in applied fields such the treatment of mental disorders.This kind of works are still in an experimental stage, focusing on the implementation of tools and protocols, validating not only the sensors and the acquisition system, but also, the process to acquire the physiological data to eventually apply them in the rehabilitation field.

authors' contrIbutIons
Martínez-Tejada performed the search, the recompilation and the analysis of the papers referenced in this article, and contribute to write the manuscript.Callejas-Cuevo and Alarcón-Aldana contributed to write the manuscript and reviewed it.All authors read and approved the final manuscript.

table 1 artiCLES
EXCLudEd for not uSinG PhySioLoGiCaL SiGnaL mEaSurEmEntS, SuCh aS ECG, GSr or

table 2 artiCLE
EXCLudEd for not uSinG vidEo GamES aS an Emotion StimuLation tooL

table 3 artiCLES
EXCLudEd for not PrESEntinG thE rESuLtS of thE mEaSurEmEntS or thE rESEarCh in thE dELimitEd arEa

table 4 artiCLES
SELECtEd for thE SyStEmatiC rEviEw