Detection of lies by facial thermal imagery analysis

An artificial vision system is presented for lie detection by analyzing face thermal image sequences. This system represents an alternative technique to the polygraph. Some of its features are: 1) it has no physical contact with the examinee, 2) it is non-intrusive, 3) it has a potential for private use, and 4) it can simultaneously analyze several persons. The proposed system is based on the detection of physiological changes in temperature in the lacrimal puncta area caused by the subtle increase in blood flow through the nearby vascular network. These changes take place when anxiety appears as a consequence of deception. Thus, the system segments the periorbital area, and tracks consecutive frames using the Kanade-Lucas-Tomasi algorithm. The results show a success rate of 79.2 % in detecting lies using a simple classification based on the comparison between the estimated temperatures in control questions


I. IntroductIon
Determining whether people are honest in their statements is useful in situations of social conflict resolution such as crimes that violate the standards set by society, or in preventing terrorist attacks, among others.Studies on psychology and physiology show a relationship between the anxiety that takes place when a person is lying, and organic responses that are difficult to control such as dilated pupils, increase of sweating, muscle tension, heart rate, blood pressure, and body temperature.However, it is possible that some of these changes may happen to honest people who have communication difficulties, or are sick, under stress, or taking medication.Therefore, there are two general lines of research on lie detection [1]: 1) identification of psychophysiological changes, and 2) identification of changes in body movements, facial expression, and voice.
The most standardized and recognized device to identify lies is the polygraph [2], which records blood pressure, heart rate and respiratory rate.Although the polygraph is highly effective (about 90 %) [3], it can only be applied to one person per equipment, and requires physical contact with the examinee to capture the body signals.Moreover, this procedure can automatically increase the anxiety level in a person, either by thinking that their honest answers will be rejected, or because the interrogation may evoke emotions.
On the other hand, thermography is a technique that records the thermal radiation generated by the functioning of physical systems, or by the heat interaction with its internal characteristics [2].For this reason, thermography is used in different fields of medicine to detect physiological changes in the human body [3].Its application in several areas has been favored by the development of thermal imaging cameras that make it possible to obtain images in the infrared spectrum with higher resolution, precision, and at an increasingly lower cost.The obtained videos provide information on changes in temperature without any physical contact.Additionally, with this technique, the number of people under analysis within the field of view can increase, they can be tested unknowingly, and there are no risk for their health.
The epidermis uses heat exchange with the outside to maintain the homeostasis of the internal organs, therefore, several internal physiological changes may modify this thermal exchange.In [6], algorithms are developed to detect, from thermal images of the skin surface, static abnormalities or temporary changes of some physiological functions such as blood flow rate, blood volume in cutaneous and subcutaneous vascular beds, and activities of the sympathetic nervous system.In 2000, research proposals on using thermography in the non-invasive analysis of the human body to detect anxiety symptoms were presented to DARPA (Defense Advanced Research Projects Agency).Pavlidis et al. [7] described a method that used thermographic videos from a heat transfer model in the skin near a main vessel to estimate blood flow velocity, and the position of blood vessels.Later, Zhu [8] developed a method to measure heart rate based on thermographic images of areas containing the main blood vessels.
Jain et al. [9] evaluated the periorbital area, the supraorbital area (by activating the corrugator muscle), neck, cheeks, and nose to detect anxiety from thermographic images.They observed a temperature increase in the first three areas, and a decrease in cheeks and nose.Most works use the periorbital area and the supraorbital area; however, the highest success rates have been achieved in the former.Pavlidis and Levine [10] reported a 76.3 % success rate in detecting lies from the thermal signature of the corrugator muscle; while Owayjan et al. [11] achieved a 83.5 % success rate in a similar system based on the thermographic readings of the two lacrimal puncta.
In order to increase the performance of the traditionally used polygraph, the possibility of adding an information channel has been studied.This information channel is based on thermography [12], which has a success rate of 85 %, and uses non-invasive techniques (without physical contact) to identifying micro-facial expressions from images in the visible spectrum [13].An alternative approach for detecting lies is to record brainwaves such as P300 [14,15], which have reached a detection rate between 82.45 % and 95.4 %.However, this is an invasive technique that may have the same polygraph disadvantages such as the examinee's emotional alteration when realizing that s/ he is being examined.This work is the first approach to a lie detector developed in Colombia by digital processing of thermographic images of the face.Section two describes the used experimental methodology, and section three, the stages of the whole system.The Detection of lies by facial thermal imagery analysis results and their analysis are presented in section four, and the main conclusions and issues for some future work are summarized at the end of this paper.

II. ExpErImEntatIon mEthodology
The proposed thermographic system is based on the analysis of the dynamic behavior of temperature on lacrimal puncta caused by specific anxiety situations (in this case, lying).These situations lead to physiological changes in blood supply to the face, which, in turn, result in changes in the heat flux [16].This feature requires recording a thermographic video, hence the following conditions were taken into account: • The place selected for recording the videos was kept at a room temperature (~25 °C) to ensure a good thermal contrast, and avoid any thermal regulation action on the human body.
• Since perspiration is a thermal insulator, all examinees rested for 5 minutes before beginning the experiment.
• In order to operate the processing algorithms properly, wearing glasses, scarves or the hair down was prohibited, and the eye area should remain uncovered throughout the video recording.
• Recordings were made with the examinee sitting in a comfortable position, with their torso upright, and directing their eyes toward the camera.Natural speech movements were allowed as long as the periorbital area remained completely into the camera field of view.
• The camera was placed approximately 1.5 m from the examinee, and at a height of 1.3 m.It had to be slightly adjusted according to the examinee's height, so that the field of view covered their face and chest.

A. Type of experiment
Another key component of the experimental methodology is the procedure that encourages lying, and thus generates anxiety [17].After evaluating the advantages and limitations of different experiments to generate anxiety, the so called mock crime was selected.In this experiment, the examinees are asked to be or not to be liars in a mock crime scenario where they allegedly steal a bill of a certain denomination; and, if "they have stolen it" they should not admit it to the examiner.The main features of this type of experiment are: • The examinees are free to lie or not.
• The examinee performs physical actions which involve them further into the experiment.
• Developing additional techniques to carry out the experiment is not necessary.
• The repeatable scenario is simple and easy.
• The absence of a real motivation to lie is its main limitation.

B. Carrying out the experiment
Given the above assumptions, students from Universidad del Valle were invited to participate in the experiment.They were told that it was an experiment on communication skills.To select the participants, the following aspects were taken into account: • The participants belonged to different schools and programs at Universidad del Valle, so the information related to the experiment was not widely disclosed.
• During the experimental session none of the participants knew or could observe what was happening with the previous participant.
• Since the participants had be recorded, they were asked to sign an acceptance clause of participation and the right to use the videos.
• Each participant answered a medical survey in order to determine whether they were "psychologically/ mentally normal", or if they had taken any substance or medication that could had altered their physiological reactions to the experiment.
In this way, it was possible to obtain a statistical sample of 27 participants between 18 and 29 years old (μ = 22.3, σ = 1.21).The following information was given to them before the experiment began: • Each participant would have a payment just for participating in the experiment.
• The participant who tells the truth in the interrogation would get double the money they would steal.
• The participant who lied and managed to fool the skilled examiner would get triple the money they would steal.
• The participant who lies and fails to deceive the skilled examiner would get only a payment for participating in the experiment.This is a modification of the original experiment to encourage the participant to deceive the examiner.
Subsequently, each person was taken to a room where they stayed for only two minutes; there, they found a box with several objects easy to remember, including a COP $ 5000 bill.In that moment, the participant decided to "steal" or not the money.Afterwards, while the participant was waiting to be interrogated, they thought about what they would say in the second phase of the experiment.Only the participant knew if they had taken the money.Afterwards, and in order to increase the emotional pressure caused by their decision of lying or telling the truth, they were told the interviewer was an expert psychologist, and that it was almost impossible to deceive them.During the interview, the participant was faced to answer an interrogatory based on the Control Question Technique (CQT), which aims to generate a thermal behavior (or syntactic model), similar to the one shown in Figure 1, which complies with the following standards: FIg. 1. Syntactical expected representation of the temperature curve of a video classified as "lies".
• The first three questions of the interrogation divert the participant's mind from the performed action (theft), and induce them to a low level of stress.To do so, the participant is requested to give answers as long as possible because temperature changes are slow.The participant's representation of the physiological temperature level during this part of the interrogation corresponds to the baseline.
• The second phase of the interrogation has three control accusatory questions.Here, the participant can respond freely to generate a considerable time window, but the interrogator has to be rude and insistent to generate the greatest possible stress.
During this phase, an increase in temperature is expected in response to the provoked stress.
• The following four questions aim to make the participant stop feeling attacked, and thus, return to a tranquility state (i.e., the thermal baseline).
These questions are addressed to generally inquire about the theft, without questioning the participant directly.
In a "no-lie" situation, temperature shows a stable or even a decreasing trend because when truth is told the stress level drops.Three people participated in the interrogation: the examinee, the interrogator, and the person who managed the equipment.The latter Detection of lies by facial thermal imagery analysis was in charge of the protocol for recording videos, record the time spent in the control questions, and pay each participant the amount of money according to their performance.Additionally, to record as much information as possible for further analysis, simultaneous audio and video recordings were made in the visible spectrum.Once the interrogation ended, information about the experiment development was checked against the verdict issued by the interrogator.

C. Description of the thermographic system for detecting lies
Temperature variations on the skin surface due to variations in blood pressure or heart rate do not exceed 1.2 °C, and depend on the participant's physical condition, age, and the activity performed prior to the recording.The ThermaCAM E320 camera, brand FLIR, available in the Artificial Vision Laboratory at Universidad del Valle, has a range sensitivity from 0.08 ºC to 30 °C.The normal adult human body temperature is between 36.5 °C and 37.5 °C.Values below 36 °C involve hypothermia, while above 38 °C indicate hyperthermia or fever; these values are considered improbable for the purpose of this paper.Xu et al. [16], for example, used a camera with a temperature range from 29 ºC to 38 °C, and a sensitivity of 0.025 ºC because their purpose was to detect the ocular temperature during the polygraph test.Although the sensitivity of the camera used by Xu et al. exceeds the ThermaCAM E320 camera, the results achieved here are sufficiently discriminant, and can improve to the extent that a camera with greater sensitivity is used.
Since the human body has generally higher temperatures than room temperatures, pixels, which exceed a dynamic threshold calculated by the Otsu method [18] on the histogram image, are kept.The face area corresponds to the largest area, and is determined by the morphological operator of majority (Figure 2).Once the face is identified, it is necessary to locate the periorbital area in both eyes, where some areas are discarded according to the anthropomorphic characteristics of the face.Following the anthropometric proportions, the face is divided vertically into three equal parts: the first third corresponds to the forehead and eyebrows, just above the eyes; the second third corresponds to the eyes, ears and nose; and the last third to the mouth and chin.Thus, the periorbital area search in each eye is limited to the first third of the face, setting a rectangle 2R wide, and 2/3R high (Fig. 3a,b).The R value is automatically calculated using (1).First, the location of all initial I i and final D i points belonging to the face (left and right sides, respectively) is determined by a scan along each i row of the segmented image G(i,j) size m x n (Fig. 3c).From (1), m distances are then obtained between each pair of endpoints of the segmented and binarized face; afterwards, their medium points (circles passing through the center of the face in Figure 3c) are found, and R is calculated as the average of these medium points.(1) where: y The location of the periorbital area, once the search space is reduced, corresponds to the higher temperature area due to the high concentration of veins and arteries in this area.At these points, temperature changes are greater than in other areas [11], so periorbital areas are identified as the points with the highest average temperature within an area of 9 x 9 pixels.
Since monitoring the temperature of the periorbital area must be extended through different frames of thermal video, which increases computing time, the tracking algorithm Kanade-Lucas-Tomasi (KLT) [19] was used, instead of performing the segmentation in each frame.This algorithm is based on the calculation of the summation of squared differences in gray levels, within the same area centered at a point of interest, between two consecutive images.
This technique uses the spatial gradient of intensity to direct the search for homologous points.With this procedure, it is possible to find the best position for the control points in a small number of operations, on the basis that in many applications the two images are aligned.However, the tracking algorithm requires the calculation of unique characteristics that distinguish the object of interest frame by frame.To distinguish the periorbital areas, the eigenvalues were used as characteristics according to the algorithm described in [20].According to this algorithm, the gradient of each considered image pixel, in this case the segmented periorbital area, is obtained first.Then, its autocorrelation 2 x 2 matrix is calculated, and its eigenvalues are found.The eigenvalues analysis can establish whether a particular pixel is a corner, an edge, or a flat region [21].Specifically, those pixels corresponding to corners represent points with intensity changes in all directions, so they are taken as control points for monitoring.Figure 4a shows an example of control points obtained with the algorithm Shi & Tomasi.Figures 4b and 4c show the rectangle corresponding to the detected area with the characteristic points found in the rectangles of periorbital areas.
The consequence of using the tracking technique to reduce processing time is the gradual deterioration in lacrimal puncta location accuracy.Therefore, every 150 frames, the procedure is repeated to obtain points of interest (lacrimal puncta) in the periorbital area, so the characteristics that must be located in the following frames are updated.The update period of 150 frames was heuristically determined, and it represented a good compromise between accuracy and computation speed.The temperature signal on lacrimal puncta can be obtained from the periorbital area location in a video sequence.In this case, the signal is noisy, but three components can be identified (Fig. 5a) [3]: • A low-frequency component, indicating the tendency of blood flow levels.
• A mid-frequency component associated to temporary changes in blood flow caused by the stress of the experiment itself.
• A high-frequency component caused mainly by monitoring instability and systematic noise.
Consequently, it was necessary to suppress the noise by eliminating the high frequency component.This was done by analyzing the 5 th level approximation coefficients of the wavelet transform, using a Meyer mother-wavelet based on the analysis of [22], and the considerations discussed in [23].In Figure 5b, the original signal wavelet is represented in green, and the approximation signal in red.This is a smoothed representation of the original curve, which includes the two components that carry the relevant information about the temperature dynamics.
Finally, a decision scheme was developed to separate and classify each signal obtained from the videos; the categories were "lies" and "does not lie".However, the increase in temperature depends not only on the participant's stress level, but also on the participant's psychophysiology; that is, on the extent and intensity of each person's reaction to the same stimuli.Therefore, an appropriate decision scheme must normalize the variability among individuals through an intra-individual measure.Given these facts, it is possible to summarize the decision scheme by ( 2). ( Where T Pcontrol is the average temperature signal during the control accusative questions, and T Total is the average temperature signal during the total length of the experiment.In other words, if the physiological change during the series of control accusative questions is higher than the baseline change, then the video is classified as "lies".

A. Face segmentation
The images supplied by the IR camera were segmented to isolate the face area and eliminate irrelevant information.A basic technique of adaptive threshold on the image histogram was used.This technique yielded an adequate segmentation for 100 % of cases, provided that recording conditions set out by the experimental protocol were maintained.

B. Periorbital area segmentation
To set the success rate of the system at the rectangle location containing the periorbital area, every situation, in which the periorbital area frame contained the lacrimal puncta inside, was defined as a successful result.Additionally, its centroid should coincide with the space between the eyebrows, with an error lower than 10 % (in pixels) of the rectangle length (about 11 pixels error, in this case).The evaluation was conducted using four videos that were randomly selected, two for each possible trial ("lies" or "does not lie").Their basic features are included in Table 1.
The results of the location (with the anthropometric proportions of the face), and monitoring (by KLT) of the periorbital area are presented below.Lies 3192

1) Face anthropometric proportions:
The algorithm performance in the four previous videos was evaluated to assess the location of the periorbital area by the face anthropometric proportions strategy.In the first frame, a polygon covering the area between the eyebrows was defined.It was expected to locate the point X in it, from which the periorbital area was defined.It was verified that when the point X was inside the polygon, a suitable location of the periorbital area and lacrimal puncta was obtained; for this reason, this condition was taken as a segmentation success.Using this method (one frame in every 150 consecutive frames of the sequence) the success rate average was 99.89 % in 136 frames.
Table 2 presents the success rates from frames that use the face anthropometric proportions to locate the periorbital area on the test videos.The videos that obtained 100% success rate were those where participants kept their head upright or made moderate head movements, and had a hairstyle that facilitate the correct face identification.In contrast, the participant Detection of lies by facial thermal imagery analysis in the video S15 kept his head tilted to the right (Fig. 6a), while the algorithm assumes that the head is always upright.Meanwhile, in the video S19, failure occurred because the participant's hairline (Fig. 6b) produced an upward movement of the upper part of the segmented face, and therefore of the periorbital area location.
2) KLT Algorithm: Success in monitoring the periorbital area depends on the result of the lacrimal puncta locating step through the face anthropometric proportions that is developed every 150 images.The criterion for defining the tracking success was that the eyebrow location should be inside the reference polygon (which contains the point X).Table 2 shows the success rates in the periorbital area location, using the face anthropometric proportions and the KLT tracking algorithm.It is important to know that while the lacrimal puncta are within the periorbital area, the next stage can be recovered from a displacement of this area.Displacement errors in this area are generally caused by cumulative errors resulting from using the tracking KLT technique to increase speed.They take place when the participants move their face considerably, the head is significantly moved, or when some conditions cause the identified area of the face to become wider at the top of the head.Since the KLT algorithm is very sensitive to sudden head movements, the participant is required to avoid rapid gestures.A sudden movement may cause a shift in the temperature location, and thus a discontinuity in the temperature graph changing the result.

C. Detecting lacrimal puncta in the periorbital area
It is assumed that lacrimal puncta correspond to those areas with higher average temperature.Figure 7 shows examples of detecting lacrimal puncta demarcated in a 9 x 9 pixels' rectangle.This detection is not affected by a head movement or by the neck contour, unless it is not contained within the detected area as being a periorbital area.The results showed a 99.8 % success on the location of the two lacrimal puncta on a total of 13746 frames from four test videos.

D. Classification
24 out of the 27 recorded videos were used to validate the system; three of the videos were invalid due to face occlusions at some point of the experiment.Table 3 shows the confusion matrix for lie detection (data number and percentage of total data).The system was always right with the videos of participants that told the truth.When the participant lied, the system showed errors: 5 out of 20 of the cases were identified as "does not lie" when they actually lied.These facts correspond to an overall system performance of 79.2 % in detecting lies.However, it is expected that in real life situations, when issues like freedom, and reputation among others are at stake, detecting deception would be easier than in an experiment that recreates a simulated theft.Detection, in this particular case, only works if the person is facing the camera, which is a major constraint in future large-scale implementations, or in a different research area.Furthermore, there are protocol requirements that need to be follow, for instance, if a person has their face wet, the system will be disabled because the temperature seen by the camera will have unexpected changes in temperature, and may even affect face tracking because it is hidden behind the sweat.

IV. conclusIons
A system for automatic detection of lies was developed by analyzing infrared images of the blood area around the lacrimal puncta.The system achieved an overall performance of 79.2 %, with 75 % success rate in detecting lies and 100 % in detecting truth.These results are very close to the 80 % reported by [24] and 87.2 % reported by [25], who used thermal imaging cameras with greater sensitivity than the ones used in this work.It is worth mentioning that the system's success rate highly depends on the protocol of video recording, and the sequence of events to which the participant is exposed to during the test.More realistic stress situations where freedom, life, or economic resources are involved can improve detection.
The experiment of a simulated theft was modified to cause greater predisposition to lie and increase the stress level in the participants by offering a payment, and double the amount if they lied and could deceive Detection of lies by facial thermal imagery analysis the human examiner.Because the psychophysiological characteristics of each participant generate different intensities and reactions to the same stimuli, a temperature profile of reference for each participant was calculated during the first phase of the interrogation in order to separate the corresponding signals "lies" and "does not lie".The profile differentiated the signals by using a fast and easy comparison.
The stepwise segmentation technique (image-faceperiorbital area-lacrimal puncta) reduced the time location of the points of interest on the lacrimal puncta by using low-complexity and fast techniques.
Although the relative subject-camera location and the rapid movements of the face are conditions that affect the periorbital area detection, they were adequately compensated with the Kanade-Lucas-Tomasi tracking algorithm, and the low-pass wavelet filtering application to the temperature signal of the lacrimal puncta.Success rates in segmentation were 100 % for the face, and for the periorbital area detection were 99.89 % by the face anthropometric proportions and 99.7 % by the KLT algorithm.
The main limitations of the system are related to the resolution of thermal imaging cameras and the lack of autofocus, which involved establishing a fixed distance between the participant and the camera during the experiment.The system does not consider any situations where the face is wet, the face image is incomplete, there is more than one face, there are sudden or abrupt head movements, or the two lacrimal puncta areas are not visible.These limitations will be addressed in future works as well as the study of other anxiety-generating sources, and the fusion with other bio-signals.Development of early warnings of suspicious situations are of great significance, and become increasingly important due to the recent terrorist attacks in France and Belgium.
Finally, the proposed system offers a set of basic features that would allow its use as a prototype thermographic system to detect other anxiety disorders [24].

tablE 1
CharaCteristiCs of the videos used for segmentation trials

tablE 2
suCCess rates in the periorbital area loCation, using the faCe anthropometriC proportions and the Klt traCKing algorithm.

tablE 3
Confusion matrix for the ClassifiCation of temperature evolution in the two deCision possibilities