Knowledge Base

Vision is one of our most important senses and enables us to see the world around us. Our eye movements contain important information, as do our facial expressions. By observing these eye movements, as well as gaze, and facial expressions, we can learn more about subconscious processes. This can be done with eye tracking technology. Combined with facial expression analysis techniques this provides valuable insights into user experience, customer preferences, desire, and appreciation.

Many fields of research use these technologies, ranging from cognitive psychology and neuro-marketing to consumer behavior and user experience research.

Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. A lot of rich data can be collected automatically using systems such as eye trackers and software such as FaceReader. By integrating these systems, you can further increase accuracy and reliability, and save time.

Both eye tracking and facial expression analysis add substantial power to your research, as non-verbal responses can provide new information about attention and emotion. This white paper shows you how FaceReader works and how eye tracking technology can be used in combination with FaceReader.

Emotion Recognition

FaceReader is robust software that is capable of automatically analyzing facial expressions, providing users with an objective assessment of a person's emotion. A total of 6 basic expressions can be recognized: happy, sad, angry, surprised, scared, and disgusted (Ekman, 1970) [1]. Additionally, FaceReader can recognize a 'neutral' state and analyze 'contempt'.

Furthermore, FaceReader automatically classifies mouth open-closed, eyes open-shut, and eyebrows raised-neutral-lowered, and it registers head and gaze orientation. Head and gaze orientation is important in communication and social encounters, and provides information about interest and attention. For instance, if a person looks away instead of straight at a person or screen, they might be bored, unhappy, or losing interest.

Analyzing Mental States in Depth

With the help of Action Units and Custom Expressions, mental or affective states like boredom or interest can be analyzed even further. Action Units are muscle groups in the face that are responsible for facial expressions.

Action Units and Custom Expressions

The Action Units are described in the Facial Action Coding System (FACS) that was published in 2002 [2] by Ekman et al. With the Action Unit Module, FaceReader can analyze 20 Action Units. Action Unit classification can add valuable information to the facial expressions classified by FaceReader.

Custom Expressions are facial expressions or mental states that you can define yourself by combining the facial expressions and Action Units that FaceReader can recognize. This enables you to measure affective states like interest, boredom or confusion.

More details about the methodology of FaceReader and how Action Units (AUs) are calculated, can be found in the Methodology Note.

FaceReader's main output is a classification of the facial expressions of your test participant. These results are visualized in several different charts that show the percentage per emotion.

Eye Tracking

Eye trackers add substantial power to your lab set-up. They produce gaze path video and large quantities of rich data, all related to attention and emotion. Gaze direction reflects attention and blink frequency is associated with mental load. Scan paths indicate how people look at websites and advertisements: which parts of an advertisement, picture, or web page they actually notice, how long they look at various items.

When a participant has previous experience with an application, the scan path of the eyes will have fewer fixations. Modern eye trackers can easily generate this information, which makes them increasingly popular in behavioral studies.

Data Collection

Data collection starts by installing the eye tracker of your choice. Choose for example a computer-monitor eye tracker that tracks the test participant's gaze on a computer screen. They are contact-free, and built into or attached to computer monitors. They all make screen captures with a gaze overlay that are suitable for analysis. Some computer-monitor eye trackers even allow you to test participant looking behavior real-time.

Researchers that are looking for on-site research options can use eye tracking glasses that enable to follow gaze directions of test participants, for instance while shopping for groceries in a supermarket.

Integrate & Visualize Your Eye Tracking and Other Data

Collect data by placing a camera on top of a computer monitor. This way you can follow the facial expressions during a test in which also an eye tracking video is made. Facial expression data can be viewed real-time in FaceReader on a control PC, but you can also record videos and analyze them afterwards.

Integrate Data

If you want to combine eye tracking and facial expression analysis with other data such as physiological data, or you would like to code behaviors, then The Observer XT can take your research to a higher level: it offers flexible integration of logged events with videos, eye tracking data, FaceReader data, and physiological data such as EEG.

The Observer XT as Integration Platform

Simply connect the eye tracking equipment to a computer that runs The Observer XT and start your data acquisition. When external equipment is connected to The Observer XT, your data sets will be synchronized. The Observer XT facilitates coding from video. You can play two or more videos simultaneously forward or backward and both at slow and fast speeds. This allows you to fast-forward to a particular point of interest in the video and carry out very detailed logging.

For example, analyze a video commercial. See exactly where the test participant was looking when smiling. The Observer XT software provides detailed visualizations that help you explore your results. Customized charts and statistics are accessible in a few mouse clicks.

Screen Capture

Register how a subject interacts with the (new) design by capturing the full screen in an audio video recording tool like Viso. With Viso you can make independent recordings in multiple rooms at once. The multiple high-end cameras allow you to view your participants from different angles and get a good view of your participants and their surroundings.

Displaying the eye tracking gaze-overlay or facial expression data is also possible. The screen captures are synchronized with the AV recordings of Viso, providing you with complete videos that help you gain insights into human performances and emotional insights.

Application Programming Interface

You can also choose to allow other programs to respond instantaneously to the emotional state of the test participant by using the FaceReader API. FaceReader produces log files after analysis, which can be accessed real-time by other applications via an Application Programming Interface (API). This means that FaceReader can be used for research into affective computing and the design of adaptive interfaces.

Analyze Your Data

Visualize and Analyze FaceReader Data

After collecting and in some cases integrating your data, you can start to investigate your results and the relationship between facial expression and eye tracking data. FaceReader provides a video analysis and a log file which shows you which expression was shown at what moment. The visualization window in FaceReader provides insight in how your video images were analyzed.

Furthermore, customized charts are accessible in just a few mouse clicks. The analysis results can be used in either The Observer XT for further analysis or a statistical program of your choice.

Export Your Data

For additional calculations and analysis, you can export all data together for backup or import elsewhere. When using The Observer XT as an integration tool, a wide range of presentation options is offered as well, to facilitate communicating your results to others. You can select important video fragments and create your own video highlights clip to illustrate your outcomes.

Example Studies and Research

By combining implicit methods as eye tracking, EEG, or physiological data with explicit methods like surveys or interviews, researchers get rich insights. Because FaceReader can serve as a non-intrusive measure of engagement, it can be used in many applications. Some examples of fields using facial expression research include:

• Ad testing and consumer behavior - FaceReader is often used to examine metrics such as consumer interest in a product, packaging, or the effectiveness of video advertisements.
• Psychological research - An understanding of facial expressions can ensure a better understanding of communication and interactions, for example between parents and children, teachers and students, or health professionals and patients.
• Usability testing - To gain understanding in users' experiences with a website or application, researchers can use eye tracking and facial expression technology to analyze participants' gaze and facial expressions during testing.

References, Publications and Resources

References

1. Ekman, P. (1970). Universal Facial Expressions of Emotions. California Mental Health Research Digest, 8(4), 151-158
2. Ekman, P.; Hager, J. (2002). Facial Action Coding System Affect Interpretation Database (FACSAID). https://en.wikipedia.org/wiki/Facial_Action_Coding_System

Blogs

• How to master automatic Facial Expression Recognition
• 5 tips to optimize your facial expression analyses
• Validation-study: Basic emotions and Action Units detection
• What can you use eye tracking for?

Other Publications and Resources

• Goldberg, J.H. (2014). Measuring Software Screen Complexity: Relating Eye Tracking, Emotional Valence, and Subjective Ratings, International Journal of Human-Computer Interaction, DOI: 10.1080/10447318.2014.906156
• Gunaratne, N.M., Fuentes, S., Gunaratne, T.M., Torrico, D.D., Ashman, H, Francis, C., Gonzalez Viejo, C., Dunshea, F.R. Consumer Acceptability, Eye Fixation, and Physiological Responses: A Study of Novel and Familiar Chocolate Packaging Designs Using Eye-Tracking Devices. Foods. 2019; 8(7):253. https://doi.org/10.3390/foods8070253
• Nguyen TA., Coursaris C.K., Leger PM., Senecal S., Fredette M. (2020). Effectiveness of Banner Ads: An Eye Tracking and Facial Expression Analysis. In: Nah FH., Siau K. (eds) HCI in Business, Government and Organizations. HCII 2020. Lecture Notes in Computer Science, vol 12204. Springer, Cham. https://doi.org/10.1007/978-3-030-50341-3_34
• Stockli, S.; Schulte-Mecklenbeck, M.; Borer, S. & Samson, A.C. (2018). Facial expression analysis with AFFDEX and FACET: A validation study. Behav Res Methods, 50(4), 1446-1460. https://doi.org/10.3758/s13428-017-0996-1

Feel free to contact us or one of our local representatives for more references or more detailed information about FaceReader, The Observer XT, Viso or our preferred suppliers Tobii Pro, Smart Eye or Eyetech.

Contact Information

International Headquarters

Noldus Information Technology bv
Wageningen, The Netherlands
Phone: +31-317-473300
Fax: +31-317-424496
E-mail: [email protected]

North American Headquarters

Noldus Information Technology Inc.
Leesburg, VA, USA
Phone: +1-703-771-0440
Toll-free: 1-800-355-9541
Fax: +1-703-771-0441
E-mail: [email protected]

Representation

We are also represented by a worldwide network of distributors and regional offices. Visit our website for contact information.

Due to our policy of continuous product improvement, information in this document is subject to change without notice. FaceReader, The Observer, and Viso are (registered) trademarks of Noldus Information Technology bv. © 2021 Noldus Information Technology bv. All rights reserved.

---

Download the original PDF: White paper - FaceReader - In combination with eye tracking (PDF)