Facial expressions are widely associated with emotions. Emotions are mental states induced by neurophysiological changes. These mental states are often seen as indicator for how a person feels or thinks.

Understanding emotions is immensely helpful for researchers and companies, as they can provide insights into behavior and user responses. Think of usability and user experience research, where products, services, or websites are being tested to discover if the intended use actually works out in practice. Or developmental psychology and parent-child interaction, for instance to study the role of mimicry in early development of social communication.

Reliable analysis with the Facial Action Coding System (FACS)

Many researchers code muscle movements to learn more about how and why the facial muscles move. They use the Facial Action Coding System (FACS) which provides them with a technique for the reliable coding and analysis of facial movements and expressions.

FACS coders describe every observable change in facial movement on the basis of Action Units (AUs). They indicate which AUs moved to produce the changes observed in the face. While this way of analyzing facial expressions is quite unbiased, it is also very time-consuming.

Automated facial expression analysis with FaceReader

Measuring facial expressions with the use of an automated tool like FaceReader can help to provide a quick and objective assessment of emotions. FaceReader is a renowned software tool for automated facial expression analysis, broadly used in scientific research.

Over the years, there have been many statements and ‘facts’ about using automated facial expression analysis to assess emotions. Let’s take a look at some of those facts, and what is actually possible and impossible to do with FaceReader, through a game of true and false.

False: Facial expression analysis and facial recognition are the same

Facial expression analysis and facial recognition are often confused and sometimes even interchanged. They are, however, not the same thing. Let’s look into the difference between facial expression analysis and facial recognition.

Facial recognition is a way of identifying or confirming an individual's identity using their face (source). Facial recognition software or systems work by mapping, analyzing, and identifying faces live or from photos or videos, and then confirming the identity of those faces by cross-referencing the images to other databases with personal photos. 

FaceReader does not do this.

Automated facial expression analysis is a computer-based technology that can detect and classify expressions that are described by Paul Ekman as the basic or universal emotions. Obviously, facial expressions vary in intensity and are often a mixture of emotions. In addition, there is quite a lot of inter-personal variation.

FaceReader has been extensively trained to classify these facial expressions, using a database of photos to discover patterns. But it does not cross-reference these against other faces in a database attempting to identify a person. So, facial recognition is impossible with FaceReader.

True: Impossible to use facial expression analysis as a lie detector or mind reader

Despite popular belief (and prime time television shows), reading minds is not a validated science. Although FaceReader can automatically analyze facial expressions, it cannot read your mind.

FaceReader can objectively assess someone’s facial expressions, muscle movements in the face, and link these to basic emotions like happy or surprised, or detect if someone is aroused. But the results of these measurements still need human interpretation.

So, FaceReader’s facial expression analysis cannot serve as a lie detector.

False: Use FaceReader to identify people

FaceReader software can automatically distinguish 6 basic emotions, 20 Action Units, and specific facial features like the state of the eyes (open or closed), eyebrows (raised or lowered) and mouth (open or closed). While FaceReader can recognize general facial features, the tool is not capable of recognizing or identifying faces or people, and therefore unsuitable for mass surveillance purposes.

Even though the software detects facial features and maps those into an artificial face model describing key points in the face, FaceReader’s main output is a classification of the facial expressions of a participant taking part in a research.

True: Recognize specific facial properties and states

FaceReader software can recognize specific facial states of test participants. For instance, the software can detect whether eyes or the mouth are open or closed, and whether eyebrows are raised or lowered. It detects if someone is wearing glasses and gives an estimate of the test participant’s gender and age.

FaceReader can also detect the test participant’s head orientation and gaze direction: does the participant look to the left or to the right, up, down, or forward? This data on gaze direction can be useful in communication and social encounters, and provides researchers with information about interest and attention.

When you combine all these measurements from FaceReader with eye tracking, you can get rich, quantitative, and reliable data for attention, mental load, and arousal. These combined measurements are often used in UX research, for instance, to determine a user’s experience with a product or website.

False: FaceReader can be used to automatically identify facial expressions in public spaces or a crowd

FaceReader is a scientific software tool that is installed on a PC and can only be used to record or analyze one person at a time. In order for FaceReader to properly analyze facial expressions, a person has to sit in front of a camera linked to a computer. When using pictures or pre-recorded videos, FaceReader can analyze only one face at a time.

Diffused frontal lighting for best results

For reliable analysis, FaceReader requires strict light conditions with diffused frontal lighting ensuring the best results. Uncontrolled light conditions in an outdoor setting – like strong reflections or shadows – have a negative impact on the use of the software.

Strict privacy protocols

The use of FaceReader also adheres to strict privacy-by-design protocols. For example, the software offers you the option not to record the test participant’s face during the analysis. In this case, all the data can be acquired but cannot be related to an identifiable person. Examples are facial expressions, head pose, age, and gender. If video is recorded, this should comply with GDPR or other privacy regulations and participants should give informed consent prior to recording.

These privacy protocols and strict conditions for good results makes the software unusable for crowd analysis or in public spaces.

Read more: 5 tips on how to measure facial expressions

True: Context is key when using automated facial expression analysis

Facial expression analysis alone can give insights into emotional states but for accurate and complete interpretations, it is important to look at the full context of someone’s behavior. For instance, what is the test participant watching on the screen and what expression does that trigger? Or which events occurred in the room while the facial expressions recognized as ‘happy’ were recorded?

To learn more about these outside factors, automated facial expression analysis is often combined with other measurements, like eye movement, heart rate (HR) and heart rate variability (HRV), and post-experiment participant interviews.

Taking the context into account can complement the strengths of facial expression analysis by revealing the environmental cues and context that shape facial expressions and emotions. Using the software outside of the restraints significantly reduces its accuracy. Human interpretation of all the combined measurements and full context is still widely important and vital to any conclusions.

The facts about FaceReader: a summary

So, these are the facts about FaceReader: facial recognition and facial expression analysis are not the same, you cannot identify people with FaceReader, and context is key when using automated facial expression analysis.

Do you want to know more about how FaceReader works? Feel free to contact us or download the FaceReader methodology note to learn more.