Knowledge Base

FaceReader 10 Technical Specifications

Functionality

FaceReader is a tool that automatically analyzes facial expressions, providing the user with an objective assessment of a person's emotion. It also offers options to enrich your data, including voice analysis, eye tracking, consumption behavior, and more.

FaceReader can recognize a number of specific properties in facial images, including the following six basic expressions:

• Happy
• Sad
• Angry
• Surprised
• Scared
• Disgusted

These are the six universal or basic emotions as described by Ekman (Universal facial expressions of emotion, California Mental Health Research Digest, 8, 151-158, 1970), which are cross cultural. Additionally, FaceReader can recognize a neutral state and analyze contempt as an emotional state. The primary output is a continuous value between 0 and 1 per expression, which corresponds with the intensity and clarity of an emotion.

Furthermore, calibration is not required in order to start analysis. All emotions are represented as bar graphs and can also be displayed as a line graph. An additional graph summarizes the valence (negativity or positivity) of the emotional status of the subject.

Happy is regarded as a positive emotion; sad, angry, scared, and disgusted are regarded as negative emotions. Valence is calculated as the intensity of happy minus the maximum value of the negative emotions.

Data are also visualized in a pie chart, showing the percentage per emotion. The line graphs and pie chart can be copied or saved as an image, for example for use in a Word report.

Valence can also be visualized in a circumplex model, as described by Russell, J. A. (A circumplex model of affect. Journal of personality and social psychology, 39(6), 1161, 1980). In this model, valence is displayed on the horizontal axis. On the vertical axis the level of arousal is shown. The calculation of arousal in FaceReader is based on activation of Action Units and volatility. Arousal can also be displayed as a line graph.

FaceReader contains a model quality bar, which gives you a good indication of how well the program is able to model the face depicted in the image. This information can also be stored in the logs created by FaceReader. When using multiple cameras, this information can be used to select the best analysis result.

In addition to emotions, FaceReader can detect subject characteristics. There are two default independent variables: Age and Gender. These are the independent variables that can also be determined automatically by FaceReader. A range is given for age. For gender a certainty level (between 0 and 100%) is given.

FaceReader also offers a number of extra classifications:

Facial States

Classification of certain parts of the face:

• Left and right eye open or closed
• Mouth open or closed
• Left and right eyebrow raised, neutral, or lowered

Global Gaze Direction

9 directions: right up, right, right down, up, forward, down, left up, left, or left down. There also is an indication of horizontal and vertical gaze angle.

Head Orientation Tracking

Enables tracking of the full 3D orientation of the head of the analyzed person. Head orientation is recorded in three angles (pitch, yaw, and roll), which denote the head rotation around the X, Y and Z axis. FaceReader also measures the distance between the face and the camera in three directions.

The extra classifications can be visualized in line graphs.

Input

FaceReader can be used with different input sources:

Video

FaceReader supports most common video codecs, including: MPEG1, MPEG2, XviD, DivX4, DivX5, DivX6, H.264, H.265, DivX, DV-AVI, and uncompressed AVI. The advantage of video analysis is that you can achieve a higher resolution. Videos can be analyzed frame-by-frame, or at a higher speed, analyzing every 2nd or every 3rd frame. When using frame-by-frame, the sample rate depends on the frame rate of the video, i.e. 30 frames/sec. This mode makes it possible to detect even short emotion changes (100 msec) and micro-expressions. For demonstration purposes, the video can run in loop (repeat) mode. When logging from video, the video time is used in the log (not the PC time). It is also possible to play back audio together with the video. FaceReader supports and has been tested with the following audio codecs: AAC, MP3, AC3, WMA. Multiple videos can be analyzed in one batch, without having to open every file separately. Analysis results of different videos are stored in separate files. The maximum advised duration for video analysis is 2 hours with 15 fps and 1 hour with 30 fps.

Live Analysis Using a Webcam

Via USB or IP camera - in this case the sample rate is dependent on processing power and on image quality. Typical sampling rates will be 5-10 frames/sec. Live analysis is also possible using an IP camera. The live image can be recorded in a DivX/MP4V format, allowing for more detailed frame-by-frame analysis afterwards in FaceReader or integration with other video and data modalities in The Observer XT. It is possible to record audio together with the video signal. The maximum duration for camera analysis is 2 hours.

Still Images

Use *.jpg, *.bmp, *.gif, *.png, or *.tga. Animated gifs are not supported - in this case the first frame of the animated gif will be used.

Images and videos to be analyzed can be rotated 90°, 180°, or 270°.

Visualization

The subject's face can be visualized in different ways:

• Show framing - draws a box around the face at the location where the face was found.
• Show mesh - shows the positions of key points on the face and the head orientation.
• Show global gaze direction.
• Show classifications of facial states.
• Show activated Action Units.
• Zoom in on face.

These visualizations can also be shown simultaneously.

Visualization of FaceReader analyses can also be displayed in a Reporting Display which enables the creation of a flexible layout with the output of the user's choice, including:

• Analysis visualization
• Expression Intensity
• Expression Summary
• Valence Line Chart
• Arousal Line Chart
• Head Orientation Line Chart
• Custom Expression Chart
• Circumplex Model of Affect
• Heart rate
• Valence Monitor
• Emoticon
• Expression Line Chart
• Head Position Line Chart
• Gaze Angles Line Chart
• Voice Expression Line Chart
• Voice Expression Intensity Bar Chart
• Voice View
• Voice Valence and Arousal Line Chart
• Fixations
• Saccades
• Vital Signs
• Heart Beat Chart
• Breathing Rate
• Consumption Behavior Statistics
• Intake events
• Chewing
• Chewing motion

Output

FaceReader can generate two types of text files or Excel files. The first is a detailed log that contains all the emotional classifier outputs. If no information was available at a certain time, the log will show the text 'Missing', or in case no correct model could be built, 'FIT-FAILED' for each column in the record. Additionally, optional classifications, model quality, arousal and the emotional valence can be logged (selectable).

It is also possible to log the X-, Y-, and Z-coordinates of all key points in the mesh in mm, or the X- and Y-coordinates of the 67 main ones in pixels. The main key points are the points around the mouth, nose, eyes, and eyebrows. The values of these coordinates are relative to the upper-left-corner of the image.

The second file is the state log. This file contains the 'emotional state' a person is in, which is an emotional category shown clearly and with a significant duration. An update to the log file is only made when the state changes. The data is tab separated, making it easily importable in most spreadsheet programs or text editors.

Combining FaceReader with The Observer XT

FaceReader can also store data in an .odx format, for direct import into The Observer XT, the software package for collection, analysis, and presentation of observational data. The data can then be synchronized with event logs, keystrokes, mouse clicks, video, screen capture, physiological data, eye-tracking data, etc. The state logs are stored as an event file showing the dominant emotional state as behaviors. Facial states, gaze direction, valence, arousal, and head orientation are also included. The detailed log is entered as numerical modifiers in a second event log, showing the exact value of every expression per sample.

FaceReader data can be visualized and analyzed in The Observer XT, enabling integration with other observational and physiological data. The advanced video editing functions of The Observer XT can be used to create highlight clips.

If you have The Observer XT 13 (or newer) with the External Data Module and you observe live, you can use the Noldus network communication protocol N-Linx to control FaceReader with The Observer XT. When you start or stop the observation in The Observer XT, the FaceReader analysis will also start or stop. The FaceReader analysis and FaceReader video can automatically be imported into the observation. In addition it is possible to create a participant with camera analysis when creating an observation in The Observer XT.

Communication with Other Applications Using the FaceReader API

Facial expressions detected by FaceReader and other output values can be accessed real-time by other applications, making the program an ideal tool for research into affective computing and the design of adaptive interfaces. In other words, FaceReader allows other software programs to respond instantaneously to the emotional state of the user. The FaceReader API can send classification results to remote programs over TCP/IP. This API can be used in a .Net application to easily make a connection with FaceReader. For more information, refer to the Technical Note 'FaceReader 9 Application Programming Interface'.

Individual Calibration

This function enables the correction of person specific biases towards a certain emotional expression. A calibration model can be created using live camera input, or images or video of the test participant showing a neutral expression. If there is no calibration model selected, there is a possibility of using continuous calibration. In this mode, FaceReader continuously adapts to the bias of a certain user.

Set-up

FaceReader achieves the best performance if it gets a good (video) image. Both the placement of the camera and the lighting of the subject's face are of crucial importance in obtaining reliable classification results.

Camera Position

The ideal position for the camera is directly in front of the subject's face. If the subject faces a computer screen, the camera can be placed either directly above or directly below the screen. Classification output might have a small bias towards the angry emotion when the camera is placed on top of the monitor and a small bias towards surprised when the camera is placed below the monitor. This is due to the fact that people tend to tilt their head when showing these emotions.

Illumination

The best results are achieved with diffuse frontal lighting. The light intensity or the color is less relevant. Strong reflections or shadows, for example caused by lights from the ceiling, should be avoided. Sideward lighting from a window will generally degrade performance. If the subject faces a computer screen, two columns of LED lights or two TL-tubes to either side of the monitor will give a good result under most circumstances.

In situations where interior lighting cannot be controlled, stronger lights (e.g. professional photo lamps) can be used to negate the effect of other undesirable light sources.

In-Depth Analysis

Project Analysis

The project explorer in FaceReader gives an overview of analyses per participant, and of participants per project. Projects which contain image analysis results cannot be analyzed here.

You can view a number of different charts and tables:

• Pie chart
• Box plot
• Bars
• Circumplex model
• Line chart
• Table

It is possible to calculate the mean, minimum, maximum, median or standard deviation of groups of participants. These values can be aggregated in the available chart types using both aggregation over time and aggregation over participants.

You can also apply Baseline Correction. This baseline can be calculated using the following options:

• All other measurements
• Other Stimuli or Event Markers, which can be selected by the user
• A specific interval before a Stimulus or Event Marker

Both Stimuli and Event Markers are mutually exclusive. They can be regarded as two different behavioral classes. For more advanced coding, it is possible to use The Observer XT.

It is possible to specify 36 Event Markers and Stimuli in total. The start of a Stimulus or the Start/Stop of an Event is triggered by a key (lower case letter or number). Markers are visualized in the Timeline; the color can be specified by the user. Event Markers or Stimuli can also be triggered externally, using the FaceReader API. Furthermore, this API can be used to generate triggers in case a Stimulus or Event Marker is scored in FaceReader.

Markers can be placed both during live and during video analysis. Time information (start and stop of Stimuli and Event Markers) can be added in the detailed logs and .odx for export to The Observer XT.

It is possible to analyze the response of separate groups of participants towards stimuli and event markers. Groups can be selected manually or automatically, based on independent variable values. Selections based on combinations of independent variables are also possible. When comparing responses, FaceReader carries out a t-test per stimulus with the participants as samples. The p-value threshold for the significance test is variable.

Charts can be copied to clipboard or saved as an image (format .png, .bmp, .jpg, .tiff, .gif). Analysis data can be exported in a .txt or .xls format.

Independent variables are participant specific variables, defined by the user. There are 2 types of independent variables:

• Numerical - can take any numerical value.
• Nominal - predefined values. Examples of nominal independent variables are native language or experience level.

There are two default independent variables, which are always present: Age and Gender. These are the independent variables that can also be determined automatically by FaceReader.

Stimulus Presentation Tool

In order to indicate and select relevant episodes and events for analysis, you can use two types of markers. First, it is possible to specify Stimuli. A stimulus has a fixed duration specified by the user, and can be accompanied by a video or image. In case of a video, the start time of the stimulus within the video file can be specified by the user. Aside from Stimuli, it is possible to specify Event Markers, for example to indicate that the test participant is drinking or gets distracted.

It is possible to synchronize stimulus presentation with the trigger of a stimulus marker in the FaceReader project. The Stimulus Presentation Tool can be used on the same computer where FaceReader is running, or on a separate computer. In this case the computers are connected via a local network. Stimulus movies or images can be presented in a fixed or random order. Test participants can enter their name and independent variable values, such as age or experience level.

Action Units

FaceReader can analyze the following 20 Action Units:

Action Unit	Description
1	Inner Brow Raiser*
2	Outer Brow Raiser*
4	Brow Lowerer**
5	Upper Lid Raiser*
6	Cheek Raiser*
7	Lid Tightener*
9	Nose Wrinkler
10	Upper Lip Raiser
12	Lip Corner Puller*
14	Dimpler*
15	Lip Corner Depressor*
17	Chin Raiser
18	Lip Pucker
20	Lip Stretcher*
23	Lip Tightener
24	Lip Pressor
25	Lips Part
26	Jaw Drop
27	Mouth Stretch
43	Eyes Closed*

* For Action Units marked with *, unilateral analysis is possible: you can choose whether the value of the Left and Right Action Units should be analyzed independently or not.

** In Baby FaceReader AU4 is replaced by AU3+4 (Brow knitting and knotting).

Intensities are annotated by appending letters A (Trace), B (Slight), C (Pronounced), D (Severe), or E (Max). Action Units and their intensities can be visualized in the Timeline, and in the Analysis Visualization, and exported in the detailed log. Export in the detailed log as numerical values is also possible.

The continuous action unit values have the following ranges:

Range	Label
0.000 to 0.100	Not Active
0.100 to 0.217	A (Trace)
0.217 to 0.334	B (Slight)
0.334 to 0.622	C (Pronounced)
0.622 to 0.910	D (Severe)
0.910 to 1.000	E (Max)

Custom Expressions

It is also possible to define and analyze your own custom expressions. Users of FaceReader can build their own algorithms, using the following measurements as inputs:

• Facial expressions
• Action Units
• Derived expressions (valence and arousal)
• Custom expressions
• Head orientation
• Head position
• Gaze angles
• Heart rate, heart rate variability
• Constant values

These inputs can be combined in a self-defined algorithm, using the following processors:

• Mathematical operations (maximum, minimum, sum, average, scale, offset, subtract, divide, multiply, scale to range, power, absolute, clip)
• Logical operations (condition, and, or, not, if..else..)
• Temporal operations (average, weighted average, sum, maximum, minimum, running average, running maximum, running minimum)

A number of custom expressions is already available in the software:

• The commonly occurring affective attitudes: interest, boredom, and confusion
• Attention
• Blink rate (AU45)
• Head turn left (AU51), Head turn right (AU52), Head up (AU53), Head down (AU54)
• Laughing and Smiling
• Leaning backward and leaning forward
• Spontaneous Laughter and Talking
• With Baby FaceReader, Baby Cry and Baby Smile are available as default custom expressions.

The intensities of custom expressions can be displayed as a line chart, and can be exported for analysis in other software packages as well.

Vital Signs

FaceReader can analyze heart rate and heart rate variability (HRV) of the test participant without additional hardware, using the FaceReader camera. Photoplethysmography (PPG) is a simple and low-cost optical technique that can be used to detect blood volume changes in the tissue under the skin. It is based on the principle that changes in the blood volume result in changes in the light reflectance of the skin. With each cardiac cycle the heart pumps blood to the periphery. Even though this pressure pulse is somewhat damped by the time it reaches the skin, it is enough to distend the arteries and arterioles in the subcutaneous tissue. PPG is often used non-invasively to make measurements at the skin surface. In remote PPG (RPPG), FaceReader can detect the change in blood volume caused by the pressure pulse when the face is properly illuminated. The amount of light reflected is then measured. When reflectance is plotted against time, each cardiac cycle appears as a peak. This information can be converted to heart rate (expressed in beats per minute). HRV is based on RMSSD (Root Mean Square of Successive Differences in msec) or SDNN (Standard Deviation of NN Intervals). The heart rate and HRV can be visualized and exported alongside other FaceReader output data.

---

Download the original PDF: FaceReader - Technical Specifications (PDF)