Knowledge Base

FaceReader 10 - Core Features and Capabilities

Functionality

FaceReader can recognize a number of specific properties in facial images, including the following six basic expressions:

• Happy
• Sad
• Angry
• Surprised
• Scared
• Disgusted

These are the six universal or basic emotions as described by Ekman (Universal Facial Expressions of Emotion, California Mental Health Research Digest, 8, 151-158, 1970), which are cross cultural. Additionally, FaceReader can recognize a neutral state and analyze contempt as an emotional state. The primary output is a continuous value between 0 and 1 per expression, which corresponds with the intensity and clarity of an emotion. Furthermore, calibration is not required in order to start analysis. All emotions are represented as bar graphs and can also be displayed as a line graph. An additional graph summarizes the valence (negativity or positivity) of the emotional status of the subject.

Happy is regarded as a positive emotion; sad, angry, scared, and disgusted as negative emotions. Valence is calculated as the intensity of happy minus the maximum value of the negative emotions.

Data are also visualized in a pie chart, showing the percentage per emotion. The line graphs and pie chart can be copied or saved as an image, for example for use in a Word report.

Valence can also be visualized in a circumplex model, as described by Russell, J. A. (A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39(6), 1161, 1980). In this model, valence is displayed on the horizontal axis. On the vertical axis the level of arousal is shown. The calculation of arousal in FaceReader is based on activation of Action Units and volatility. Arousal can also be displayed as a line graph.

FaceReader contains a model quality bar, which gives you a good indication of how well the program is able to model the face depicted in the image. This information can also be stored in the logs created by FaceReader. When using multiple cameras, this information can be used to select the best analysis result.

In addition to emotions, FaceReader can detect subject characteristics. There are two default independent variables: Age and Gender. These are the independent variables that can also be determined automatically by FaceReader. A range is given for age. For gender, a certainty level (between 0 and 100%) is given.

Finally, FaceReader offers a number of extra classifications:

Facial States

Classification of certain parts of the face:

• Left and right eye open or closed
• Mouth open or closed
• Left and right eyebrow raised, neutral, or lowered

Global Gaze Direction

Nine directions: right up, right, right down, up, forward, down, left up, left, or left down. There also is an indication of horizontal and vertical gaze angle.

Head Orientation Tracking

Enables tracking of the full 3D orientation of the head of the analyzed person. Head orientation is recorded in three angles (pitch, yaw, and roll), which denote the head rotation around the X, Y, and Z axis. FaceReader also measures the distance between the face and the camera in three directions.

The extra classifications can be visualized in line graphs.

Input

FaceReader can be used with different input sources:

Video

FaceReader supports most common video codecs, including: MPEG1, MPEG2, XviD, DivX4, DivX5, DivX6, H.264, H.265, DivX, DV-AVI, and uncompressed AVI. The advantage of video analysis is that you can achieve a higher resolution. Videos can be analyzed frame-by-frame, or at a higher speed, analyzing every 2nd or every 3rd frame. When using frame-by-frame, the sample rate depends on the frame rate of the video, i.e. 30 frames/sec. This mode makes it possible to detect even short emotion changes (100 msec) and micro-expressions. For demonstration purposes, the video can run in loop (repeat) mode. When logging from video, the video time is used in the log (not the PC time). It is also possible to play back audio together with the video. FaceReader supports and has been tested with the following audio codecs: AAC, MP3, AC3, WMA. Multiple videos can be analyzed in one batch, without having to open every file separately. Analysis results of different videos are stored in separate files. The maximum advised duration for video analysis is 2 hours with 15 fps and 1 hour with 30 fps.

Output

FaceReader can generate two types of text files or Excel files, the first being a detailed log that contains all the emotional classifier outputs. If no information was available at a certain time, the log will show the text 'Missing', or in case no correct model could be built, 'FIT-FAILED' for each column in the record. Additionally, optional classifications, model quality, arousal and the emotional valence can be logged (selectable).

It is also possible to log the X-, Y-, and Z-coordinates of all key points in the mesh in mm, or the X- and Y-coordinates of the 67 main ones in pixels. The main key points are the points around the mouth, nose, eyes, and eyebrows. The values of these coordinates are relative to the upper-left-corner of the image.

The second file is the state log. This file contains the 'emotional state' a person is in, which is an emotional category shown clearly and with a significant duration. An update to the log file is only made when the state changes. The data is tab separated, making it easily importable in most spreadsheet programs or text editors.

Combining FaceReader with The Observer XT

FaceReader can also store data in an .odx format, for direct import into The Observer XT, the software package for collection, analysis, and presentation of observational data. The data can then be synchronized with event logs, keystrokes, mouse clicks, video, screen capture, physiological data, eye-tracking data, etc. The state logs are stored as an event file showing the dominant emotional state as behaviors. Facial states, gaze direction, valence, arousal, and head orientation are also included. The detailed log is entered as numerical modifiers in a second event log, showing the exact value of every expression per sample.

• Voice Expression Intensity Bar Chart
• Voice View
• Voice Valence and Arousal Line Chart
• Fixations
• Saccades
• Vital Signs
• Heart Beat Chart
• Breathing Rate
• Consumption Behavior Statistics
  • Intake Events
  • Chewing
  • Chewing Motion

2. Live analysis using a webcam (via USB or IP camera) — In this case the sample rate is dependent on processing power and on image quality. Typical sampling rates will be 5–10 frames/sec. Live analysis is also possible using an IP-camera. The live image can be recorded in a DivX/MP4V format, allowing for more detailed frame-by-frame analysis afterwards in FaceReader or integration with other video and data modalities in The Observer® XT. It is possible to record audio together with the video signal. The maximum duration for camera analysis is 2 hours.

3. Still images — Use *.jpg, *.bmp, *.gif, *.png, or *.tga. Animated gifs are not supported - in this case the first frame of the animated gif will be used.

Images and videos to be analyzed can be rotated 90°, 180°, or 270°.

Visualization

The subject's face can be visualized in different ways:

• Show Framing — draws a box around the face at the location where the face was found.
• Show Mesh — shows the positions of key points on the face and the head orientation.
• Show Global Gaze Direction.
• Show Classifications of Facial States.
• Show Activated Action Units.
• Zoom In on Face.

These visualizations can also be shown simultaneously.

Visualization of FaceReader analyses can also be displayed in a Reporting Display which enables the creation of a flexible layout with the output of the user's choice, including:

• Analysis Visualization
• Expression Intensity
• Expression Summary
• Valence Line Chart
• Arousal Line Chart
• Head Orientation Line Chart
• Custom Expression Chart
• Circumplex Model of Affect
• Heart Rate
• Valence Monitor
• Emoticon
• Expression Line Chart
• Head Position Line Chart
• Gaze Angles Line Chart
• Voice Expression Line Chart

Source: FaceReader 10 Technical Specifications, Noldus Information Technology