Knowledge Base

FaceReader 10 - Analysis Modules

In-Depth Analysis

Project Analysis

The project explorer in FaceReader gives an overview of analyses per participant, and of participants per project. Projects which contain image analysis results cannot be analyzed here.

You can view a number of different charts and tables:

• Pie Chart
• Box Plot
• Bars
• Circumplex Model
• Line Chart
• Table

It is possible to calculate the mean, minimum, maximum, median or standard deviation of groups of participants. These values can be aggregated in the available chart types using both aggregation over time and aggregation over participants.

You can also apply Baseline Correction. This baseline can be calculated using the following options:

• All Other Measurements
• Other Stimuli or Event Markers, which can be selected by the user
• A specific interval before a Stimulus or Event Marker

Both Stimuli and Event Markers are mutually exclusive. They can be regarded as two different behavioral classes. For more advanced coding, it is possible to use The Observer XT.

It is possible to specify 36 Event Markers and Stimuli in total. The start of a Stimulus or the Start/Stop of an Event is triggered by a key (lower case letter or number). Markers are visualized in the Timeline, the color can be specified by the user. Event Markers or Stimuli can also be triggered externally, using the FaceReader API. Furthermore, this API can be used to generate triggers in case a Stimulus or Event Marker is scored in FaceReader.

Markers can be placed both during live and during video analysis. Time information (start and stop of Stimuli and Event Markers) can be added in the detailed logs and .odx for export to The Observer XT.

It is possible to analyze the response of separate groups of participants towards stimuli and event markers. Groups can be selected manually or automatically, based on independent variable values. Selections based on combinations of independent variables are also possible. When comparing responses, FaceReader carries out a t-test per stimulus with the participants as samples. The p-value threshold for the significance test is variable.

Charts can be copied to clipboard or saved as an image (format .png, .bmp, .jpg, .tiff, .gif). Analysis data can be exported in a .txt or .xls format.

Independent variables are participant specific variables, defined by the user. There are 2 types of independent variables:

• Numerical — can take any numerical value.
• Nominal — predefined values. Examples of nominal independent variables are native language or experience level.

There are two default independent variables, which are always present: Age & Gender. These are the independent variables that can also be determined automatically by FaceReader.

Stimulus Presentation Tool

In order to indicate and select relevant episodes and events for analysis, you can use two types of markers. First, it is possible to specify Stimuli. A stimulus has a fixed duration specified by the user, and can be accompanied by a video or image. In case of a video, the start time of the stimulus within the video file can be specified by the user. Aside from Stimuli, it is possible to specify Event Markers, for example to indicate that the test participant is drinking or gets distracted.

It is possible to synchronize stimulus presentation with the trigger of a stimulus marker in the FaceReader project. The Stimulus Presentation Tool can be used on the same computer where FaceReader is running, or on a separate computer. In this case the computers are connected via a local network.

Action Units

FaceReader can analyze the following 20 Action Units:

• 1 – Inner Brow Raiser*
• 2 – Outer Brow Raiser*
• 4 – Brow Lowerer**
• 5 – Upper Lid Raiser*
• 6 – Cheek Raiser*
• 7 – Lid Tightener*
• 9 – Nose Wrinkler
• 10 – Upper Lip Raiser
• 12 – Lip Corner Puller*
• 14 – Dimpler*
• 15 – Lip Corner Depressor*
• 17 – Chin Raiser
• 18 – Lip Pucker
• 20 – Lip Stretcher*
• 23 – Lip Tightener
• 24 – Lip Pressor
• 25 – Lips Part
• 26 – Jaw Drop
• 27 – Mouth Stretch
• 43 – Eyes Closed*

*For Action Units marked with an asterisk, unilateral analysis is possible: you can choose whether the value of the Left and Right Action Units should be analyzed independently or not.

**In Baby FaceReader, AU4 is replaced by AU3+4 (Brow Knitting and Knotting).

Intensities are annotated by appending letters A (Trace), B (Slight), C (Pronounced), D (Severe), or E (Max). Action Units and their intensities can be visualized in the Timeline and in the Analysis Visualization, and exported in the detailed log. Export in the detailed log as numerical values is also possible.

The continuous action unit values have the following ranges:

• 0.000 – 0.100: Not Active
• 0.100 – 0.217: A (Trace)
• 0.217 – 0.334: B (Slight)
• 0.334 – 0.622: C (Pronounced)
• 0.622 – 0.910: D (Severe)
• 0.910 – 1.000: E (Max)

Custom Expressions

It is also possible to define and analyze your own custom expressions. Users of FaceReader can build their own algorithms, using the following measurements as inputs:

• Facial expressions
• Action Units
• Derived expressions (valence & arousal)
• Custom expressions
• Head orientation
• Head position
• Gaze angles
• Heart rate, heart rate variability
• Constant values

These inputs can be combined in a self-defined algorithm, using the following processors:

• Mathematical operations (maximum, minimum, sum, average, scale, offset, subtract, divide, multiply, scale to range, power, absolute, clip)
• Logical operations (condition, and, or, not, if…else…)
• Temporal operations (average, weighted average, sum, maximum, minimum, running average, running maximum, running minimum)

A number of custom expressions is already available in the software:

• The commonly occurring affective attitudes: interest, boredom, and confusion
• Attention
• Blink rate (AU45)
• Head turn left (AU51), head turn right (AU52), head up (AU53), head down (AU54)
• Laughing and smiling
• Leaning backward and leaning forward
• Spontaneous laughter and talking
• With Baby FaceReader, Baby Cry and Baby Smile are available as default custom expressions

The intensities of custom expressions can be displayed as a line chart, and can be exported for analysis in other software packages as well.

Vital Signs

Heart Rate and Heart Rate Variability

Analyze heart rate and heart rate variability (HRV) of the test participant without additional hardware, using the FaceReader camera. Photoplethysmography (PPG) is a simple and low-cost optical technique that can be used to detect blood volume changes in the tissue under the skin. It is based on the principle that changes in the blood volume result in changes in the light reflectance of the skin. With each cardiac cycle the heart pumps blood to the periphery. Even though this pressure pulse is somewhat damped by the time it reaches the skin, it is enough to distend the arteries and arterioles in the subcutaneous tissue. PPG is often used non-invasively to make measurements at the skin surface. In remote PPG (RPPG), FaceReader can detect the change in blood volume caused by the pressure pulse when the face is properly illuminated. The amount of light reflected is then measured. When reflectance is plotted against time, each cardiac cycle appears as a peak. This information can be converted to heart rate (expressed in beats per minute). HRV is based on RMSSD (Root Mean Square of Successive Differences in msec) or SDNN (Standard Deviation of NN Intervals). The heart rate and HRV can be visualized as a line chart, and exported for analysis in other packages.

The European Patent Office has granted patent EP2960862A1 to Vicarious Perception Technologies (VicarVision) for the method for stabilizing vital sign measurements using parametric facial appearance models via remote sensors. For more information, see: https://patents.google.com/patent/EP2960862A1/en.

For an overview of the accuracy of the estimated heart rate, see: Gudi, Amogh, Marian Bittner, and Jan van Gemert. "Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation." Applied Sciences 10.23 (2020): 8630.

Breathing Rate

FaceReader also supports the measurement of breathing rate by detecting upper-body movements associated with respiration. As breathing is a relatively slow process, the software needs 15 seconds for calibration to show values, and another 15 seconds to see reliable measurements. Note that the upper body should be visible in the video for the breathing rate measurement to work, and that excessive movement will reset calibration. If chest motions are very shallow or if there is no breathing detected, a warning will show in the vital signs panel.

Voice Analysis

Using English-language data, including both scripted and natural speech. Preliminary tests show potential applicability to other languages, particularly those with close linguistic or cultural similarities, such as Germanic languages.

A volume threshold is used to decide whether to classify the audio signal (as to not classify background noise). Unlike video, audio cannot be segmented into discrete frames. Therefore, an audio buffer is used to collect approximately one second of data before starting analysis.

The Voice Expression Line Chart shows the detected emotions over time. If at any point in time the volume is below the threshold, gaps will appear. The Voice Expression Intensity bar chart shows the currently detected emotions. The Voice View shows a four-second audio waveform, with different colors to indicated different detected emotions. Loudness and Speech Rate will appear alongside it. The Voice Valence and Arousal Line Chart shows valence and arousal over time.

Using a high-quality microphone and limiting background noise is recommended to improve the accuracy of the voice analysis. It might be necessary to adjust the microphone's sensitivity to achieve the best results. For more information, download the White Paper about Voice Analysis on the FaceReader resources page.

Source: FaceReader 10 Technical Specifications, Noldus Information Technology