Knowledge Base

How Does FaceReader Work?

FaceReader classifies facial expressions in several steps which is explained in the figure below. Please see [1] - [3] and the references mentioned in the text below for more information. FaceReader 10 uses new methods that are different from the ones used in previous versions of the software. We recommend using the same FaceReader version within one project.

1. Face finding — The position of the face in an image is found using a deep learning based face finding algorithm [1], which searches for areas in the image having the appearance of a face.
2. Face modeling — FaceReader uses a facial modeling technique based on deep neural networks [2]. It synthesizes an artificial face model, which describes the location of almost 500 key points in the face. The predicted key points of such a model are learned from a database of annotated images. It is a single pass quick method to directly estimate the full collection of landmarks in the face. After the initial estimation, the key points are compressed using Principal Component Analysis. This leads to a highly compressed vector representation describing the state of the face.
3. Facial expression classification — Classification of the facial expressions is done by a trained deep artificial neural network to recognize patterns in the face [3]. FaceReader directly classifies the facial expressions from image pixels. Over 100,000 images of people faces from all over the world, that were manually annotated, were used to train the artificial neural network. The network was trained to classify the six basic or universal emotions described by Ekman [4]: happy, sad, angry, surprised, scared, disgusted and a neutral state. Additionally, the network was trained to classify the set of facial Action Units available in FaceReader, as well as to estimate other facial attributes such as age, gender, and the presence of glasses.

The diagram above illustrates the FaceReader processing pipeline: Input image → Face finding → Face image → Face modeling (2A) → Face model → Deep neural network (2B) → Face classification → Final output (3).

Validation

Facial expression analysis of FaceReader 10 is validated using the Amsterdam Dynamic Facial Expression Set (ADFES) [5] and the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP) [6], wherein expressions are manually scored by FACS certified annotators. In addition, relevant validation was also performed on other datasets including Baby FACS manual dataset [7] and Taiwanese Facial Expression Image Database (TFEID) [8].

1. For facial expression classification on ADFES and WSEFEP FaceReader 10 has an accuracy of 98.7% and 97.2%, respectively. For FaceReader 9 these values were 99.3% and 95.7%, resp. That means that facial expression estimation under ideal conditions is improved in FaceReader 10. For Action Unit classification on ADFES and WSEFEP FaceReader 10 has an F1 score* of 0.83 and 0.80, resp. For FaceReader 9 these values were ~0.78 and ~0.76, resp. That means that facial expression estimation under ideal conditions is improved in FaceReader 10.
2. For Action Unit estimation using the baby model, FaceReader 10 has an F1 score of 0.69 on the Baby FACS Manual dataset [7]. FaceReader 9 also had a score of 0.63. This means that the baby model in FaceReader 10 performs even better on baby faces compared to FaceReader 9.
3. The East Asian model on average performed approximately 10% better on Asian faces of our internal test data set (1200 East Asian faces) than the General model. The East Asian model and General model had an accuracy of 93% and 91%, resp. when the Taiwanese Facial Expression Image Database (TFEID, 268 images, [7]) was tested.
4. Gender classification in FaceReader 10 achieves and accuracy of 94% on an internal test set.
5. The detection of (eye) glasses on faces achieves an accuracy of 99.2% on an internal test set.

* The F1 score is the weighted average of precision and recall. This score takes both false positives and false negatives into account.

References

1. Zafeiriou, S.; Zhang, C.; Zhang, Z. (2015) A survey on face detection in the wild: Past, present and future. Computer Vision and Image Understanding, 138: 1-24.
2. Bulat, A.; Tzimiropoulos, G. (2017) How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). Proceedings of the IEEE International Conference on Computer Vision, pp. 1021-1030.
3. Gudi, A.; Tasli, H.E.; den Uyl, T.M.; Maroulis, A. (2015) Deep learning based FACS action unit occurrence and intensity estimation. International Conference and Workshops on automatic face and gesture recognition (FG), 6: 1-5.
4. P. Ekman. Universal facial expressions of emotion. California Mental Health Research Digest, 8: 151-158, 1970.
5. Van der Schalk, J., Hawk, S. T., Fischer, A. H., & Doosje, B. J. (2011). Moving faces, looking places: The Amsterdam Dynamic Facial Expressions Set (ADFES), Emotion, 11, 907-920. DOI: 10.1037/a0023853
6. Olszanowski, M.; Pochwatko, G.; Kuklinski, K.; Scibor-Rylski, M.; Lewinski, P.; Ohme, R.K. (2014) Warsaw Set of Emotional Facial Expression Pictures: A validation study of facial display photographs. Frontiers in Psychology 5, DOI: 10.3389/fpsyg.2014.01516.
7. Oster, H. (2016). Baby FACS: Facial Action Coding System for infants and young children. Unpublished monograph and coding manual. New York University.
8. Chen, L.F.; Yen, Y.S. (2007) Taiwanese facial expression image database. Ph.D. dissertation, Brain Mapping Lab, Inst. Brain Sci., Nat. Yang-Ming Univ., Taipei, Taiwan.

To cite the use of FaceReader in publications you can use the following reference:

Noldus (2021). FaceReader: Tool for automatic analysis of facial expressions: Version 10 [Software]. Wageningen, The Netherlands: Noldus Information Technology B.V.

It is important to include the version number of the software in the reference since FaceReader's algorithms are improved with every upgrade.

Source: FaceReader 10 Reference Manual (Help), Noldus Information Technology