Knowledge Base

What Is an API?

An application programming interface (API) is a particular set of rules and specifications that software programs can follow to communicate with each other. It serves as an interface between different software programs and facilitates their interaction, similar to the way the user interface facilitates interaction between humans and computers.

This document describes how the FaceReader API can be used to integrate FaceReader into another application or framework. You do NOT need to read this document if you want to use FaceReader as a standalone tool.

If you want to use the API, please contact our support department for the possibilities. Our support department can also supply you with an example program that can receive FaceReader data. You can contact our support department via the Help menu of the FaceReader program, or via our website www.noldus.com/support-center.

What Can the FaceReader API Be Used For?

The FaceReader API allows you to integrate FaceReader with third-party software or your own software. It allows you to:

• Start and stop FaceReader analysis.
• Score Stimuli and Event Markers (only with the Project Analysis Module).
• Receive the classification results from FaceReader (live as they become available).

Whenever FaceReader has classified an image, video frame or camera frame, a network package containing the emotional state or detailed values is sent out to connected applications (TCP/IP) that can then use these results. This allows application developers to, for example, adjust the content of a website or application based on the emotions of a user or to log FaceReader values together with input values from other sensors.

In early versions of FaceReader (3 or lower) this was already possible by constantly reading the last line from the "live log" that FaceReader can produce. However, the API offers a number of advantages:

• Lower I/O overhead; no hard disk operations are necessary anymore and results are only sent once they become available.
• Integrating with applications on platforms other than Windows becomes easier as reading large log files from shared network locations is not always feasible.
• Integration with applications running on PCs in a completely different network, possibly on the other side of the world.
• Packages are sent in XML format which can be parsed consistently, regardless of the settings you choose in FaceReader.

How to Use the API

To make use of the API there are two options:

• Use the supplied FaceReaderAPI.dll. This is a .NET dll containing all functionality needed to receive from and send messages to FaceReader. The way to use this dll is explained by the sample source code of the FaceReaderExternalControlSample application.
• Build your own TCP communication application and send the messages defined in the API. This is explained in this document.

Receiving FaceReader Data in Microsoft .NET Applications

For developers who use Microsoft .NET technology a library (FaceReaderAPI.dll) has been included that allows you to communicate with FaceReader with only a few lines of code.

Preparation – Start FaceReader

1. Install and start FaceReader.
2. Choose File > Settings > Data Export.
3. Under External Communication (API and Stimulus Presentation Tool), select Enable External Control.
4. Use FaceReader as you normally would. You may close all visualization windows to speed up the analysis.

Adding FaceReaderAPI.dll to Your Project

1. Make sure the .NET 4.8 framework is installed on the computer where this dll is used. This framework can be downloaded from the Microsoft Download website, using Automatic Updates (check the Optional components). Alternatively, install FaceReader. This will automatically install the .NET framework if it is missing.
2. Add FaceReaderAPI.dll as a reference in your project.

Build Your Own TCP Communication Application

The FaceReader API uses TCP messages to talk to FaceReader and to receive messages from FaceReader. These messages are XML documents describing the information. There are three types of messages:

• Action Message – Used to make FaceReader perform an action. Possible actions are:
  • Start Analyzing – makes FaceReader start the currently open Analysis.
  • Stop Analyzing – makes FaceReader stop the currently open Analysis.
  • Start Receiving Detailed Logs – tells FaceReader to start sending detailed logs to this client.
  • Start Receiving State Logs – tells FaceReader to start sending state logs to this client.
  • Stop Receiving Detailed Logs – tells FaceReader to stop sending detailed logs to this client.
  • Stop Receiving State Logs – tells FaceReader to stop sending state logs to this client.
  • Get Stimuli – makes FaceReader send a Response Message containing a list of the available stimuli in the project.
  • Get Event Markers – makes FaceReader send a Response Message containing a list of the available event markers in the project.
  • Score Stimulus – makes FaceReader score a certain stimulus at the current timestamp.
  • Score Event Marker – makes FaceReader score a certain event marker at the current timestamp.
• Response Message – Used by FaceReader to send a response corresponding to a received Action Message. Possible responses are:
  • Error – the action defined by the Action Message was not possible or another error occurred.
  • Success – the Action Message was handled correctly.
  • Sends Stimuli – FaceReader sends the available stimuli in the project as a response to a Get Stimuli Action Message.
  • Sends Event Markers – FaceReader sends the available event markers in the project as a response to a Get Event Markers Action Message.
• Classification Message – Contains the results of a frame/image classified by FaceReader. Can be either:
  • State Log – contains the dominant expressions only.
  • Detailed Log – contains all the classifications enabled in the Logging Settings.

Action Message

An action message contains:

• Id – The Id can be any string. Can be used to identify a Response Message. FaceReader will send the Response Message corresponding to this action with the same Id.
• Action Type:
  • FaceReader_Start_Analyzing
  • FaceReader_Stop_Analyzing
  • FaceReader_Start_DetailedLogSending
  • FaceReader_Start_StateLogSending
  • FaceReader_Stop_DetailedLogSending
  • FaceReader_Stop_StateLogSending
  • FaceReader_Get_Stimulus
  • FaceReader_Get_EventMarker
• Information – Can be any amount of strings. Currently only the messages for scoring a stimulus or scoring an event marker should contain the name of this marker in the Information.

Example of an Action Message Without Extra Information

<?xml version="1.0" encoding="utf-8"?>
<ActionMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Id>ID01</Id>
  <ActionType>FaceReader_Start_Analyzing</ActionType>
</ActionMessage>

Example of an Action Message With Extra Information

<?xml version="1.0" encoding="utf-8"?>
<ActionMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Id>ID02</Id>
  <ActionType>FaceReader_Score_Stimulus</ActionType>
  <Information>
    <string>Stimulus 1</string>
  </Information>
</ActionMessage>

Response Message

A response message contains:

• Id – The Id corresponds to the Action Message this message is responding to.
• ResponseType:
  • FaceReader_Sends_Error
  • FaceReader_Sends_Success
  • FaceReader_Sends_Stimuli
  • FaceReader_Sends_EventMarkers
• Information – Can be any amount of strings. Contains for example an error message, a success message, or the Stimuli/Event Markers that the project contains.

Example of a Response Message With Information

<?xml version="1.0" encoding="utf-8"?>
<ResponseMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ResponseType>FaceReader_Sends_Error</ResponseType>
  <Information>
    <string>Already analyzing</string>
  </Information>
</ResponseMessage>

Example of a Response Message With Multiple Information Items

<?xml version="1.0" encoding="utf-8"?>
<ResponseMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ResponseType>FaceReader_Sends_Stimuli</ResponseType>
  <Information>
    <string>Stimulus 1</string>
    <string>Stimulus 2</string>
    <string>Stimulus 3</string>
  </Information>
</ResponseMessage>

Classification Message

A classification message contains:

• Id – The Id corresponds to the Action Message this message is responding to.
• LogType:
  • DetailedLog
  • StateLog
• AnalysisType:
  • Video
  • Camera
• FrameNumber
• AnalysisStartTime – Start time of the Analysis, when the first result was analyzed.
• FrameTimeTicks – Timestamp of the frame in ticks (1 second = 10,000,000 ticks).
  • In case of video, the FrameTimeTicks is the frame time in the video.
  • In case of camera, the FrameTimeTicks is the frame time past the AnalysisStartTime.
• ClassificationValues – List of ClassificationValue objects:
  • Classification Value
    • Type
      • Value (in case of a numeric value)
      • State (in case of a string value)
    • Label – Name of the ClassificationValue.
    • Value – Only has a value if Type == Value. Can be any amount of floats.
    • State – Only has a value if Type == State. Can be any amount of strings.

Example of a Value-Typed ClassificationValue

<ClassificationValue>
  <Type>Value</Type>
  <Label>Neutral</Label>
  <Value>
    <float>0.201352075</float>
  </Value>
  <State />
</ClassificationValue>

Example of a State-Typed ClassificationValue

<ClassificationValue>
  <Type>State</Type>
  <Label>Gender</Label>
  <Value />
  <State>
    <string>Female</string>
  </State>
</ClassificationValue>

Example of a MultiValued-Typed ClassificationValue

<ClassificationValue>
  <Type>Value</Type>
  <Label>Example</Label>
  <Value>
    <float>282.03125</float>
    <float>301.93515</float>
    <float>290.5138</float>
    <float>300.786652</float>
    <float>300.093658</float>
    <float>302.1726</float>
  </Value>
  <State />
</ClassificationValue>

Example of a Small Detailed Log

Note that the ClassificationValues reported by the API are selected with File > Settings > Data Export > Export (Detailed Log, ODX, N-Linx and API).

<?xml version="1.0" encoding="utf-8"?>
<Classification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <LogType>DetailedLog</LogType>
  <AnalysisType>Video</AnalysisType>
  <FrameNumber>9244</FrameNumber>
  <AnalysisStartTime>2012-08-01T06:54:24.4580478+02:00</AnalysisStartTime>
  <FrameTimeTicks>3697600000</FrameTimeTicks>
  <ClassificationValues>
    <ClassificationValue>
      <Type>Value</Type>
      <Label>Neutral</Label>
      <Value>
        <float>0.201352075</float>
      </Value>
      <State />
    </ClassificationValue>
    <ClassificationValue>
      <Type>Value</Type>
      <Label>Happy</Label>
      <Value>
        <float>0.00125725183</float>
      </Value>
      <State />
    </ClassificationValue>
    <ClassificationValue>
      <Type>Value</Type>
      <Label>Sad</Label>
      <Value>
        <float>0.151250109</float>
      </Value>
      <State />
    </ClassificationValue>
    <ClassificationValue>
      <Type>Value</Type>
      <Label>Angry</Label>
      <Value>
        <float>0.2761046</float>
      </Value>
      <State />
    </ClassificationValue>
  </ClassificationValues>
</Classification>

Example of a State Log

<?xml version="1.0" encoding="utf-8"?>
<Classification xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <LogType>StateLog</LogType>
  <AnalysisType>Video</AnalysisType>
  <FrameNumber>9601</FrameNumber>
  <AnalysisStartTime>2012-08-01T06:54:24.4580478+02:00</AnalysisStartTime>
  <FrameTimeTicks>3840400000</FrameTimeTicks>
  <ClassificationValues>
    <ClassificationValue>
      <Type>State</Type>
      <Label>Dominant Expression</Label>
      <Value />
      <State>
        <string>Happy</string>
      </State>
    </ClassificationValue>
  </ClassificationValues>
</Classification>

Source: Application Programming Interface - FaceReader 9 Technical Note, Noldus Information Technology