Knowledge Base

Reliability Analysis

Main Topics

• What is Reliability Analysis?
• Reliability Analysis - Four Methods
• Reliability Analysis General Procedure
• Reliability Analysis Settings
• Reliability Analysis Methods in Detail
• The Reliability Analysis Result
• Effect of the Coding Scheme on Reliability Analysis
• Reliability Analysis and Gaps
• Frequently Asked Questions about Reliability Analysis

What Is Reliability Analysis?

Consistency

In The Observer XT, Reliability Analysis assesses the extent to which two observations are in agreement with each other. This indicates how free the observations are from random errors and to which extent repeated measurements of the same event produce the same results.

Reliability Analysis always works with pairs of observations. If you want to compare three or more observations with each other, you have to specify each combination of observations as the pairs for comparison.

Intra-Observer Reliability

Check your own consistency. At the beginning of a study, make a video of your study subjects, and code it with The Observer XT. You keep both the video and the observational data file as a reference. Periodically, for example once every six months, you score the same video again and measure the agreement between the new data set and the reference data set.

Inter-Observer Reliability

Train observers. If a new student joins your research group, he/she has to adopt the coding scheme used in your research. After, for example, one month of practice, test the trainee's data with a reference data file. Reliability analysis helps determining when the trainee is ready for the real work.

Reliability Analysis - Four Methods

Available Methods

Frequency/Sequence

This method compares all event types. It compares the frequency of events and timing and overlap between the two observations.

Duration/Sequence

This method compares events with duration. It compares whether events overlap in two observations. It does not take into account events without duration and instantaneous sampling events.

Frequency

This method compares all event types. It compares the total number of occurrences of each event in the two observations. The duration or timing of events is not taken into account.

Duration

This method compares events with duration. It compares the total duration of each event between two observations. The frequency or timing of events is not taken into account. It does not take into account events without duration and instantaneous sampling events.

Which Method Should I Choose?

• Frequency/Sequence – A detailed indicator of the correspondence between two observations when timing of the events is important.
• Duration/Sequence – When you want to know to what extent two observers scored the same events with the same durations.
• Frequency – When timing and duration of events are not important in your study. This method is suited for observations with a relatively high rate of transitions between events.
• Duration – When you want to compare the duration of events but the timing is not important in your study.

Note: Just like in any other analysis, only the events selected in your active data profile will be used. If you run reliability analysis with the default data profile, all events in your observations will be used for comparison.

Reliability Analysis General Procedure

Aim

To determine the reliability of annotations, either between coders or between observations of the same coder.

Prerequisites

• Your license for The Observer XT includes the Advanced Analysis Module.
• You have at least two observations, scored from the same video or live scene.

Procedure

1. Make sure that the data profile specifying the data you want to analyze is active, that is, highlighted in blue in the Project Explorer.
2. Do one of the following:
   • Choose Analyze > Reliability Analysis > New.
   • In the Project Explorer, click the Analyses folder. Then choose Analyze data > Reliability Analysis.
   • In the Project Explorer, expand the Analyses item. Click Reliability Analysis > New Reliability Analysis.
   The Reliability Analysis Settings window appears.
3. In the Pairs tab, add the pairs of observations you want to compare. See Choose the Observations to Compare.
4. In the Settings tab, select the Comparison method and additional settings. See Choose the Comparison Settings.
5. Click OK to calculate the reliability statistics.
6. Optionally, edit the table layout. See Choose Table Layout Options.

The results of the selected comparison method are shown in the Reliability Analysis Result. The Statistics and Confusion Matrix are displayed. If you selected the Frequency/Sequence method, the Results also contains the Comparison List.

Save Your Analysis Settings

To save the settings specified in your analysis, click the Set default button. If this window is gone, click Settings on top of the result window. The settings for the Settings and Layout tab are saved.

Apply Settings to a New Analysis

If you are about to run reliability analysis and want to recall the settings you have specified earlier, click the Reset to default button. The settings for the Settings and Layout tab are restored.

Reliability Analysis Settings

What Do You Want to Do?

• Choose the Observations to Compare
• Choose the Comparison Settings
• Choose Table Layout Options