Knowledge Base

Reliability Analysis Settings

Synchronization Options

Align first scored events – Select this option if you want to take the first event you scored in each observation as a synchronization point. For example, if a classroom carried out live scoring of a projected video, previously scored by an expert.

Align video start time – Select this option if you score from video, but observer 1 started coding at another moment in the video than observer 2. For example, observer 1 started to code the video at t=0:00:00 and coded an event at t=0:01:00 and observer 2 started at t=0:00:50 and coded the same event about 10 seconds after that start.

Show Extra Kappa Statistics

Select this option to calculate the minimum, maximum and average Kappa of all the combined observation pairs. This enables you to easily see how much Kappa varies between your observation pairs.

Note that the average Kappa is not the same as the Kappa of the combined results of two or more pairs of observations. See Why are the Combined Kappa and Average Kappa Not the Same?

Combine Multiple Pairs of Observations

Select Include all results combined to get an additional row in the Reliability Analysis results sheet that sums up all agreements and disagreements from all pairs of observations.

When you select Include all results combined, each observation should occur in only one pair in the list. An overall Kappa or Rho value obtained with many pairs of observations is likely to be affected by bias and prevalence due to the large numbers in some cells of the matrix, not in others. Furthermore, an overall Kappa means losing information on single pairs of observations. In addition, it is always a good idea to export the Kappa values for each pair of observations, and then make a histogram of those values to see how they are distributed. Some observations or observers may be more associated with a lower Kappa than others. There may be observations in which some behaviors which are difficult to observe are prevalent, so they result in low Kappa values.

Modifiers

If you select Include Modifiers, these are included in the events to compare.

If some of the modifiers are numerical and you use the Frequency/Sequence comparison method, you can enter a Margin on numericals as a tolerance value. Modifiers in two events of the same type that differ by less than the margin result in an agreement.

See also the example in Confusion Matrix.

Choose Table Layout Options

Aim

To select the number of decimals in the results table and specify how to display elements that are in the coding scheme, but are not scored in any of the observations.

Prerequisite

You carried out at least one pair of observations.

Number of Decimals

1. Open the Settings tab of the Reliability Analysis Settings window (choose Analyze > Reliability Analysis > New > Settings).
2. Choose a number from the Number of decimals list.

Show Elements Not Scored

1. Open the Layout tab of the Reliability Analysis Settings window (choose Analyze > Reliability Analysis > New > Layout).
2. Select Show not scored elements if you want the result to include the event types that were not scored in any observation. If you do so, the matrix of agreements and disagreements will be larger. The cells corresponding to not scored events will contain the missing value symbol. See Show Missing Values As.

If the option is not selected (default), only the event types that were scored in at least one observation are included in the result.

Show Missing Values As

1. Open the Settings tab of the Reliability Analysis Settings window (choose Analyze > Reliability Analysis > New > Settings).
2. Choose the character you want to have displayed for missing values in the confusion matrix.

Reliability Analysis Methods in Detail

• The Frequency/Sequence Method in Detail
• The Duration/Sequence Method in Detail
• The Frequency Method in Detail
• The Duration Method in Detail