Knowledge Base

Statistics

How to Display the Statistics

1. Choose Analyze > Reliability Analysis > New.
2. Select the settings and click OK.
3. Click the Statistics button on the toolbar of the Reliability Analysis Results.

Available Statistics

• Agreements
• Disagreements
• Index of concordance
• Percentage of agreements
• Cohen's Kappa (k)
• Significance of Kappa
• Kappa max
• Pearson's Rho (ρ)
• Significance of Rho
• Prevalence index
• Confidence interval
• Extra Kappa statistics

Agreements

The duration of agreement (Duration, or Duration/Sequence method) or the number of agreements (Frequency, or Frequency/Sequence method) in the observation pair. It is the sum of the values in the diagonal of the confusion matrix of the Reliability Analysis. The way the Agreements are calculated depends on the comparison method.

Disagreements

The duration of disagreement (Duration, or Duration/Sequence method) or the number of disagreements (Frequency, or Frequency/Sequence method) in the observation pair. It is the total of the values in the off-diagonal cells, including Window Error, Modifier Error and No Records in the Confusion Matrix of the Reliability Analysis. The way the Disagreements are calculated depends on the comparison method.

Index of Concordance

The proportion of agreements between events in Reliability Analysis, calculated as Agreements / (Agreements + Disagreements). The values range between 0 (no agreements) and 1 (full agreement).

Percentage of Agreements

The percentage of agreements between events in Reliability Analysis, calculated as (Agreements / (Agreements + Disagreements)) * 100%. The values range between 0 (no agreements) and 100 (full agreement).

Cohen's Kappa (k)

An overall measure of agreement in Reliability Analysis, from Cohen J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37-46.

The formula is: κ = (po - pc) / (1 - pc)

Where po is the observed proportion of agreements and pc is the proportion of agreements expected by chance. The term aij is the value of the matrix cell at row i and column j. The terms ain and anj are the values of the cells of row i and column j (n ranges from 1 to the last item of the row/column). The term wij is the weight of the value at row i and column j, with two possible values: 0 for agreements (in the diagonal and in the light blue cells) and 1 for disagreements (the remaining cells).

When the events do not include numerical modifiers or the margin set for comparison is zero, the proportion of expected agreement is calculated from the row and column marginal totals.

The values of k range between -1 (non-random full disagreement) and +1 (non-random full agreement), but for practical purposes the range from 0 to 1.00 is of interest. A k of zero means that there is no agreement beyond chance, and a k of 1.00 means that there is perfect agreement. Interpretations of intermediate values are subjective.

Kappa is scored as Invalid if it is based on one comparison, such as one event in each observation, in which case agreement is 100%.

Significance of Kappa

To test the significance of κ, a standard score z is calculated and a one-tailed test on this score is carried out. The probability is shown next to k. A one-tailed test is considered appropriate when the null hypothesis states a value of zero for kappa, because a negative value of kappa does not normally have a meaningful interpretation.

The common statement that kappa is a "chance-corrected measure of agreement" may be misleading. As a test statistic, kappa can verify that agreement exceeds chance levels. But as a measure of the level of agreement, kappa is not "chance-corrected"; indeed, in the absence of some explicit model of rater decision making, one cannot know whether or not a specific agreement was achieved by chance.

Kappa Max

Kappa max is the maximum Kappa that can be obtained with your data. It is based on the maximum number of agreements possible in your observation pair. For example, if you have event B that is scored by two observers, and Observer 1 scores B three times while Observer 2 scores it nine times, then the maximum number of agreements in your observation pair is 3.

Kappa max is calculated as: κmax = (Amax - Ae) / (1 - Ae)

Where Amax is the maximum possible proportion of agreements and Ae is the proportion of agreements expected by chance, both derived from the row and column marginal totals of the confusion matrix.

Pearson's Rho (ρ)

A measure of the strength and direction of the linear relationship between the row totals and the column totals in the confusion matrix of a Reliability Analysis. If there are no disagreements in the confusion matrix, then the total for row 1 is equal to the total for column 1, the total for row 2 is equal to the total for column 2, and so on, meaning there is perfect correlation between the two observations.

Rho values range between -1.0 and +1.0, where -1.0 is the perfect negative (inverse) correlation, 0.0 means no correlation at all, and +1.0 is the perfect positive correlation.