Knowledge Base

Confusion Matrix

How to Display the Confusion Matrix

1. Choose Analyze > Reliability Analysis > New.
2. Select the settings and click OK. See Reliability Analysis General Procedure.
3. Click the Confusion Matrix button on the toolbar of the Reliability Analysis Results.

The Confusion Matrix

The Confusion Matrix displays agreements and disagreements of the observation pairs in a matrix. This matrix has one row and one column for each combination of Subject, Behavior, and Modifiers scored in those observations. If you have multiple observation pairs, you can select the pairs from the list on the toolbar.

If you selected the option Include All Results Combined in the Comparison Settings (see Combine Multiple Pairs of Observations), there is an additional pair in the list on the toolbar: Observations A Combined - Observations B Combined.

Cell Values

The values of the confusion matrix can be understood as follows:

• Dark blue cells are agreements.
• White cells including Window Error, Modifier Error and No Records are disagreements.
• Light blue cells along the diagonals are displayed when your coding scheme contains numerical modifiers and you set a margin to the modifiers in the Reliability Analysis Settings. You can select a margin if you choose Frequency/Sequence or Duration/Sequence as the comparison method. The light blue cells contain agreements of events with modifier values that are not exactly the same in both observations but differ less than the margin.
• Additional rows and columns indicate special cases, including <Gap "Behavior Name">, No Records, Error, and Window Error.

<Gap "Behavior Name">

<Gap "Behavior Name"> can appear in the table when you chose Frequency/Sequence or Duration/Sequence as the comparison method and you chose to analyze gaps. It appears in the following cases:

• When no behavior is scored at the start of the observation. A <Gap "Behavior Name"> state is added at the start of the observation for each behavior of a Start-Stop group, ending with the start of the first behavior of that group.
• When a behavior with duration ends, that is, when you press the stop code for a behavior. A <Gap "Behavior Name"> state is added that starts at the end of the event and ends at the start of the next occurrence of that behavior.
• When the behavior group type is set to Behaviors Cannot Occur at the Same Time, it contains both events with and without duration, and you have only scored events without duration. A <Gap "Behavior Group Name"> state is added at the start of the observation, ending at the end of the observation.

No Records

Rows and columns named No Records can appear in the table when you chose Frequency/Sequence or Duration/Sequence as the comparison method. It shows the occurrences of events in one observation that cannot be paired with any event in the second observation. No Records appear in the following cases:

• There is no data from the same behavior group in the other observation.
• When you have selected Compare All Events from the Comparison Scope list and one or both of the following is true:
  • One observation is longer than the other.
  • You have aligned the two observations in such a way that one starts earlier than the other. For example, two live observations started at different times, and you have selected Use the Start of Observations as the Synchronization method.

Error

Rows and columns named Error appear in the table when you chose Frequency or Duration as the comparison method. It shows the number of occurrences (frequency method) or seconds (duration method) that events differ in both observations.

• Numbers in the Error row indicate that events occurred more often, or longer, in the first observation of a pair.
• Numbers in the Error column indicate that events occurred more often, or longer, in the second observation of a pair.

Window Error

Rows and columns named Window Error can appear in the Reliability Analysis table when you chose Frequency/Sequence as the comparison method. Rows and columns named Window Error show the disagreements between the events of the same type whose onset times differ by more than the Tolerance window.

• The Window Error row shows the disagreements between the same event types where the event in observation 1 (left of the diagonal) occurs earlier than that in observation 2.
• The Window Error column shows the disagreements between the same event types where the event in observation 1 occurs later than that in observation 2.

Notes

• If you have defined two or more observation pairs for comparison, the size of the matrix is constant across pairs of observations, because it includes the combinations of events in all pairs regardless of whether a specific event was scored in a specific pair of observations. This means that if event A was scored in an observation of Pair 1 and in neither observation of Pair 2, the matrix of Pair 2 still contains the row and column for event A.
• If you have selected Show Elements Not Scored, the matrix also includes those combinations of Subject, Behavior and their Modifiers that were not scored in those observations. Therefore, the matrix can be much larger.
• If you select Include All Results Combined for multiple comparisons (see Combine Multiple Pairs of Observations), the confusion matrix of the combined results page sums up all agreements and disagreements from all pairs of observations.