Knowledge Base

The Frequency/Sequence Method in Detail

Calculation

For each pair of observations selected in the Pairs tab of the Reliability Analysis Settings window, The Observer XT analyzes the two observations in five runs. The Observer XT searches for matches between state events and matches between point events. State events are never matched to point events.

Gaps between events are only included in the calculation if you select the option Analyze gaps between events. See also Effect of Behavior Group – Frequency/Sequence Method and Reliability Analysis and Gaps.

Example

See the example below of two observations. B, C, and E are events with duration. F and G are events without duration. A tolerance window of 1 s was used and gaps between events were not analyzed.

Run 1 – Find Overlapping Events

The program searches for events of the same type in the two observations that overlap in time at least partially. The onset and offset times of the events need not be the same.

The following agreements are scored:

• 1.6 Event E (Obs 1) – 0.8 Event E (Obs 2)
• 4.5 Event C (Obs 1) – 4.1 Event C (Obs 2)
• 12.3 Event E (Obs 1) – 10.7 Event E (Obs 2)

Run 2 – Find Overlapping Events Within the Tolerance Window

The program searches for events of the same type among all those that have not been considered in the previous run, and scores agreements between those which do not overlap, but whose onset time differs less than the tolerance window (dotted lines).

The onset times of Event B differ by less than the tolerance window. The following agreement is scored:

• 0.9 Event B (Obs 1) – 0.2 Event B (Obs 2)

Run 3 – Find Disagreements: First Event Within Tolerance Window

The program considers the events left out in the previous runs, and searches for any event in the other observation within the tolerance window (dotted lines) that overlaps the event in the first observation. If multiple events are available in the other observation, then the first event is considered. Pairing in run 3 always results in disagreements, because if the events are of the same type they would have been considered as an agreement in run 1 or 2.

The following disagreement is scored:

• 11.0 Event G (Obs 1) – 11.2 Event F (Obs 2)

Run 4 – Find Disagreements: Any Event Within Tolerance Window

The program considers the events left out in the previous runs; however, it searches for any event in the other observation within the tolerance window (dotted lines), even if that has been scored as agreement or disagreement in a previous run. If multiple events are available in the other observation, then the last event is considered. Pairing results in disagreements.

The following disagreements are scored:

• 11.0 Event B (Obs 1) – 10.7 Event E (Obs 2)
• 11.0 Event G (Obs 1) – 11.9 Event F (Obs 2)

Please note that 10.7 Event E (Obs 2) was also paired as an agreement in run 1. This is one of the frequent cases in which an event produces one agreement and one or more disagreements. 11.0 Event G (Obs 1) was also paired as a disagreement in run 3.

Run 5 – Find Disagreements: Events Outside the Tolerance Window

The program considers the events that have not been considered in the previous runs, and searches for the most nearby event in the other observation that has yet to be scored as agreement or disagreement. If two events are equally far in time from the focal event, the first of the two is considered. Pairing results in disagreements. If the paired events are of the same type, which is the case in the example below, they are scored as Window Error.

The following disagreement is scored:

• 8.0 Event F (Obs 1) – 11.7 Event F (Obs 2)

Please note that 11.2 Event F (Obs 2) was also paired as a disagreement in run 3.

References

For more information, please see the following paper: Jansen, R. G., Wiertz, L. F., Meyer, E. S., & Noldus, L. P. (2003). Reliability analysis of observational data: Problems, solutions, and software implementation. Behavior Research Methods, Instruments, & Computers, 35(3), 391–399.

For general information about reliability analysis see Haccou, P., & Meelis, E. (1992). Statistical analysis of behavioural data: An approach based on time-structured models. Oxford University Press.

Agreements and Disagreements

The number of Agreements (A) and Disagreements (D) are used in calculation of the reliability statistics. Each count represents an event pair. See Statistics. You find these values by adding up the cells in the Confusion Matrix. The number of Agreements is the sum of the cells in the diagonal. The number of Disagreements is the sum of off-diagonal cells, including Window Error, Modifier Error, and No Records.

Stop Times

The stop times are not compared directly; however, they are compared indirectly. When you score the stop of a behavior without starting a new one, there are gaps between events. If you selected Analyze gaps between events, The Observer XT treats such gaps as <Gap "Behavior name"> and finds matches between them just like events. For more information, see <Gap "Behavior name"> and Reliability Analysis and Gaps.

Analyze Modifiers

• If behavioral modifiers are not defined in your coding scheme, the Behavior Modifiers options in the settings window are not available.
• If you select the Include Modifiers option, comparison of events is more discriminating, as two events with the same subject or behavior but different modifiers will result in a disagreement. De-selecting this option makes the comparison of events less selective.

Choose Specific Event Logs for Comparison

In the Reliability Analysis Settings window, you choose observations for comparison. If your observations include two or more event logs, this window does not distinguish between them. To select a specific event log, open your data profile (or create a new one) and create a Result container for each event log. Next, filter event logs in such a way that the event logs you want to compare end up in different Results boxes. Then, in the Reliability Analysis Settings window, choose the combination Observation * Result container to specify an event log for comparison.

Comments

If you have scored comments without associated behaviors, these rows in the event log result in <Missing behavior> in the analysis, which are treated as events without duration in the reliability analysis. This affects the outcome of the reliability analysis. To avoid this, create a data profile (see Select Data for Analysis) and use a filter box in which you select all behaviors. This filter box removes the lines in your event log that contain only comments. Then carry out the reliability analysis.