Knowledge Base

Choose the Observations to Compare

Aim

To select which pairs of observations to use for the Reliability Analysis.

Prerequisites

• You carried out at least one pair of observations.
• The Settings tab of the Reliability Analysis Settings window is open (Choose Analyze > Reliability Analysis > New > Settings).

Procedure

Select an observation in the list with observations and click the Add>> button. Add as many pairs as you want.

Tip: If you have many observations, type part of the name in the Filter Observations field. For example, if you have a large number of observations with the type of patients file (on paper or on tablet) in the observation name, type the name of the patients file type in the Filter Observations field. As a result, the list of observations only shows the observations with that type in the name. To reset the filter, click the 'x' icon at the right of the Filter Observations field.

Notes

• If your observations contain two or more event logs, you must select single event logs for comparison. To do so, create a data profile with a filter on the event logs. See Filter Observations.
• If the list does not include an observation, it could be due to that observation containing errors. Open the observation and check for errors. See Check the Event Log for Errors.
• Reliability analysis is meant to be applied to whole observations. It is recommended not to apply a data profile where data are merged or split in intervals or time bins.
• If your data profile contains more than one result container (see Create Groups of Data), you need to specify one of them for each observation. The lists show all the combinations of observations and result containers.
• You can also add an observation pair in the Statistics view of the reliability analysis results by clicking the empty row at the bottom (indicated by Click to Add Observation).
• Always select the observation that is the 'gold standard' as the first observation in a pair. When you view the paired events in the comparison list, you see the video image associated with the first observation, not the second.

Edit the Selection

To Replace Observations

To replace observations in a pair in the Observation Pairs list, click on the observation you want to replace. The observation name becomes highlighted in blue. Now add the new observation; this will replace the old one.

To Remove Pairs

To remove a specific observation pair, click on one of the observations of that pair and click the Remove pair button. To remove the whole list of observation pairs, click the Remove all pairs button.

Choose the Comparison Settings

Aim

To choose the way The Observer XT compares the observation pairs.

Prerequisites

• You carried out at least one pair of observations.
• The Settings tab of the Reliability Analysis Settings window is open (Choose Analyze > Reliability Analysis > New > Settings).

Tip: Click on one of the Quick Help buttons to read more about an option or setting.

Settings

• Comparison method
• Tolerance window
• Analyze gaps between events
• Comparison scope
• Synchronization
• Show extra Kappa statistics
• Combine multiple pairs of observations
• Modifiers

Comparison Method

Select a comparison method from the list. See Reliability Analysis - Four Methods.

Tolerance Window

The Tolerance window setting is only available if you have selected the Frequency/Sequence comparison method. The tolerance window defines how accurate the timing of an event must be (in the comparison between two observations in Reliability Analysis) to be considered an agreement or not. For example, if in Observation 1 the event 'Gaze at screen' occurs at 4.0 s and stops at 6.0 s while in Observation 2 it starts at 6.5 s, that is counted as an agreement when the tolerance window is set to 3 s. However, if the tolerance window is set to 1 s, the two events are counted as a disagreement as the mismatch between their start times (2.5 s) is greater than the tolerance window.

Analyze Gaps Between Events

Analyze gaps between events is only available if you select Frequency/Sequence or Duration/Sequence as a comparison method. With this option, The Observer XT also takes gaps between events into account when comparing two observations. A gap is a time-interval in an observation in which no event was scored for a specific behavior group.

Important: Analyzing gaps between events may give unexpected results. We advise you to only select it if the stop of events are important for your analysis.

The example in the figure below shows the effect of including gaps in a reliability analysis with the Frequency/Sequence method. The figure shows two observations scored with two events, A and B. Observation 1 contains gaps, Observation 2 does not. The way gaps are compared differs between the comparison methods and also between behavior group types. See Reliability Analysis and Gaps.

Comparison Scope

With comparison scope, you define what section of the observations is compared for analysis. Select one of the following:

• Compare all events (default) – To compare all the events no matter whether one observation is longer than the other.
• Compare time based intersection – To compare two observations that differ in length, comparing only the events in the time period shared by both observations. For example, two observers score data from a video file. One observer starts scoring later and stops earlier in the video than the other observer. The result is that the second observation will be longer than the other. If you select Compare time based intersection, The Observer compares the video segments that both observers scored.

Synchronization

To select the way two observations in a pair are aligned:

• Use absolute start times (clock times) – Select this option if your observations were scored live, and you want to align them according to their 'true' start time (for example, 14h 23min 56s).
• Align the start of observations – Select this option if you coded from video and started your observations at the same moment in the video. Do not use this option when you score live and observers do not start scoring at the exact same time.