|| Author: Duncan Riley|

How To Measure Interobserver Agreement

Line diagrams for study 2 of the patient and observers. Display of individual measurements after the patient (N = 14), with all three observers displayed in different colors. The first series consists of plots for patients 1 to 3, the second series for patients 4 to 6, etc. Note that the scan was not available for patient 3 at time 4 due to a technical error. Reliability refers to a test`s ability to distinguish patients despite measurement errors, while compliance focuses on the measurement error itself [11]. The assessment of reliability is well established and is generally done using intraclassical correlation coefficients (ICC) [12, 13]. Duration-based IOA algorithms evaluate the concordance between the timing data of two observers. These measures consist of (a) the total duration and (b) the average duration per presence. Table 3 summarizes the strengths of both algorithms. Consider as a common example of a time-based IOA the hypothetical data flow shown in Figure 3, in which two independent observers record the duration of a target response across four deposits. IOA at the point interval. One approach to improving the accuracy of two observers` compliance with interval recording is simply to limit compliance testing to cases where at least one of the observers recorded a target response at an interval.

Intervals during which none of the observers reported a target response are excluded from the calculation in order to obtain stricter concordance statistics. Cooper et al. (2007) suggest that IOA with a point interval (also called “presence agreement” in the research literature) is most advantageous when target reactions appear at low rates.

Comments are closed