AIMS: Intraclass correlation coefficients (ICCs) are commonly used to assess the interrater reliability of ratings on a set of targets (e.g., individuals) across various raters. Different ICCs are used depending on whether the raters are treated as random or fixed. Typically, raters are treated as random when the researcher wishes to generalize the reliability findings to an arbitrarily larger population of potential raters, whereas raters are treated as fixed when the researcher is only interested in the specific raters within the study. The aim of this presentation is to introduce an agreement-based ICC for defining interrater reliability when the raters in the study are viewed as being sampled from a relatively small, finite population of raters.
METHODS: Suppose a clinical trial plans to use K=5 raters to provide ratings on a clinical outcome assessment. Prior to the trial a small-scale study is conducted to ensure the ratings are reliable. However, within this study, only a sample of k=3 of the 5 raters could provide ratings used to estimate the interrater reliability. Treating the raters as fixed is inappropriate, as the researcher would like to generalize beyond the k=3 raters to the K=5 raters that will be used in the trial. However, the random effect model is also inappropriate, as the researcher is not interested in generalizing beyond the K=5 raters that will be used within the clinical trial. A finite population ICC for interrater reliability is proposed to account for the sampled raters belonging to a small population of potential raters.
RESULTS: Interestingly, the fixed and random rater ICCs can be viewed as special cases of the finite population ICC. That is, when the number of sampled raters k is equal to the finite population of raters K, the ICC reduces to the ICC with a fixed rater effect. As the population of raters K tends towards infinity, the ICC reduces to the ICC with a random rater effect.
CONCLUSION: The proposed finite sample ICC is a more general ICC formulation that is applicable to fixed raters, random raters, and everything in between.
