Measuring agreement in medical informatics reliability studies

George Hripcsak, Daniel F. Heitjan

Research output: Contribution to journalArticlepeer-review

171 Scopus citations


Agreement measures are used frequently in reliability studies that involve categorical data. Simple measures like observed agreement and specific agreement can reveal a good deal about the sample. Chance-corrected agreement in the form of the kappa statistic is used frequently based on its correspondence to an intraclass correlation coefficient and the ease of calculating it, but its magnitude depends on the tasks and categories in the experiment. It is helpful to separate the components of disagreement when the goal is to improve the reliability of an instrument or of the raters. Approaches based on modeling the decision making process can be helpful here, including tetrachoric correlation, polychoric correlation, latent trait models, and latent class models. Decision making models can also be used to better understand the behavior of different agreement metrics. For example, if the observed prevalence of responses in one of two available categories is low, then there is insufficient information in the sample to judge raters' ability to discriminate cases, and kappa may underestimate the true agreement and observed agreement may overestimate it.

Original languageEnglish (US)
Pages (from-to)99-110
Number of pages12
JournalJournal of Biomedical Informatics
Issue number2
StatePublished - 2002


  • Agreement
  • Kappa
  • Latent structure analysis
  • Prevalence
  • Reliability
  • Tetrachoric correlation

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics


Dive into the research topics of 'Measuring agreement in medical informatics reliability studies'. Together they form a unique fingerprint.

Cite this