Improving handover competency in preclinical medical and health professions students: establishing the reliability and construct validity of an assessment instrument

Background: As part of the worldwide call to enhance the safety of patient handovers of care, the Association of American Medical Colleges (AAMC) requires that all graduating students “give or receive a patient handover to transition care responsibly” as one of its Core Entrustable Professional Activities (EPAs) for Entering Residency. Students therefore require educational activities that build the necessary teamwork skills to perform structured handovers. To date, a reliable instrument designed to assess teamwork competencies, like structured communication, throughout their preclinical and clinical years does not exist. Method: Our team developed an assessment instrument that evaluates both the use of structured communication and two additional teamwork competencies necessary to perform safe patient handovers. This instrument was utilized to assess 192 handovers that were recorded from a sample of 229 preclinical medical students and 25 health professions students who participated in a virtual course on safe patient handovers. Five raters were trained on utilization of the assessment instrument, and consensus was established. Each handover was reviewed independently by two separate raters. Results: The raters achieved 72.22 % agreement across items in the reviewed handovers. Krippendorff’s alpha coefficient to assess inter-rater reliability was 0.6245, indicating substantial agreement among the raters. A confirmatory factor analysis (CFA) demonstrated the orthogonal characteristics of items in this instrument with rotated item loadings onto three distinct factors providing preliminary evidence of construct validity. Conclusions: We present an assessment instrument with substantial reliability and preliminary evidence of construct validity designed to evaluate both use of structured handover format as well as two team competencies necessary for safe patient handovers. Our assessment instrument can be used by educators to evaluate learners’ handoff performance as early as their preclinical years and is broadly applicable in the clinical context in which it is utilized. In the journey to optimize safe patient care through improved teamwork during handovers, our instrument achieves a critical step in the process of developing a validated assessment instrument to evaluate learners as they seek to accomplish this goal.