Comparison of simulation-based assessments and faculty ratings for general surgery resident milestone evaluation: Are they telling the same story?

Background: The goal of this article is to examine the role of simulation in documenting resident nontechnical competencies outlined by the general surgery milestones, and compare those assessments to faculty ratings. Methods: Trainees completed a multiphasic simulation scenario to assess ability to obtain informed consent, lead a preoperative time out, crisis management, communication, and delivering bad news. Assessments from this scenario were compared with ratings of these same competencies collected from clinical rotations. Results: Twenty-six PGY1 trainees participated in the training program. Results revealed no significant correlations between simulation performance and faculty rotation ratings for any of the 5 competencies. With the exception of communication in the operating room, faculty reported an overall inability to reliably observe these competencies 26% to 41% of the time. Conclusions: This work suggests that traditional end-of-rotation evaluations may not be the most feasible and valid method to evaluate resident nontechnical skills for milestone assessment. Simulation-based assessments should be considered to fill this gap.