Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

Juan C. Tellez, Imad Radi, Rodrigo E. Alterio, Madhuri B. Nagaraj, Haley B. Baker, Daniel J. Scott, Herbert J. Zeh, Patricio M. Polanco

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Objective: Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks. Design: Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity. Setting: This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital. Participants: Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance. Results: Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels. Conclusions: This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.

Original languageEnglish (US)
Pages (from-to)589-596
Number of pages8
JournalJournal of Surgical Education
Volume81
Issue number4
DOIs
StatePublished - Apr 2024

Keywords

  • Patient Care
  • Practice-Based Learning and Improvement
  • SimNow
  • construct validity
  • proficiency-based training
  • robotic surgery
  • simulation
  • virtual reality

ASJC Scopus subject areas

  • Surgery
  • Education

Fingerprint

Dive into the research topics of 'Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum'. Together they form a unique fingerprint.

Cite this