Remote Assessment of Real-World Surgical Safety Checklist Performance Using the OR Black Box: A Multi-Institutional Evaluation

BACKGROUND: Large-scale evaluation of surgical safety checklist performance has been limited by the need for direct observation. The operating room (OR) Black Box is a multichannel surgical data capture platform that may allow for the holistic evaluation of checklist performance at scale. STUDY DESIGN: In this retrospective cohort study, data from 7 North American academic medical centers using the OR Black Box were collected between August 2020 and January 2022. All cases captured during this period were analyzed. Measures of checklist compliance, team engagement, and quality of checklist content review were investigated. RESULTS: Data from 7,243 surgical procedures were evaluated. A time-out was performed during most surgical procedures (98.4%, n = 7,127), whereas a debrief was performed during 62.3% (n = 4,510) of procedures. The mean percentage of OR staff who paused and participated during the time-out and debrief was 75.5% (SD 25.1%) and 54.6% (SD 36.4%), respectively. A team introduction (performed 42.6% of the time) was associated with more prompts completed (31.3% vs 18.7%, p < 0.001), a higher engagement score (0.90 vs 0.86, p < 0.001), and a higher percentage of team members who ceased other activities (80.3% vs 72%, p < 0.001) during the time-out. CONCLUSIONS: Remote assessment using OR Black Box data provides useful insight into surgical safety checklist performance. Many items included in the time-out and debrief were not routinely discussed. Completion of a team introduction was associated with improved time-out performance. There is potential to use OR Black Box metrics to improve intraoperative process measures.