Towards Better Debriefing through Context-aware Video Segmentation in Standardized Patient Encounter Ear Exams

Evaluating medical students’ physical skills is vital for assessing and validating the acquisition and utilization of their medical knowledge. However, the conventional method of human-based grading for standardized patient encounter videos is costly and error-prone, requires the participation of experts, and may suffer from inter-rater reliability issues and long processing times. Here we propose a deep learning pipeline to identify, extract and score ear exams in over a thousand Simulation Center COSCE medical encounter videos. Our three-stage approach consists of audio-based pre-segmentation with Whisper and Silero VAD, tool detection with CLIP and body node detection with Detectron2. The results of our pipeline are then compared to human graded output and used for automatic extraction of the most relevant video segments. This approach represents a first step toward our overarching goal of expediting and enhancing the quality of the debriefing process following standardized assessments.