Meaningful Assessment of Robotic Surgical Style using the Wisdom of Crowds

M. Ershad, Robert V Rege, A. Majewicz Fey

Research output: Contribution to journalArticlepeer-review

23 Scopus citations


Objective: Quantitative assessment of surgical skills is an important aspect of surgical training; however, the proposed metrics are sometimes difficult to interpret and may not capture the stylistic characteristics that define expertise. This study proposes a methodology for evaluating the surgical skill, based on metrics associated with stylistic adjectives, and evaluates the ability of this method to differentiate expertise levels. Methods: We recruited subjects from different expertise levels to perform training tasks on a surgical simulator. A lexicon of contrasting adjective pairs, based on important skills for robotic surgery, inspired by the global evaluative assessment of robotic skills tool, was developed. To validate the use of stylistic adjectives for surgical skill assessment, posture videos of the subjects performing the task, as well as videos of the task were rated by crowd-workers. Metrics associated with each adjective were found using kinematic and physiological measurements through correlation with the crowd-sourced adjective assignment ratings. To evaluate the chosen metrics’ ability in distinguishing expertise levels, two classifiers were trained and tested using these metrics. Results: Crowd-assignment ratings for all adjectives were significantly correlated with expertise levels. The results indicate that naive Bayes classifier performs the best, with an accuracy of 89 ± 12 , 94 ± 8 , 95 ± 7 , and 100 ± 0 % when classifying into four, three, and two levels of expertise, respectively. Conclusion: The proposed method is effective at mapping understandable adjectives of expertise to the stylistic movements and physiological response of trainees.

Original languageEnglish (US)
Pages (from-to)1037-1048
Number of pages12
JournalInternational Journal of Computer Assisted Radiology and Surgery
Issue number7
StatePublished - Jul 1 2018


  • Crowd-sourcing
  • Motion analysis
  • Robotic surgery
  • Surgical skill assessment

ASJC Scopus subject areas

  • Surgery
  • Biomedical Engineering
  • Radiology Nuclear Medicine and imaging
  • Computer Vision and Pattern Recognition
  • Health Informatics
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Meaningful Assessment of Robotic Surgical Style using the Wisdom of Crowds'. Together they form a unique fingerprint.

Cite this