Evaluation of previously embolized intracranial aneurysms: Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists

Scott L. Zuckerman, Nikita Lakomkin, Jordan A. Magarik, Jan Vargas, Marcus Stephens, Babatunde Akinpelu, Alejandro M. Spiotta, Azam Ahmed, Adam S. Arthur, David Fiorella, Ricardo Hanel, Joshua A. Hirsch, Ferdinand K. Hui, Robert F. James, David F. Kallmes, Philip M. Meyers, David B. Niemann, Peter Rasmussen, Raymond D. Turner, Babu G. WelchJ. Mocco

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


Background The angiographic evaluation of previously coiled aneurysms can be difficult yet remains critical for determining re-treatment. Objective The main objective of this study was to determine the inter-rater reliability for both the Raymond Scale and per cent embolization among a group of neurointerventionalists evaluating previously embolized aneurysms. Methods A panel of 15 neurointerventionalists examined 92 distinct cases of immediate post-coil embolization and 1 year post-embolization angiographs. Each case was presented four times throughout the study, along with alterations in demographics in order to evaluate intra-rater reliability. All respondents were asked to provide the per cent embolization (0-100%) and Raymond Scale grade (1-3) for each aneurysm. Inter-rater reliability was evaluated by computing weighted kappa values (for the Raymond Scale) and intraclass correlation coefficients (ICC) for per cent embolization. Results 10 neurosurgeons and 5 interventional neuroradiologists evaluated 368 simulated cases. The agreement among all readers employing the Raymond Scale was fair (κ =0.35) while concordance in per cent embolization was good (ICC=0.64). Clinicians with fewer than 10 years of experience demonstrated a significantly greater level of agreement than the group with greater than 10 years (κ =0.39 and ICC=0.70 vs κ =0.28 and ICC=0.58). When the same aneurysm was presented multiple times, clinicians demonstrated excellent consistency when assessing per cent embolization (ICC=0.82), but moderate agreement when employing the Raymond classification (κ =0.58). Conclusions Identifying the per cent embolization in previously coiled aneurysms resulted in good inter- and intra-rater agreement, regardless of years of experience. The strong agreement among providers employing per cent embolization may make it a valuable tool for embolization assessment in this patient population.

Original languageEnglish (US)
Pages (from-to)464-468
Number of pages5
JournalJournal of NeuroInterventional Surgery
Issue number5
StatePublished - 2018


  • aneurysm
  • angiography
  • coil

ASJC Scopus subject areas

  • Surgery
  • Clinical Neurology


Dive into the research topics of 'Evaluation of previously embolized intracranial aneurysms: Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists'. Together they form a unique fingerprint.

Cite this