TY - JOUR
T1 - A comparison of Monte Carlo dropout and bootstrap aggregation on the performance and uncertainty estimation in radiation therapy dose prediction with deep learning neural networks
AU - Nguyen, Dan
AU - Sadeghnejad Barkousaraie, Azar
AU - Bohara, Gyanendra
AU - Balagopal, Anjali
AU - McBeth, Rafe
AU - Lin, Mu Han
AU - Jiang, Steve
N1 - Publisher Copyright:
© 2021 Institute of Physics and Engineering in Medicine.
PY - 2021/3/7
Y1 - 2021/3/7
N2 - Recently, artificial intelligence technologies and algorithms have become a major focus for advancements in treatment planning for radiation therapy. As these are starting to become incorporated into the clinical workflow, a major concern from clinicians is not whether the model is accurate, but whether the model can express to a human operator when it does not know if its answer is correct. We propose to use Monte Carlo Dropout (MCDO) and the bootstrap aggregation (bagging) technique on deep learning (DL) models to produce uncertainty estimations for radiation therapy dose prediction. We show that both models are capable of generating a reasonable uncertainty map, and, with our proposed scaling technique, creating interpretable uncertainties and bounds on the prediction and any relevant metrics. Performance-wise, bagging provides statistically significant reduced loss value and errors in most of the metrics investigated in this study. The addition of bagging was able to further reduce errors by another 0.34% for on average, when compared to the baseline model. Overall, the bagging framework provided significantly lower mean absolute error (MAE) of 2.62, as opposed to the baseline model's MAE of 2.87. The usefulness of bagging, from solely a performance standpoint, does highly depend on the problem and the acceptable predictive error, and its high upfront computational cost during training should be factored in to deciding whether it is advantageous to use it. In terms of deployment with uncertainty estimations turned on, both methods offer the same performance time of about 12 s. As an ensemble-based metaheuristic, bagging can be used with existing machine learning architectures to improve stability and performance, and MCDO can be applied to any DL models that have dropout as part of their architecture.
AB - Recently, artificial intelligence technologies and algorithms have become a major focus for advancements in treatment planning for radiation therapy. As these are starting to become incorporated into the clinical workflow, a major concern from clinicians is not whether the model is accurate, but whether the model can express to a human operator when it does not know if its answer is correct. We propose to use Monte Carlo Dropout (MCDO) and the bootstrap aggregation (bagging) technique on deep learning (DL) models to produce uncertainty estimations for radiation therapy dose prediction. We show that both models are capable of generating a reasonable uncertainty map, and, with our proposed scaling technique, creating interpretable uncertainties and bounds on the prediction and any relevant metrics. Performance-wise, bagging provides statistically significant reduced loss value and errors in most of the metrics investigated in this study. The addition of bagging was able to further reduce errors by another 0.34% for on average, when compared to the baseline model. Overall, the bagging framework provided significantly lower mean absolute error (MAE) of 2.62, as opposed to the baseline model's MAE of 2.87. The usefulness of bagging, from solely a performance standpoint, does highly depend on the problem and the acceptable predictive error, and its high upfront computational cost during training should be factored in to deciding whether it is advantageous to use it. In terms of deployment with uncertainty estimations turned on, both methods offer the same performance time of about 12 s. As an ensemble-based metaheuristic, bagging can be used with existing machine learning architectures to improve stability and performance, and MCDO can be applied to any DL models that have dropout as part of their architecture.
KW - Bootstrap aggregation
KW - Deep learning
KW - Dose prediction
KW - Monte carlo dropout
KW - Radiation therapy
KW - Treatment planning
KW - Uncertainty
UR - http://www.scopus.com/inward/record.url?scp=85101856263&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85101856263&partnerID=8YFLogxK
U2 - 10.1088/1361-6560/abe04f
DO - 10.1088/1361-6560/abe04f
M3 - Article
C2 - 33503599
AN - SCOPUS:85101856263
SN - 0031-9155
VL - 66
JO - Physics in medicine and biology
JF - Physics in medicine and biology
IS - 5
M1 - 054002
ER -