Multi-Modality and Multi-View 2D CNN to Predict Locoregional Recurrence in Head Neck Cancer

Locoregional recurrence (LRR) remains one of leading causes in head and neck (HN) cancer treatment failure despite the advancement of multidisciplinary management. Accurately predicting LRR in early stage can help physicians make an optimal personalized treatment strategy. In this study, we propose an end-to-end multi-modality and multi-view convolutional neural network model (mMmV-CNN) for LRR prediction in HN cancer. In mMmV, a dimension reduction operator is designed, projecting the 3D volume onto 2D images in different directions, and a multi-view strategy is used to replace the original 3D method, which reduces the complexity of the algorithm while preserving important 3D information. Meanwhile, multi-modal data is used for the classification by making full use of the complementary information from cross modality data. Furthermore, we design a multi-modality deep neural network which is trained in an end-to-end manner and jointly optimize the deep features of CT, PET and clinical features. A HN dataset which consists of 206 patients was used to evaluate the performance. Experimental results demonstrated that mMm V-CNN can obtain an AUC value of 0.81 and outperform a state of the art CNN-based method.