Cell Segmentation With Globally Optimized Boundaries (CSGO): A Deep Learning Pipeline for Whole-Cell Segmentation in Hematoxylin-and-Eosin–Stained Tissues

Accurate whole-cell segmentation is essential in various biomedical applications, particularly in studying the tumor microenvironment. Despite advancements in machine learning for nuclei segmentation in hematoxylin and eosin (H&E)-stained images, there remains a need for effective whole-cell segmentation methods. This study aimed to develop a deep learning–based pipeline to automatically segment cells in H&E-stained tissues, thereby advancing the capabilities of pathological image analysis. The Cell Segmentation with Globally Optimized boundaries (CSGO) framework integrates nuclei and membrane segmentation algorithms, followed by postprocessing using an energy-based watershed method. Specifically, we used the You Only Look Once (YOLO) object detection algorithm for nuclei segmentation and U-Net for membrane segmentation. The membrane detection model was trained on a data set of 7 hepatocellular carcinomas and 11 normal liver tissue patches. The cell segmentation performance was extensively evaluated on 5 external data sets, including liver, lung, and oral disease cases. CSGO demonstrated superior performance over the state-of-the-art method Cellpose, achieving higher F1 scores ranging from 0.37 to 0.53 at an intersection over union threshold of 0.5 in 4 of the 5 external datasets, compared to that of Cellpose from 0.21 to 0.36. These results underscore the robustness and accuracy of our approach in various tissue types. A web-based application is available at https://ai.swmed.edu/projects/csgo, providing a user-friendly platform for researchers to apply our method to their own data sets. Our method exhibits remarkable versatility in whole-cell segmentation across diverse cancer subtypes, serving as an accurate and reliable tool to facilitate tumor microenvironment studies. The advancements presented in this study have the potential to significantly enhance the precision and efficiency of pathologic image analysis, contributing to better understanding and treatment of cancer.