Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening.
Authors
Affiliations (10)
Affiliations (10)
- Department of Computer Information Systems, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
- Advanced Computing (AdvCOMP), Centre of Excellence, Universiti Malaysia Perlis (UniMAP), Pauh Putra Campus, Arau 02600, Perlis, Malaysia.
- Medical Imaging Department, School of Integrative Medicine & Life Sciences, KPJ Healthcare University, Nilai 71800, Negeri Sembilan, Malaysia.
- Department of Biomedical Systems and Informatics Engineering, Yarmouk University, P.O. Box 566, Irbid 21163, Jordan.
- Department of Pathology, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
- Department of Radiology and Nuclear Medicine, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
- Public Health & Preventive Medicine Division, Directorate of the Royal Medical Services, P.O. Box 855005, Amman 11855, Jordan.
- Hematology and Blood Banking, Ministry of Health, Princess Basma Teaching Hospital, Irbid 21110, Jordan.
- Department of Obstetrics and Gynecology, King Abdullah University Hospital, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
- Department of General Surgery, King Abdullah University Hospital, P.O. Box 3030, Irbid 22110, Jordan.
Abstract
<b>Background</b>: Cervical cancer remains one of the leading causes of cancer-related morbidity among women worldwide. The Papanicolaou (Pap) smear is widely used for early detection; however, its manual interpretation is time-consuming, requires substantial expertise, and is often affected by inter-observer variability, particularly in cases with dense and overlapping cells. <b>Methods</b>: This study developed an optimized deep learning framework for cervical cell instance segmentation, specifically targeting the separation of overlapping cells in Pap smear images. The proposed framework was based on Mask R-CNN with a ResNet-50 backbone and Feature Pyramid Network. A public development dataset of 460 cervical smear images was used for model development and internal evaluation, while an independent 210-image dataset collected from King Abdullah University Hospital was reserved as an external held-out clinical assessment set. To improve small-cell detection and mask refinement, the Region Proposal Network was adapted using biologically informed anchor scales (16, 32, 64, 128, 256), followed by soft calibration-based post-processing. <b>Results</b>: The proposed framework achieved an AP50 of 89.90% and a Mask IoU of 90.44%. Small-cell performance, measured using APs under the COCO small-object convention, reached 22.80%, while APm and APl reached 68.45% and 81.42%, respectively. Clinical cell-count evaluation on the independent KAUH held-out set was performed using the mean count of five physicians as the reference standard and achieved an overall clinical detection accuracy of 93.00%. <b>Conclusions</b>: The optimized Mask R-CNN framework improved the detection and separation of overlapping cervical cells in Pap smear images and may serve as a supportive tool for cytopathology workflows. The results suggest that biologically informed anchor optimization and soft calibration can improve cell-level instance segmentation, particularly for small and overlapping cells. Further validation on larger multi-center datasets remains necessary before routine clinical deployment.