Research on Tongue Image Segmentation and Classification Methods Based on Deep Learning and Machine Learning

Bin Liu, Zeya Wang, Kang Yu, Yunfeng Wang, Haiying Zhang, Tingting Song, Hao Yang

Tongue diagnosis is a crucial method in traditional Chinese medicine (TCM) for obtaining information about a patient’s health condition. In this study, we propose a tongue image segmentation method based on deep learning and a pixel-level tongue color classification method utilizing machine learning techniques such as support vector machine (SVM) and ridge regression. These two approaches together form a comprehensive framework that spans from tongue image acquisition to segmentation and analysis. This framework provides an objective and visualized representation of pixel-wise classification and proportion distribution within tongue images, effectively assisting TCM practitioners in diagnosing tongue conditions. It mitigates the reliance on subjective observations in traditional tongue diagnosis, reducing human bias and enhancing the objectivity of TCM diagnosis. The proposed framework consists of three main components: tongue image segmentation, pixel-wise classification, and tongue color classification. In the segmentation stage, we integrate the Segment Anything Model (SAM) into the overall segmentation network. This approach not only achieves an intersection over union (IoU) score above 0.95 across three tongue image datasets but also significantly reduces the labor-intensive annotation process required for training traditional segmentation models while improving the generalization capability of the segmentation model. For pixel-wise classification, we propose a lightweight pixel classification model based on SVM, achieving a classification accuracy of 92%. In the tongue color classification stage, we introduce a ridge regression model that classifies tongue color based on the proportion of different pixel categories. Using this method, the classification accuracy reaches 91.80%. The proposed approach enables accurate and efficient tongue image segmentation, provides an intuitive visualization of tongue color distribution, and objectively analyzes and quantifies the proportion of different tongue color categories. In the future, this framework holds potential for validation and optimization in clinical practice.