(Publisher of Peer Reviewed Open Access Journals)
Navigation

International Journal of Advanced Computer Research (IJACR)

ISSN (Print):2249-7277    ISSN (Online):2277-7970
Volume-14 Issue-66 March-2024
Full-Text PDF
Paper Title : Greek sign language recognition using machine learning
Author Name : Mohamed Aqib Abid, Neetu Pillai and Nahsam Ahammed
Abstract :

This research develops a real-time Greek sign language (GSL) recognition system using machine learning and computer vision techniques to assist communication for the deaf community. The hand tracking module from the CVZone library was utilized for hand detection and tracking in video feeds. A dataset of 37,589 images capturing 24 Greek alphabet letter gestures was collected from different users under varying conditions. This dataset was used to train a deep neural network model based on the MobileNetV2 architecture. The model achieved 96.86% accuracy on the validation set during training. On the final test set, it obtained 95.66% accuracy in classifying and recognizing GSL hand gestures. Compared to prior rule-based and fuzzy clustering approaches, the model demonstrated significantly improved recognition performance. The high accuracy indicates that combining robust computer vision techniques for hand tracking with deep convolutional neural networks can be an effective approach for real-time GSL recognition. This system has the potential to facilitate communication and accessibility for the deaf and hard-of-hearing community. Further work should focus on expanding the vocabulary beyond individual letters to incorporate common words and phrases, as well as optimizing the system's real-time performance by reducing processing lags. Overall, this research demonstrates a promising machine learning and computer vision-based system for automatic GSL recognition that can aid the deaf community.
Keywords : SLR, ASLR, RNN and CNN.
Cite this article : Abid MA, Pillai N, Ahammed N. Greek sign language recognition using machine learning. International Journal of Advanced Computer Research. 2024; 14(66):1-17. DOI:10.19101/IJACR.2023.1362022.
References :
[1]Obi Y, Claudio KS, Budiman VM, Achmad S, Kurniawan A. Sign language recognition system for communicating to people with disabilities. Procedia Computer Science. 2023; 216:13-20.
[Crossref] [Google Scholar]
[2]Riad AM, Elminir HK, Shohieb SM. Hand gesture recognition system based on a geometric model and rule based classifier. British Journal of Applied Science & Technology. 2014; 4(9):1432-44.
[Google Scholar]
[3]Mariappan HM, Gomathi V. Real-time recognition of Indian sign language. In international conference on computational intelligence in data science (ICCIDS) 2019 (pp. 1-6). IEEE.
[Crossref] [Google Scholar]
[4]Wu J, Sun L, Jafari R. A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors. IEEE Journal of Biomedical and Health Informatics. 2016; 20(5):1281-90.
[Crossref] [Google Scholar]
[5]Rekha J, Bhattacharya J, Majumder S. Hand gesture recognition for sign language: a new hybrid approach. In proceedings of the international conference on image processing, computer vision, and pattern recognition (IPCV) 2011 (pp. 1-7). WorldComp.
[Google Scholar]
[6]Huu PN, Phung NT. Hand gesture recognition algorithm using SVM and HOG model for control of robotic system. Journal of Robotics. 2021; 2021:1-3.
[Crossref] [Google Scholar]
[7]Shinde P, Shinde P, Shinde S, Shinde S, Shinde S. Augmented reptile feeder. In Pune section international conference (PuneCon) 2022 (pp. 1-4). IEEE.
[Crossref] [Google Scholar]
[8]Ismail MH, Dawwd SA, Ali FH. A review on Arabic sign language recognition. Journal of Advances in Computer and Electronics Engineering. 2021; 6(12):1-12.
[Google Scholar]
[9]Madhubala B, Sarathambekai S, Vairam T, Seelan KS, Sathya RS, Swathy AR. Pre-trained convolutional neural network model based pneumonia classification from chest X-Ray images. In proceedings of the international conference on smart data intelligence (ICSMDI 2021) 2021(pp.1-8).
[Crossref]
[10]Michele A, Colin V, Santika DD. Mobilenet convolutional neural networks and support vector machines for palmprint recognition. Procedia Computer Science. 2019; 157:110-7.
[Crossref] [Google Scholar]
[11]Carney M, Webster B, Alvarado I, Phillips K, Howell N, Griffith J, et al. Teachable machine: approachable web-based tool for exploring machine learning classification. In extended abstracts of the 2020 CHI conference on human factors in computing systems 2020 (pp. 1-8). ACM.
[Crossref] [Google Scholar]
[12]Dogo EM, Afolabi OJ, Nwulu NI, Twala B, Aigbavboa CO. A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. In international conference on computational techniques, electronics and mechanical systems (CTEMS) 2018 (pp. 92-9). IEEE.
[Crossref] [Google Scholar]
[13]Patro VM, Patra MR. Augmenting weighted average with confusion matrix to enhance classification accuracy. Transactions on Machine Learning and Artificial Intelligence. 2014; 2(4):77-91.
[Google Scholar]
[14]Agarwal A, Sharma P, Alshehri M, Mohamed AA, Alfarraj O. Classification model for accuracy and intrusion detection using machine learning approach. Peer J Computer Science. 2021; 7:e437.
[Crossref] [Google Scholar]
[15]Wadhawan A, Kumar P. Deep learning-based sign language recognition system for static signs. Neural Computing and Applications. 2020; 32:7957-68.
[Crossref] [Google Scholar]
[16]Wen F, Zhang Z, He T, Lee C. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nature Communications. 2021; 12(1):1-13.
[Crossref] [Google Scholar]
[17]Rastgoo R, Kiani K, Escalera S. Sign language recognition: a deep survey. Expert Systems with Applications. 2021; 164:113794.
[Crossref] [Google Scholar]
[18]Rastgoo R, Kiani K, Escalera S. Hand sign language recognition using multi-view hand skeleton. Expert Systems with Applications. 2020; 150:113336.
[Crossref] [Google Scholar]
[19]Wadhawan A, Kumar P. Sign language recognition systems: a decade systematic literature review. Archives of Computational Methods in Engineering. 2021; 28:785-813.
[Crossref] [Google Scholar]
[20]Al-hammadi M, Muhammad G, Abdul W, Alsulaiman M, Bencherif MA, Alrayes TS, et al. Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access. 2020; 8:192527-42.
[Google Scholar]
[21]Corballis MC. From mouth to hand: gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences. 2003; 26(2):199-208.
[Crossref] [Google Scholar]
[22]Oyedotun OK, Khashman A. Deep learning in vision-based static hand gesture recognition. Neural Computing and Applications. 2017; 28(12):3941-51.
[Crossref] [Google Scholar]
[23]Nagi J, Ducatelle F, Di CGA, Cireşan D, Meier U, Giusti A, et al. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In international conference on signal and image processing applications (ICSIPA) 2011 (pp. 342-7). IEEE.
[Crossref] [Google Scholar]
[24]Rioux-maldague L, Giguere P. Sign language fingerspelling classification from depth and color images using a deep belief network. In Canadian conference on computer and robot vision 2014 (pp. 92-7). IEEE.
[Crossref] [Google Scholar]
[25]Huang J, Zhou W, Li H, Li W. Sign language recognition using 3d convolutional neural networks. In international conference on multimedia and expo (ICME) 2015 (pp. 1-6). IEEE.
[Crossref] [Google Scholar]
[26]Huang J, Zhou W, Li H, Li W. Sign language recognition using real-sense. In China summit and international conference on signal and information processing (ChinaSIP) 2015 (pp. 166-70). IEEE.
[Crossref] [Google Scholar]
[27]Pigou L, Dieleman S, Kindermans PJ, Schrauwen B. Sign language recognition using convolutional neural networks. In computer vision-ECCV workshops: Zurich, Switzerland, Proceedings, Part I 13 2015 (pp. 572-8). Springer International Publishing.
[Crossref] [Google Scholar]
[28]Molchanov P, Gupta S, Kim K, Pulli K. Multi-sensor system for drivers hand-gesture recognition. In 11th IEEE international conference and workshops on automatic face and gesture recognition (FG) 2015 (pp. 1-8). IEEE.
[Crossref] [Google Scholar]
[29]Tang A, Lu K, Wang Y, Huang J, Li H. A real-time hand posture recognition system using deep neural networks. ACM Transactions on Intelligent Systems and Technology (TIST). 2015; 6(2):1-23.
[Crossref] [Google Scholar]
[30]Yang S, Zhu Q. Video-based Chinese sign language recognition using convolutional neural network. In 9th international conference on communication software and networks (ICCSN) 2017 (pp. 929-34). IEEE.
[Crossref] [Google Scholar]
[31]Tushar AK, Ashiquzzaman A, Islam MR. Faster convergence and reduction of overfitting in numerical hand sign recognition using DCNN. In region 10 humanitarian technology conference (R10-HTC) 2017 (pp. 638-41). IEEE.
[Crossref] [Google Scholar]
[32]Bheda V, Radpour D. Using deep convolutional networks for gesture recognition in american sign language. arXiv preprint arXiv:1710.06836. 2017.
[Crossref] [Google Scholar]
[33]Rao GA, Syamala K, Kishore PV, Sastry AS. Deep convolutional neural networks for sign language recognition. In conference on signal processing and communication engineering systems (SPACES) 2018 (pp. 194-7). IEEE.
[Crossref] [Google Scholar]
[34]Koller O, Zargaran S, Ney H, Bowden R. Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. International Journal of Computer Vision. 2018; 126:1311-25.
[Crossref] [Google Scholar]
[35]Kumar EK, Kishore PV, Kumar MT, Kumar DA. 3D sign language recognition with joint distance and angular coded color topographical descriptor on a 2–stream CNN. Neurocomputing. 2020; 372:40-54.
[Crossref] [Google Scholar]
[36]Prabhu R. Understanding of convolutional neural network (CNN)-deep learning. Medium Com. 2018:1-11.
[Google Scholar]
[37]Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M. Real-time computer vision-based Bengali sign language recognition. In 17th international conference on computer and information technology (ICCIT) 2014 (pp. 192-7). IEEE.
[Crossref] [Google Scholar]
[38]Uddin MA, Chowdhury SA. Hand sign language recognition for Bangla alphabet using support vector machine. In international conference on innovations in science, engineering and technology (ICISET) 2016 (pp. 1-4). IEEE.
[Crossref] [Google Scholar]
[39]Rao GA, Kishore PV. Selfie video based continuous Indian sign language recognition system. Ain Shams Engineering Journal. 2018; 9(4):1929-39.
[Crossref] [Google Scholar]
[40]Zhou H, Zhou W, Zhou Y, Li H. Spatial-temporal multi-cue network for continuous sign language recognition. In proceedings of the AAAI conference on artificial intelligence 2020 (pp. 13009-16).
[Crossref] [Google Scholar]
[41]Sarhan N, Wilms C, Closius V, Brefeld U, Frintrop S. Hands in focus: sign language recognition via top-down attention. In international conference on image processing (ICIP) 2023 (pp. 2555-9). IEEE.
[Crossref] [Google Scholar]
[42]Marais M, Brown D, Connan J, Boby A. Spatiotemporal convolutions and video vision transformers for signer-independent sign language recognition. In international conference on artificial intelligence, big data, computing and data communication systems (icABCD) 2023 (pp. 1-6). IEEE.
[Crossref] [Google Scholar]
[43]Tarrés L, Gállego GI, Duarte A, Torres J, Giró-i-nieto X. Sign language translation from instructional videos. In proceedings of the conference on computer vision and pattern recognition 2023 (pp. 5624-34). IEEE.
[Google Scholar]
[44]Laines D, Gonzalez-mendoza M, Ochoa-ruiz G, Bejarano G. Isolated sign language recognition based on tree structure skeleton images. In proceedings of the conference on computer vision and pattern recognition 2023 (pp. 276-84). IEEE.
[Google Scholar]
[45]Arkushin RS, Moryossef A, Fried O. Ham2pose: Animating sign language notation into pose sequences. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2023 (pp. 21046-56).
[Crossref] [Google Scholar]
[46]Papadimitriou K, Sapountzaki G, Vasilaki K, Efthimiou E, Fotinea SE, Potamianos G. SL-REDU GSL: a large greek sign language recognition corpus. In international conference on acoustics, speech, and signal processing workshops (ICASSPW) 2023 (pp. 1-5). IEEE.
[Crossref] [Google Scholar]
[47]Papadimitriou K, Potamianos G. Sign language recognition via deformable 3d convolutions and modulated graph convolutional networks. In international conference on acoustics, speech and signal processing (ICASSP) 2023 (pp. 1-5). IEEE.
[Crossref] [Google Scholar]
[48]Naz N, Sajid H, Ali S, Hasan O, Ehsan MK. Signgraph: an efficient and accurate pose-based graph convolution approach toward sign language recognition. IEEE Access. 2023; 11:19135-47.
[Crossref] [Google Scholar]
[49]Al-qaisy NE, Al-kaseem BR, Al-dunainawi Y. AI-based portable gesture recognition system for hearing impaired people using wearable sensors. In 15th international conference on developments in eSystems engineering (DeSE) 2023 (pp. 33-8). IEEE.
[Crossref] [Google Scholar]
[50]Mahyoub M, Natalia F, Sudirman S, Mustafina J. Sign language recognition using deep learning. In 15th international conference on developments in eSystems engineering (DeSE) 2023 (pp. 184-9). IEEE.
[Crossref] [Google Scholar]