Machine Learning Used to Close the Communication Gap through a Linguistic Tool for Deaf People
![](https://revistas.uptc.edu.co/public/journals/15/cover_issue_959_en_US.jpg)
Abstract
Currently, there is a wide variety of resources and tools on the internet for integrating artificial intelligence into languagerelated projects. This article proposes the use of machine learning to address and reduce the communication gap between hearing people and deaf people who use sign language as a form of communication. For this purpose, the Edge Impulse tool, which facilitates the implementation of machine learning models on mobile devices, was employed. The SinSeñas2.0 project emerged as a response to this need and is based on the Extreme Programming (XP) methodology to understand user needs and offer an effective solution. A dataset of 3102 images of Colombian signs was collected, divided into 80% for training and 20% for testing. Convolutional neural networks (CNN) and deep learning techniques were used to train the model, which improved the accuracy in recognizing signs. The results showed that model configuration 1, with an accuracy of 99% and a loss of 0.03%, was the most effective. This configuration used an input size of 96x96 and employed transfer learning with the MobileNet V2 neural network. The tool also included data augmentation techniques to create a balanced and diversified dataset, thereby improving the model’s robustness against different capture conditions. The research demonstrates that SinSeñas2.0 significantly improves the accuracy and efficiency of sign language recognition compared to previous approaches that did not use machine learning. This advancement not only facilitates communication between hearing and deaf people but also represents a significant contribution to sign language translation technology, promoting the social inclusion of deaf individuals.
Keywords
artificial intelligence, Colombian sign language translation, communication gap, deaf people, edge Impulse, machine Learning
References
- C. Kwakye-Ndlovu, “The Impact of Technological Innovation On Deaf and Hard of Hearing Communities,” Honors College Theses, no. 987, Georgia Southern University, 2024. [Online]. https://digitalcommons.georgiasouthern.edu/honors-theses/987
- L. A. Tovar, and L. López, “La clasificación en la lengua de señas colombiana (LSC),” Lenguaje, vol. 46, no. 1, e6194, 2018. https://doi.org/10.25100/lenguaje.v46i1.6194
- World Federation of the Deaf, “Deaf People and Employment in the Developing World,” Global Report. Together, We have Achieved Much, 2019-2023. [Online]. https://wfdeaf.org/wp-content/uploads/2023/09/2019-2023-WFD-Report.pdf
- P. A. Rodríguez-Correa, A. Valencia-Arias, O. N. Patiño-Toro, Y. Oblitas Díaz, R. Teodori De la Puente, “Benefits and development of assistive technologies for Deaf people’s communication: A systematic review”, Frontiers Education, vol. 8, e1121597, 2023.
- https://doi.org/10.3389/feduc.2023.1121597
- C. F. Bravo Mosquera, M. F. Silva Joaqui, K. Márceles Villalba, S. Amador Donado, “SINSEÑAS: Mobile application for learning and translation of colombian sign language”, Ingenieria y Competitividad, vol. 26, no. 1, e12815, 2024. https://doi.org/10.25100/iyc.v26i1.12815
- J. A. Meriño Guzmán, “Diseño de un guante electrónico para la interpretación y traducción del lenguaje de señas en personas con discapacidad auditiva mediante tecnología Arduino e interfaz de visualización por medio de una aplicación Android,” Thesis, Unidades Tecnológicas de Santander, Bucaramanga, Colombia, 2022. http://repositorio.uts.edu.co:8080/xmlui/handle/123456789/3302
- I. Papastratis, C. Chatzikonstantinou, D. Konstantinidis, K. Dimitropoulos, P. Daras, “Artificial Intelligence Technologies for Sign Language”, Sensors, vol. 21, no. 17, e5843, 2021. https://doi.org/10.3390/s21175843
- R. Sreemathy, M. Turuk, I. Kulkarni y S. Khurana, “Sign language recognition using artificial intelligence”, Educ. Inf. Technol., 2022. https://doi.org/10.1007/s10639-022-11391-z
- T. Ananthanarayana et al., “Deep Learning Methods for Sign Language Translation”, ACM Trans. Accessible Comput., vol. 14, no. 4, pp. 1–30, 2021. https://doi.org/10.1145/3477498
- J. Bora, S. Dehingia, A. Boruah, A. A. Chetia, D. Gogoi, “Real-time Assamese Sign Language Recognition using MediaPipe and Deep Learning”, Procedia Comput. Sci., vol. 218, pp. 1384–1393, 2023. https://doi.org/10.1016/j.procs.2023.01.117
- A. L. Cavalcante Carneiro, L. Brito Silva, D. H. Pinheiro Salvadeo, “Efficient sign language recognition system and dataset creation method based on deep learning and image processing,” in Proc. SPIE 11878, Thirteenth International Conference on Digital Image Processing (ICDIP 2021), e1187803, 2021. https://doi.org/10.1117/12.2601018
- S. Gan, Y. Yin, Z. Jiang, L. Xie, S. Lu, “Towards Real-Time Sign Language Recognition and Translation on Edge Devices,” in Proceedings of the 31st ACM International Conference on Multimedia (MM ‘23), october 2023, pp. 4502-4512. https://doi.org/10.1145/3581783.3611820
- A. Kanavos, O. Papadimitriou, P. Mylonas, M. Maragoudakis, “Enhancing Sign Language Recognition Using Deep Convolutional Neural Networks,” in 2023 14th International Conference on Information, Intelligence, Systems & Applications (IISA), 2023, pp. 1-4. https://doi.org/10.1109/IISA59645.2023.10345865
- R. Astya, A. Agrawal, A. Chauhan, A. K. Sony, K. R. Thakur, P. Nand, “Enhancing Sign Language Detection using Tensor Flow,” in 2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N), Greater Noida, India, 2023, pp. 1-6. https://doi.org/10.1109/ICAC3N60023.2023.10541807.
- M. D. Dere, R. O. Dere, A. Adesina, A. R. Yauri, “Un marco integral para la traducción del lenguaje de señas americano a idiomas de bajos recursos en Nigeria,” Científico Africano, vol. 21, e01809, 2023. https://doi.org/10.1016/j.sciaf.2023.e01809.
- I. A. Adeyanju, O. O. Bello, M. A. Adegboye, “Machine learning methods for sign language recognition: A critical review and analysis,” Intelligent Systems with Applications, vol. 12, e200056, 2021. https://doi.org/10.1016/j.iswa.2021.200056.
- S. Ambler, Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process. Editorial Theresa Hudson, 2002. https://msoo.pbworks.com/f/Scott+W.+Ambler+-+Agile+Modeling.pdf
- Ministerio de Salud y Protección Social, Boletines poblacionales: Personas con Discapacidad - PCD. Bogotá: Oficina de Promoción Social del Ministerio de Salud y Protección Social, 2020. https://www.minsalud.gov.co/sites/rid/Lists/BibliotecaDigital/RIDE/DE/PS/boletinespoblacionales-
- personas-discapacidadI-2020.pdf
- J.J. Caiza, K. Márceles Villalba, “Estrategia inclusiva mediada por TIC para el mejoramiento de la comunicación de personas en condición de discapacidad auditiva y vocal,” Iberian Journal of information Systems and Technologies, vol. 23, pp. 184-193, 2019.