A method of multimodal machine sign language translation for natural human-computer interaction
Journal: Scientific and Technical Journal of Information Technologies, Mechanics and Optics (Vol.22, No. 3)Publication Date: 2022-06-23
Authors : Axyonov A.A. Kagirov I.A. Ryumin D.A.;
Page : 585-593
Keywords : body language; gesticulation; machine sign language translation; naturalness of a communication medium;
Abstract
This paper aims to investigate the possibility of robustness enhancement as applied to an automatic system for isolated signs and sign languages recognition, through the use of the most informative spatiotemporal visual features. The authors present a method for the automatic recognition of gestural information, based on an integrated neural network model, which analyses spatiotemporal visual features: 2D and 3D distances between the palm and the face; the area of the hand and the face intersection; hand configuration; the gender and the age of signers. A 3DResNet-18-based neural network model for hand configuration data extraction was elaborated. Deepface software platform neural network models were embedded in the method in order to extract gender and age-related data. The proposed method was tested on the data from the multimodal corpus of sign language elements TheRuSLan, with the accuracy of 91.14 %. The results of this investigation not only improve the accuracy and robustness of machine sign language translation, but also enhance the naturalness of human-machine interaction in general. Besides that, the results have application in various fields of social services, medicine, education and robotics, as well as different public service centers.
Other Latest Articles
- The Green Energy Transition and Energy Security in Mexico, 1980–2016 Expansion and Intensification of Extractivism
- Modelling of basic Indonesian Sign Language translator based on Raspberry Pi technology
- Quantum-probabilistic SVD: complex-valued factorization of matrix data
- Improving sign language processing via few-shot machine learning
- Method for generating masks on face images and systems for their recognition
Last modified: 2022-06-23 20:21:10