Sign Language Recognition Based Communication System Using Machine Learning Algorithm for Vocally Impaired People

Our research presents a system designed to empower individuals who are deaf or vocally impaired by enabling seamless communication through sign language recognition. The system integrates advanced sensor technology, data processing, and machine learning to translate hand and finger movements into understandable gestures. One accelerometer and five flex sensors are strategically placed on the fingers to capture precise movements, which are then transmitted to a receiver unit. The data is processed using a MATLAB-based application that employs various machine learning models, including Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Linear Discriminant Analysis (LDA), and Ensemble methods. The system is trained on a dataset generated from these sensor readings, with each model evaluated for its accuracy in gesture recognition. Among the tested models, the Ensemble method achieved the highest classification accuracy of 94.6%, making it the most effective for real-time sign language recognition. This system not only bridges the communication gap for deaf- mute individuals but also represents a significant step forward in creating more inclusive technologies for society.

Introduction

There are many deaf and vocally impaired (VI) people in the world. Sign language (SL) becomes their primary means to express feelings, ideas, and thoughts, using a combination of hand movements, postures, and facial expressions to facilitate communication. Sign language comprises three elements: manual features (hand gestures), non-manual features (facial expressions and body posture), and finger spelling (spelling out words from the local language) [1]–[3]. There are two main approaches to sign language which are based on inputs. They are vision based and sensor- based methods [4]. In vision-based techniques use camera-captured images, eliminating the need for additional sensors or specialized gloves. In contrast, sensor-based methods rely on gloves or sensors to recognize signs. Sign language gestures fall into two categories: image based and motion-based. Key recognition elements include finger-spelling, non-manual features (e.g., facial expressions), and word-level signs [5]. In sign language, communication primarily involves the upper body (waist up), especially hand and finger movements. Globally, 70–90 million people have speech impairments, while in Bangladesh alone, around 2.4 million use sign language—a number that continues to rise [6]. The World Federation of the Deaf estimates that more than 70 million people worldwide use more than 300 sign languages [7]. On the other hand, it is estimated that there are approximately 466 million deaf and mute people in the world, with 34 million of them being children [8]. The World Health Organization estimates that there are over 70 million deaf and silent people in the world. 32 million of the 360 million deaf people in the globe are young children. Moreover, by 2050, it is predicted that one in four people would have hearing loss of some kind [9]. For Deaf-mute people, some have made models by direct object detection or object movement. Like controlling any object through DC motor controlling [10], Home appliance [11]–[13] etc. Besides, among the vision-based models, the most used models are hand posture recognition [1], [14], lip posture, body posture [15], head movement [16], image processing [17] etc. There is so many research has been conducted on designing smart systems that can convert sign language into speech. Approximately 20 years ago, researchers began studying automatic sign language recognition (SLR), with a focus on American Sign Language [18], Australian [19], and Korean [20] sign languages. Since then, numerous Arabic [21], British [22],Chinese [23], French [24] and German [25] systems have been created. But very few attempts are made for the conversion from Bangla Sign Language (BSL) to Speech.

Previous research related to sign language is based on either sensors or algorithms. The authors in [26] developed a smart hand gesture recognition system using the wearable Magic Ring, which translates gestures into Japanese sign language. Using the KNN algorithm, the system achieves 85.2% accuracy with both hands, compared to 77.4% with just the right hand [27]. Starner and Pentland [28] developed a hand motion detection-based machine learning algorithm where a webcam captures input video. The video is pre-processed to enhance the RGB color space and convert it to YCbCr. However, sometimes the system fails to detect the motion of the hand, leap, or other parts of the body. To identify Indian Sign Language, an sEMG system uses a CRF algorithm, achieving 74.33% accuracy for hand gestures with five sensors [29]. A wearable glove-based device was proposed by the authors in [30], enabling real-time recognition of English alphabet hand gestures in sign language. This research develops an Android app that translates American Sign Language into text in real time. The app captures images via the smartphone camera, segments the skin using YCbCr, extracts feature with HOG, and classifies signs using an SVM, achieving 89.54% accuracy [31]. This Hatibaruah et al. [32] developed a developed a sign language interpreter that translates gestures into text on a display. Using histogram backprojection for image segmentation and CNNs trained on an Indian Sign Language database of 26 alphabets and 10 digits, the system achieved a testing accuracy of 89.89%. Abiyev et al. [33] developed an American Sign Language translator to assist mute individuals, using SSD for hand detection and a CNN algorithm to translate signs into text using a fingerspelling dataset. The author proposed a real-time sign language translator using R-CNN, achieving 92.25% accuracy in converting diverse sign language gestures into voice outputs [34]. This research presents a bidirectional sign language translation system using an NLP-based deep learning approach to convert gestures to audio and spoken words to animated 3D signs [35]. Soji and Kamalakannan [36] enhanced a CNN-based model for Indian Sign Language recognition, focusing on message-conveying gestures, and achieved 90.1% accuracy. To enhanced communication a hand gesture recognition system for sign language integrates FFNN and HMM algorithms with voice processing capabilities [37]. Lu et al. [38] developed a YoBu glove to identify gesture movements, using an ELM kernel-based algorithm with 18 IMU sensors. The system identified gestures from 54 extracted features and achieved 91.2% accuracy. Sriram and Nithiyanandham [39] introduced introduced a glove with 5 accelerometers and Bluetooth that decodes ASL gestures through axis orientation and mobile app converting motions to text and speech. This research uses a data glove with tilt sensors, an accelerometer and Bluetooth translates Malaysian Sign Language gestures [40]. The author proposed the YOLOv5s algorithm to detect hand gesture movements, achieving 92.30% accuracy in recognizing 29 letters of Turkish Sign Language [41]. The author proposed the realization of Turkish Sign Language expressions using a humanoid robotic hand [42]. The author used flashcards with Turkish words and pictures to elicit sign language expressions. Results showed that 13-year-olds understood both the words and pictures, while 8-year-olds only recognized the pictures and not the written words or their meanings [43].

Most of the previous research has largely focused on image processing for sign language recognition, but sensor-based models, despite their challenges, offer greater accuracy by avoiding issues like background noise and gesture errors. We designed a system for speech-impaired users, using a machine learning model trained on data from flex sensors and an accelerometer. The receiver unit contains two microcontrollers: the first microcontroller receives data from the transmitter, which are then communicated to the second microcontroller via the I2C protocol. The data is collected and plotted in MATLAB for specific signs, and this process is repeated to gather sufficient training data.

Research Method

System Block Diagram

In Fig. 1 the system block diagram and in Fig. 2, the simplified block diagram for the data acquisition system is presented. The system utilizes flex sensors, a 3-axis accelerometer, two Arduino Nanos, an Arduino Uno, a push button, a buzzer, two Bluetooth modules, and a MATLAB app. The system is divided into two main parts: the transmitter and the receiver units. The transmitter unit includes a flex sensor, a 3-axis accelerometer, an Arduino Nano, a push button, a buzzer, and a Bluetooth master module. It captures data from the sensors, which is then processed by Microcontroller-1 and transmitted via Bluetooth. The receiver unit, equipped with a Bluetooth slave module, receives the data and forwards it to Microcontroller-2, which then passes it to Microcontroller-3 using the I2C protocol. The data is subsequently sent to the MATLAB app for visualization, where it’s plotted as curves. If the MATLAB App program is correctly configured, the data is automatically saved to an Excel file. This data is then used to train a machine learning model, making the system fully functional.

Fig. 1. Block diagram of our system.

Fig. 2. Simplified block diagram for the data acquisition system.

Circuit Diagram of the System

In Fig. 3, the transmitter circuit includes a push button, buzzer, Bluetooth master module, Arduino Nano, five flex sensors, an accelerometer, and a battery. The flex sensors and accelerometer are mounted on the arm; the accelerometer captures hand movements, while the flex sensors detect finger positions. Together, they generate data sent to Microcontroller-1, which transmits it via its TX pin to the Bluetooth module. The Bluetooth master module then sends this data to the receiver unit’s slave module. The transmitter is powered by a battery, though it was initially connected directly to a laptop. Pressing the push button activates the buzzer, signaling the start of data collection. A second beep indicates the end of data input, after which the data is transmitted to the receiver. In Fig. 4, in the receiver section consists of two microcontrollers: an Arduino Nano and an Arduino Uno. The Arduino Nano is connected to the Bluetooth slave module via its RX pin. When the transmitter’s Bluetooth master sends data, the receiver’s Bluetooth module captures it and sends it to Microcontroller-2, which then passes it to Microcontroller-3 via the I2C protocol. The data is then transmitted in digital format to the MATLAB App for further processing.

Fig. 3. Circuit diagram for transmitter unit.

Fig. 4. Circuit diagram for receiver unit.

System Algorithm

In Fig. 5, the transmitter processes 257 data points from sensors: 256 from the accelerometer and one from the flex sensor. For analysis, 25 data points are sampled for various accelerometer positions, while the flex sensor value remains nearly constant. After collecting the data, the Bluetooth master module transmits it to the receiver. Processing starts when data arrive from the transmitter unit to the receiver unit, as illustrated in Fig. 6. The data is then sent to the MATLAB app through the I2C protocol.

Fig. 5. Flow chart of data acquisition and transmitter unit.

Fig. 6. Flow chart of receiver unit.

Implementation, Results and Discussion

This section describes the development of the prototype, the testing of the system, and the analysis of the results.

Protype

In Fig. 7, illustrates the transmitter section of our system. The image on the right showcases the wearable device, while the image on the left highlights the transmitter section. Vocally impaired individuals use a wearable glove equipped with a 3-axis accelerometer to detect gesture movements. This sophisticated sensing device measures acceleration across three perpendicular axes (X, Y, Z), enabling precise detection of motion, orientation, and vibration in three-dimensional space. The core of the device is a MEMS (Micro-Electro-Mechanical System) sensor that converts physical movement into electrical signals, which are processed by a microcontroller. A flex sensor is also included to measure the amount of bending. The prototype typically includes a power management system and communication interfaces such as I2C or SPI. The accelerometer data is sampled across various positions, with the flex sensor value remaining relatively stable. Once the data is collected, the Bluetooth master module receives it from the transmitter section for further analysis. In Fig. 8, depicts the transmitter section of our system. The data is transmitted in digital format to the MATLAB app for subsequent processing.

Fig. 7. Transmitter section.

Fig. 8. Receiver section.

Training of Classification Model

Each sign language gesture in the dataset contains 25 occurrences. Since there are 20 different sign languages, the dataset comprises a total of 500 occurrences.

Each sign language gesture in the dataset contains 25 occurrences. Since there are 20 different sign languages, the data set comprises a total of 500 occurrences. Each data instance includes a flex sensor vector along with three vectors that provide acceleration data in the x, y, and z directions. Each vector contains 257 samples. The graphically depicts information about the sign language gestures for “Hi,” “How are you,” “Good,” “Watch Me,” “You,” “Come,” “Nice,” “Thirsty,” “Sit,” “Stand Up,” “No,” “Water,” “Call Me,” “Goodbye,” “Smile,” “Victory,” “Sorry,” “Sick,” “Change,” and “Love.” In Figs. 9–13 show four situations in that order.

Fig. 9. Case 1 HI (left) Raw sensor data and generated pulses after the thresholding operation (right) execution of hand finger commands in the hardware.

Fig. 10. Case 2 How are you (left) Generated pulses (right) Hand finger command.

Fig. 11. Case 3 good (left) Generated pulses (right) Hand finger command.

Fig. 12. Case 4 watch me (left) Generated pulses (right) Hand finger command.

Fig. 13. Case 5 You (left) Generated pulses (right) Hand finger command.

Data Acquisition Process in MATLAB

This section details the data acquisition process in MAT- LAB. We employed a dataset comprising 20 distinct sign languages, resulting in 20 unique data curves for each corresponding signal. For demonstration purposes, we include representative curves: Fig. 14, “Thirsty.” Each sign language gesture was sampled 25 times, leading to a total dataset of 20 × 25 = 500 samples are demonstrate.

Fig. 14. MATLAB date receiving for thirsty.

Feature Plot

In Fig. 15, presents a plot comparing two features: DCx, representing the average x-value, and DCy, representing the average y-value. This plot illustrates the relationship between the average x and y coordinates. The dataset consists of 20 sign languages, with the samples displayed as clusters. In most cases, the data points are tightly grouped, indicating that the output is accurate. However, a few outliers have deviated from their respective clusters, representing errors. Despite these outliers, the plot suggests that the overall accuracy of our model is robust.

Fig. 15. Feature plot.

Predicting Plot

Predictive modeling is a widely used statistical method for forecasting behavior. This data-mining technique, known as predictive modeling solutions, constructs a model by analyzing historical and current data to predict future outcomes. In Fig. 16, some data points are marked with cross symbols, indicating errors in our model. However, the majority of commands were executed correctly, demonstrating that most samples were accurately identified. Data points that appear distant from the clusters are typically considered inaccurate, as they represent instances where identification was unsuccessful.

Fig. 16. Prediction model.

ROC Curve

The ROC curves for two of the 20 sign languages analyzed. Fig. 17, illustrates the ROC curve for the “Victory” sign, which achieved 100% accuracy, as all decisions were correct. In contrast, Fig. 18, shows the ROC curve for the “Come” sign, where some decisions were incorrect, resulting in less than 100% accuracy. This is reflected in the slight bend in the curve.

Fig. 17. ROC curve for ‘Victory.’

Fig. 18. ROC curve for ‘Come.’

Confusion Matrix

In Fig. 19, presents the confusion matrix, which pro- vides insight into the accuracy of our model by showing the frequency of correct and incorrect classifications. For instance, out of 25 samples for the “Come” sign, the model correctly identified 24 instances but mistakenly classified one as “Smile.” Similarly, the matrix shows how the model performed for other signs like “Call Me,” “Change,” “Good,” and “Goodbye.” Notably, the “Victory” sign was accurately detected in all 25 instances, while the “Watch Me” sign had the most misclassifications, with the model incorrectly identifying it as “Good” twice, “Love” once, “Nice” once, “Sick” twice, “Smile” twice, “Thirsty” once, and “Water” once. The confusion matrix is crucial for evaluating the accuracy and reliability of a model.

Fig. 19. Confusion matrix.

Discussion

Our model addresses the challenges faced by the deaf- mute community, achieving a remarkable accuracy of 94.6%, which significantly surpasses the 75%–90% accuracy range of existing models. We employed five flex sensors per finger to capture hand bending and an accelerometer for hand motion detection, covering 20 sign languages with 25 samples each, totaling 500 samples. In Table I, the gesture recognition accuracy of the tested models is demonstrated, and Table II shows a comparison of previous research accuracies, indicating that our model has the highest accuracy among all.

Table I. Gesture Recognition Accuracy

SL	Model	Classification model	Accuracy (%)
1	SVM	Linear SVM	90.6
2	SVM	Quadratic SVM	90.2
3	SVM	Cubic SVM	88.6
4	SVM	Medium Gaussian SVM	88.0
5	KNN	Weighted KNN	87.0
6	Linear discriminant	Linear discriminant	94.0
7	Ensemble	Bagged trees	94.6
8	Ensemble	Subspace discriminant	91.8

Table II. Comparison with Previous Research

Sign language	Models	Interface	Type	Accuracy (%)
Australian [19]	IBL	Glove	Gestures	80.0
Arabic [21]	HMM	Image	Word	90.6
British [22]	FVs	Image	Word	87.67
Chinese [23]	HMM	Glove, Image	Word	86.3
French [24]	HMM	Glove, Image	Gestures	81.6
German [25]	HMM	Image	Word	81.0
Japanese [26]	KNN	Ring, Motion	Word	85.20
Indian [28]	CRF	Glove	Word	74.33
English [30]	–	Glove	Word	–
American [31]	SVM	Image	Word	89.54
Indian [32]	CNN	Image	Word	89.89
American [33]	CNN	Image	Word	–
[34]	R-CNN	Image	Voice	92.25
Indian [36]	CNN	Image	Word	90.1
[37]	HMM	Image	Voice	80.0
Lu et al. [38]	ELM	Glove, Motion	Numbers	91.2
American [39]	OpenCV	Glove, Motion	Alphabet	–
Malaysian [40]	HMM	Glove, tilt	Gestures	78.3
Turkish [41]	YOLOv5s	Hand gesture	Word	92.3
Turkish [42]	–	Hand gesture	Gestures	–
Turkish [43]	–	Hand gesture	Gestures	–
Propose	EBT	Glove, Motion	Sign	94.6

Conclusion

Our Research are developed a sign language recognition system that incorporates flex sensors on each finger and an accelerometer on the hand. These sensors capture hand gestures, translating them into sign language by measuring finger angles and hand slopes. It has some limitations, including sensitivity to temperature changes, which can affect the accuracy of the output. Furthermore, the Blue- tooth module used presents challenges such as slower data transfer rates, limited range, and interference, which affects the overall system performance. A data acquisition circuit for one hand, scalable to dual-hand for advanced functionality in the future.