Computing driver tiredness and fatigue in automobile via eye tracking and body movements

The aim of this paper is to classify the driver tiredness and fatigue in automobile via eye tracking and body movements using deep learning based Convolutional Neural Network (CNN) algorithm. Vehicle driver face localization serves as one of the most widely used real-world applications in fields like toll control, traffic accident scene analysis, and suspected vehicle tracking. The research proposed a CNN classifier for simultaneously localizing the region of human face and eye positioning. The classifier, rather than bounding rectangles, gives bounding quadrilaterals, which gives a more precise indication for vehicle driver face localization. The adjusted regions are preprocessed to remove noise and passed to the CNN classifier for real time processing. The preprocessing of the face features extracts connected components, filters them by size, and groups them into face expressions. The employed CNN is the well-known technology for human face recognition. One we aim to extract the facial landmarks from the frames, we will then leverage classification models and deep learning based convolutional neural networks that predict the state of the driver as 'Alert' or 'Drowsy' for each of the frames extracted. The CNN model could predict the output state labels (Alert/Drowsy) for each frame, but we wanted to take care of sequential image frames as that is extremely important while predicting the state of an individual. The process completes, if all regions have a sufficiently high score or a fixed number of retries are exhausted. The output consists of the detected human face type, the list of regions including the extracted mouth and eyes with recognition reliability through CNN with an accuracy of 98.57% with 100 epochs of training and testing.


Introduction
Drowsiness can be defined as a feeling of being lethargic and sleepy. A state of drowsiness when coupled with driving can be life threatening. Drowsiness can be caused due to several reasons like sleep deprivation, excess fatigue due to long road trips, excessive workload etc. Driving vehicles in such situations can lead to brief states of unconsciousness which can be very dangerous. Every year, a lot of people lose their lives due to road accidents. This includes both the drivers as well as the pedestrians as mentioned in [1]. Fatigue is an underestimated, but still a major problem in traffic of around 2.5 million traffic accidents in Germany in 2015 [2], were 2,898 accidents, with a total of 59 deaths (~ 1.7% of deaths), are due to fatigue attributed [3]. However, estimates are based on an unreported number of up to 20% from [4]. Various driving errors are often the official cause of accidents indicated, which, however, is causally due to an overtired driver are due. These accidents are also above average often serious accidents with personal injuries. Not only to look at car accidents caused by micro sleep, but also those caused by a prolonged reaction time or misjudgment of dangerous situations will. To warn the driver in good time of such situations and these to prevent this, the presented research was carried out.
Drowsy driving is a major societal problem which needs to be addressed. Feeling drowsy while driving is not just sleeping, but it could also be brief spans in which we do not have full concentration on driving and our eyes are closed or distracted. This small amount of distraction could prove fatal in many situations as given in [5].
Logistic Regression, Nave Bayes Classifier, Decision Trees, Random Forest, Gradient Boosting Machine, K-Nearest Neighbors, and Linear Support Vector Machines are examples of classification-based deep learning methods [6]. To combat driver drowsiness, researchers used Simple Feedforward Neural Networks, Recurrent Neural Networks, Long Short-Term Memory, Gated Recurrent Unit, and Convolutional Neural Networks. Our models were assessed using two evaluation metrics: recall and accuracy. The study establishes the issue statement as well as the 'Real-Life Drowsiness Dataset,' which we used to train and test our deep learning models. The introduces the approach and algorithms for detecting tiredness in EEG records, as described in [7]. for creating a sleepiness detection system that employs deep learning models to identify drowsiness in drivers Based on their facial landmarks derived from the films as indicated in [8], our drowsiness detection problem tries to anticipate all those time frames in the video where the driver is drowsy for a particular input video of a driver.

Aim of study
the energy efficiency in computing driver tiredness that detects the drowsiness of a driver based on facial cues and behavioral patterns with fatigue in automobile. The measures adopted by our research work can be summarized: i.
Face detection using pre-trained deep learning model, in our case (CNN). ii.
Landmark detection, identifying eyes, nose, mouth, and jaw line. iii.
Calculation of aspect ratios. Once the features of a face are extracted, we need to calculate the aspect ratios of eyes (sleepy) and mouth (yawn) using the CNN algorithm. iv.
Comparison of live aspect ratios with the threshold for the detection of drowsiness. v.
Alarming the driver. Once the drowsiness is detected we need to alert the driver using a sound/ alarm. vi.
Computing the driver tiredness with deep learning based distributors and accidents must reduce CNN annually 10.5% their performance with high detection accuracy. vii.
Evaluating the efficiency and renovation of least tired actions of driver or calculate it by following the expressions of driver on DDD dataset. viii.
Evaluating the incentives of drowsiness detection system with renovation, i.e. adding deep learning based insulation to driver to improve their driving performance.

Problem statement
The analysis of driving behavior is the most common variant used by automobile manufacturers. The ATTENTION ASSIST from Mercedes Benz as mentioned in [9] evaluates them steering movements (steering torque, angle and speed), the vehicle longitudinal and lateral acceleration as well as the driver's operating handles and combines their values with additional environmental parameters such as the time of day or the duration of the journey as given in [10]. For each driver there is an individual one during a 20-minute initialization phase driver profile created. On this basis, the system gives an attention value on a five-point scale from low to high. Volkswagen and Bosch use a similar approach as presented in [11]. Since these are internal developments more detailed information is not available. The researchers in [12] carried out a review for older studies but could not produce clear results present. They basically questioned whether the results of the studies are generally transferable. The researchers in [13] ran their own tests with 34 test persons through and combined driving behavior (lane keeping, speed) with body signals. The combination gave better results, than driving behavior alone. Another approach without being built into the vehicle sensors tracked by [14]. They use gyroscope and accelerometer a smartwatch and deduced the steering was movement. The driving fatigue in [15] was recognized to 98%.

Background
Engineers missing wayside signals, having diminished situation awareness, or falling asleep while operating the train are all examples of fatigue-related performance degradation in the rail sector. As indicated in [16], every train engineer has experienced weariness, putting themselves, passengers, and places next to the rail line at danger of accidents. The most common system used to keep locomotive crew engaged is the alerted system, which utilizes a reset timer to verify engineers are still awake and responsive. If an engineer does not respond to the alert in time, the system will apply brakes to the locomotive as mentioned in [20]. Although the current locomotive alerted system has contributed to keeping the engineer awake, nothing can determine whether the engineer is truly mentally engaged, as it is deactivated upon any interaction with the control system as mentioned in [17]. Engineers may still be awake enough to press a button yet fatiguedlacking situation awareness of their surroundings or current operations. A deadly accident in Macdona, Texas in 2020 [18] involved an engineer demonstrating automatic behavior, defined as a state where one is mentally fatigued but physically awake enough to continue providing input to the train control system. Because motor reflex responses typically require a lower level of cognitive effort [19], the operator could operate the train despite his impairment, which subsequently reset the alert it never triggered to rouse the engineer to a more alert state. However, the report also states that the engineer's control interactions were inappropriate given the task context, as the engineer increased the speed of the train when he should have been decreasing it. If MEFA had been in use before this accident, the engineer would have received continued alerts of inappropriate cab interaction activity upon engaging the throttle in the wrong direction as mentioned in [20]. While the use of Positive Train Control (PTC) would have also prevented this accident, there are still multiple dark areas or rail line parts (e.g., grade crossings) that do not employ PTC, and nationwide implementation may be delayed by several years. Local obstructions or rail infrastructure failures, which cannot be captured by autonomous systems such as PTC, require immediate human intervention as mentioned in [21]. The fatigue state was assessed on a subjective scale and overall classification sensitivity was almost 89 percent. A classifier trained with both cues had better performance for most subjects while the single-cue trained classifiers were slightly less accurate but similar to each other in performance. The characterization in different categories is not absolute so they do overlap in some cases as mentioned in [22]. Researcher in [23] similarly found that pupil size variability, measured by standard deviation, Shannon entropy, and wavelength entropy, was higher just before sleep onset. They were able to correctly identify 83 percent of the pre-sleep data segments (30 sec. duration) using linear discriminant analysis of all three variables. The Micro-Nod Detection System from [25] attempted to classify patterns of head movements through three non-contact capacitive sensors mounted above the head in the car ceiling headliner with PERCLOS used as ground truth for fatigue state. A 15-to 20-minute training period was needed to establish "normative" head behavior for an individual driver. If the observed behavior exceeded normal (threshold not defined in report), the system would trigger a visual alert. No details about how the measured capacitance was processed to label processed head movements. The author in [26] claimed that drowsiness detection in advance of a driver error (e.g., road departure) using this system was higher than the performance of a generic PERCLOS system. There is no evidence that Micro-Nod system was ever developed into a commercial product or tested in a rail or automobile system. Figure 2. Recurrence of alert sign is higher than recurrence of drowsy sign [26].
Researcher in [27] developed a set of fuzzy-classifier of wavelet-based features using data from EEG, EOG, and ECG, and compared classification performance with subjective raters evaluating video recordings of the subjects' faces using criteria based on EEG, EOG, and ECG signals were collected using skin electrodes during two 25-minute driving sessions performed between 9 a.m. and 1 p.m. [28]. The first session was in higherdensity traffic environment, the second in a low-density monotonous environment. Instead of using group statistics of frequency or power derived from FFTs of the signals, they instead used the coefficients from the WPT of the signals to construct feature sets for the classifier as given in [29]. They achieved recognition accuracy of 95 to 97 percent with a combination of all three data types. Figure 3. Driver's drowsiness detection system general workflow representation [29] This is especially true for user-selected news subjects. The rationale-augmented convolutional neural network (CNN) [30] was used to construct this work. The YOLO (You Look Only Once) computational technique was used in this study to recognize vehicles from infrared photos [31]. Data clustering is an essential machinelearning issue in this study. It can be used for a variety of purposes, one of which being image segmentation. [32]. SVM (support vector machine), KNN (K-Nearest Neighbors), Decision tree, Logistic Regression, and ANN (Artificial Neural Network) back propagation are just a few of the many categorization models available.
We will look at several procedures and methods for early diagnosis of glaucoma disease using the MATLAB Deep Convolutional Neural Network (DCNN) [33] in this research. The enhancement of medical diagnostic procedures is one of the issues that has occupied many publications, with many studies beginning to elicit algorithms to raise the efficiency of illness diagnosis [34]. We report an accuracy of 88.4 percent in this study for a 5-class grouping assignment. At the high-affectability working point, we report 92.3 percent exactness, 96.2 percent, and affectability 94.5 by 87.2 percent for the 4-class grouping attempting to distinguish carcinomas. In the opinion of everyone [35]. Attention Assist primarily measures small changes in driver steering behavior to gauge fatigue. It calibrates a baseline pattern from the initial driving behavior on a trip and compares current behavior in real time as mentioned in [36]. Driving conditions, other activity such as interacting with the controls, and car state are also considered before an alert is issued. The automotive market is currently the strongest driver for the development of fatigue detection technologies. As noted above, two general approaches have been taken for commercial products. Automotive manufacturers are developing systems that primarily use car-based information, including driver control behavior, to infer the driver's state as given in [37].

Methodology
The goal of this research assignment is to implement a driver tiredness detection learning and CNN based prediction model in the transportation sector that allows cars to intelligently "learn" and "predict" driver tiredness behavior. There are many motivations to pursue research in this area, some due to technological advancements in the autonomous vehicle industry that encourage research in V2I/V2V communications to receive information about driver drowsiness real-time while driving (with or without a human driver). Other motivations include keeping a driver informed on the road to prevent accidents due to "missing a driver tiredness detection", reducing a driver's transportation costs and commute time by increasing fuel economy, energy efficiency, and vehicle part lifetime through idling prevention, and contributing to less damage to the environment by cutting down vehicle emissions.
The model developed in this research assignment can be divided into two main algorithms; one accomplishes "learning" driver drowsiness using a computer vision system, and the other uses the learned data to develop predictions on traffic signal lengths and provides other important information on the driver expression back to another driver. The vision system developed in the first algorithm is a driver tiredness detection object recognition system that allows vehicles to detect different driver expression configurations and the right arrow on live stream image capture. These state detections, along with the timestamp of detection and the signal I.D. of the driver tiredness detection, are uploaded as learning data into a Driver Drowsiness Detection (DDD) dataset to be used later to form signal length predictions. The algorithm can perform three separate functions that are integrated to create a driver tiredness detection prediction system: • Notifying the driver of the upcoming tiredness detection on their straight route.
• Determining the last known state of that particular driver tiredness (if it exists).
• Offering a prediction on when the state of the driver tiredness detection will change if prior learning data exists at the time period of crossing the driver expression (Eyes and Mouth).
All of these functions are performed simultaneously, displaying updates to the driver on a GUI when in proximity to the driver tiredness detection. All of the records are stored in a knowledgebase (K.B.) that is contacted in real-time to predict the driver when passing through the driver expression at a known signal length prediction time period. The learning system has proven extremely accurate in identifying all signal configurations on live stream image capture without mistaking other similar objects on the road to be driver drowsiness.
A dataset is composed of images for driver drowsiness detection is obtained from an open-source repository known as Driver Drowsiness Detection (DDD) Dataset, where, usually, each row represents an observation and each column a different variable. Object recognition has turned into an emerging technology in autonomous vehicles, where collision avoidance and advanced driver assistance systems are becoming crucial to the success of vehicles operating without user input. The human eye can perceive and process multiple objects in a frame at once, allowing for drivers to make informative decisions in a jiffy on the road. Object recognition models are striving to achieve performance that can mimic the human eye as closely as possible (it is impossible to get the same level of detection). To attain this, models must be trained on a large set of similar images using a neural network framework, until losses diminish and stay constant at a small value. Training is performed one iteration at a time, where an iteration consists of a forward and backward propagation through the layers of nodes in the network (input, hidden, and output). An example neural network, with calculations to find the output node values (forward propagation) and update the weights for a new iteration (backward propagation) based on the error between the actual values and the output values found [38].   Driver drowsiness in very busy downtown or inner-city areas have the ability to offer strong This last known state could potentially be uploaded hours or days ago, but since the assumption in this project is that driver drowsiness follows a fixed schedule every day of the week, the drivers can still use the information to predict when the state will change. The system is trained on 70% of the dataset, the testing is done on 20% of dataset however the validation of done with the remaining 10% of dataset with 10-cross fold validation in 100 epochs of training. Figure 8. The graph represents the loss and accuracy for training-validation process on DDD dataset

Results
paper is to classify the driver tiredness and fatigue in automobile via eye tracking and body movements using deep learning based Convolutional Neural Network (CNN) algorithm. Vehicle driver face localization serves as one of the most widely used real-world applications in fields like toll control, traffic accident scene analysis, and suspected vehicle tracking. The research proposed a CNN classifier for simultaneously localizing the region of human face and eye positioning. The classifier, rather than bounding rectangles, gives bounding quadrilaterals, which gives a more precise indication for vehicle driver face localization. Because learning tree design is more advanced than traditional optimization, where gradient is simply taken, we can utilize an additive method to reduce quality by adding new trees one at a time rather than all at once. The CNN model applied features on the single depth image refinement methods have a chance to acquire satisfactory results only when the entropy is simple with several big color patches. For instance, many approaches use anisotropic neural network to impose the piecewise smoothness on the entropy, which is not commonly correct for the real-world objects. Range estimation can be done using time of flight in several ways. The CNN works well for the feature extraction estimation of both features-like method and feature extraction model is highly dependent on the regularization terms which emphasize the piecewise smoothness. This is a typical method for estimating entropy in practically all state-of-the-art depth refinement methods. When the entropy is simple, with large regions of patterns and only a few prominent colors, they perform well. However, there are many real-world items with sophisticated arrangement colors and patterns for which all of these entropy estimation approaches will fail. If the entropy estimation fails, the final depth refinement result has little chance of being right. The output consists of the detected driver face type, the list of regions including the extracted face and the reliability through CNN with an accuracy of 98.57% with 100 epochs of training and testing. The proposed system has been trained on 70% of data while 20% is used for testing and remaining 10% for validation.

Discussion
We developed two new signal processing algorithms to determine sleepiness degree based on bio signals in this advanced research. We gather eye blink data when a patient is awake as the normal pattern in the eye blink signal analysis approach. The difference in eye blink signal voltage and times can be used to diagnose drowsiness using drowsy eye blink signal patterns. The accuracy rate is greater than 98.00%. We extract three aspects of brain wave signals using the brain wave analysis method. When people are drowsy or not drowsy, these characteristics differ. CNN uses a pattern recognition algorithm using three features. The pattern identification accuracy is 98.57 percent, which is higher than the single feature drowsiness/tiredness detection accuracy. The PNN classifier was also used to investigate the effects of alcohol and some anti-drowsiness medicines. In one hour, alcohol causes sleepiness, and anti-drowsiness medications can relieve drowsiness quickly. However, the effects of anti-drowsiness products are just temporary. When I do all the collection myself, it's difficult to be entirely drowsy, therefore "awake state" and "drowsy state" are the only options. In the future, further levels of drowsiness will be required, such as "Awake," "Medium Drowsy," and "Very Drowsy." In CNN, higher levels result in more categories. The CNN classifier is able to split patterns in a more thorough and accurate manner. The visual system is one of the most investigated and well understood sections of the brain during driving, according to CNN. The prevalence of visual input among other sensory systems in humans could justify a natural desire for scientists to examine the processes of vision. Another reason why studying the visual system is appealing is that vision is one of the most straightforward senses to modify in a controlled experiment. It is feasible to functionally describe the distinct sections of the visual system by linking the brain activity of specific regions with the exposition of a subject to diverse instances. Table 1. The comparison between existing technique for classifying the tiredness/fatigue in terms of accuracy Article Technique Accuracy [39] Support Vector Machine (SVM) 95.20% [40] Long-Term Short Memory (LSTM) 92.71% [41] Fuzzy-Recurrent Neural Network (F-RNN) 87.30% Proposed Convolutional Neural Network (CNN) 98.57% The CNN network with two retrained layers was able to achieve lower training losses than the model with only one retrained layer. When comparing the left red plot to the right blue plot, it is clear that networks with lower training loss could achieve superior results in that particular visibility region. After the enormous success of deep convolutional networks with traditional comparison recognition, the scientists turned to machine learning. It appeared that this component of vision could be well imitated by feedforward network topologies.

Conclusion
In this paper, a method to detect, classify, and extract expression from a known set of vehicle driver face using deep leaning technique was presented. The intended use case involved an operator that captures human face attached to driver real time driving, such as cropping the human face or generating the region of interest (ROI), using an image processing technique where the extracted content is processing with deep learning-based CNN technique. The output consists of the detected driver face type, the list of regions including the extracted face and the reliability through CNN with an accuracy of 98.57% with 100 epochs of training and testing. The proposed system has been trained on 70% of data while 20% is used for testing and remaining 10% for validation. To show the significance of the designed feature vector, we used a DDD dataset consisting of multiple classes which represented a subset of all existing types of faces. There are both old and modern varieties in it. The results revealed that the local binary pattern histograms have the greatest influence on the classification outcome, followed by color characteristics, and finally the driver face size ratio. The resilience of the classification under diverse lighting situations, such as the presence of shadows and specular reflections, as well as blurriness and wrong colors induced by insufficient illumination, was demonstrated by evaluating the composite feature vector.