Detection of yawning in driver behavior based on a convolutional neural network

Among the factors that usually cause road accidents in the world is driver fatigue, which accumulates during the trip or is present even before it begins. One of the most common signs of fatigue or tiredness of a vehicle driver is yawning. The detection of signs of yawning in human behavior is potentially able to further characterize its state of fatigue. Computer image processing methods are actively used to detect the openness of the mouth and yawning for a person. However, this approach has many disadvantages, which include different environmental conditions and a variety of situational yawning options for different people. The paper presents a scheme of a detector for determining signs of yawning, which is focused on processing images of the driver's face using data analysis methods, computer image processing, and a convolutional neural network model. The essence of the proposed method is to detect yawning in the driver's behavior in the cabin of a vehicle based on the analysis of a sequence of images obtained from a video camera. It is shown that the driver's yawning state is accompanied by a wide and prolonged openness of the mouth. Prolonged openness of the mouth signals the appearance of signs of yawning. A conceptual model for detecting the openness of the mouth for a vehicle driver is presented and a scheme for processing and labeling the YawDD and Kaggle Drowsiness Dataset datasets is developed. The developed convolutional neural network model showed an accuracy of 0.992 and recall of 0.871 on a test 10 % data set. The proposed scheme for detecting the yawning state has been validated on a test video subset extracted from the YawDD: Yawning Detection Dataset. This detection scheme successfully detected 124 yawns among all video files from the test dataset. The proportion of correctly classified objects is 98.2 % accuracy, precision is equal to 96.1 %, recall is 98.4 %, and F score is 97.3 % while detecting signs of yawning in driver behavior. Detecting signs of yawning in the driver's behavior allows one to clarify information about the driver and thereby to increase the effectiveness of existing driver monitoring systems in the vehicle cabin, aimed at preventing and reducing the risk of road accidents. The proposed approach can be combined with other technologies for monitoring driver behavior when building an intelligent driver support system.