Real-Time Face Mask Detection to Ensure COVID-19 Precautionary Measures in the Developing Countries

Abstract

Recently, the rapid transmission of Coronavirus 2019 (COVID-19) is causing a significant health crisis worldwide. The World Health Organization (WHO) issued several guidelines for protection against the spreading of COVID-19. According to the WHO, the most effective preventive measure against COVID-19 is wearing a mask in public and crowded areas. It is quite difficult to manually monitor and determine people with masks and no masks. In this paper, different deep learning architectures were used for better results evaluations. After extensive experimentation, we selected a custom model having the best performance to identify whether people wear a face mask or not and allowing an easy deployment on a small device such as a Jetson Nano. The experimental evaluation is performed on the custom dataset that is developed from the website (See data collection section) after applying different masks on those images. The proposed model in comparison with other methods produced higher accuracy (99% for training accuracy and 99% for validation accuracy). Moreover, the proposed method can be deployed on resource-constrained devices.

1. Introduction

Recently, in December of 2019, a pandemic situation was raised in China named COVID-19. It is a disease caused by the SARS-CoV-2 virus, and the World Health Organization (WHO) declared it a global pandemic on 11 March 2020. Throughout history, humanity has observed several pandemic situations such as the African obesity pandemic [1], the pandemic flu [2], the HIV/AIDS pandemic [3], etc. Humans are currently facing a crucial time to fight with an invisible enemy, namely, coronavirus (COVID-19). Due to this deadly disease, millions of people are infected by COVID-19 worldwide, and many have died. At the start of January 2020, several patients reported having pneumonia of unidentified etiology in Wuhan city [4], the cause of which was linked with a wet seafood wholesale market in the region [5]. The rapid spread of the COVID-19 epidemic poses major challenges to virus control [6].

According to Google News, from December of 2019 to date (accessed on 2 September 2021), the COVID-19 pandemic grew and evolved, which infected more than 216,026,420 people, and 4,495,014 have died [7]. Researchers are scrambling to learn about the virus and recommend effective responses. According to Worldometer, the total confirmed coronavirus cases globally are 216,918,733, there are 4,511,302 deaths, and 193,849,589 are recovered [8].

Table 1. Illustrated country-wise information about COVID-19 such as confirmed cases, recovered cases, and total number of deaths (accessed on 2 September 2021).

Worldometer [8]				Google News COVID-19 [7]
Countries	Confirmed	Recovered	Deaths	Confirmed	Deaths
Worldwide	216,918,733	193,849,589	4,511,302	216,026,420	4,495,014
United States	39,617,417	30,812,242	654,381	38,830,051	637,066
Italy	4,524,292	4,255,808	129,056	4,524,292	129,056
Germany	3,933,569	3,726,700	92,631	3,933,585	92,136
India	32,695,030	31,888,642	437,860	32,695,030	437,830
Saudi Arabia	543,796	531,733	8526	543,318	8512
UAE	716,381	702,102	2038	715,394	2036
China	94,819	88,924	4636	94,765	4636
Pakistan	1,152,481	1,033,373	25,604	1,152,481	25,604

shows COVID-19-related information such as confirmed cases, recovered cases, and deaths, which were collected by Worldometer [8] and Google News COVID-19 [7].

COVID-19 symptoms can lead to a disease that can range from benign to extremely dangerous. COVID-19 symptoms appear between 2 and 15 days after one is infected, named as the incubation period. Anyone having symptoms of COVID-19 is kept in quarantine (isolation wards) in active medical custody. The major symptom of the disease includes shortness of breath, sore throat, cough, rapid heartbeat, chest pain, and fever. Commonly, the virus can be spread from one person to another through respiratory droplets produced during coughing and can also be spread from touching, i.e., unclean surfaces and then touching someone’s face (Centers for Disease Control and Prevention, 2020).

Given the transmission of COVID-19, it was advised by the WHO to several countries to confirm that their nationals are wearing masks in any public places [9,10]. Before COVID-19, there are only a few people who used to wear masks for health protection from air pollution, and health specialists also used them while practicing at clinics. The WHO has declared it as a global pandemic due to a large number of positive cases reported, where most of the cases are found in crowded areas. Therefore, it was recommended by the experts to wear a mask in crowed as well as any public places for the prevention of disease transmission [11]. The French government started an initiative to detect passengers without wearing a mask in the metro station. For this purpose, Artificial intelligence (AI) algorithms were deployed on the CCTV cameras in Paris metro stations [12]. AI algorithms such as deep learning (DL) and machine learning (ML) can be used in several ways to prevent COVID-19 transmission [13].

Face-mask detection using ML and DL achieved promising results; however, the existing methods need to be more robust in a real-time environment. Furthermore, the lack of suitable datasets and DL-based models for face mask detection [14] that can detect a person without a face-mask in real-time to avoid the spread of COVID-19 is still ongoing. For this purpose, we created a large-scale dataset that consists of two classes, i.e., face mask and not mask, and also developed a lightweight CNN model for efficient face-mask detection to provide a safe and secure environment for every individual by minimizing virus spread. The proposed work will currently be implemented in targeted areas such as schools, colleges, universities, mosques, and superstores. It will be implemented at the main entry point, where the system will check each individual’s face. If the system detects any person without a mask, they would not be allowed to enter. The contributions of the proposed work are as follows:

• A new face dataset is developed containing images of different people generated by a generative adversarial network, and then different face masks are designed that are applied on these face images to create a custom dataset consisting of two classes (mask and non-mask).
• A lightweight DL-based method consisting of four convolutional, one fully connected, and one output layer has been developed to accurately detect face masks.
• To validate the performance of the proposed lightweight DL model over the custom dataset, we trained different pre-trained DL models such as AlexNet, VGG16, VGG19, ResNet101, NesNetMobile, and MobileNet. The proposed model outperforms the existing DL-based models in terms of a better balance among time complexity, model size, and accuracy.

The remaining work is as follows. The literature review is presented in Section 2. Section 3 explains the proposed technique and the information about the dataset, system configuration, and performance parameters, while the experimental results are discussed in Section 4. Finally, the paper is concluded in Section 5 with key findings and various future directions.

2. Literature Review

Nowadays, Convolutional Neural Networks (CNN) are used in many ways to facilitate humans and also prevent their lives from damages such as fire disasters [15,16], facial feature analysis [17,18], healthcare [19,20,21], and many others fields [22,23,24]; in addition, in several studies, a resource-constrained device has been used to interact in real-time for human lives facilitation [25,26]. In this work, our main focus is on face mask detection for COVID-19 prevention using a CNN. Generally, in most related publications, the focus is on face recognition and construction when wearing face masks. By contrast, this research focuses on the detection of those individuals who are not wearing face masks in public places to help in reducing the transmission of COVID-19. Scientists and researchers have proved that wearing face masks in public places significantly reduces the spreading rate of COVID-19. Bosheng Qin et al. [27] proposed a new method to detect the face mask wearing condition. Their trained model can classify the face mask condition into incorrect, correct, and no face mask wearing conditions. Their trained model has achieved 98.70% accuracy. Chong Li et al. [28] proposed a YOLOv3-based framework for face detection. They have verified their method by training on the WIDER FACE [29], CelebA [30], and FDDB [31] datasets. Their model has achieved 93.9% accuracy. In [32], Din et al. proposed a GAN-based novel framework that can detect and segment the face mask from the face image and regenerate the face image using a GAN. Their trained model-generated images look like the actual images. Nieto-Rodríguez et al. [33] proposed a framework to detect the special face mask in the medical room. They have used real-time image processing for detecting face masks. The objective of their work is to minimize the false-positive rate. They have achieved a 95% recall and a 5% false-positive rate for the detection of the surgical mask. Muhammad et al. [34] proposed a framework called MRGAN to segment the microphone from the face image and used the GAN to reconstruct the segmented holes into the face image. They have trained their model on their synthetic dataset. The trained model works better than the state-of-the-art methods.

Mingijie Jiang et al. [35] proposed a face mask detector called RetinaFaceMask. Their method used a novel object-removal algorithm to remove predictions with low confidence. They have achieved 1.5% and 2.3% higher precision and 5.9% and 11.0% higher recall than state-of-the-art results on face mask and face detection, respectively. On the other hand, they also explored the performance of the proposed method on light-weight neural networks such as MobileNet. Shashi Yadav et al. [36] proposed deep learning mixed with a geometric-based technique to monitor social distancing and face masks in public areas. They have implemented their model on Raspberry pi4. Their model detects violations of social distancing and face mask wearing by receiving input from cameras. When a violation occurs in a specific area, the developed system sends alerts to the control room and notifies the public by an alarm. The paper presented in [37] used a transfer learning strategy for the automatic identification of persons who are not wearing a face mask. In this approach, the researchers used pre-trained InceptionV3 with fine-tuning. They used Simulated Masked Face Dataset (SMFD) for training and testing and also used image augmentation methods to increase the number of training samples for better results evaluation.

In another work presented in [38], the authors used a hybrid model such as deep learning and conventional machine learning to identify a person’s face mask wearing condition. Their model consists of two main components. The first one is designed to extract prominent features from facial images using Resnet50, and the second component is a different classifier used for classification such as an ensemble algorithm, Support Vector Machine (SVM), and decision trees. For experimental evaluation, three different datasets were used, namely, Labeled Faces in the Wild (LFW) [39], SMFD [38], and World Masked Face Dataset (RMFD) [38]. The SVM classifier reached an accuracy of 99.64% during training. It appears from the literature review that several researchers used different mechanisms for face mask detection and also achieved superior results; however, the limited amount of training data and computational complexity restrict their systems from a real-time implementation. Therefore, firstly, we create a novel large-scale dataset that consists of two classes, i.e., face mask and no mask. Secondly, we developed a lightweight CNN model that can be easily deployed over resource-constrained (edge devices) to efficiently and accurately detect face masks in a real-time scenario.

3. Methodology

Face-mask detection using ML-based methods is tedious and time-consuming work due to ML requiring hand-crafted features engineering, which requires domain experts. Particularly, in ML-based methods, early face-mask detection and response generation are also challenging due to low detection accuracy. Considering these challenges, DL-based models are efficiently utilized for face mask detection in surveillance systems. DL provides end to end features extraction mechanism but it requires a huge amount of training data and high computational power. The main motive behind this work is to enhance the performance of the DL-based model and its deployability over edge devices, for which we proposed a new framework as shown in Figure 1. The proposed framework is divided into two major steps such as (1) face cropping and (2) face mask detection.

Figure 1. The prevention and precaution steps of COVID-19 [40].

Figure 1. The prevention and precaution steps of COVID-19 [40].

3.1. Face Cropping Using Face-Detection Algorithms

The first step in the proposed work is face detection (FD), where accurate FD is very significant for face mask detection. FD is a computer vision algorithm to detect frontal faces from images. For this purpose, Zhang et al. [41] proposed a Multi-task Cascaded Convolutional Network (MTCNN) to receive an input image and give seven outputs points to each detected face. The first two points are related to the bounding box such as the bottom right point and the top-left point. The others five points correspond to the right mouth corner, left mouth corner, nose tip, right eye pupil, and left eye pupil. The authors claim that they attain a mean error of 6.9% on the AFLW benchmark. The main benefit of MTCNN includes accuracy regarding different poses, occlusion, and illumination. Figure 2, show the output performance of MTCNN during different illuminations and positions [42].

3.2. Lightweight CNN Model for Face Mask Detection

After the face detection, an efficient CNN-based architecture is used for the training and testing of face mask detection. There are several already available architectures for the training purpose, which were already discussed in the literature review section. In this work, we presented a custom architecture to detect whether the person wears a face mask or not. In the proposed work, face mask detection is performed by facial feature analysis. The proposed lightweight CNN model consists of four convolutions, one fully connected and the output layer, and after each convolution layer, a nonlinear ReLU activation function is used to perform thresholding operation. The ReLU activation function drops neurons from the network whose values are less than zero, and the neuron with positive values are unchanged [42]. In the proposed model, we employ two maximum pooling layers for dimensionality reduction, which affects the training duration of the network. During training the model, the problem of overfitting can be occurred due to the model and data simplicity. To reduce the overfitting problem, we used the dropout and batch normalization mechanism. The output neurons in the last fully connected layer are equal to the number of classes recognized by the network, and lastly, the softmax classifier is used to classify the given input into the corresponding class such as face mask detected and no face mask detected.

The input image size of the first convolution layer in the proposed architecture is 128 × 128, having RGB channels with 32 different filters (kernels). The size of these filters is 3 × 3 × 3 with a 1-pixel stride. The output of the first convolution layer is the input of the second convolutional layer after normalization and pooling layers. There is a total of 64 kernels with a size of 3 × 3 in the second convolutional layer. The rest of the convolutional layers are connected without any intervening subsampling or batch normalization. In the third convolutional layer, 128 kernels are used, where each kernel has a size of 3 × 3 × 3. The final convolutional layer has 256 kernels having of size 3 × 3 × 3. The fully connected layers have a total of 128 neurons. The output of the fully connected layer is fed into a softmax layer, which produces a distribution over the 2-class labels, namely, mask detected and mask not detected. The complete detail of the proposed system is shown in Figure 3 along with subsequent layers descriptions.

4. Results and Discussions

The dataset used for training and testing purposes and the achieved results are discussed. Different experiments were performed for evaluating the proposed method using Python 3.6.4, OpenCV 3+ on a Core i3 PC to acquire the stream from the visual sensor.

4.1. Dataset

Data collection is the most important part of any research. For this purpose, we have collected a dataset from an internet source, from the website (https://thispersondoesnotexist.com/ (accessed 30 December 2020)). After the collection of these images, we have designed different face masks and applied them on each face to generate another class for face mask detection. This dataset consists of two classes: mask and non-mask. The non-mask class consists of 3076 images, whereas the mask class consists of 4664 images, and all the images have 3 channels (Red, Green, and Blue) with sizes of 224 × 224 pixels. The dataset is divided into three subcategories: training, validation, and testing. In the proposed work, 70% of the data is used for training, whereas 20% is for validation, and the remaining 10% is for testing purposes. In Figure 4, we demonstrate the sample images of the proposed dataset.

4.2. Evaluation Metrics

In this research, for performance evaluation of our proposed framework, we used the following performance metrics: Accuracy, Recall, Precision, and F1-measure, which are briefly explained in [43], and presented from Equations (1) to (4):

$$\mathrm{Accuracy}=\frac{(\mathrm{TP}+\mathrm{TN})}{(\mathrm{TN}+\mathrm{FN})+(\mathrm{FP}+\mathrm{TP})}$$

(1)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{(\mathrm{FN}+\mathrm{TP})}$$

(2)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{(\mathrm{FP}+\mathrm{TP})}$$

(3)

$$\mathrm{F}1-\mathrm{measure}=2\times \frac{(\mathrm{Precision}\times \mathrm{Recall})}{(\mathrm{Precision}+\mathrm{Recall})}$$

(4)

where TP, TN, FP, and FN are the True Positive, True Negative, False Positive, and False Negative samples, respectively, from a confusion matrix.

4.3. Results of the Proposed Model

In Figure 5, the line graph provides information about the training and validation of the proposed model on 20 epochs. Overall, it is clear that although the proposed model reached higher training and validation accuracy, similarly, the training and validation loss reached a minimum level, which can be seen in Figure 5. Here in this graph, it can be seen that the accuracy of training on the first epoch started from 74%, while the validation accuracy started from 87%. After each epoch, the accuracy of the training and validation gradually increases, which indicates that the learning rate is suitable. Finally, after 20 epochs, the level of training accuracy and validation accuracy reached 99%, as shown in Figure 5. Similarly, the training and validation loss can also be seen in the same Figure 5, which reached almost 0.02% on the last epoch.

4.4. Performance of the Proposed Model

The confusion matrix is the base of the performance matrices, where TP, TN, FP, and FN are extracted to calculate the Accuracy, Recall, Precision, and F1-Measure. In the proposed work, a detail of the testing set is provided in

Table 2. Result of the confusion matrix for both validation and testing sets.

Confusion Matrix of Test Set
Mask	No mask	Per class accuracy
251	5	98.04%
2	181	98.90%

. We discussed the result of the testing set, wherein the mask detection class presents 251 images that are accurately classified as persons wearing face masks and 5 are incorrectly classified. In the other class, 2 images are classified incorrectly while 181 are correctly classified. In the testing set, the accuracy reached 98.04% and 99.90% for the mask class and the non-mask class, respectively.

Table 3. Classification Report of Test Set.

	Precision	Recall	F1-Measure
Mask Detected	0.99	0.98	0.99
Mask Not Detected	0.97	0.99	0.98
Accuracy			98.47
Macro Avg	0.98	0.98	0.98
Weighted Avg	0.98	0.98	0.98

contains information about the performance metrics (precision, recall, and F1-measure). Here, it can be seen that the precision, recall, and F1-measure are 0.99, 0.98, and 0.99, respectively for mask detection, while the precision, recall, and F1-measure for the non-mask class are 0.97, 0.99, and 0.98 respectively. In Table 3, the average accuracy, Macro Avg, and Weighted Avg is 98%.

4.5. Comparisons

The proposed work is compared with state-of-the-art, pre-trained benchmark models to evaluate the performance based on accuracy, model size, number of parameters, and training time as given in

Table 4. Comparison with different deep learning architectures on custom drowsiness detection dataset.

Technique	Dataset	Model Size	Parameters (Million)	Accuracy (%)	Training Time (M:S)	FPS (CPU)
Proposed	Custom dataset	16 MB	2.2	98.47	30:38	28.07
AlexNet	-	233 MB	60	98.07	34:04	5.33
VGG16	-	528 MB	138	98.75	35:77	2.98
VGG19	-	574 MB	143	99.22	37:35	1.83
ResNet101	-	98 MB	20	99.00	55:13	7.43
MobileNet	-	13 MB	4.2	95.13	34:27	20.23
NesNetMobile		23 MB	5.3	97.05	44:13	14.22

. The proposed model reached an accuracy of 98.47%, and the accuracy of AlexNet, VGG16, ResNet50, MobileNet, and NesNetMobile are 98.07%, 98.75%, 99.22%, 99.00%, 95.13%, and 97.05% respectively. Table 4 represents the model size, learning parameters, and training time. Considering the model size, MobileNet is the smallest model, where the size of VGG19 is the largest. In this experiment, our proposed model takes the second position (16MB) in the smallest models, but the proposed model takes less time during training, and its inference time is also given in Table 4. The inference time represents the frame per second (FPS). The proposed system is efficient due to fewer training parameters as compared to all mentioned models.

4.6. Visualized Results

In this section, the visual results of face mask detection in real-time, including face detection and recognizing whether the person wears a face mask or not are demonstrated, as shown in Figure 6. It can be seen that a green color bounding box represents the detected face, and on the top of each bounding box, a corresponding label is assigned with two different colors. The blue color indicates those individuals who are wearing face masks as shown in the first row of Figure 6, where the red color is used for those who are without face mask.

5. Conclusions and Future Work

Due to COVID-19, a huge health crisis has arisen globally. The governments of several countries around the globe are trying to control the rapid spread of COVID-19. According to the statistics of Worldometer [8] and Google News COVID-19 [7], a rapid spread of COVID-19 is found in crowded areas. Several research studies have shown that wearing a face mask in crowded areas can reduce the rapid transmission of COVID-19. Therefore, in most countries, it is compulsory to wear face masks in crowded areas. It is very difficult to monitor crowds in these places. Therefore, in this work, we proposed a lightweight CNN model that detects people without a face mask. The proposed model achieved an average accuracy of 98.47% during testing. In the future, we can improve the proposed work by using a large amount of data, and it can also be extended to classify the type of mask and implement a facial recognition system, deployed at various workplaces, to support person identification while wearing the mask. We are aiming to test the proposed model in hazy and foggy environments in the future, where face detection can be challenging. We will use a generative adversarial network to enhance the quality of a given image for better performance evaluation. Furthermore, the proposed work can also be extended by using efficient object detection algorithms such as the various versions of Yolo, Faster-RCNN, etc., which can efficiently crop faces from the given input images and correspondingly recognize whether the person is wearing a face mask or not in one blended step.