What An Impact Wearing A Mask Has On Facial Recognition Models!
3 main points
✔️ A more exhaustive study of the impact of masks on face recognition than previous studies
✔️ Using three datasets and six machine learning models
✔️ A unique study using a dataset that includes half of all facial images with and without masks
An Exploratory Study of Masked Face Recognition with Machine Learning Algorithms
written by Megh Pudyel, Mustafa Atay
(Submitted on 14 Jun 2023)
Comments: Published in IEEE SoutheastCon 2023
Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
code：
The images used in this article are from the paper, the introductory slides, or were created based on them.
First of all
Facial recognition is a machine learning technology that is widely used for a variety of applications, including immigration control, secure login to electronic devices, community surveillance, school attendance, workplace attendance, and criminal investigation. However, when COVID-19 caused a global pandemic, the wearing of masks became essential in everyday life. Facial recognition technology, which had been used extensively, was severely impacted, and the performance of traditional facial recognition technology was significantly degraded. Although mask-compatible face recognition models have been reported in the past few years and have achieved a certain level of accuracy, the impact of mask wearing on face recognition has not yet been fully studied.
Therefore, this paper compares and evaluates the performance of multiple face recognition models in discriminating "masked" and "unmasked" face images. The paper uses six traditional machine learning algorithms: support vector machine (SVC), K nearest neighbor (KNN), linear discriminant analysis (LDA), decision tree (DT), logistic regression (LR), and naïve Bayes (NB). (Deep learning models have not been validated.)
It investigates the best performing and the worst performing models for masked face images. The paper also evaluates performance on a dataset of masked and unmasked face images, as well as on a dataset of half masked face images.
Compared to previous studies, this paper is unique in that it investigates a wide range of mask-wearing data and machine learning models.
Machine Learning Models and Datasets
As mentioned in the introduction, this paper evaluates six models: Support Vector Classifier (SVC), Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Decision Trees (DT/Decision Trees), Logistic Regression (LR/Logistic Regression), and Naïve Bayes (NB/Naïve Bayes) models are evaluated.
The dataset also uses Our Database of Faces (ORL), which contains 41 subjects and 10 images per subject, for a total of 410 unmasked face images. The figure below shows the sample data from ORL.
In this paper, a mask is added to the face image using an open source software called MaskTheFace for ORL face images, and a mask-wearing face image is prepared. The mask to be worn is randomly selected from six mask templates and added. The figure below shows sample data with masks added.
Experimental Details
In this paper, six experiments are conducted using the machine learning model and dataset described above. Note that in all experiments, the Local Binary Pattern (LBP) algorithm is used for feature extraction.
(Experiment 1) Of the 10 images prepared for each of the 41 subjects, 9 are used as unmasked face images to train 6 machine learning models. Then, of the 10 images prepared for each of the 41 subjects, the remaining one is also used as an unmasked face image to test each machine learning model.
(Experiment 2 ) Of the 10 images prepared for each of the 41 subjects, 9 are used as unmasked face images to train 6 machine learning models. Then, of the 10 images prepared for each of the 41 subjects, the remaining one is used as a face image with a mask on, using MaskTheFace to test each model. Figure 3.
(Experiment 3) Of the 10 images prepared for each of the 41 subjects, 9 are used as face images with masks on and 6 machine learning models are trained. The remaining one of the 10 images prepared for each of the 41 subjects is then also used as a non-masked face image to test each machine learning model.
(Experiment 4 ) Of the 10 images prepared for each of the 41 subjects, 9 are used as face images with masks on to train 6 machine learning models. The remaining one of the 10 images prepared for each of the 41 subjects is then used with MaskTheFace to test each model as a face image with the mask on. Figure 4.
(Experiment 5) From the 10 images prepared for each of the 41 subjects, one image is set aside for testing, and from the remaining 9 images, 4 are combined with the MaskTheFace software to create a total of 8 face images, 4 with the mask worn and 4 without the mask worn. This creates a dataset with half of the face images wearing masks. We use this dataset to train six machine learning models and test each model on a single unmasked face image that we have set aside.
(Experiment 6) Six machine learning models were trained using the dataset created in Experiment 5, which consisted of half masked face images, and each model was tested on a single masked face image. Figure 5.
Experimental results
The table below shows the Accuracy of the six machine learning models in all six experiments. For the machine learning models trained on unmasked face images (UM) and tested on masked face images (M) (UM/M), LDA has the lowest drop in accuracy at 61% and KNN has the largest drop in accuracy at 24%. We also see that for the machine learning model trained on the masked face image (M) and tested on the masked image (M), LR shows the highest accuracy of 80% and KNN the lowest accuracy of 37%.
The table below shows the F1 scores of the six machine learning models in all six experiments. For the machine learning models trained on unmasked face images (UM) and tested on masked face images (M), LDA has the highest F1 score of 76% and KNN has the lowest score of 39%. Also, for the model where half is trained on the masked face image (HM) and tested on the masked image (M), LR shows the highest F1 score of 89% and KNN shows the lowest score of 54%. For the model trained on the masked image (M) and tested on the masked image (M), LR shows the highest F1 score of 89% and DT shows the lowest score of 54%.
Reviewing the table below (again), we see that the best average performance is 81% when trained on unmasked face images (UM) and tested on unmasked face images (UM). This is a natural result because the machine learning model is built on the assumption that the face image is not wearing a mask. On the other hand, we see that the lowest average performance is 45% when trained on unmasked faces and tested on masked face images. This indicates that models trained on unmasked face images, as reported in the Corona Disaster, are not suitable for recognizing masked face images. This is reproducible with the results already reported.
We also find that the average accuracy of the machine learning model decreases for tests on masked face images when trained on a dataset consisting of either unmasked face images or face images that are half masked. When the machine learning model is trained on the masked face images, we see an increase in accuracy in testing the masked face images.
As shown in the table above, LR outperforms the other models in identifying unmasked face images on all three types of training data. LR outperforms the other models in identifying masked images when trained on datasets with masked or half masked images.
For systems that need to recognize both masked and unmasked facial images, it is better to train on a dataset consisting of half masked facial images and use LR, as shown in the table above.
Summary
In this paper, in order to investigate the effect of wearing masks on machine learning models, we used a "dataset of face images with masks", a "dataset of face images without masks", and a "dataset of face images with half wearing masks and half not wearing masks", as well as a support vector machine ( SVC/Support Vector Classifier), Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Decision Trees (DT), Logistic Regression ( LR/Logistic Regression), and Naïve Bayes (NB/Naïve Bayes).
Experimental results report that LR performs best as a system that recognizes both masked and unmasked facial images when trained on "a dataset of half masked and half unmasked facial images".
A trend toward increased accuracy of models trained on more masked face images has been observed when identifying masked face images, but at the same time a decreasing trend has been observed in identifying unmasked images.
Categories related to this article
