Can Facial Expressions Detect Parkinson's Disease? What Facial Expressions Affect It The Most?

Image Recognition 10/03/2021

3main points
✔️ Discrimination by facial expression facilitates diagnosis
✔️ Propose a reliable method as a biomarker
✔️ What is the most influential facial expression among Smiling, Disgusted, and Surprised?

Facial expressions can detect Parkinson's disease: preliminary evidence from videos collected online
written by Mohammad Rafayet Ali, Taylor Myers, Ellen Wagner, Harshil Ratnu, E. Ray Dorsey, Ehsan Hoque
(Submitted on 9 Dec 2020)
Comments: Accepted to arXiv.
Subjects: Human-Computer Interaction (cs.HC); Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV)

code：

First of all

Is there a way to make it easier to determine if you have Parkinson's?

This paper proposes that it is possible to determine the presence or absence of symptoms from images and video data of facial expressions. In fact, one of the symptoms of Parkinson's disease is a decrease in the change of facial expression. In this paper, we focused on facial expressions and analyzed and studied smiling, disgusted, and surprised faces.

If it were possible to diagnose Parkinson's disease from facial expressions, it would not only make it easier to diagnose symptoms but would also be a help to people in areas where there are few neurologists around or where it is difficult to see a doctor. In addition, one of the attractive features of the diagnosis and classification methods proposed in this study is that they are able to achieve high classification accuracy using basic methods such as SVM training, clustering by k-means after PCA, and application of logistic regression models.

What is Parkinson's disease?

Parkinson's disease can cause tremors in the limbs, muscle stiffness, and non-rhythmic head movements, but how can these symptoms be detected and caught early?

hypomimia

Hypomimia is important for the early diagnosis of this disease.

One of the main symptoms of Parkinson's disease is paralysis, which causes stiffness of facial muscle movement and decreased facial expression. In this case, "Hypomimia" can be a very sensitive biomarker, and the following shows that it is suitable for early diagnosis.

Wearable sensors, often used as existing biomarkers, are reliable but expensive
Facial expression analysis is inexpensive (only a camera is needed for analysis, so implementation costs are low)
Can get a diagnosis even if a neurologist is not nearby (regardless of location)

Benefits of Telemedicine

One of the advantages of this camera image-based diagnosis is that you can get a diagnosis wherever you are, which is

Patients who need to be physically separated due to COVID-19 or other
People who have difficulty moving on their own
Areas where there are no neurologists in the vicinity

People with Parkinson's disease are particularly likely to benefit from this treatment. In addition, early detection of Parkinson's disease is very important because the decreased facial expression has been linked to social well-being and depression.

Methods

1.Dataset

The dataset consists of 604 participants (61 with Parkinson's disease symptoms and 543 without), each with 3 video recordings, for a total of 1812 videos. An online Parkinson's disease recording tool, PARK (Parkinson's Analysis with Remote Kinetic tasks), was used to acquire the videos, and this paper presents the results of the analysis of the facial imitation task collected using this framework. This paper presents the results of the analysis of the face imitation task collected using this framework. The facial imitation task includes three facial expressions: smiling (Smiling ), disgusted (Disgusted ), and surprised (Surprise ), and each video contains one of these expressions.

An example of the dataset is shown in Figure 3, where we collected 10-12 seconds of each video by asking participants to make each expression, hold that expression for a few seconds, and then make no expression.

Recruitment of participants

For those without Parkinson's, Facebook ads and Amazon Mechanical Turks and Amazon Mechanical Turks. For those with Parkinson's, they are being treated at the University of Rochester Medical Center, where the authors of the paper are based or have agreed to participate in the study. These participants must have been diagnosed by a specialist at the medical center as having some degree of Parkinson's disease symptoms.

2. feature extraction and calculation tools

The acquired video is analyzed using the OpenFace software which automatically calculates the facial action unit (AU) value for each frame. This facial action unit (AU) (Table 2) is available, and the current study is based on relating which action units are affected by smiling, disgusted, and surprised faces.

Smiling: AU01, AU06, AU12
Disgusted: AU04, AU07, AU09
Surprised face: AU01, AU02, AU04

As shown in Table 2 below, since the above facial action units (AUs) were found to be associated with each facial expression, the variance of the AUs was calculated and this variance is considered as a feature in this study.

From http://multicomp.cs.cmu.edu/wp-content/uploads/2018/11/OpenFace.pdf, Table II

3. analysis

The first analysis focuses on the distribution of characteristics between people with and without Parkinson's disease symptoms. We first performed the Mann-Whitney U test, and since the data were not normally distributed, we used a nonparametric significance test. Also, in the repeated significance test, we performed a Bonferroni correction for all p-values here.

Result

Statistical information of research collaborators

Processing of each expression

The differences in variance in each action unit (AU) of the face for people with and without Parkinson's disease symptoms can be summarized as in Table 2 below. Significant differences were found in smiling (Smiling) and surprised (Surprise) facial expressions, specifically in AU01 and AU06, especially in smiling facial expressions.

These results also show that the expression of a smile can be important in distinguishing the presence or absence of Parkinson's disease symptoms.

Differences in AU variance between people with and without Parkinson's disease symptoms

In addition, SVM was actually applied to nine facial action units (AUs) to classify the presence or absence of Parkinson's disease symptoms, and the results were as follows.

Correct response rate:95.6%, F1:0.95, AUC:0.94, Precision 95.8%,Recall 94.3

Figure 1: Feature weights from logistic regression.

People with Parkinson's disease symptoms were classified as 1, people without Parkinson's disease symptoms were classified as 0, and people with Parkinson's disease symptoms were classified as 1. The results of the binary classification are shown in the table below. The green bar indicates a characteristic with p<0.05, which can be said to be a significant characteristic. In addition, seven of the nine features have negative weights, which means that the lower the variance of the facial action unit (AU), the higher the probability of having Parkinson's disease symptoms, and it can also be said that the stiffness of the facial muscles affects the negative values.

Figure 2: Two-dimensional visualization of nine facial action units (AUs).

The nine features are transformed into two-dimensional data using PCA, and then K-means clustering is applied to the figure, which is clustered into three clusters. The red cluster has the highest PD% (percentage of participants with Parkinson's disease) of 76%, and the center of this red cluster is close to the (0,0) coordinate, indicating that the effect of AU variance among participants with Parkinson's disease is similar.

Discussion

The distributions showed that the variances of the two AUs were significantly different between people with and without Parkinson's disease symptoms. Furthermore, both of these were associated with smiling expressions, suggesting that smiling expressions, more than other expressions, are most affected by Parkinson's disease.

Results of logistic regression

The results of the logistic regression (Figure 1) show that The green area is the area where significance was found and of the four types

　Three weights derived from smiling expressions and one weight derived from disgusted expressions.
　Three types of negative weights

The three negative weights indicate that the relationship with Parkinson's disease symptoms is inverse. This means that low-level (Frontalis、pars medialis、Depressor Glabellae、Depressor Supercilli、Currugator、Orbicularis oculi、pars orbitalis、Zygomatic Major) muscle It can be said that the movement is related to the

AU01(Inner Brow Raiser)

The medial part of the eyebrows is raised, a characteristic that shows greater variability in those with Parkinson's disease than in those without it. In the past, eyebrow tremor has been found to be an early symptom of Parkinson's disease, and it is thought that Parkinson's disease was a factor in the increased movement of the eyebrows.

Analytical accuracy

Although Parkinson's disease symptoms are often characterized by several different modalities, including limb tremor, head movement, voice, memory, sleep, and gait, the following results indicate that facial expression scanning can be used as a reliable biomarker.

Simple SVM classifier : 95%.
Video analysis tool: 92% ( depends on limb shaking and head movement )

Implications for neurologists

By adding an algorithm that analyzes features that a neurosurgeon can see, plus subtle features that are often not visible to the naked eye, important new information can be added.

Important point

It is important to note that patients with Parkinson's disease do not exhibit all symptoms; only some symptoms may occur, or conversely, none may occur. Therefore, rather than relying on one Instead of relying on one modality, the use of several different modalities may provide more reliable diagnostic results. In this study, facial expression (especially smiling) was used as one of the modalities with high reliability.

Use of mobile phones

Smartphones and other devices can be used to create tools that automatically screen patients by taking and sending short videos like the one in this study, and refer them to a neurologist if they are at risk.

Also, globally, the number of neurologists per population is significantly lower in Africa, Asia, and some parts of South America. On the other hand, about 75% of people in Africa (and more than 90% in South Africa) have access to mobile phones, and the use of facial expression-based analysis proposed in this study could further contribute to improving health equity and access.

Data Availability

Due to IRB requirements, we are unable to share videos containing personally identifiable information, but the post-extraction features and Python scripts used in the analysis can be obtained from the cited GitHub.

Summary

In this paper, we propose a method to discriminate the presence or absence of Parkinson's disease symptoms based on facial expressions and show that it is a reliable biomarker. It is also shown that smile expression is an important factor in discriminating In addition, we showed that smile expression is an important factor in discrimination. This leads to the realization of easier diagnosis. This could make diagnosis easier for people who are currently difficult to diagnose and could also be an aid in areas where psychiatrists cannot reach.

Categories related to this article