Facial expression differences indicate pain improvement at the emergency department

1. Introduction

Pain is an unpleasant feeling after experiencing trauma and is a major reason for seeking medical services [1, 2, 3]. Pain intensity is a major modifier of triage classification in emergency departments (EDs) [4, 5, 6]. Pain can be measured using several methods, including verbal rating scales, visual analog scales, numeric rating scales (NRSs), and face scales. Pain intensity is typically measured during triage through self-reporting on a 10-point NRS [7, 8]. Patients unable to understand the description of pain or communicate effectively may undergo an incomplete pain evaluation-particularly if they are elderly, very young, foreign, or unconscious [9, 10]. In addition, subjective ratings of pain raise concerns regarding stoic concealment or pain exaggeration [11].

Previous studies have attempted to measure pain by using behavioral assessments. For example, facial expressions of pain consist of a combination of emotions, including anger, fear, and disgust [12, 13, 14]. Identifying specific cues (e.g., brow lowering, eye closing, nose wrinkling, and mouth opening) can improve the subjective judgment of deception in pain expression [13, 14]. Facial action units, including brow lowering, orbital tightening, levator contraction, and eye closing, are considered prototypic expressions of pain [15]. The facial action detection technology had progressed from the first beginning as human coders to computers. Automatic tracking of facial landmarks became one of crucial development in this field. The accuracy and person independent facial landmark detection system is a major concern. A novel method is the Constrained Local Neural Field (CLNF). The CLNF capture more complex information and exploit spatial relationship between pixels. CLNF had shown outperforming across different conditions. Despite improvement of the face tracking technology, evidence supporting the efficacy of facial expression tracking for pain measurement in the emergency clinical setting is limited. The validity of previous findings in patients who cannot effectively communicate their pain is questionable.

Pain is a major symptom and major reason for patients to seek medical services, but limited evidence supports the applicability and usage of facial expressions as a pain measurement strategy in the ED. To facilitate pain measurement in the ED, the behavioral and physiological presentations of patients in pain should be investigated. The feasibility of using facial expressions as an indicator of pain in the ED is uncertain. In this study, we explored possible differences in facial expressions before and after pain management or treatment and compared these differences with those in self-reported NRSs. Moreover, the facial expressions with the highest correlation with clinically significant reductions in NRS pain scores were analyzed.

2. Methods

2.3 Data collection

2.3.1 Facial parameter extraction from videos

The development of facial feature detection algorithms has progressed. However, barriers, including poor lighting conditions and occlusions from extreme poses, still exist. The constrained local neural field (CLNF) model is a facial feature detection algorithm based on a point-distribution model; the positions of feature points in an image are determined by calculating the probability of aligned facial features. This algorithm was extended and its reliability was improved using a nonuniform regularized landmark mean shift fitting approach, optimizing the detection of a landmark’s position [16, 17]. The CLNF method is an automated analytical method. The lowest error rate has been reported for the CLNF method with higher reliability than that of other methods [16].

In this study, we tracked the positions of 68 facial landmarks in each postprocessed recording by using the CLNF method. To identify the features of facial expressions of pain, the characteristics of such expressions were computed directly from the x and y positions of tracked key points instead of using action units (Fig. 1). Twenty-five facial parameters were extracted per frame to represent patients’ facial expressions, which involved the eyebrows, eyes, mouth, nose, and head. To eliminate interindividual differences, we examined these features as ratios (divided by the distance from the nose tip to the root; Fig. 1), as follows:

Fig. 1.Twenty-five facial parameters were extracted per frame to represent the facial expressions of the patient.

Eyebrow distance ratio: The distance to the medial eyebrows divided by the distance to the lateral edge of the eyebrows (parameter 1).

Eyebrow fall: The quadratic polynomial coefficients of the right and left eyebrows divided by the distance from the nose tip to the root (parameters 2-5).

Eye opening: The sagittal length of both eyes divided by the distance from the nose tip to the root (parameters 6 and 9).

Eye squinting: The sagittal length of the bilateral upper and lower eyelids to the medial-lateral canthus linedivided by the distance from the nose tip to the root (parameters 7, 8, 10, and 11).

Eye distance: The distance between the medial canthi divided by the distance between the lateral canthi (parameter 12).

Mouth height and width: The sagittal and horizontal lengths of the lips divided by the distance between the medial canthi (parameters 13 and 14).

Lip corner: The distance between the medial canthi and the mouth angle of both sides divided by the distance from the nose tip to the root (parameters 15 and 16).

Mouth variation: The distance from the nose to the upper and lower lips divided by the distance from the nose tip to the root (parameters 17 and 18).

Mouth aspect ratio: The sagittal length between the lips divided by the horizontal length of the lips (parameter 19).

Eye aspect ratio: The sagittal length of both eyes divided by the medial-lateral canthus length (parameters 20 and 21).

Philtrum variation: The distance between the nose tip and upper lip divided by the distance from the nose tip to the root (parameter 22).

Nasolabial fold variation: The gap between the nasolabial folds divided by the distance from the nose tip to root (parameter 23).

Head pose and position: Head pose was estimated by calculating the movement between the head and the camera by using the perspective-n-point method based on the coordinates (parameter 24) and rotation angle (parameter 25).

In research on emotion recognition [18], neutral features were subtracted from emotional features to characterize significant differences in facial motion and enhance classification accuracy. Based on this concept, we defined the neutral condition as patients with a pain level of 0 and used the following function: $Featur{e}_i=\frac{X_i-mea{n}_{neutral}}{S_{neutral}}(1)$

2.3.2 Assessment of covariates

The patients’ age in years, gender, chief complaint, body part affected, NRS or pain intensity scores during the two sessions, duration of video recording during the initial and follow-up sessions, and vital signs in ED triage (body temperature, heart rate, blood pressure, and respiration rate) were recorded for analysis. Analgesics provided, if any, to alleviate pain were identified. Information on admission to a ward or intensive care unit, surgery, and date of discharge from the ED was obtained from medical records.

3. Results

Among 241 participants, 163 completed both recording sessions and were enrolled in the study. Most of the study participants were men (58.9%); the mean age was 51.9 years (Table 1). Abdominal pain (53.3%) was the most common complaint at ED triage, followed by limb pain (12.3%). The mean duration of the interval between the two sessions was 68.6 (SD: 30.5) min. The initial and follow-up mean NRS scores were 6.4 and 3.7, respectively. Moreover, 112 (68.7%) participants had taken analgesics to alleviate pain, and 30.1% were admitted to the hospital, with a mean hospital stay of 2.5 (SD: 5.8). Surgery was conducted in 11.0% of participants. Mortality and admission to the intensive care unit did not occur. Only eight patients revisited the ED within 72 h after discharge, with three additional hospital admissions.

Table 2 shows the correlations between changes in NRS scores and differences in facial parameters and physiological parameters between the initial and follow-up sessions. A stronger reduction in NRS scores was associated with differences in systolic blood pressure (sBP; r = 0.247, P = 0.011) and changes in the following facial features: eye opening (left: r = -0.210, P = 0.007; right: r = -0.206, P = 0.008), eye aspect ratio (left: r = -0.382, P $<$ 0.001; right: r= -0.305, P $<$ 0.001), and head pose by rotation angle (r = 0.218, P = 0.005).

Fifty participants exhibited significant improvement in pain levels (i.e., a difference of $\ge$ 4 in NRS scores) between the two sessions (pain improvement group). The pain improvement group showed differences in (sBP, OR = 0.973, 95% confidence interval (CI): 0.949-0.998, P = 0.034; diastolic blood pressure, OR = 1.078, 95% CI: 1.026-1.113, P = 0.003), eye aspect ratio (left: $\beta$ = 5.613, 95% CI: 2.234-14.104, P $<$ 0.001; right: $\beta$ = 2.743, 95% CI: 1.395-5.391, P = 0.003), and nasolabial fold variation ($\beta$ = 0.548, 95% CI: 0.306-0.982, P = 0.043) after adjustment for initial pain level, gender, and age (Table 3). The scatter plots in Fig. 2 show the correlation of the differences in NRS scores with the differences in measurements of facial expressions.

Fig. 2.Scatter plots showing the correlations between changes in NRS scores and differences in measurements of facial expressions of pain (P20, 21: eye aspect ratio, P23: nasolabial folds, P25: head angle).

4. Discussion

We found a significant correlation between the changes in NRS pain scores and the differences in facial parameters in the ED. Facial expressions of pain comprise various eye parameters and head rotation angles. Subgroup analysis indicated an association of additional parameters over the nose area with significant improvement in pain. Previous studies have emphasized the association between facial expressions and pain induced by stimuli or exercise [19, 20, 21, 22, 23]. The presentation of patients in prior studies might differ from that of patients in the ED. Our findings indicate that intraindividual changes in facial expressions can be used to track clinically relevant differences in pain. Facial expressions alone cannot be used as a pain measurement strategy in the ED.

This study identified facial parameters associated with pain in an ED population of a tertiary care center. These findings echo those of previous studies that have characterized features of the eyes, head posture, and nasolabial folds of patients in pain [13, 15, 24, 25, 26]. Thus, in addition to changes in the eyes, head posture, and nose area, activity in the brow and the activity in mouth areas are essential elements of facial expressions of pain. By contrast, we found a nonsignificant correlation of brow- and mouth-related parameters with changes in NRS scores. High-frequency mouth opening and residual eyebrow movement have been reported to be associated with the masking of pain expression [14]. Our findings suggest the importance of parameters related to the eyes, head angle, and nasolabial folds. In addition, lower blood pressure was associated with improved pain intensity. The responses to pain stimuli involve the neuroendocrine system, which maintains adequate physiological function against the imbalance [27]. Acute pain tends to increase blood pressure through the activation of the sympathetic nervous system [28, 29]. Increased peripheral resistance, heart rate, and stroke volume all contribute to the increased resting blood pressure during pain stimulus [30]. Previous studies have proposed that hypertension-related acute pain modulation reduces pain sensitivity through hemostatic feedback [31]. The excitation of the central and peripheral nervous systems decreases with pain relief, followed by a decline in blood pressure.

Second, instead of recognizing facial action units, we computed features that characterize these expressions directly from the positions of tracked key points. Artificial intelligence and machine learning have been applied to the analysis of relevant action expression records [15, 32, 33, 34, 35]. For example, Ashraf et al. used an active appearance model to identify frame-level pain [32], and Kaltwang et al. used a relevance vector regression model to discriminate between real and false pain [33]. Littlewort et al. reported that an automated facial coding system that involved machine learning could correctly identify real pain [34]. Another automatic continuous facial feature landmark tracking system enabled the monitoring of pain over time [35]. Action units use a layer combined with the computation of facial landmarks, which are easily communicated after being translated into different facial movements. This study analyzed several parameters by using primitive information to facilitate the transfer of the study results to a machine instead of post-processed action coding system.

Third, self-reporting is considered to be the primary pain measurement. A comprehensive assessment of pain includes the assessment of behavioral and physiologic presentations [36]. Although there were different starting points for self-reporting and distinct facial expressions for different types of pain stimuli, their association was evident in the presence of pain [37]. In addition to sensory discriminative factors, motivational affective and cognitive factors are important [38]. The interpretation of painful stimuli may be affected by culture, personality, experiences, environment, and emotional state [39, 40].

5. Limitations

The present study has several limitations. First, sampling and selection bias could not be ruled out because the study involved convenience sampling. Second, one in three patients refused to participate in this study. Third, this study was conducted with a limited number of participants from a university-affiliated hospital. Furthermore, the study population comprised patients with triage levels of 3 to 5, with few patients at levels 1 and 2. Patients with triage levels 1 and 2 patients were in critical condition and triaged according to modifiers other than NRS scores (e.g., unconsciousness, respiratory distress, and circulation collapse). Therefore, the study population may not be representative of all patients in pain; this limits the generalizability of our findings.

Several confounders, including the number of ED revisits after discharge, were not evaluated. Only revisit data for the ED of the study hospital were available. Some patients may have been lost to follow-up after discharge from the ED. Some patients may have left the ED without follow-up. Thus, some data on follow-up vital signs were missing. Nonetheless, we believe that the study reflected reality and provided adequate conclusions.

6. Conclusions

Using facial expressions to monitor the pain changes of a patient is a novel application of such technology in the ED. Gaps exist that prevent this method from being feasible in clinical practice, such as limitations related to facial recognition, multiface tracking, and data management process. Pain measurement in the ED should involve objective and automatic detectors either continuously or in a point-of-care style. In addition to the development of a feasible technological application, several aspects require further clinical research. First, outcome research is the most crucial field. As current technology is applied to daily ED care, the evolution of pain management should be observed. It will be interesting to check how clinical practice adjusted after revealing the results of dynamic facial expressions to the physicians and nurses in a timely manner. Second, a large database for subgroup analysis and cross-region comparison should be established. In subgroup analysis, patients who are unable to communicate their pain due to language barriers post physical/mental illness or those from different backgrounds should be included. In addition, the database would enable the exploration of the basics of pain expression and modulation under different causes of pain stimuli.

In short, we reported a significant correlation between changes in NRS pain scores and differences in facial parameters in ED patients. Facial parameters related to the eyes, head pose, and nasolabial folds were associated with changes in pain intensity. Our findings indicate that intraindividual changes in facial expressions can be used to track clinically relevant differences in pain. Facial expressions alone cannot be used as a pain measurement strategy in the ED.

Acknowledgements

This study was supported by grants from Chang Gung Memorial Hospital, Linkou, Taiwan (CMRPG3E1791) and the Ministry of Science and Technology, Taiwan (MOST 106-2221-E-007-027)

Funding

This study was funded by Chang Gung Memorial Hospital, Linkou, Taiwan (CMRPG3E1791).

Conflict of interest

The authors declare that they have no competing interests.