1. Introduction

Pain is a prevalent and debilitating symptom in cancer patients. Research indicates that approximately 55% of patients encounter pain during anticancer treatment, a percentage that escalates to 66% for individuals grappling with metastatic, advanced, or terminal diseases [1]. While accurate symptom assessment is crucial for effective pain management [2], conventional unidimensional pain rating scales, such as the 0–10 numeric rating scale (NRS) and the Visual Analog Scale (VAS), exhibit significant limitations. These scales are susceptible to reporting bias, influenced by psychosocial factors like tendencies to catastrophize or underreport pain [3]. Furthermore, although multidimensional assessment tools prove useful in evaluating various features of a patient’s pain experience, they fail to provide an objective and exhaustive measurement of pain [4]. Additionally, the inherently subjective nature of pain, coupled with self-reported assessments relying on individual interpretation and communication, introduces another paramount bias [5].

In this complex landscape, the exploration of pain assessment can be enriched through multifaceted evaluations, harnessing the capabilities of artificial intelligence (AI) methodologies for comprehensive pain analysis [6]. Notably, AI has the potential to significantly impact pain assessment by offering innovative solutions for personalized treatment strategies. Therefore, efforts are directed toward advancing automatic pain assessment (APA) systems [7]. For instance, there have been successful applications of computer vision and machine learning (ML) techniques in assessing pain through facial expressions [8], alongside other advancements centered around neurophysiology-based pain detection methods [7].

Since pain encompasses both sensory and emotional dimensions, the careful assessment of the emotional component has a significant impact on the evaluation of the whole painful experience [9]. Significantly, speech analysis could be used for this purpose. Speech analysis, the systematic examination of spoken language, aims to extract meaningful information, recognize patterns, and derive valuable insights [10]. This comprehensive process encompasses various speech aspects, including acoustic features, linguistic content, intonation, rhythm and more. Therefore, speech analysis has numerous applications such as speech recognition, speaker identification, language identification, speech pathology, and emotion recognition (i.e., speech emotion recognition, SER) [11]. In the context of vocal emotion recognition, the focus of analysis revolves around the acoustic features of the voice. These features, including pitch, tone, rhythm and intensity, are scrutinized to identify distinctive patterns associated with different emotional states [12].

By implementing AI strategies, automated systems can more efficiently process and interpret speech data. Currently, speech analysis is regarded as a specialized branch of Natural Language Processing (NLP), a subfield of AI that focuses on human-computer interaction through spoken and written natural language [13] both spoken and written. Indeed, some very specific speech analysis applications deal with the interpretation and understanding of natural spoken language and related nuances. Among other tasks, speech analysis contributes to emotion recognition, identifying and understanding the speaker’s emotional state. This approach is commonly used for sentiment analysis and emotional well-being assessments [14]. In medicine, computational language analyses have been performed for multiple purposes such as functional evaluation after laryngeal surgery [15], respiratory disease detection from voice [16, 17, 18], automatic depression detection [19], and investigation in neurodegenerative diseases [20].

Previously, we developed and validated an APA model based on facial expression analysis. By analyzing videos obtained from patients with oncological pain, the binary classifier demonstrated excellent performance in recognizing the presence or absence of pain. The classifier achieved 94% accuracy, and the area under the receiver operating characteristic Curve (AUROC) value was 0.98 [8].

Given these considerations, we introduce an AI framework for APA in cancer patients. In the first part of the study, we describe the computational language analysis for emotion recognition. Subsequently, the APA approach entails the concurrent use of two distinct models, one for automatic emotion recognition from speech recordings and another for analyzing facial expressions of pain. Two case studies are utilized to demonstrate the viability of the framework. The aim is to pave the way for a holistic pain assessment and, ultimately, personalized pain management strategies.

2. Materials and methods

This study is integral to a broader multimodal APA project, incorporating computer vision and natural language processing methodologies, alongside the analysis of physiological signals.

2.4 Neural network

The developed neural network is a Multi-Layered Perceptron that uses a Rectified Linear Unit (ReLU) activation function and is trained with the Adam (Adaptive Moment Estimation) algorithm for weight optimization to minimize the Mean Square Error. Adam combines principles from other optimization algorithms such as Root Mean Square Propagation (RMSprop) and Momentum, rendering it especially well-suited for tasks of this nature. Specifically, this stochastic optimization method excels in addressing challenges posed by substantial data and/or parameter sizes. The approach is also fitting for objectives that evolve and problems featuring exceedingly noisy and/or sparse gradients [25].

The Neural Network is composed of an input layer with 181 prosodic features, followed by three hidden layers of 300, 200 and 100 neurons respectively, and an output layer that maps the resulting stimuli over six emotional states (disgust, joy, fear, anger, surprise, sadness) plus a neutral state. The resulting emotional state is the one with the highest probability (Fig. 1).

Fig. 1.Multilayer perceptron classifier. The classifier is made up of five dense layers. The first layer (input) consists of 181 nodes associated with prosodic features extracted for the audio clip of each image. There are three hidden layers of 300, 200 and 100 nodes respectively. The output encompasses six emotional states (disgust, joy, fear, anger, surprise, sadness) plus a neutral state. We implemented the Rectified Linear Unit (ReLU) activation function and the Adaptive Moment Estimation (Adam) algorithm for weight optimization. Softmax function was applied to the final (output) layer to obtain the probability distribution of the output across the 7 possibilities. The training process was guided by the cross-entropy loss. Speech emotion analysis is performed through the Multilayer Perceptron Classifier.

2.8 Application of continuous audio-video analysis to use cases

We used the previously created APA model [8] for facial expression analysis from video recordings as well as the emotion recognition model from audio recordings to combine the emotion recognition model and the cancer pain recognition model within a larger framework for the assessment of emotional states and pain in oncological patients. Therefore, the approach adopts a continuous audio-video analysis through the two models. Fig. 2 shows a screenshot of an ongoing analysis that captures both pain from facial expressions and emotions from audio traces within the same video acquisition.

Fig. 2.The evaluation of emotional states related to automatic pain analysis performed through a model for facial expressions investigation [8]. The underlying Neural Network classifier for video analysis is based on the Shallow Network architecture. It consists of 17 input nodes (17 action units related to pain expressions) and a single output node that provides the probability of it being Pain versus No Pain. The Eudico Linguistic Annotator (ELAN) version 6.7 was implemented.

For this study, two cases of patients referred to the Pain Medicine Unit (National Cancer Center, Napoli, Italy) were examined to demonstrate the application of the model in real-world scenarios. The first case involves an oncological patient without cancer pain, while the second case pertains to a patient experiencing cancer-related pain. Comprehensive clinical descriptions for each case are provided in the subsequent sections.

3. Results

3.1 Model performances

The model performed exceptionally well on the test set, effectively mitigating overfitting. The details of cross-validation accuracies are shown in Fig. 3.

Fig. 3.Cross-validation results. Accuracy obtained per each fold. The hyperparameter K was set to 15, a value that was confirmed through subsequent optimization processes. This resulted in the creation of 14 folds for training the network and a single fold for its validation.

In addition, the overall model performance provided 84% average accuracy across the folds of the cross-validation and demonstrated promising metrics values for all the classes (Table 2).

Table 2.Overall model performances.

	Precision	Recall	F1-score	Support
Anger	0.87	0.80	0.83	25
Disgust	0.69	0.85	0.76	13
Fear	0.82	0.86	0.84	21
Joy	0.71	0.73	0.67	19
Sadness	0.90	1.00	0.95	26
Surprise	0.84	0.84	0.84	19
Neutral	1.00	0.88	0.93	24
Accuracy	0.84	147

It is observed that neutral and sadness states achieved the highest metrics, with precision, recall, and F1-score ranging from 90% to 100%. Disgust and joy states reached the lowest values, probably due to the lower number of occurrences supporting the model. Finally, anger, fear and surprise states achieved the most stable and balanced metrics within 80%–87%, with surprise reaching a stable value of 84% for all the performance metrics considered.

3.2 Continuous audio-video analysis in cancer patients: use cases

3.2.1 Case study No. 1: clinical presentation of a patient without cancer pain

A 67-year-old female was referred to the Pain Medicine Unit due to chronic postoperative pain. The patient reported moderate pain, assessed using the Numeric Rating Scale (NRS 0–10) with ratings of 4–5. She also described a significant neuropathic component, including tactile allodynia, dysesthesias, and paresthesias, at the site of a surgical procedure performed approximately 2 years earlier for the removal of a lung lesion. A non-invasive method (percutaneous electrical nerve stimulation, PENS) was utilized to manage chronic post-thoracotomy pain. However, at the time of the video recording and interview, the patient reported being pain-free (NRS 0), with neuropathic symptoms nearly completely resolved, except for persistent paresthesias.

3.2.2 Case study No. 2: clinical presentation of a patient with cancer pain

A 72-year-old male presented to the Pain Medicine Unit with severe pain (NRS 0–10, 8) attributed to multiple bone lesions resulting from prostate cancer. He underwent multimodal treatment involving pharmacological approaches (including opioids and adjuvant medications) and non-pharmacological interventions (such as palliative radiotherapy). Despite these interventions, the patient reported insufficient pain control, especially when triggered by movement (known as incident breakthrough cancer pain [29]).

3.2.3 Implementation of audio-video analysis and comparison of results

To implement the emotion recognition model in the framework of oncological pain assessment, we adopted both the previously developed APA model [8] for facial expression analysis from video recordings (binary classifier: pain/no pain) along the emotion recognition model from audio recordings. The models were applied together within the same framework of continuous audio-video analysis for both the use cases here described, with and without cancer pain respectively. As far as emotion recognition, we computed emotional states throughout the entire video, with each label assigned per 500 ms frame. In the analysis of the audio file, we excluded dialogue pauses and interviewer speech phases.

In addition to accurately identifying the presence or absence of pain in both patients through video analysis, the audio recording results also offered clear insights into the emotional characteristics of the two considered patients, as depicted in the following Fig. 4.

Fig. 4.Comparison of emotional state analyses for the two examined use cases: with and without cancer pain.

As from Fig. 4, the continuous analysis of the emotional status and pain expression in patients with and without cancer pain revealed marked differences in terms of emotion prevalence.

The neutral and surprise states were predominant for the patient who did not suffer from cancer pain, accounting for 56% and 41%, respectively. On the other hand, when the assessment framework was applied to the patient reporting cancer-related pain, the analysis highlighted a predominance of the neutral and anger states, accounting for 49% and 20%, respectively, followed by sadness (17%) and surprise (12%), which, in this case, was less represented (12%) compared to the case of non-cancer pain (41%).

The differences in terms of the percentage of emotions between the two presented clinical cases of non-cancer pain and cancer-related pain are reported in the following Table 3.

Table 3.Differences in the percentage of emotional state prevalence between the two presented clinical cases of non-cancer pain and cancer-related pain.

	Differences in emotional state prevalence
Neutral	+7%
Surprise	+29%
Joy	+1%
Disgust	0%
Anger	−19%
Fear	−2%
Sadness	−17%
Legend: Computes as Emotion in the “No cancer pain” Class—“Cancer pain” Class.

The proposed approach can detect the highest prevalence of negative emotions (anger, fear, sadness) in the case of cancer-related pain and the highest prevalence of neutral or positive emotions in the absence of cancer pain (neutral, surprise, joy). No differences were detected for the disgust state (Fig. 5).

Fig. 5.Framework of the audio-video analyses in the two case studies. The video-audio analysis allows for simultaneous assessment of facial expressions of pain (action unit-based frame-by-frame assessment) and speech emotion recognition with the output of the full emotional status. For the assessment of audio and video tracks, we employed an Annotator (Eudico Linguistic Annotator, ELAN version 6.7). Although effective for ongoing evaluation, the tool itself is not an integrated platform.

4. Discussion

The implementation of AI methodologies in pain assessment, particularly in the context of cancer patients, holds significant promise for advancing our understanding and improving personalized pain management strategies [30]. AI and APA systems present innovative solutions for objective measurement [31]. Our study explored the application of AI in the form of computational language analysis for extracting emotional content from speech records. In particular, we focused on the simultaneous automatic emotion recognition and video analysis, aimed at an AI-based multifaceted approach to pain assessment. The implementation of APA processes aligns with the broader trend in the field directed toward leveraging AI strategies to enhance the efficiency and objectivity of pain assessment [7, 32, 33]. Speech analysis represents a valuable strategy for APA investigations. In response to advancements in signal processing and machine learning, researchers are now creating algorithms to automate the assessment of pain levels through speech. For example, Tsai et al. [34] embedded stacked bottleneck vocal features in a Long short-term memory (LSTM) architecture to detect pain, concentrating on prosodic signals within a subset of the triage dataset. Additionally, Li et al. [35] enhanced the variational acoustic model by introducing age and gender factors. Other projects focused on the analysis of bioacoustic pain markers related to the speech signal [36]. Therefore, we propose a step towards the fusion of different information channels aimed at pain assessment based on specific factors tied to the channel.

The central premise is that prosody and speaking features can significantly convey pain as a complex set of emotions, often referred to as the “Big Six” [9, 10, 11, 22]. Nevertheless, the difficulties hidden in this approach lie in the scarce availability of speech data annotated along any pain scale. Consequently, we are working towards the development of multimodal audio/video integrated models. The preliminary step involves defining the different frameworks to be implemented. Within our frameworks, the video analysis was performed by designing an AU-based classifier. The speech analysis was performed by designing a neural architecture. Therefore, by examining acoustic features, linguistic content and prosody, we aimed to capture emotional states that were subsequently associated with video-based APA analysis (binary classifier: Pain/No Pain) [8]. The developed neural network, trained on prosodic features extracted from speech signals, demonstrated promising results in emotion recognition, with accuracy reaching 0.84. Concerning the model’s performances, we utilized accuracy as primary metric due to the balanced nature of the classes. This metric provides a holistic view of the s overall correctness in predicting emotions across all classes. Precision measures the proportion of correctly identified emotions among all instances where the model predicted a certain emotion. Recall, on the other hand, quantifies the ability of the model to correctly identify instances of a specific emotion out of all instances where that emotion truly occurred. The F1-score, which is the harmonic mean of precision and recall, gives a balanced assessment of the model’s performance, particularly useful when dealing with imbalanced datasets or when precision and recall need to be considered together [37].

An important aspect is the definition of the set of prosodic features to include for the training and testing of the model. The number of prosodic features that can be extracted from speech is not fixed or standardized [38]. It can vary depending on the specific research or application, and different studies or systems may use different sets of features. Researchers might choose a subset of relevant features based on the goals of their study or the requirements of their application. For instance, in some studies focused on emotion recognition, researchers may extract features related to pitch, duration, and intensity variations associated with different emotional states. In contrast, applications like speaker identification might prioritize features related to pitch patterns, rhythm, and voice quality [39, 40]. For this reason, we have opted to incorporate a diverse array of features based on the Librosa library.

Given these preliminary results, the continuous evaluation revealed different emotional states, showcasing the potential of our approach for real-time assessment and personalized pain management pathways. The diversified emotional states and pain features unveiled through continuous evaluation underscore the versatility and potential applications of our method, suggesting its adaptability in dynamically responding to individuals’ changing pain experiences and needs.

To demonstrate the practical applications of our SER framework, we have selected two exemplary cases for analysis. In the first case, involving a patient who did not report pain, our assessment revealed predominant emotional states of neutrality and surprise. Since the patient did not describe pain issues, feelings of neutrality might indicate a lack of emotional distress related to discomfort or suffering. The presence of surprise could be linked to unexpected events or interactions during the assessment process. Conversely, in the second case, our analysis identified a significant component of anger and sadness. These emotions may stem from the patient’s experience of pain and associated distress. Chronic pain can evoke feelings of frustration, helplessness and sadness [41]. Additionally, the patient’s ongoing struggle with poorly controlled breakthrough cancer pain despite multimodal treatment strategies, as mentioned in the case description, could contribute to heightened emotional responses such as anger and sadness [29]. These emotional states may also reflect the patient’s psychological and emotional resilience or vulnerability, as well as their coping mechanisms and support systems. Factors such as past experiences, personality traits, and social support networks can influence how individuals perceive and respond to pain and distress [42]. These findings underscore the complex interplay between physical symptoms, psychological factors, and emotional well-being in cancer patients, highlighting the importance of holistic and patient-centered approaches to pain management [41, 42, 43].

Finally, from our initial findings emerges that a crucial aspect demanding in-depth exploration is the correlation between facial expressions and emotional responses through computational methodologies. Physiologically, emotions manifest through different facial expressions. For example, surprise may be expressed as widened eyes, raised eyebrows, and open mouth. These physical reactions are often involuntary and are part of the body’s automatic response to unexpected stimuli. Despite the conducted experiments within the medical domain [44], particularly in the field of pain medicine, this realm remains largely untapped for comprehensive investigation and understanding. For instance, an intriguing avenue of research involves studying the correlations between emotion circuits and the neurobiology of chronic pain [45].

This investigation acknowledges several limitations that warrant consideration. An important limitation is the model’s lack of verification across a large population sample. It is emphasized that the objective of the paper was not the generalization and validation of the model across a large sample; instead, it focused on the potential preliminary application of the proposed framework in a clinical setting of oncological patients. In defining the methods, we selected two case studies of oncological patients, analogously to what has been done in another explorative study on APA through biosignal analyses [46]. Although the obtained results may be potentially influenced by cultural and language biases and lack external validation, they show promising performance metrics of the emotion recognition classifier and possible applicability in clinical settings. This encourages further assessment to investigate the reliability and validity of the model on a larger scale, which will be addressed in future works.

Concerning model building, a key constraint revolves around the utilization of datasets not explicitly designed for the study of cancer pain. The predictor expresses outputs (i.e., emotion) for audio traces that reflect what it has learned from the EMOVO dataset. This aspect could significantly impact the reliability of the outcomes, as the datasets might lack essential variables for a thorough analysis of cancer pain phenomena. Consequently, caution is advised in concluding the results, given that this main limitation may influence the generalizability and applicability of the findings. Future research endeavors should prioritize the collection of more targeted and comprehensive datasets to overcome this limitation, thereby enhancing the accuracy and reliability of the results. Furthermore, the model employed in the study has limitations in conducting temporal analysis as it focuses on a single time instant without considering contextual parameters.

In the context of facial expression for APA, the pain phenomenon was evaluated using a binary classification system, distinguishing between its presence and absence [8]. However, it is essential to acknowledge that this study did not delve into more granular analyses regarding the various types or gradations of pain. Another limitation of the analysis is the overall quantization of emotions, considering the entire video. For increased reliability, it would be advisable to segment the video into smaller sections instead of computing based on the overall duration. However, this limitation is also associated with the need for precise labeling within the context of audio and video content. These limitations suggest the necessity for a multimodal model to enhance the accuracy of the study’s findings and address gaps in the analysis. Significant improvements could involve procedures to label text and integrate various behavioral analyses, such as facial emotion recognition [47, 48, 49] as well as to adopt further algorithms and models to cluster [50, 51, 52] and classify [53, 54, 55] the patients based on the facial, vocal patterns and additional multimodal patterns. Our forthcoming research in the field of APA will delve deeper into this aspect.

5. Conclusions

In APA research, exploring automatic emotion recognition offers valuable insights. These initial findings indicate the promising application of AI models to continuously estimate emotional states from video recordings. This strategy has the potential to complement other APA approaches as part of a multimodal analysis. Further research is warranted to refine this process, particularly in integrating this data with multi-parameter analyses, including text analysis, and multimodal physiological parameters. In the subsequent stages, the model will need to anticipate the integration of individual approaches and ultimately its practical implementation.

Availability of data and materials

The raw data can be obtained on request from the corresponding author. The dataset implemented for the binary classifier is available at: https://doi.org/10.5281/zenodo.7557362 [28].

Author contributions

MC, FC and VNV—designed the research study. MC, FM, MI and VNV—performed the research. AC, SB, FSa and VC—provided help and advice on the clinical evaluations. AV, MI, AMP, JM, VB and FSe—analyzed the data. MC, EGB and OP—wrote the manuscript. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

The research adhered to the principles of the Declaration of Helsinki, and written informed consent was obtained from all patients. The Medical Ethics Committee of the Istituto Nazionale Tumori, Fondazione Pascale (Napoli, Italy), approved the PASCALE study (Pain ASsessment in CAncer Patients by Machine Learning), under the protocol code 41/20 Oss (26 November 2020). (ClinicalTrials.gov Identifier: NCT04726228). Patient consent was acquired for the study and scientific divulgation of personal identifiable information.

Acknowledgment

The authors thank Cristina Romano and Mariacinzia Grizzuti from the National Cancer Institute of Naples for the provided administrative and technical support.

Funding

This research received no external funding.

Conflict of interest

JM is a co-founder and shareholder of Callisia srl University Spin-off at Università Politecnica delle Marche developing a smart bracelet collecting patient data intelligently for real-time visualization and data analysis. The rest of the authors have no relevant financial or non-financial interests to disclose. Marco Cascella is serving as one of the Editorial Board members of this journal. We declare that Marco Cascella had no involvement in the peer review of this article and has no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to MAM.