Title
Author
DOI
Article Type
Special Issue
Volume
Issue
1Department of Emergency Medicine, Korea University Guro Hospital, 08308 Seoul, Republic of Korea
2Department of Pediatrics, Korea University Guro Hospital, 08308 Seoul, Republic of Korea
3Department of Emergency Medicine, Korea University Ansan Hospital, 15355 Ansan, Republic of Korea
4Institute for trauma research, Korea University, 02841 Seoul, Republic of Korea
5Poderosa Co., Ltd, 06626 Seoul, Republic of Korea
6WeAreFriends, Co., Ltd, 05635 Seoul, Republic of Korea
7InnoRules Co., Ltd, 05855 Seoul, Republic of Korea
8Waycen Inc., 06182 Seoul, Republic of Korea
*Corresponding Author(s):kuedchoi@korea.ac.kr (Sung-Hyuk Choi)
| History | Submitted: 13 September 2022 | Accepted: 25 November 2022 | Published: 08 September 2023 |
| Copyright: | ©2023 The Author(s). Published by MRE Press. |
The initial severity triage of patients in the emergency department (ED) is implemented differently worldwide, according to the characteristics of each country. However, better classification methods are being studied due to various problems with the current system. Therefore, the aim of this study was to determine the usefulness of patients’ severity assessment in a new way that gives appropriate values to factors that can be obtained in the ED. We collected data from 158,246 patients who visited the ED from 01 January 2016 to 31 December 2020. Using the appropriate values of various factors obtained using the Rasch analysis method, the cut-off values for predicting hospitalization and discharge at the ED of patients were determined. Furthermore, using artificial intelligence, the patients who were hospitalized and discharged from the ED were classified and compared with the results of the Rasch analysis. The accuracy of the algorithms was analyzed as a combination of factors that could be obtained during the initial stage of the patient’s visits. The area under the curve (AUC) value for the prediction of hospitalization and discharge by a combination of factors immediately obtained from the visit was 0.611. In addition, using the factors that could be obtained after a certain period, the AUC value of hospitalization and discharge prediction was 0.767. The results of analysis using artificial intelligence were similar to or slightly higher than the AUC value of the Rasch analysis. The prediction of hospitalization and discharge in the ED using clinical parameters with Rasch analysis can be used for medical assistance, which is expected to help in the efficient operation of the ED.
Cite this article
Sung-Joon Park, Sung-Hyuk Choi, Dae-Jin Song, Jong-Hak Park, Ju-Hyun Song, Han-Jin Cho, Sun-Hong Lee, Byung-Chul Ko, Kyu-Hwan Ahn, Gil-Gon Kim, Won-Seok Choi, Kyung-Nam Kim. Improving triage accuracy of hospitalization and discharge decisions in the emergency department. Signa Vitae. 2023; 19(5): 75-90. doi: 10.22514/sv.2023.015
1. Introduction
The severity triage in the emergency department determines the priority for treatment. It is used to provide appropriate treatments for cases of greater urgency when many patients visit at the same time. In Korea and around the world, this is often the case—many patients visit the emergency department (ED) at the same time and not at regular intervals. This delays appropriate treatment and increases patient complaints owing to overcrowding and lack of medical staff. Therefore, severity triage of patients is essential for proper management and patient satisfaction, as it enables the efficient use of medical staff and eliminates overcrowding [1]. As a result, in many countries, severity triage of patients is implemented with slight differences depending on the country’s circumstances.
Various severity triage systems such as the Emergency Severity Index (ESI), Canadian Triage and Acuity Scale (CTAS), Manchester Triage System (MTS), and Australasian Triage System (ATS) are used. In Korea, we use the Korean Triage and Acuity Scale (KTAS) based on the CTAS to classify the severity of emergency patients into five classes [2]. However, this requires continuous training of the medical staff for accurate severity triage and is ineffective at eliminating overcrowding and reducing workload and patient complaints.
Hospitalization and discharge are generally determined by the severity of patients, but patients who require hospitalization are often transferred due to a lack of hospital beds. As the number of patients waiting for hospitalization in EDs increases, proper management is difficult owing to a lack of space to treat new emergency patients. Therefore, patients should be quickly transferred to other hospitals.
The rapid prediction of hospitalization and discharge in the ED can increase patient satisfaction by eliminating overcrowding and effectively utilizing medical resources such as hospital beds and medical personnel. Recently, Rasch analysis, presented as a weighted value (WV) with the frequency of factors [3], was used as a statistical method, and its usefulness has been reported in mental illness screening and the prognosis evaluation of chronic diseases [4, 5]. There are also reports on prediction of disease deterioration using artificial intelligence (AI) [6, 7]. Therefore, we aimed to investigate the usefulness of predicting hospitalization and discharge through Rasch analysis using initial factors of patients presenting to the ED as well as values obtained from various blood tests. In addition, we compared and verified the usefulness of Rasch analysis using AI results.
2. Methods
2.1 Participants and distribution of patient characteristics
We investigated various factors that could be obtained from 158,246 patients aged 15 years or older who visited the ED at one hospital from 01 January 2016 to 31 December 2020. Using the medical records, the data were completely anonymized before access and used the necessary factors. We collected data from nine factors immediately obtained at the beginning of the visit and data from 18 variables obtained from blood tests. These factors can be easily obtained in most emergency centers in South Korea. In addition, patients were classified into groups based on their initial characteristics.
2.2 Composition and weight values (WVs) of collected factors
We examined initial factors that could be obtained in the early stages for rapid assessment at the ED. Demographic characteristics such as age and sex as well as medical factors such as past history, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, and state of consciousness were examined. In addition, blood sugar level, hydrogen ion concentration (pH), partial pressure of arterial oxygen (PaO2), and lactic acid level, all of which can be obtained immediately from point-of-care testing (POCT), were examined depending on the symptoms of the patients visiting the ED. The WV for the range of each factor was determined using the Rasch analysis (Tables 1 and 2). The WV was used such that the higher the severity of the patient, the higher the value. We also examined hemoglobin level, hematocrit, white blood cell (WBC) count, prothrombin time (PT), and aspartate aminotransferase (AST), alanine aminotransferase (ALT), bilirubin, creatinine, amylase, sodium, potassium, C-reactive protein (CRP), creatine kinase-myocardial band (CK-MB), and troponin T levels, which can be obtained from most patients after a certain period of time.
| Factor | Range | Weighted value |
| Age (yrs) | ||
| 15 ≤ n < 18 | 6.2 | |
| 18 ≤ n < 66 | 1.1 | |
| 66 ≤ n < 80 | 12.9 | |
| 80 ≤ n < 100 | 22.1 | |
| 100 ≤ n | 98.9 | |
| Sex | ||
| Male | 47.7 | |
| Female | 52.3 | |
| Past history | ||
| Diabetes (Y/N ) | 29.5/3.9 | |
| Hypertension (Y/N ) | 20.6/6.7 | |
| Tubeculosis (Y/N ) | 98.8/1.2 | |
| Hepatitis (Y/N ) | 61.7/1.4 | |
| Malignancy (Y/N ) | 45.1/1.9 | |
| Chronic Obstructive Pulmonary Disease (COPD) (Y/N ) | 61.7/1.4 | |
| None | 6.7 | |
| More than two diseases | 100.0 | |
| SBP (mmHg) | ||
| n < 70 | 97.9 | |
| 70 ≤ n < 90 | 62.7 | |
| 90 ≤ n < 120 | 2.1 | |
| 120 ≤ n < 140 | 10.2 | |
| 140 ≤ n < 160 | 17.2 | |
| 160 ≤ n < 180 | 37.2 | |
| 180 ≤ n | 53.6 | |
| DBP (mmHg) | ||
| n < 40 | 98.3 | |
| 40 ≤ n < 60 | 38.0 | |
| 60 ≤ n < 80 | 1.7 | |
| 80 ≤ n < 100 | 4.2 | |
| 100 ≤ n < 120 | 28.6 | |
| 120 ≤ n | 69.2 | |
| Heart rate (/minute) | ||
| n < 30 | 98.6 | |
| 30 ≤ n < 40 | 92.9 | |
| 40 ≤ n < 60 | 55.8 | |
| 60 ≤ n < 100 | 1.4 | |
| 100 ≤ n < 120 | 15.3 | |
| 120 ≤ n < 150 | 14.6 | |
| 150 ≤ n < 180 | 29.7 | |
| 180 ≤ n | 54.6 | |
| Respiratory rate (/minute) | ||
| n < 8 | 99.2 | |
| 8 ≤ n < 12 | 86.0 | |
| 12 ≤ n < 25 | 0.8 | |
| 25 ≤ n < 35 | 13.4 | |
| 35 ≤ n | 24.8 | |
| Body temperature (°C) | ||
| n < 35 | 98.3 | |
| 35 ≤ n < 36 | 79.0 | |
| 36 ≤ n < 38 | 1.7 | |
| 38 ≤ n < 40 | 10.2 | |
| 40 ≤ n | 50.2 | |
| Consciousness | ||
| Alert response | 1.9 | |
| Verbal response | 74.8 | |
| Painful response | 68.0 | |
| Unresponsive | 98.1 | |
| SBP, systolic blood pressure; DBP, diastolic blood pressure; weighted value, value according to the weighted value of the factor. |
| Factor | Range | Weighted value |
| Hemoglobin (g/dL) | ||
| n < 5 | 98.2 | |
| 5 ≤ n < 7 | 78.0 | |
| 7 ≤ n < 12 | 18.1 | |
| 12 ≤ n < 16 | 1.8 | |
| 16 ≤ n | 52.6 | |
| Hematocrit (%) | ||
| n < 15 | 66.9 | |
| 15 ≤ n < 25 | 39.2 | |
| 25 ≤ n < 36 | 12.7 | |
| 36 ≤ n < 46 | 1.2 | |
| 46 ≤ n < 60 | 27.7 | |
| 60 < n | 98.8 | |
| WBC count (103/µL) | ||
| n < 3.0 | 73.5 | |
| 3 ≤ n < 4.5 | 54.8 | |
| 4.5 ≤ n < 11.0 | 2.0 | |
| 11.0 ≤ n < 20.0 | 25.8 | |
| 20.0 ≤ n < 30.0 | 68.3 | |
| 30.0 ≤ n | 98.0 | |
| Prothrombin time (International normalised ratio (INR)) | ||
| n < 0.75 | 0.0 | |
| 0.75 ≤ n < 1.25 | 1.8 | |
| 1.25 ≤ n < 3.00 | 47.0 | |
| 3.00 ≤ n | 98.2 | |
| AST level (IU/L) | ||
| n < 36 | 2.3 | |
| 36 ≤ n < 60 | 39.3 | |
| 60 ≤ n < 100 | 68.5 | |
| 100 ≤ n < 300 | 77.4 | |
| 300 ≤ n | 97.7 | |
| ALT level (IU/L) | ||
| n < 36 | 2.1 | |
| 36 ≤ n < 60 | 46.8 | |
| 60 ≤ n < 100 | 65.1 | |
| 100 ≤ n < 300 | 74.7 | |
| 300 ≤ n | 97.9 | |
| Bilirubin level (mg/dL) | ||
| n < 1.2 | 1.6 | |
| 1.2 ≤ n < 2.0 | 48.2 | |
| 2.0 ≤ n < 6.0 | 60.7 | |
| 6.0 ≤ n < 9.0 | 95.0 | |
| 9.0 ≤ n | 98.4 | |
| Creatinine level (mg/dL) | ||
| n < 1.1 | 1.9 | |
| 1.1 ≤ n < 3.0 | 38.2 | |
| 3.0 ≤ n < 6.0 | 72.0 | |
| 6.0 ≤ n < 10.0 | 84.3 | |
| 10.0 ≤ n | 98.1 | |
| Amylase level (U/L) | ||
| n < 100 | 1.4 | |
| 100 ≤ n < 250 | 41.0 | |
| 250 ≤ n < 500 | 73.0 | |
| 500 ≤ n < 1000 | 88.8 | |
| 1000 ≤ n | 98.6 | |
| Sodium level (mmol/L) | ||
| n < 100 | 0.9 | |
| 100 ≤ n < 120 | 51.4 | |
| 120 ≤ n < 135 | 19.8 | |
| 135 ≤ n < 145 | 1.0 | |
| 145 ≤ n < 160 | 43.5 | |
| 160 ≤ n < 180 | 65.8 | |
| 180 ≤ n | 99 | |
| Potassium level (mmol/L) | ||
| n < 2.5 | 98.4 | |
| 2.5 ≤ n < 3.5 | 35.8 | |
| 3.5 ≤ n < 5.5 | 1.6 | |
| 5.5 ≤ n < 7.0 | 57.5 | |
| 7.0 ≤ n | 87.3 | |
| CRP level (mg/L) | ||
| n < 5 | 1.8 | |
| 5 ≤ n < 10 | 37.8 | |
| 10 ≤ n < 20 | 60.9 | |
| 20 ≤ n < 30 | 98.2 | |
| 30 ≤ n | 98.2 | |
| Glucose level (mg/dL) | ||
| n < 60 | 97.9 | |
| 60 ≤ n < 80 | 83.7 | |
| 80 ≤ n < 140 | 2.1 | |
| 140 ≤ n < 200 | 39.4 | |
| 200 ≤ n < 300 | 55.7 | |
| 300 ≤ n | 75.5 | |
| pH | ||
| n < 7.00 | 97.4 | |
| 7.00 ≤ n < 7.20 | 87.0 | |
| 7.20 ≤ n < 7.35 | 44.4 | |
| 7.35 ≤ n < 7.45 | 2.6 | |
| 7.45 ≤ n < 7.50 | 22.3 | |
| 7.50 ≤ n | 37.6 | |
| PaO2 (mmHg) | ||
| n < 40 | 94.5 | |
| 40 ≤ n < 60 | 63.6 | |
| 60 ≤ n < 80 | 32.7 | |
| 80 ≤ n | 5.5 | |
| Lactic acid level | ||
| n < 2 | 2.7 | |
| 2 ≤ n < 4 | 22.7 | |
| 4 ≤ n < 6 | 69.4 | |
| 6 ≤ n < 10 | 82.9 | |
| 10 ≤ n | 97.3 | |
| CK-MB level (ng/mL) | ||
| n < 3.6 | 1.8 | |
| 3.6 ≤ n < 5.0 | 49.6 | |
| 5.0 ≤ n < 10.0 | 51.0 | |
| 10.0 ≤ n < 30.0 | 65.8 | |
| 30.0 ≤ n | 98.2 | |
| Troponin T level (ng/mL) | ||
| n < 0.1 | 1.7 | |
| 0.1 ≤ n < 0.3 | 57.2 | |
| 0.3 ≤ n < 0.6 | 80.0 | |
| 0.6 ≤ n < 1.0 | 98.0 | |
| 1.0 ≤ n | 98.3 | |
| WBC, white blood cell; AST, aspartate aminotransferase; ALT, alanine aminotransferase; CRP, C-reactive protein; pH, hydrogen ion concentration; PaO2, partial pressure of arterial oxygen; CK-MB, creatine kinase myocardial band; weighted value, value according to the weighted value of the factor. |
2.3 Accuracy of WV in combination with initial factors
The sum of WVs for various combinations of factors that can be obtained in the early stages following presentation was calculated (Table 3). In addition, sensitivity, specificity, accuracy, positive predictive value, and negative predictive value were investigated by determining the cutoff values for hospitalization and discharge at the ED.
| Algorithms | Factors |
| A | Age, Sex, Past history, SBP, DBP, HR, RR, BT, AVPU |
| B | HBG, HCT, WBC, PT (INR), Bilirubin, AST, ALT, Creatinine, Amylase, Sodium, Potassium, CRP |
| Type 1 | A |
| Type 2 | A + PaO2, pH, Lactic acid |
| Type 3 | A + Glucose |
| Type 4 | A + Glucose, PaO2, pH, Lactic acid |
| Type 5 | A + B + Glucose |
| Type 6 | A + B + Glucose, PaO2, pH, Lactic acid |
| Type 7 | A + B + PaO2, pH, Lactic acid |
| Type 8 | A + B + Glucose, CK-MB, Troponin |
| Type 9 | A + B + PaO2, pH, Lactic acid, CK-MB, Troponin |
| Type 10 | A + B + Glucose, PaO2, pH, Lactic acid, CK-MB, Troponin |
| A, defined as immediate characteristics; B, blood sample results; SBP, systolic blood pressure; HR, heart rate/min; RR, respiratory rate; BT, body temperature; AVPU, alert/verbal/painful/unresponsive (index of consciousness); HBG, hemoglobin; HCT, hematocrit; WBC, white blood cell; PT, prothrombin time; CRP, C-reactive protein; Glucose, blood glucose; DBP, diastolic blood pressure; AST, aspartate aminotransferase; ALT, alanine aminotransferase; PaO2 partial pressure of arterial oxygen; pH, hydrogen ion concentration; CK-MB, creatine kinase myocardial band; INR: International normalised ratio. |
2.4 Accuracy of WV in combination with initial and blood factors
The sum of WVs in the various combinations of factors that can be obtained from immediate examination, POCT in the initial stage, and from blood tests after a certain period of time was calculated (Table 3). We also investigated the sensitivity, specificity, accuracy, positive predictive value, and negative predictive value to determine the cutoffs for hospitalization and discharge at the ED. In addition, we selected the most appropriate combination of factors with the highest accuracy that can be clinically useful for the highest number of patients.
2.5 Comparison between Rasch analysis and artificial intelligence analysis
The prediction of hospitalization and discharge of patients was analyzed and compared between Rasch analysis and AI using the same factors. From the data, the triage result, symptom, and gender columns were converted to binary columns: Triage result (hospitalization, death, intensive care unit (ICU): 1, discharge: 0), symptom (with symptoms: 1, none: 0), gender (male: 1, female: −1). Past history and AVPU columns were converted into one-hot encoding vectors and the remaining factors were normalized through the standard score (z-score), and values outside the standard score range (−4.0 ~ 4.0) were replaced with the standard score −4.0 or 4.0. Samples with at least one missing value among all columns were removed. The training set and the test set were randomly sampled and separated by 7:3. The MLP model consists of three hidden layers (128-128-128), dropout of each layer was applied, the activation function was ReLU, and the output layer was Sigmoid. The training hyper parameters of the two models were as follows (XGBooster model; classifier: XGBClassifier, booster: gbtree, learning_rate: 0.001, eval_metric: AUC, max_depth: 10, n_estimators: 200, positive_class_weight = ((number of negative samples) × 1.5)/(number of positive samples), MLP model; Optimizer: Adam, Batch size: 256, Learning Rate: 0.001, Loss: Cross Entropy, Class weight: (1/(neg or pos)) × (total/2.0)). In the learning process, we used sklearn’s GridSearchCV module to find the hyper-parameter value. 5-fold cross validation is performed on the training set separated from the test set. (XGBoost parameter space param_xgb = (“learning_rate”: (0.001, 0.005) “max_depth”: (10, 30, 50), “n_estimators”: (100, 200, 300, 500, 1000)). MLP parameter space param_mlp = (“learning_rate”: (0.0001, 0.0005, 0.001, 0.005))). In order to compare and verify the results of the Rasch analysis, the test set was fixed and the machine learning model was learned, and the training set of the machine learning model was separated into training: validation = 8:2 to go through the learning process. In addition, the prediction of hospitalization and discharge in the ED with AI was analyzed using direct raw clinical data values as well as the results of WV analysis [8].
2.6 Statistical analysis and software
Descriptive analysis was performed using SPSS v.22 software (SPSS Inc, Chicago, IL, USA). Rasch analysis was performed using Winsteps® (version 3.8.10) (Winsteps.com, Beaverton, OR, USA). AI results were evaluated using a gradient boosting tree (XGBoost) and multilayer perceptron (MLP).
3. Results
We investigated age, sex, past history, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, state of consciousness, hemoglobin level, hematocrit, WBC count, arterial pH, PaO2, and PT, AST, ALT, bilirubin, creatinine, amylase, sodium, potassium, CRP, blood sugar, lactic acid, CK-MB, and troponin T levels, obtained from 158,246 patients who visited the ED from 01 January 2016, to 31 December 2020. Each factor was assigned an interval for the value based on its normal and abnormal ranges. In addition, the effect of each factor on the severity of the patient was obtained by calculating the WV using Rasch analysis with the frequency of the factors. In addition, the appropriate cut-off values of hospitalization and discharge were determined using the sum of WVs of the combination of various factors for the prediction of hospitalization and discharge by statistical methods. Furthermore, the Machine Learning was used to classify the severity of patients according to the actual value of the factors and the WV of the factors, and the results were compared with the results of the Rasch analysis. The significance of the study was determined by sensitivity, specificity, accuracy, positive predictive value, negative predictive value, and AUC values compared with actual hospitalization and discharge data.
3.1 Patient characteristics
In the medical records of 158,246 patients (15 years of age or older), Type 1, representing a combination of factors that can be conveniently obtained in the early stages of the visit, was assessed in 154,383 patients. Among these patients, 38,963 were hospitalized, and 115,420 were discharged. In addition, Type 5, representing the combination of factors that can be obtained at the beginning of the visit and after a certain period of time, was studied in 78,552 patients, including 27,428 hospitalized and 51,124 discharged patients (Table 4).
| Type | Total patients | Adm. patients | Dis. patients | AUC (95% CI) | Cut-off values | Sensitivity(%) | Specificity (%) | Accuracy (%) | Positive predictive value (%) | Negative predictive value (%) |
| Type 1 | 154,383 | 38,963 | 115,420 | 0.611 (0.608–0.614) | 106.5 | 65.22 | 54.35 | 57.10 | 32.54 | 82.24 |
| 107.5 | 64.54 | 55.12 | 57.50 | 32.68 | 82.16 | |||||
| 108.5 | 63.87 | 56.27 | 57.91 | 32.64 | 81.75 | |||||
| 109.5 | 62.79 | 56.96 | 58.43 | 32.99 | 81.93 | |||||
| 110.5 | 62.50 | 57.24 | 58.56 | 33.04 | 81.89 | |||||
| Type 2 | 35,188 | 18,842 | 16,346 | 0.612 (0.606–0.618) | 174.5 | 68.85 | 49.39 | 59.81 | 61.06 | 57.90 |
| 175.5 | 68.41 | 49.97 | 59.84 | 61.18 | 57.84 | |||||
| 176.5 | 67.93 | 50.62 | 59.89 | 61.33 | 57.80 | |||||
| 177.5 | 67.45 | 51.10 | 59.85 | 61.39 | 57.66 | |||||
| 178.5 | 66.96 | 51.49 | 59.77 | 61.41 | 57.48 | |||||
| Type 3 | 93,089 | 35,005 | 58,084 | 0.685 (0.682–0.689) | 132.5 | 61.05 | 67.06 | 64.80 | 52.77 | 74.08 |
| 133.5 | 60.40 | 67.60 | 64.89 | 52.91 | 73.91 | |||||
| 134.5 | 59.70 | 68.72 | 65.32 | 53.49 | 73.88 | |||||
| 135.5 | 59.19 | 69.22 | 65.45 | 53.68 | 73.78 | |||||
| 136.5 | 58.53 | 69.80 | 65.56 | 53.88 | 73.64 | |||||
| Type 4 | 32,408 | 18,030 | 14,378 | 0.627 (0.621–0.633) | 198.5 | 67.59 | 53.41 | 61.30 | 64.53 | 56.79 |
| 199.5 | 67.30 | 53.82 | 61.32 | 64.63 | 56.76 | |||||
| 200.5 | 66.85 | 54.27 | 61.27 | 64.70 | 56.63 | |||||
| 201.5 | 66.41 | 54.63 | 61.19 | 64.73 | 56.47 | |||||
| 202.5 | 65.97 | 55.06 | 61.13 | 64.80 | 56.33 | |||||
| Type 5 | 78,552 | 27,428 | 51,124 | 0.767 (0.764–0.771) | 299.5 | 64.14 | 77.13 | 72.60 | 60.08 | 80.04 |
| 300.5 | 63.89 | 77.41 | 72.69 | 60.27 | 79.98 | |||||
| 301.5 | 63.67 | 77.66 | 72.77 | 60.45 | 79.94 | |||||
| 302.5 | 63.44 | 77.88 | 72.84 | 60.61 | 79.88 | |||||
| 303.5 | 63.13 | 78.11 | 72.88 | 60.74 | 79.79 | |||||
| Type 6 | 22,535 | 13,008 | 9527 | 0.773 (0.767–0.779) | 372.5 | 69.37 | 71.48 | 70.26 | 76.86 | 63.08 |
| 373.5 | 69.13 | 71.75 | 70.24 | 76.97 | 63.00 | |||||
| 374.5 | 68.90 | 72.15 | 70.28 | 77.16 | 62.95 | |||||
| 375.5 | 68.53 | 72.35 | 70.14 | 77.19 | 62.74 | |||||
| 376.5 | 68.31 | 72.58 | 70.12 | 77.28 | 62.65 | |||||
| Type 7 | 22,864 | 13,186 | 9678 | 0.774 (0.768–0.780) | 350.5 | 69.64 | 71.55 | 70.45 | 76.94 | 63.37 |
| 351.5 | 69.35 | 71.90 | 70.43 | 77.08 | 63.26 | |||||
| 352.5 | 69.09 | 72.19 | 70.40 | 77.20 | 63.16 | |||||
| 353.5 | 68.80 | 72.39 | 70.32 | 77.25 | 63.00 | |||||
| 354.5 | 68.56 | 72.65 | 70.29 | 77.35 | 62.91 | |||||
| Type 8 | 40,380 | 16,893 | 23,487 | 0.804 0.799–0.808) | 315.5 | 71.20 | 76.05 | 74.02 | 68.14 | 78.59 |
| 316.5 | 71.04 | 76.30 | 74.10 | 68.32 | 78.56 | |||||
| 317.5 | 70.85 | 76.51 | 74.14 | 68.45 | 78.49 | |||||
| 318.5 | 70.57 | 76.71 | 74.14 | 68.55 | 78.38 | |||||
| 319.5 | 70.34 | 76.96 | 74.19 | 68.71 | 78.29 | |||||
| Type 9 | 18,149 | 10,580 | 7569 | 0.795 (0.789–0.802) | 354.5 | 74.97 | 69.77 | 72.80 | 77.61 | 66.60 |
| 355.5 | 74.68 | 70.10 | 72.77 | 77.74 | 66.45 | |||||
| 356.5 | 74.49 | 70.37 | 72.77 | 77.84 | 66.37 | |||||
| 357.5 | 74.23 | 70.54 | 72.69 | 77.89 | 66.20 | |||||
| 358.5 | 73.97 | 70.83 | 72.66 | 77.99 | 66.06 | |||||
| Type 10 | 17,961 | 10,475 | 7486 | 0.795 (0.788–0.801) | 398.5 | 69.64 | 74.97 | 71.86 | 79.56 | 63.83 |
| 399.5 | 69.45 | 75.17 | 71.83 | 79.65 | 63.75 | |||||
| 400.5 | 69.24 | 75.47 | 71.84 | 79.80 | 63.68 | |||||
| 401.5 | 69.02 | 75.69 | 71.80 | 79.89 | 63.58 | |||||
| 402.5 | 68.80 | 75.79 | 71.72 | 79.91 | 63.45 | |||||
| Adm, admission; Dis, discharge; AUC, area under the curve; CI, confidence interval. |
3.2 The accuracy of the prediction for hospitalization and emergency department discharge based on the initial factors of the visit
The AUC values of types 1, 2, 3 and 4 among the different combinations using 13 factors immediately available for examination and from POCT at the beginning of the ED visit were 0.611, 0.612, 0.685, and 0.627, respectively, showing a slightly increasing tendency in the combinations of more factors, but this was not statistically significant (Table 4). Therefore, Type 1, which combined the nine factors that can be most easily obtained immediately following presentation, was determined. When setting the cut-off value to 108.5, the sensitivity was 63.87%, and the specificity was 56.27%, but the negative predictive value was 81.75%, which helps predict safe discharge (Fig. 1).
Fig. 1.ROC curve of admission and discharge using various combinations of factors in Type 1. The AUC value of Type 1 was 0.611, consisting of nine factors available by immediate examination at presentation to the emergency department. When the cut-off value was set to 108.5, the sensitivity of hospitalization was 63.87% and negative predictive value of discharge from a hospital was 81.75%. ROC, receiver operating characteristic.
3.3 The accuracy of the prediction for hospitalization and emergency department discharge based on the initial and blood factors
The AUC values of types 5, 6, 7, 8, 9 and 10 were 0.767, 0.773, 0.774, 0.804, 0.795 and 0.795, respectively, consisting of 9 factors immediately obtained by initial examination, 4 factors immediately obtained by POCT, and 14 factors obtained by the blood test after a certain period of time (Table 4). Therefore, Type 5 was determined because it was easy to use in combination with 22 factors measured in most emergency patients, with a cut-off value of 301.5, sensitivity of 63.67%, and specificity of 77.69%; however, the negative predictive value of emergency patients was 79.94% (Fig. 2).
Fig. 2.ROC curve of admission and discharge using various combinations of factors in Type 5. AUC value of Type 5 was 0.767, consisting of nine factors immediately obtained by initial examination, one factor immediately obtained by POCT, and 12 factors obtained by the blood tests after a certain period of time from presentation to the emergency department. When the cutoff value was set to 301.5, the sensitivity of hospitalization was 63.77% and negative predicitive value of discharge from a hospital was 79.94%. ROC, receiver operating characteristic; POCT, point-of-care testing.
3.4 Comparison between Rasch analysis and artificial intelligence analysis
To evaluate the usefulness of predicting the hospitalization and discharge of patients using Rasch analysis with factors selected by medical experts, we compared them with the predictions of hospitalization and discharge of patients’using AI of Types 1 and 5. The usefulness of XGBoost AI and MLP AI was investigated using the WVs obtained from Rasch analysis and raw data, respectively. In XGBoost AI, the algorithms provided by the AI that were tested on data from 46,313 patients after training on the data from 108,070 patients in Type 1 using the WVs showed an AUC value of 0.661. In addition, the AI algorithms showed an AUC value of 0.727 in Type 5, which was tested on data from 23,565 patients after training on the data from 54,987 patients. The AUC values using raw data were 0.676 for Type 1 and 0.734 for Type 5 (Table 5). However, the AUC values in the WVs using the MLP AI were 0.728 for Type 1 and 0.810 for Type 5, and the AUC values in the raw data were 0.750 and 0.845 for Type 1 and Type 5, respectively. This was similar to or somewhat higher than the AUC values of 0.611 and 0.767 for Types 1 and 5 in the classification by medical experts using Rasch analysis. The factors that significantly affected the results were age and past medical history in the Type 1 group (Fig. 3) and CRP level in the Type 5 group (Fig. 4). As AI has disadvantages that may partially explain the process, it is difficult to know the internal progress and the prediction of hospitalization and discharge of patients using Rasch analysis, which can adjust the WV of each factor, has medical usefulness. In the future, it is expected to be classified based on a combination of AI and the selection of factors based on medical decision making by experts.
| Algorithm | Type | Train | Test | Data Type | AUC | Recall (Sensivity) | Specificity | Accuracy | Precision (Positive Predictive value) | Negative predictive value |
| XGBoost | Type 1 | 108,070 | 46,313 | WV | 0.661 | 0.750 | 0.571 | 0.616 | 0.371 | 0.871 |
| Raw | 0.676 | 0.702 | 0.649 | 0.663 | 0.420 | 0.866 | ||||
| Type 5 | 54,987 | 23,565 | WV | 0.727 | 0.766 | 0.687 | 0.715 | 0.568 | 0.845 | |
| Raw | 0.734 | 0.756 | 0.729 | 0.739 | 0.600 | 0.848 | ||||
| MLP | Type 1 | 108,070 | 30,875 | WV | 0.728 | 0.314 | 0.942 | 0.783 | 0.645 | 0.803 |
| (validation: 15,438) | Raw | 0.750 | 0.566 | 0.813 | 0.751 | 0.506 | 0.847 | |||
| Type 5 | 54,987 | 15,709 | WV | 0.810 | 0.651 | 0.829 | 0.767 | 0.671 | 0.919 | |
| (validation: 7856) | Raw | 0.845 | 0.660 | 0.863 | 0.792 | 0.721 | 0.825 | |||
| XGBoost, extreme gradient boosting tree; MLP, multilayer perceptron; weighted value, value according to weighted value of factor; raw value, value according to raw data; AUC, area under the curve; WV, weighted value. |
Fig. 3.Distribution of factors influencing admission and discharge decision using artificial intelligence analysis in Type 1. The factors affecting the results were significantly influenced by age and past history in Type 1. weighted value, value according to weighted value of factor; Raw value, value according to raw data; SAGE_V, the factor affecting the result was influenced by age; SRR_V, respiratory rate; SBT_V, body temperature; SHR_V, heart rate; SPASTHIST_V2, past history (hypertension); SSEX_V, SEX; SAVPU_VA, conscious (alert); SPASTHIST_V5, past history (malignancy); SPASTHIST_V1, past history (diabetic); SDBP_V, diastolic blood pressure; SSBP_V, systolic blood pressure; SPASTHIST_V3, past history (tuberculosis); SPASTHIST_V6, past history (chronic obstructive pulmonary disease); SAVPU_VP, conscious (painful response); SPASTHIST_V4, past history (hepatitis); SAVPU_VV, conscious (verbal response); SAVPU_VU, conscious (unresponsive); SPASTHIST_SD, weighted value of past history; SAGE_SD, weighted value of age; SHR_SD, weighted value of heart rate; SSEX_SD, weighted value of sex; SSBP_SD, weighted value of systolic blood pressure; SAVPU_SD, weighted value of AVPU; SRR_SD, weighted value of respiratory rate; SBT_SD, weighted value of body temperature; SDBP_SD, weighted value of diastolic pressure.
Fig. 4.Distribution of factors influencing admission and discharge decision using artificial intelligence analysis in Type 5. The factors affecting the results were significantly influenced by C-reactive protein (CRP) level in Type 5. weighted value, value according to weighted value of factor; raw value, value according to raw data; SCRP_V, the factor affecting the result was influenced by CRP level; SAGE_V, age; SHCT_V, HCT (hematocrit); SAST_V, AST (aspartate aminotransferase) level; SWBC_V, WBC (white blood cell); SPTINR_V, PT (prothrombin time (INR)); SPASTHIST_V2, past history (hypertension); SSEX_V, sex; SAVPU_VA, conscious (alert); SSODIUM_V, sodium level; SRR_V, respiratory rate; STBIL_V, bilirubin level; SHGB_V, hemoglobin level; SBT_V, body temperature; SCR_V, creatinine level; SPOTASS_V, potassium level; SBS_V, blood sugar (glucose) level; SPASTHIST_V3, past history (tuberculosis); SALT_V, ALT (alanine aminotransferase) level; SHR_V, heart rate; SCRP_SD, weighted value of CRP; SPASTHIST_SD, weighted value of past history; SHGB_SD, weighted value of hemoglobin level; SAGE_SD, weighted value of age; SAST_SD, weighted value of AST (aspartate aminotransferase) level; SSEX_SD, weighted value of sex; SWBC_SD, weighted value of WBC; SAVPU_SD, weighted value of AVPU; SSODIUM_SD, weighted value of sodium; SHCT_SD, weighted value of HCT; STBIL_SD, weighted value of bilirubin level; SRR_SD, weighted value of respiratory rate; SBS_SD, weighted value of blood sugar (glucose) level; SPTINR_SD, weighted value of PT (INR); SCR_SD, weighted value of creatinine level; SBT_SD, weighted value of body temperature; SAMYLASE_SD, weighted value of amylase level; SSBP_SD, weighted value of systolic blood pressure; SHR_SD, weighted value of heart rate; SALT_SD, weighted value of ALT level.
4. Discussion
Severity triage of emergency patients is essential for rapid treatment and life-saving of severe patients by facilitating the reduction of overcrowding and efficient deployment of resources and medical staff [1]. Thus, in many countries, severity triage of emergency patients has been implemented. Various countries employ different severity triage systems, including ESI, CTAS, MTS, a modified version of MTS, and ATS. In addition, various triage systems based on reference standards have been used [2, 9, 10]. These severity triage systems determine the rapid treatment ranking according to the patient’s condition. However, from the hospital’s perspective, hospitalization and discharge are determined when emergency patients visit, and if it is difficult to secure hospital beds, patients should be transferred to other medical institutions. Consequently, the existing severity triage lacks usefulness and causes disputes between patients and medical staff [11]. In overcrowded hospitals, the development of a safe discharge system to prioritize hospitalization will improve patient safety and reduce repeat visits [12]. It has also been reported that clinical evaluation is superior to the use of various severity triage systems [13].
To improve the accuracy and usefulness of the severity triage systems, repetitive and diverse training is required to resolve the problem of discrepancies in the triage results and to achieve quick and accurate severity triage [14, 15, 16]. In addition, severity triage systems are being developed in a variety of ways, including reducing wait time to triage [17], developing computer triage programs [18, 19], and making pre-hospital emergency calls [20]. These all have the potential to improve patient prognosis and increase the accuracy of triage. The World Health Organization reports various guidelines on the triage system and management of emergency patients, while also highlighting the importance of new research for improved guidelines [21].
Korea’s emergency medical system comprises emergency rooms, local emergency medical institutions, local emergency medical centers, and regional emergency medical centers. However, owing to the ambiguous functions of the institutions, approximately 10 million patients per year are treated at 150 local emergency medical centers and 38 regional emergency medical centers. The KTAS used in Korea comprises a five-tier classification system. The first level requires immediate treatment by the medical staff due to immediate threat to life, and the second level implies that prognosis can be improved with immediate treatment and indicates typically patients requiring hospitalization. Although the third stage is an emergency, hospitalization or outpatient treatment will usually be the result, leading to overcrowding of emergency centers, such as waiting for hospitalization. In addition, decisions of admissions or discharge differ between institutions depending on protocols and hospital bed capacity.
Recently, the usefulness of the statistical method using Rasch analysis has been demonstrated [3]. This presents the importance of the frequency of each factor in the data as a WV. From a medical perspective, Rasch analysis is useful not only in screening for psychiatric diseases [4] but also in assessing the prognosis of chronic diseases [5, 22]. Therefore, we used the WV of Rasch analysis, which can be quantified based on the importance of useful factors obtained from EDs, to determine hospitalization and outpatient treatment.
The measurement factor was determined to quantify the importance of each factor by selecting 27 factors that are measured in most EDs in Korea, including 9 factors that can be obtained immediately by examination, 4 factors that can be obtained immediately by POCT, and 14 factors that can be obtained from blood tests. The WV of each factor was calculated using Rasch analysis after evaluation by dividing them into normal and abnormal ranges based on the expertise of medical personnel. The distribution of the WV of the factors was mostly parabolic, showing a pattern of increased risk when low or high, matching the characteristics of biological homeostasis.
Types 1, 2, 3 and 4 consisted of a combination of 13 factors, including 9 factors that can be obtained immediately by examination and 4 factors that can be obtained immediately by initial POCT. Although the AUC values of the four groups were similar, the Type 1 group, which can be evaluated with only nine factors measurable by examination, had a negative predictive value of approximately 81% and could be evaluated initially without invasive measurements of blood glucose levels and arterial blood gas analysis. In addition, for types 5, 6, 7, 8, 9 and 10, which consisted of a combination of various factors used for accurate evaluation of hospitalization and discharge, the AUC values were similar, in the range of 0.767–0.795. In Korea, the Type 5 group, which is a combination of blood test parameters (without arterial blood gas analysis and cardiac enzyme tests) and initial measurement factors, is considered the simplest to use. Therefore, Types 1 and 5 will help in the efficient operation of the ED by predicting the hospitalization and discharge decisions for patients. In addition, the criteria for hospitalization and discharge vary depending on the hospital’s environment and capabilities; therefore, adjusting the appropriate cut-off values for hospitalization and discharge would be more useful for the operation of the ED in each hospital.
Recently, owing to the development of AI, various studies have compared triage systems using AI to the existing triage system. Machine learning-based electronic triage is more accurate than ESI for hospitalization, discharge, critical care, and emergency procedures incorporating patient vital signs, chief complaints, and medical history [9]. It has also been reported that repeated visits to the ED are an important factor for obtaining high accuracy when triaging patients [23]. As many factors such as demographic data including sex and age of patients, vital signs, chief complaint, and past history were used, the accuracy of results using AI was higher. Compared to ESI, the triage through AI is more accurate, with less undertriage in ESI 3–5 (low severity) and less overtriage in ESI 1–3 (high severity) [7].
We selected a model that explains the characteristics and distribution of this research data well by comparing the decision tree type algorithm and an algorithm using the neural network, which are among of the various machine learning algorithms types commonly used for data-based problem solving. Therefore, the experiments were conducted using XGBoost, a representative model of the decision tree, and MLP, a basic model of the neural network. And we conducted a study using AI for classification based on hospitalization and discharge using Type 1, the initial triage as a combination of immediately obtainable factors, and Type 5, a later triage as a combination of factors that can be obtained immediately and factors that can be obtained from blood tests. The accuracy of hospitalization and discharge was evaluated using the WVs according to the range of factors used in the Rasch analysis and actual measurements of each factor. In the severity triage using AI, MLP showed somewhat higher accuracy than Rasch analysis, as examined in Type 1 and Type 5, but the difference between the WVs and actual values of the factors was minimal. In comparison between AI algorithms, the performance of complex deep learning algorithm (MLP) showed higher accuracy than simple machine learning algorithms (XGBoost), and it is estimated that better results will be achieved if sufficient data is provided. As the AI method analyzed the effect of factors on the results of hospitalization and discharge, age and CRP levels were considered to have had a significant impact. Although it is important to choose factors based on the impact of the factors used to determine hospitalization and discharge, nine factors were selected for the initial triage because it is easy to select factors that can be obtained from most emergency centers.
The effects and accuracy of factors used in KTAS in Korea were studied, and the pain score was evaluated differently by the medical staff, who reported that modification was needed [11]. In our study, we supported the adequacy of the selected factors by excluding the pain score. The composition of the combination of factors is considered to be easy to use. Types 1 and 5 use a small number of factors that can be easily obtained, and it is expected to improve if the combination of factors is applied to patients with similar symptoms and not used on all patients. In addition, the accuracy of the cut-off values for hospitalization and discharge by Rasch analysis will be improved if the cut-off values are adjusted according to the capabilities and environment of the hospital. The severity triage of AI using WVs also showed similar results to those of Rasch analysis, and if hospitalization and discharge decisions using Rasch analysis are used as medical assistance measures, it will help alleviate overcrowding.
This study has limitations. As this study was conducted with data from a single institution, multi-institutional research is needed in the future. However, it will be useful because each hospital can adjust the cut-off values for hospitalization and discharge. Additionally, the value of each factor was measured by a medical device at one hospital. Although there may be some differences in the value of the factor or addition of the factor at another hospital in another country, it can be used because the WV can be corrected according to the value of the factor. It is somewhat less accurate for direct clinical use. However, it can be used as a means of medical assistance by medical staff, and when combined with various methods, which can be conducted in the future, such as analysis of each symptom, can provide high accuracy. Therefore, this study is sufficient as a cornerstone. The usefulness of Rasch analysis was compared to that of AI, but the accuracy was similar or somewhat low. In addition, MLP AI method, not deep learning AI, was used, and cross-validation was not performed. However, a meaningful result was obtained using Rasch analysis. In the future, research on AI is required. Finally, this was a retrospective study that used existing data. However, if the program of this study is currently entered and used in a hospital information computer system, it will be possible to prove its effectiveness via a prospective study in the future by comparing patient wait times and satisfaction as results.
5. Conclusions
The rapid prediction of hospitalization and discharge using Rasch analysis in ED was highly accurate when combined with more efficient factors, similar to the analysis of artificial intelligence. Therefore, it will contribute to the effective operation of ED in favor of providing appropriate treatment for patients and eliminating overcrowding. It is expected to be more useful if it is computerized and applied to the hospital information system in the future.
Availability of data and materials
Not applicable.
Author contributions
SJP, SHC and DJS—implemented conceptualization and provided ideas. SHC, DJS, JHP, JHS and HJC—conducted data curation, formal analysis and fund acquisition. SHL, BJK and KHA—investigated the data and proposed Rasch analysis method execution and compared with AI. GGK, WSC and KNK—performed project management and provided equipment for Rasch analysis statistics and verification with AI. SHC—performed supervision, verification and visualization. SJP and SHC—wrote the manuscript. All authors read, reviewed, edited and approved the final manuscript.
Ethics approval and consent to participate
This study was carried out with the approval of the Korea University Guro Hospital Institutional Review Board (2021GR0239). Ethics committee approved waiver of informed consent.
Acknowledgment
We are grateful to our colleague Qiuyu C and Bang IY for participation in this study and helpful discussion and we appreciated Min CG and Park SJ who conducted a statistical review.
Funding
This work was supported by National IT Industry Promotion Agency (NIPA) and Korean Health Industry Development Institute (KHIDI) grant funded by the Korea government (MIST, MOHW) (ITAS1906210110010001000500200) and partially supported by a Korea University Grant.
Conflict of interest
The authors declare no conflict of interest.
