Signa Vitae. 2023; 19(6): 144-151. doi: 10.22514/sv.2023.108
Original Research

Effect of an artificial-intelligent chest radiographs reporting system in an emergency department

Do Hyeok Yoon1, Sejin Heo1,2, Jae Yong Yu3, Se Uk Lee1, Sung Yeon Hwang1, Hee Yoon1, Tae Gun Shin1, Gun Tak Lee1, Jong Eun Park1, Hansol Chang1,2, Taerim Kim1, Won Chul Cha1,2,*,

1Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, 06351 Seoul, Republic of Korea

2Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, 06355 Seoul, Republic of Korea

3Department of Biomedical Systems Informatics, Yonsei University College of Medicine, 03722 Seoul, Republic of Korea

*Corresponding Author(s):wc.cha@samsung.com (Won Chul Cha)

History Submitted: 09 March 2023 | Accepted: 25 April 2023 | Published: 08 November 2023
Copyright:  ©2023  The Author(s). Published by MRE Press.
This is an open access article under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/).

Collapse table of contents

Abstract

Though chest radiography is a first-line diagnostic tool in the emergency department (ED), interpretation has a high error rate. We aimed to evaluate the usability and acceptability of deep learning-based computer-aided detection for chest radiography (DeepCADCR) in an ED environment. We conducted a single-institution survey of emergency physicians (EPs) who had used DeepCADCR (Lunit INSIGHT Chest Xray (CXR), version 3.1.4.1) as part of their ED workflow for at least three months. We developed 22 questions that assessed the subscales of effectiveness, efficiency, safety, satisfaction, and reliability. A seven-point Likert agreement scale was used to rate the responses. A total of 23 EPs who completed the survey was enrolled in the study. When averaged by subscale, satisfaction scores were highest (mean 4.71, standard deviation (SD) 1.43), and safety scores were lowest (mean 4.3, SD 0.72). When scores were converted to acceptability, the total average acceptance of DeepCADCR was 86.0%, with higher scores in ED residents than ED specialists for all subscales. Use of DeepCADCR in the ED workflow was well accepted by EPs.

Keywords:Artificial intelligence;Deep learning;Chest radiography;Emergency department;Survey;Computer-aided detection
PDF(916.55 kB)|EndNote (RIS)|BibTeX|RefMan|RefWorks

Cite this article

Do Hyeok Yoon, Sejin Heo, Jae Yong Yu, Se Uk Lee, Sung Yeon Hwang, Hee Yoon, Tae Gun Shin, Gun Tak Lee, Jong Eun Park, Hansol Chang, Taerim Kim, Won Chul Cha. Effect of an artificial-intelligent chest radiographs reporting system in an emergency department. Signa Vitae. 2023; 19(6): 144-151. doi: 10.22514/sv.2023.108

1. Introduction

Chest radiography is a first-line diagnostic tool in the emergency department (ED) [1, 2]. However, interpretation has a high error rate (up to 22%) [3]. Moreover, interpretation of Chest xray (CR) images of patients requiring immediate procedures, such as pneumothorax or large pleural effusion, is often delayed [4, 5]. Although it is recommended that radiologists interpret all diagnostic radiology images in the ED [6, 7], coverage by radiologists is limited, especially on nights or weekends [8].

Recently, there have been major breakthroughs in the development of deep learning-based computer-aided detection of chest radiography (DeepCADCR) [6, 9, 10, 11]. Several studies have demonstrated that DeepCADCR can contribute to prompt and accurate interpretation as well increased work efficiency [12, 13, 14, 15]. These benefits have been studied mostly in simulated studies with radiologists [14, 16, 17]. Implementation of DeepCADCR by non-radiologists in a real clinical environment has only been investigated by a single study [18].

Although promising, the impact of DeepCADCR for EPs in real practice remains unclear [19]. Patients present various symptoms and signs in the ED environment, and timely interpretation of all CRs and prompt decision-making can be difficult [20, 21]. Moreover, CR conducted in the ED generally was interpreted by radiologists after ED care was complete and was not used for ED decisions. Implementation of DeepCADCR in the ED has been expected to result in more accurate and prompt interpretations, but its usability and acceptability have not been evaluated. Therefore, we aimed to evaluate the usability and acceptability of DeepCADCR in the clinical workflow of EPs.

2. Methods

2.1 Study design

We conducted a survey study of EPs in a tertiary academic center ED in Seoul, Korea. The survey was carried out after the EPs had used DeepCADCR in their ED workflow for three months.

2.2 Study settings

The ED of a tertiary academic hospital in a metropolitan area with an average of 250 ED visits per day implemented DeepCADCR from June 2022 to September 2022. When patients visit this ED, EPs examine the patient and then transcribe initial laboratory orders, including CR if indicated. The EP interprets the CR images and consults with a radiologist if necessary. Radiologists are stationed in the hospital 24–7 and interpret CR images for ED patients, but it usually takes hours for specific consults.

2.3 Radiology, Picture archiving and communication system (PACS) and the DeepCADCR system

Chest radiographs were obtained with the patients in the posteroanterior projection or in the supine position according to condition. All CRs were obtained using an XGEO GC80 (Digital Radiography System, Samsung Medison, Seoul, South Korea) or GM60A-32 (Mobile Digital Radiography System, Samsung Medison, Seoul, South Korea) digital radiography system.

DeepCADCR (Lunit INSIGHT CXR, version 3.1.4.1) was implemented to assist the interpretation of ED CRs with a focus on nine abnormal findings: pulmonary nodules, calcification, fibrosis, pneumothorax, pleural effusion, atelectasis, pneumothorax, cardiomegaly, and consolidation. The system provides a probability score between 0% and 100%, with a heat map of each original chest radiograph to identify the location of the abnormality when the probability score is 15% or greater.

Routine ED practice during the study period is illustrated in Fig. 1. When CR was performed, the DeepCADCR system received the image from PACS storage and presented results in the electronic medical records (EMR) integrated view within a few seconds. Results were presented in the form of an INSIGHT map and/or INSIGHT report. Location(s) of abnormalities and abnormality scores were provided in the heat map of each chest radiograph. The INSIGHT report includes the anatomical location of the detected lesion along with information about the INSIGHT map.

Implementation of computer-aided detection system on chest radiograph into emergency department workflow. PACS, picture archiving and communication system; EMR, electronic medical records; CXR, chest Xray.

Fig. 1.Implementation of computer-aided detection system on chest radiograph into emergency department workflow. PACS, picture archiving and communication system; EMR, electronic medical records; CXR, chest Xray.

Original CRs were examined side by side with INSIGHT maps by EPs to receive assistance from DeepCADCR in real time. An example image of a DeepCADCR INSIGHT map is provided in Fig. 2.

Example images of the computer aided detection system on chest radiograph. Heat map on abnormal findings. The system provided abnormality score below the chest radiograph with probability score. When probability score as 15% or greater, the system interpreted a chest radiograph as an abnormal.

Fig. 2.Example images of the computer aided detection system on chest radiograph. Heat map on abnormal findings. The system provided abnormality score below the chest radiograph with probability score. When probability score as 15% or greater, the system interpreted a chest radiograph as an abnormal.

2.4 Participants

Fig. 3 describes the inclusion flow of study participants. ED residents and board-certified ED specialists affiliated with the ED were enrolled as participants. Only EPs with clinical experience with DeepCADCR during the study period were included; EPs who were on rotation in other clinical departments were excluded.

Flow diagram of study participants. AI, artificial intelligence.

Fig. 3.Flow diagram of study participants. AI, artificial intelligence.

2.5 Survey development

There is as no consensus on standards for assessing deep learning-based clinical decision support systems. We developed a systematic questionnaire format based on review of previous studies [18, 19]. A total of 22 questions was developed to assess effectiveness, efficiency, safety, satisfaction and reliability. A 7-point Likert agreement scale was used to rate the responses as “Strongly disagree = 1”, “Disagree = 2”, “Somewhat disagree = 3”, “Neither agree nor disagree = 4”, “Somewhat agree = 5”, “Agree = 6”, or “Strongly agree = 7”. Averages of scores were calculated to determine the overall user experience, along with the system usability scale (SUS). More specific questions were developed for each feature of the DeepCADCR to assess the level of agreement among EPs and the impact of the CAD system on clinical practice.

2.6 Statistical analysis

All continuous variables are reported as mean (SD) or median (IQR). Categorical variables are described as number and percentage. We calculated Cronbach’s alpha to measure the internal consistency of the developed questions. We defined a score of 4 or higher as an “acceptable” response. For all statistical analyses, p < 0.05 was considered statistically significant. Statistical analyses were performed using R software (version 4.1.2, R Foundation for Statistical Computing, Boston, Messachusetts, USA).

3. Results

3.1 Participants

A total of 31 EPs used DeepCADCR during the study period. We excluded 8 EPs who left or who rotated to other departments. Finally, 23 EPs were enrolled and completed the survey. Table 1 describes the baseline characteristics of participants. Average age of the participants was 33.2 years, and 12 (52.2%) participants were female. Fifteen (65.2%) participants were emergency residents, and 10 (43.5%) participants had more than 5 years of work experience.

Table 1.Baseline characteristics of the study participants.
VariablesParticipants (N = 23)
Age, mean (SD)33.2 (7.4)
Sex, n (%)
Male11 (47.8)
Female12 (52.2)
Physician experience, n (%)
Resident15 (65.2)
Specialist8 (34.8)
Work years, n (%)
≤513 (56.5)
>510 (43.5)
Experience with AI clinical decision support system, n (%)
Yes11 (47.8)
No12 (52.2)
Experience with attending a lecture or seminar regarding medical AI, n (%)
Yes9 (39.1)
No14 (60.9)
AI, artificial intelligence; SD, standard deviation.

3.2 Chest radiograph interpretation by DeepCADCR

A total of 19,649 patients visited our ED during the study period. Of these, 2882 (14.7%) were younger than 19 years and 5373 (27.3%) did not undergo CR. As a result, a total of 14,745 CR images from 11,394 ED patients were collected during the study period.

The DeepCADCR interpreted 6138 (41.6%) CR images as normal and 8607 (58.3%) as abnormal. The proportions of specific abnormal findings are shown in Supplementary Fig. 1. The most common abnormal finding was pulmonary nodule (24.5%), followed by consolidation (22.8%), fibrosis (16.4%), and pleural effusion (13.2%).

3.3 User experience with DeepCADCR

The full survey questions are listed in Table 2. Response distribution for each subscale of user experience with DeepCADCR is presented in Supplementary Fig. 2. When averaged by subscale, satisfaction scores were highest (mean 4.71, SD 1.43), and safety scores were lowest (mean 4.3, SD 0.72). Average user experience scores for each subscale are presented in Fig. 4.

Survey questions and acceptance level of DeepCADCR by participants are presented in Table 2. When scores were converted to acceptability, the total average acceptance was 86.0%.

Table 2.Acceptability of DeepCADCR for chest radiograph interpretation.
Survey questionsAcceptability (%)
Effectiveness80.4
Q1. My chest CR interpretation improved after CADCR implementation.69.6
Q2. DeepCADCR helped plan future diagnoses and treatment of ER patients.78.3
Q3. DeepCADCR made me more confident in my reading.91.3
Q4. Overall, CADCR helped me interpret chest images of ER patients.82.6
Efficiency72.8
Q5. It is convenient that the DeepCADCR reading is shown along with the image.91.3
Q6. It took more time to read the CR after DeepCADCR implementation.26.1
Q7. Due to DeepCADCR, I was able to reduce the number of requests for CR readings by radiologists.43.5
Q8. DeepCADCR improved the effectiveness of care overall.82.6
Safety69.5
Q9. I was able to recommend appropriate outpatient follow-up for abnormal findings incidentally discovered by CADCR.69.6
Q10. DeepCADCR readings may lead to additional tests for patients that they would not have undergone without CADCR.60.9
Q11. DeepCADCR may delay patient care.43.5
Q12. DeepCADCR can cause unnecessary confusion.26.1
Q13. DeepCADCR was able to reduce the number of missed cases requiring emergency treatment.69.6
Q14. Implementation of DeepCADCR in the ER could increase patient safety overall.87.0
Satisfaction85.9
Q15. The benefit of the DeepCADCR system is worth the cost.82.6
Q16. I am willing to continue using the DeepCADCR reading system.87.0
Q17. I would recommend DeepCADCR use to my colleagues and other EPs.87.0
Q18. Overall, I am satisfied with the performance of the DeepCADCR reading system.87.0
Reliability76.1
Q19. I have a good understanding of the DeepCADCR reading algorithm.34.8
Q20. DeepCADCR readings were easy to interpret.95.7
Q21. I trust the DeepCADCR readings.82.6
Q22. DeepCADCR readings were mostly consistent with my readings.91.3
Average86.0
DeepCADCR, deep learning-based computer-aided detection system for chest radiographs. Scores over 4 were regarded as acceptance of DeepCADCR. Q6, Q11, Q12 were negative items. The 100-value was used for average acceptability. CR, chest radiography; EPs, emergency physicians; ER, emergency room.
Summary of user experience responses.

Fig. 4.Summary of user experience responses.

3.3.1 Effectiveness and efficiency

Effectiveness of DeepCADCR was 80.4%, and efficiency was 72.8%. In terms of effectiveness, the statement “DeepCADCR made me more confident in my reading” (Q3) received the highest acceptance. In the efficiency subscale, participants were least likely to agree with DeepCADCR to not request a CR reading by a radiologist (Q7, 43.5%).

3.3.2 Safety, satisfaction and reliability

Safety of DeepCADCR was 69.5%, satisfaction was 85.9%, and reliability was 76.1%. Acceptance of safety was lowest among all subscales, indicating that participants were not convinced that DeepCADCR would improve patient safety. Nevertheless, participants were largely satisfied with DeepCADCR and intended to use (Q16, 87.0%) and recommend it to their colleagues and other EPs (Q17, 87.0%). Most participants were satisfied with DeepCADCR presentation of results (Q22, 95.7%) but did not understand the model algorithm (Q19, 34.8%).

3.3.3 Subgroup analysis

We analyzed the results of user experience in terms of EP experience. Average scores were higher in the resident group than in the specialist group for all subscales; however, the difference in reliability was not significant (Supplementary Fig. 3). The mean (SD) for all questions was 4.7 (SD 1.3) in the resident group and 3.9 (SD 1.6) in the specialist group. The largest difference was in perceived effectiveness of DeepCADCR (5.3 (SD 1.0) in the resident group vs. 3.7 (SD 1.6) in the specialist group, p < 0.001), and the smallest difference was in perceived reliability (4.5 (SD 1.4) in the resident group vs. 4.2 (SD 1.5) in the specialist group, p = 0.308).

We calculated Cronbach’s alpha for the five-subscale questionnaire. The effectiveness subscale of four questions had an α value of 0.94, the efficiency subscale of 4 questions had an α value of 0.78, the safety subscale of 6 questions had an α value of 0 .74, the satisfaction subscale of 4 questions had an α value of 0.96, and the reliability subscale of 4 questions had an α value of 0.84.

3.4 System usability scale of DeepCADCR

The mean (SD) SUS score was 64.5 (7.7). Table 3 provides details of each statement. Of all statements, “the system was easy to use” received the best evaluation from participants, with the highest mean score and lowest SD. Other statements such as “ease of learning the system” (mean 4.0, SD 0.6), “confident in the system” (mean 3.4, SD 0.8), and “intend to use the system” (mean 3.4, SD 0.8) obtained relatively high scores.

Table 3.System usability scale results.
Standard questionsMean (SD)
I think that I would like to use this system frequently3.4 (0.8)
I found the system unnecessarily complex1.0 (0.5)
I thought the system was easy to use4.1 (0.5)
I think that I would need the support of a technical person to be able to use this system2.0 (0.7)
I found that the various functions in this system were well integrated3.3 (0.9)
I thought there were too many inconsistencies in this system1.9 (0.7)
I imagine that most people would learn to use this system very quickly4.0 (0.6)
I found the system very cumbersome to use1.8 (0.7)
I felt very confident using the system3.4 (0.8)
I needed to learn a lot of things before I could start using the system1.8 (0.6)
Range 1–5; 1, strongly disagree and 5, strongly agree. SD, standard deviation.

3.5 Survey results by specific abnormal findings

Participants showed the highest agreement with the DeepCADCR interpretation for pleural effusion (mean 5.5, SD 0.5) and pneumothorax (mean 5.5, SD 0.8). Areas of lower agreement were atelectasis (mean 4.3, SD 1.0) and fibrosis (mean 4.4, SD 1.1). DeepCADCR had the greatest impact when participants were diagnosed or required further treatment for pneumothorax (mean 4.7, SD 1.8) and pneumoperitoneum (mean 4.5, SD 1.8) (Supplementary Table 1).

3.6 Future DeepCADCR application

Seven questions about the one-year use of DeepCADCR were answered by participants. The results are described in Table 4. Although patient safety received the lowest score based on user evaluation, participants felt that continuous use of DeepCADCR would help improve patient safety (mean 4.9, SD 1.2). Other questions such as “help in chest radiograph interpretation” (mean 4.6, SD 1.3), “satisfaction with performance” (mean 4.6, SD 1.4), and “trust the CADCR interpretation” (mean 4.6, SD 1.4) obtained relatively high scores.

Table 4.Perceptions of DeepCADCR use in the future.
Survey QuestionsScores, mean (SD)
After using DeepCADCR for a year, my ability to read chest radiographs will have improved4.3 (1.5)
After using DeepCADCR for a year, I will use it to help read chest radiographs4.6 (1.3)
After using DeepCADCR for a year, the efficiency of treatment will have improved4.5 (1.3)
After using DeepCADCR for a year, patient safety will have improved4.9 (1.2)
After using DeepCADCR for a year, I will be satisfied with the performance of the AI reading system4.6 (1.4)
After using DeepCADCR for a year, I will trust the AI readings4.6 (1.4)
After using DeepCADCR for a year, I will use the AI reading system frequently4.5 (1.5)
Range 1–7; 1, strongly disagree and 7, strongly agree. DeepCADCR, deep learning-based computer-aided detection system for chest radiographs; AI, artificial intelligence; SD, standard deviation.

4. Discussion

This is the first study to evaluate user experience with DeepCADCR in the workflow of an ED. The study showed high acceptance of CADCR among EPs; its acceptance level varied from 69.5% (patient safety category) to 85.9% (satisfaction category) (Table 2). Residents were more positive toward DeepCADCR than were board-certified ED specialists.

In previous works, DeepCADCR was only evaluated in simulation settings or in general wards or outpatient departments. This is the first study to evaluate DeepCADCR implementation in clinical practice in the ED. Simulation-based studies have previously been used to evaluate the performance of EPs with or without DeepCADCR [6, 19]. Because CR interpretation is influenced by various factors such as severity of disease, ED crowding, and EP workload, it is unclear whether the results of simulation studies are generalizable to real-world clinical practice. Moreover, when implementing machine learning-based CDSS, user acceptance needs to be high in addition to acceptance of the system [22].

We also provided evidence that CAD systems can positively impact non-radiologists in a clinical setting. The effect of DeepCADCR on clinical decision-making is unclear. Most studies have targeted radiologists in related fields rather than non-radiologists; however, radiologists do not interact with patients directly or make decisions regarding care. We believe that DeepCADCR can improve patient treatment and prognosis [23, 24]. We implemented DeepCADCR into clinicians’ workflows and found that EPs made most use of the system when they had to make a diagnosis or develop a treatment plan. Residents were more satisfied and influenced by DeepCADCR than were board-certified ED specialists.

Generally, a resident’s ability to interpret radiology images is lowest when they are in Post graduate year (PGY1) and improves as they become more advanced residents and board-certified specialists [25]. Novice residents typically lack knowledge to interpretate radiology images and time to assess images, which are possible reasons why higher satisfaction was reported in the resident group. DeepCADCR can assist residents not only in terms of interpretation accuracy, but also aid in visualization to result in higher satisfaction and usability.

Only 34.8% of participants said they had a good understanding of the DeepCADCR algorithm. Before implementing this system in our ED, we described the function and instructions to our participants. However, we did not cover all knowledge and mechanisms of the deep learning algorithms because the participants had various backgrounds in medical artificial intelligence (AI). According to one study, education has a positive influence on physician acceptance of AI [26]. However, less than half of the participants in our study had prior experience with medical AI. We suggest that aggressive medical AI education can improve physicians’ acceptance, especially with regard to safety and reliability.

User acceptance could vary by work environment [27]. Because DeepCADCR does not differentiate interval changes between abnormal CRs, it is less useful for chronic diseases. The proportions of normal and abnormal interpretations by DeepCADCR would likely impact EP acceptance; a multicenter prospective study should be performed to assess this.

EPs provided high scores for DeepCADCR for abnormal findings requiring emergent management such as pneumothorax, pneumoperitoneum, and pleural effusion. One study reported similar results to our study; when a CAD system was used to detect pneumothorax in CR after lung biopsy, physicians gave a high rating to the interpretation performance. Additional significant factors should be discovered and investigated to implement and maximize the usability of DeepCADCR.

First, we conducted this study in a single-center ED. Multicenter prospective studies are recommended to evaluate the generalizability of our findings. Second, we used only one type of DeepCADCR; the results could differ with another type of DeepCADCR with a different user interface and display. Third, we only conducted the survey once, so we were not able to analyze outcomes over time. Changes in learning, acceptance, and user experience over time should be evaluated in future studies. Fourth, we evaluated the user experience and the impact of DeepCADCR through a survey. However, in real-world clinical settings, many confounding factors can affect the interpretation of chest radiographs and therapeutic procedures. Finally, we did not directly investigate the effectiveness of DeepCADCR in the ED setting.

5. Conclusions

DeepCADCR implemented in the ED workflow was well accepted by EPs. They were highly satisfied with the system, with residents being more positive toward DeepCADCR than board-certified ED specialists.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author contributions

WCC—designed the research study and supervised the overall study process. DHY, SJH and SUL—performed the survey. SYH and GTL—provided help and advice on the research. SJH, HY, JYY and TGS—curated and analyzed the data. DHY—wrote the manuscript. JEP, HSC and TRK—reviewed and edited the manuscript.

Ethics approval and consent to participate

The methods were performed in accordance with relevant guidelines and regulations and approved by the Samsung Medical Center institutional review board (IRB no. 2022-06-116-001). All participants provided written informed consent prior to inclusion in the study.

Acknowledgment

The authors would like to thank A Yeon Kim, Department of Digital Health, Samsung Advanced Institute for Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Korea, for her statistical assistance.

Funding

This research was supported by the Korean Society of Emergency Medicine (2020-01).

Conflict of interest

The authors declare no conflict of interest. Won Chul Cha is serving as one of the Editorial Board members of this journal. We declare that Won Chul Cha had no involvement in the peer review of this article and has no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to MAM.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at https://oss.signavitae.com/mre-signavitae/article/1722143294146658304/attachment/Supplementary%20material.docx.

References

Aronchick J, Epstein D, Gefter WB, Miller WT. Evaluation of the chest radiograph in the emergency department patient. Emergency Medicine Clinics of North America. 1985; 3: 491–505.

[Google Scholar]

Chung JH, Duszak R, Hemingway J, Hughes DR, Rosenkrantz AB. Increasing utilization of chest imaging in us emergency departments from 1994 to 2015. Journal of the American College of Radiology. 2019; 16: 674–682.

[Google Scholar]

Donald JJ, Barnard SA. Common patterns in 558 diagnostic radiology errors. Journal of Medical Imaging and Radiation Oncology. 2012; 56: 173–178.

[Google Scholar]

Al aseri Z. Accuracy of chest radiograph interpretation by emergency physicians. Emergency Radiology. 2009; 16: 111–114.

[Google Scholar]

Kim JH, Kim JY, Kim GH, Kang D, Kim IJ, Seo J, et al. Clinical validation of a deep learning algorithm for detection of pneumonia on chest radiographs in emergency department patients with acute febrile respiratory illness. Journal of Clinical Medicine. 2020; 9: 1981.

[Google Scholar]

Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep learning for chest radiograph diagnosis in the emergency department. Radiology. 2019; 293: 573–580.

[Google Scholar]

Jin KN, Jae HJ, Shin CI, Chai JW, Chun SR, Shin SD, et al. Overnight preliminary interpretations of CT and MR images by radiology residents in ER: how accurate are they? Journal of The Korean Society of Emergency Medicine. 2008; 19: 205–210.

[Google Scholar]

Sellers A, Hillman BJ, Wintermark M. Survey of after-hours coverage of emergency department imaging studies by US academic radiology departments. Journal of the American College of Radiology. 2014; 11: 725–730.

[Google Scholar]

Hwang EJ, Park S, Jin K, Kim JI, Choi SY, Lee JH, et al. Development and validation of a deep learning—based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs. Clinical Infectious Diseases. 2019; 69: 739–747.

[Google Scholar]

Nam JG, Park S, Hwang EJ, Lee JH, Jin K, Lim KY, et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology. 2019; 290: 218–228.

[Google Scholar]

Nam JG, Kim M, Park J, Hwang EJ, Lee JH, Hong JH, et al. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. The European Respiratory Journal. 2021; 57: 2003061.

[Google Scholar]

Kozuka T, Matsukubo Y, Kadoba T, Oda T, Suzuki A, Hyodo T, et al. Efficiency of a computer-aided diagnosis (CAD) system with deep learning in detection of pulmonary nodules on 1-mm-thick images of computed tomography. Japanese Journal of Radiology. 2020; 38: 1052–1061.

[Google Scholar]

Summers RM. Improving the accuracy of CTC interpretation: computer-aided detection. Gastrointestinal Endoscopy Clinics of North America. 2010; 20: 245–257.

[Google Scholar]

van Zelst JC, Tan T, Mann RM, Karssemeijer N. Validation of radiologists’ findings by computer-aided detection (CAD) software in breast cancer detection with automated 3D breast ultrasound: a concept study in implementation of artificial intelligence software. Acta Radiologica. 2020; 61: 312–320.

[Google Scholar]

Wani IM, Arora S. Computer-aided diagnosis systems for osteoporosis detection: a comprehensive survey. Medical & Biological Engineering & Computing. 2020; 58: 1873–1917.

[Google Scholar]

Fujita R, Iwasawa T, Aoki T, Iwao Y, Ogura T, Utsunomiya D. Detection of the usual interstitial pneumonia pattern in chest CT: effect of computer-aided diagnosis on radiologist diagnostic performance. To be published in Acta Radiologica. 2020. [Preprint].

[Google Scholar]

Seah JCY, Tang CHM, Buchlak QD, Holt XG, Wardman JB, Aimoldin A, et al. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. The Lancet Digital Health. 2021; 3: e496–e506.

[Google Scholar]

Jones CM, Danaher L, Milne MR, Tang C, Seah J, Oakden-Rayner L, et al. Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study. BMJ Open. 2021; 11: e052902.

[Google Scholar]

Kim JH, Han SG, Cho A, Shin HJ, Baek S. Effect of deep learning-based assistive technology use on chest radiograph interpretation by emergency department physicians: a prospective interventional simulation-based study. BMC Medical Informatics and Decision Making. 2021; 21: 311.

[Google Scholar]

Pinto A, Reginelli A, Pinto F, Lo Re G, Midiri F, Muzj C, et al. Errors in imaging patients in the emergency setting. The British Journal of Radiology. 2016; 89: 20150914.

[Google Scholar]

Thakkalpalli M. Reducing diagnostic errors in emergency department with the help of radiographers. Journal of Medical Radiation Sciences. 2019; 66: 152–153.

[Google Scholar]

Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digital Medicine. 2020; 3: 17.

[Google Scholar]

Yoo H, Kim EY, Kim H, Choi YR, Kim MY, Hwang SH, et al. Artificial intelligence-based identification of normal chest radiographs: a simulation study in a multicenter health screening cohort. Korean Journal of Radiology. 2022; 23: 1009–1018.

[Google Scholar]

Chabi M, Borget I, Ardiles R, Aboud G, Boussouar S, Vilar V, et al. Evaluation of the accuracy of a computer-aided diagnosis (CAD) system in breast ultrasound according to the radiologist’s experience. Academic Radiology. 2012; 19: 311–319.

[Google Scholar]

Seagull FJ, Bailey JE, Trout A, Cohan RH, Lypson ML. Residents’ ability to interpret radiology images. Academic Radiology. 2014; 21: 909–915.

[Google Scholar]

Civaner MM, Uncu Y, Bulut F, Chalil EG, Tatli A. Artificial intelligence in medical education: a cross-sectional needs assessment. BMC Medical Education. 2022; 22: 772.

[Google Scholar]

Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library—resource projects to increase what is learned. Journal of the American Medical Informatics Association. 1994; 1: 28–33.

[Google Scholar]