A real-world intelligent system for the diagnosis and triage of COVID-19 in the emergency department

1.1 The role of radiography and artificial intelligence in the diagnosis of COVID-19

One of the main effects of SARS-CoV-2 in the human body is a particular form of pneumonia. Thus, proper visualization of the lungs using medical imaging techniques (e.g., CXR or CT) should reveal COVID-19 infection when there is pulmonary involvement. Accordingly, an automatic procedure to identify the particular pattern of this disease on a CXR seems an efficient way for patient triage in the ED. The advantages of CXRs include its speed, wide availability, accessibility, and portability. Therefore, automatic analysis of CXR images seeking specific imaging patterns would suffice to obtain a rapid diagnosis. The automation of this specific diagnosis can be approached by using AI systems, the so-called ‘intelligent systems’.

One of the most outstanding fields in which AI has been applied in the last decade is computer vision, and the diagnosis of COVID-19 from CXR images seems a proper task to be tackled using this approach. In fact, many groups around the world have explored this idea, and several articles have been published on the subject to date [7, 8, 9, 10]. Unfortunately, the quality of reported research is quite diverse and most proposals cannot be reproduced due to several issues, including lack of data availability, missing details in the description of processes or parameters, or incomplete descriptions of the methods followed. Hence, it is very difficult to build an intelligent system for this endeavor solely based on the information published in the scientific literature.

1.2 Artificial intelligence for medical diagnosis

The use of AI in medical diagnosis began with the use of expert systems. Instead of designing an ‘all-in-one’ algorithm, expert systems combine knowledge with a general algorithmic problem-solving method to reach a required solution, e.g., a diagnosis in medical problems. By using this architecture, expert systems provide transparency (explain their solutions), flexibility (knowledge can be improved), user-friendliness (no programming skills are needed), and competence (high capability of solving problems) [11]. Expert systems raise the work of a medical expert to a new qualitative level, reducing the risk of misdiagnosis [12].

Machine learning (ML) techniques focus on increasing knowledge through automated application of statistics. The underlying idea is to make the computer learn from data. Since large amounts of medical data exist, new knowledge can be acquired and applied in medicine. For instance, in prognostics, an ML model can learn the patterns of health trajectories of a vast number of patients and this can help physicians to anticipate the course of a given condition [13]. Regarding disease detection and diagnosis, the utilization of several ML artifacts, like Bayesian networks, K-means clustering or Support vector machines, has proved to be very successful [14].

Artificial neural networks (ANN) are part of ML techniques. These networks try to mimic the structure of the brain and are formed by a set of layers of neurons. Each neuron is connected to some of the neurons of the previous layer (in fact, to all of them if it is fully connected) and produces an output using a weighted sum of its inputs (which are the outputs of neurons in the previous layer). These weights are the model parameters to be adjusted during the training phase. The larger the number of neurons, the more parameters have to be learned, especially in the case of fully connected networks, where each neuron is connected to all the neurons of the previous layer.

With the recent advances in parallel computing, the capabilities of ANNs have improved significantly, and larger networks with many layers of artificial neurons are built and trained. Deep Learning (DL) studies these networks along with all the algorithms that work with them [15]. There are many applications of DL in medicine, but the most common ones concern medical image analysis [16]. Traditional ML techniques can achieve results in the range of small-sized ANNs, although it is difficult to reach the performance of state-of-art DL networks in the field of image analysis [17]. Such performance levels are obtained at the cost of computationally intensive model creation processes.

All these ML techniques address different problems. One of them is classification, i.e., the process of assigning a class to each object under consideration. These objects are usually described in data terms by the value of some of their features (e.g., size, weight, age, color). However, sometimes the available data about the object is an image, described by a matrix of the color value of each pixel (points of the image). This is called ‘image classification’, which should not be confused with object detection or image labeling, where more than one label is assigned to a given image. Medical image classification aims to differentiate different types of medical images according to a certain criterion, such as clinical conditions [18].

Many methods have been proposed to classify images. Image classification systems usually require extensive preprocessing to prepare images for analysis, including edge detection, color feature extraction or normalization [19]. With the advent of DL, some of this preprocessing is addressed directly within the network layers [20]. Auto-encoders and convolutional ANNs address this preprocessing, being able to improve image classification and obtain striking results. This automatic annotation process has been used for a wide variety of medical image classification analyses, including histopathological classification of colon cancer [18] or skin cancer [21].

Despite a very high number of papers have been published presenting different attempts to create automatic systems for image detection of COVID-19 [10, 18, 22], unfortunately, most of them suffer from several weak points and usually fail to provide enough information to allow for their reproduction [23]. In this article, we strive to provide not only a comprehensive description of the process followed to build a robust and effective system for that purpose, but also to make it easy to understand and fully reproducible. This article reports the work performed, as well as the results obtained by our research team, and provides all the necessary information to allow interested researchers to reproduce all the work carried out. It also discusses the issues we have run through and provides hints to avoid or mitigate them.

The diagnosis of COVID-19 can be approached as a normal data science problem: it can be formulated as a classification problem and different models can be built by applying any of a diverse set of possible ML techniques. However, a rigorous process involving data pre-processing and many technical details need to be undertaken to eventually achieve an effective system. Such process can be structured in the following steps: problem formulation, data collection and pre-processing, model definition and construction, validation, and, finally, deployment.

2.1 Problem formulation

The goal of the present work was to develop an automatic diagnostic system for COVID-19 based on CXRs. The gold standard diagnostic test for COVID-19 is the RT-PCR test, but several factors may hinder its application, including the shortage of test kits or the time required to obtain the result. Since one of the main effects of COVID-19 is lung involvement, a faster diagnostic approach is possible by analyzing CXRs. If an automatic procedure without the intervention of a radiologist could be developed, the approach would take shorter times and demand fewer resources.

The resulting system would take a CXR as the only input and feed it to a computer system which, in turn, would provide an answer in terms of whether the patient is affected by COVID-19 or not. The diagnostic challenge boils down to an image classification problem. However, the task is not that simple and several issues need to be addressed:

• Enough data to train the AI system are required, which entails having a large amount of correctly labeled CXRs.

• The system should properly distinguish COVID-19 lung involvement from other types of effects produced by other diseases.

• Real-world CXRs are not perfect. They may suffer from several technical deficiencies that must be addressed before images are processed.

The starting point for the model we need relies on recent ML-based solutions for problem classification. These systems have been built out of the last generation of ANNs, an extensive set of models collectively known as DL models [24].

DL generically refers to the whole area, including models, architectures, training algorithms and roughly anything related to them. The most relevant early models for images were built by Google researchers on the ImageNet dataset [25]. ImageNet is a research project that has developed a large database of images with diverse annotations. The dataset contains over 14 million images (out of 1 million annotated) for 21,000 classes. This dataset also served as the basis for an image classification challenge from which several related but different models have been developed using the dataset as a starting point. Some of the most notable models are VGG16-VGG19 [26], MobileNet [27], ResNet [28], Inception [29], DenseNet [30] or Xception [31].

Creating a brand new model for image classification imposes a high tax in terms of necessary data and resources. These requirements are usually hard to satisfy. However, the knowledge extracted by the networks along their training is embedded within their structure (both topology and weight values). Most of their general knowledge can be useful for other classification problems. Based on this premise, a neural network able to address other classification problems can be built. This requires making some adaptation to the network topology, particularly, input and output layers, and a relevant dataset.

2.2 Data collection and preprocessing

The availability of a high-quality dataset is a key ingredient in any data-driven solution. Research groups all over the world have been gathering their own datasets as the COVID-19 pandemic evolved. However, only a few have been published for general use. To face the challenge posed by the present research work, two datasets have been considered: a publicly available one and a locally gathered one.

During the pandemic time, a few repositories with CXRs (or other kinds of radiological images) have been published on the Internet. However, the quality of the data or the representativeness of the sample hinder their final effectivity. They can be used for exploring purposes, but not to develop a robust real-world system. Among all the public repositories, the one that has probably received the most attention is COVIDx from the COVID-Net Open Source Initiative [32]. COVIDx is a dataset of CXR images that comprises 15,190 samples. It has been generated by combining five different publicly available data repositories. The images in the dataset are labeled and therefore divided into three groups: “Normal”, “Pneumonia” and “COVID-19”. For the purpose of our problem, we aggregated the first two classes into a single one (“Non-COVID”). The distribution for each group are: Non-COVID-19 CXR images (91.5%) and COVID-19 CXR images (8.5%). This is the original version of the dataset, as published in [32]. We will refer to it as COVIDx 1. The dataset, nevertheless, has been updated in successive versions. The last one was published in November 2021. It is now composed of ~30,000 images divided into two categories (Non-COVID-19 and COVID-19), extracted from several health institutions all over the world. The percentages of the two classes are 14,192 COVID-19-negative CXR images and 16,690 COVID-19-positive CXR images.

The second dataset to consider was collected locally in the province of Granada, Andalusia, southern Spain. It is composed of 114,251 anonymized Digital Imaging and Communication in Medicine (DICOM) files from patients that attended several public hospitals in the province of Granada. More specifically, these hospitals were: Hospital Universitario Virgen de las Nieves of Granada, Hospital Comarcal Santa Ana de Motril and Hospital Comarcal de Baza. These DICOM files contain, among other information, high resolution (4000 × 3000 pixels) CXR images. We will refer to this dataset as GranaCov.

Since CXR images are not in perfect condition, a thorough cleaning and preprocessing step was necessary, and a subsequent filtering process had to be applied. The images can be both lateral and frontal radiographs showing the lung area. During the extraction of images from the DICOM files, only the frontal images were selected, discarding the lateral ones, resulting in a total of 68,360 frontal images. Moreover, because labels (frontal, lateral) assigned to the images were not completely consistent, a separate image classification neural network was trained to distinguish between frontal CXR images from the rest. This not only allowed us to discard images that showed a lateral view (i.e., mislabeled), but also some other images not related to this project which were erroneously included in the dataset. The accuracy level obtained on this specific task was 98.6%. This model automatically filtered out images of children, mislabeled images, and images that do not correspond to the lung area, such as arms, hips or undefined objects. After this process, the total number of available frontal images was 64,790.

From the filtered frontal images, due to the poor quality of some CXRs, a process of image quality improvement was necessary. In this regard, two types of problems were identified: some images with a white background presented inverted color tones and some were excessively saturated. The images with the white background were improved by inverting the colors, while the saturated ones underwent a process of histogram equalization to increase contrast. Both types of problems are automatically detected by our preprocessing pipeline and do not require human intervention. Fig. 1 shows examples of these cases.

Fig. 1.Image preprocessing examples of chest X-rays (CXR). a. Color inversion. a1. CXR with white background (i.e., negative). a2. Result from color inversion. b and c. histogram equalization. b1. Black saturated CXR (i.e., overexposure). b2. Histogram equalization. c1. White saturated CXR (i.e., underexposure). c2. Histogram equalization.

The labels for the CXR images were obtained from an anonymized hospital database. In this database, each patient can have more than one associated CXR and the COVID-19 infection identification criterion is based on PCR test results. It contains 40,751 records of PCR, Immunoglobulin G (IgG) and Immunoglobulin M (IgM) antibody tests (antibody tests were not used in this study). On this subject, we encountered several problems as we learned that the database also contained CXRs associated with previous, non-COVID-19 related, episodes. This required another filtering process to select only images acquired during the same episode in which the PCR test outcome was obtained. Once these problems were solved, the database for the study was limited to 6569 COVID-19 negative and 1267 COVID-19 positive images, yielding a final dataset with 7836 instances (Table 1).

2.3 Base Convolutional Neural Networks

A review of the scientific literature reveals that Convolutional Neural Networks (CNN) are generally used to address image classification problems. These networks are a special type of neural network where each layer performs a convolutional operation (or set of operations) on the input image or the result of the previous layer. In contrast to neurons of regular fully connected networks, convolutional neurons perform a convolutional operation (whose associated weights have to be learned during the training phase) only on a region of the input image or the result produced by the previous layer. This reduces the number of parameters to be estimated in comparison to a fully connected network applied to the same type of data. It has to be noted that the dimensionality of inputs like images is very high even for low-resolution input samples as each pixel is an input element. If no additional classification model is used, the last layers of CNNs are fully connected ones that are only geared to produce the output, i.e., a classification label. In summary, the layers of a CNN are trained to learn a set of convolutional filters which should be helpful to extract relevant features for the classification task.

CNNs are not usually built from scratch but rather pre-trained networks are employed. To this end, Transfer Learning approaches are used. Transfer Learning provides the possibility to re-use the previous parameter values acquired by training one network and adapt it to another—similar—problem. The knowledge acquired by the network is, thus, re-used to solve a different problem. The reasoning behind this procedure is that this type of neural network, during the training phase, learns a set of filters that are useful for extracting features from any image and that these filters can be re-used, although a fine-tuning step is always required (network re-training step). In other words, a neural network that can classify regular images can be re-engineered to classify CXR images. Therefore, it is a good starting point and helps to save a great amount of computing time.

After some initial tests with different models, MobileNetV2 [33] and VGG16 [26] were the state-of-art neural network models chosen as the base to build our diagnosis system. Trained versions of these models are available to download from the official TensorFlow site¹ (¹https://www.tensorflow.org/api_docs/python/tf/keras/applications/mobilenet_v2/MobileNetV2(Accessed February 2022))² (²https://www.tensorflow.org/api_docs/python/tf/keras/applications/vgg16/VGG16 (Accessed February 2022)) (version core 2.8.0 at the time of writing). VGG16 is a representative of models with a very large number of parameters (~138 million). The network is composed of five convolutional blocks. The first two blocks consist of two convolutional layers and a Pooling layer (224 × 224 × 64 and 112 × 112 × 128), while the others consist of three convolutional layers (56 × 56 × 256, 28 × 28 × 512, 14 × 14 × 512). Three Fully-Connected layers of different depths (4096, 4096, and 1000) follow the stacked convolutional blocks. The final layer is a soft-max layer.

MobileNetV2, with ~3.5 million parameters is a network of smaller size. The architecture contains an initial convolutional layer (224 × 224 × 3), followed by seven bottleneck blocks (112 × 112 × 32, 112 × 112 × 16, 56 × 56 × 24, 28 × 28 × 32, 14 × 14 × 64, 12 × 12 × 96, 7 × 7 × 160), to finish again with two convolutional layers (7 × 7 × 320, 1 × 1 × 1280). The same parameters have been used as in the previous model. Both models had already been pre-trained using the ImageNet dataset with images of size 224 × 224.

2.4 Building of the network

As mentioned above, the networks used are not trained from scratch, but rather a Transfer Learning approach is used. Our starting points were ImageNet pre-trained networks. From the different choices, VGG16 and MobileNetV2 were the models finally selected. The original formulations of the model had to be adapted to the task of classifying CXR images to detect COVID-19 lung involvement. Accordingly, the last two layers were removed and some new trainable layers were added on top of the remaining ones: a dense layer with 1024 units, a dropout layer with 0.5 rate of units to drop and, finally, a new dense layer, as it is completely dependent on the number of classes. In the original problem, the number of classes was over 1000, but in our case, it is just two.

The detailed architecture of the VGG16 network is provided (Fig. 2). The first two layers are convolutional, with 64 3 × 3 filters (per layer). The stride value used is 1. The convolutional filter size and the stride value are maintained constant for the whole network. The third layer is a max-pooling layer of size 2 × 2 with a 2 × 2 stride. This layer down-samples input data computing the maximum over a 2 × 2 window. The fourth and the fifth layers are also convolutional, with 128 filters (per layer). The sixth layer is another max-pooling layer of the same type as the preceding one. The next three layers are convolutional, with 256 3 × 3 filters. Max-pooling is applied by the tenth layer. The previous scheme (three convolutional layers followed by a max-pooling one) is repeated twice with 512 filters per convolutional layer. Finally, the data is flattened and passed to two fully connected layers of size 4096, each followed by a dropout layer. Dropout layers randomly set some of the inputs to 0 to prevent overfitting.

Fig. 2.Schematic layer representation of a VGG16 neural network used for classifying CXR images of COVID-19. The type and size of each layer is included. Conv, convolutional.

As the last layer, a fully connected one is used to produce the classification result. Its output size depends on the number of classes of the classification problem. This layer uses the softmax function to produce a result in the (0, 1) range associated with each possible class. The sum of all outputs of a layer of this type is equal to 1.

The architecture of the VGG16 net can be summarized as follows: a series of convolutional layers followed by a max-pooling one. The dimension of the data gets progressively reduced while the number of filters per convolutional layer is progressively increased before data reaches a pair of fully connected layers and the final softmax one.

Regarding domain adaptation, since our datasets were not large enough for these networks to learn, we used various openly available CXR image datasets. The first dataset used was the one made available to the scientific community by the National Institute of Health (NIH) Clinical Center [34]. The dataset contains more than 130,000 anonymized CXR images with 14 categories of lung diseases. Remarkably, it does not contain any COVID-19 cases. Nevertheless, it serves the purpose of letting the network discover features specific to CXR images. The networks trained on this CXRs of non-COVID-19 images were then further trained to build the actual models that would diagnose suspected COVID-19 patients.

The tasks performed for model adaptation include the following:

• Network architecture tuning: light adjustments to the number of layers/blocks to learn how the network size was related to the results.

• The pre-trained layers were frozen to allow only the initial and last layers to be modified during the re-training phase to keep intact most of the pre-learned features. This test was carried out to account for the possibility that the number of CXR images was not large enough to re-train networks of this size.

• Different parameters related to the re-training phase were tested, e.g., optimizer method or learning rate.

• Data augmentation: the dataset was artificially modified by adding modified versions of the original data. These modifications included horizontal and vertical flips, zoom and up to 20 degrees of rotation.

Then the networks were trained on the COVIDx 1 dataset since this was the only one available at the early stages of the research. By following the established empirical methodology, the dataset was randomly split into two independent parts: training set (70%) and test set (30%). Only the training set was used in the model building tasks, including hyperparameter selection identification, reserving the test set for the exclusive task of model testing. The COVIDx 1 dataset showed a clear class imbalance problem as the number of COVID-19 negative samples was clearly higher than the positive ones. This required the use of the well-known majority class (i.e., COVID-19 negative) under-sampling technique. By having a similar number of samples of both classes, the learning process does not get biased towards mostly generating majority class predictions because it is sufficient to achieve global accuracy levels.

The employed training procedure used the following parameters: as an optimizing procedure, both Stochastic Gradient Descent (SGD) and Adam were tested. SGD always yielded higher accuracy values. After some initial test-and-error experiments, the remaining parameters were set as learning rate at 0.001, decay at 0.000001, and momentum at 0.9. The network performance is measured based on its accuracy, defined as the rate between right answers and the total number of instances. A deeper insight is offered by the confusion matrix: each row represents the instances of each actual class while each column represents how the test samples of each class were labeled by the classification system.

3.1 Experimental results

While data augmentation was applied at different intensification levels, no improvement in the results was observed. Thus, the process was eventually not applied, and only the original set was used for training.

The results obtained in terms of classification accuracy for the VGG-16 and MobileNetV2 were 92% and 91%, respectively (both on the COVIDx 1 test set). The confusion matrices are shown in Table 2 and Table 3.

F1-scores for both networks were 0.9163 and 0.91282, respectively, pointing to a lightly better performance of VGG16 over MobileNetV2.

COVIDx dataset has been updated several times, the last one in November 2021. This version is composed of 30,000 CXR images with 16,690 positive cases, the rest being negative. We named this updated version as COVIDx 2. We reproduced the same procedure applied to COVIDx 1 with this enlarged version of the dataset. On this occasion, VGG16 achieved a 91% accuracy rate in the test, while MobileNetV2 showed 95% accuracy. The confusion matrices are shown in Table 4 and Table 5.

F1-scores for both networks were 0.9072 and 0.9526, respectively, resulting in a better result for the simpler architecture of MobileNetV2.

The experiments have been replicated by using different random partitions of the dataset into training and test sets. The accuracy values and confusion matrices obtained are quite similar to those reported above, with a standard deviation of 2.0 units of accuracy level.

Our final goal was to have a system tailored to the patients undergoing a CXR with suspicion of COVID-19 in the province of Granada. Thus, a local CXR image dataset was needed. The gathering of the local dataset was severely delayed due to administrative and technical reasons, but after one year the dataset, GranaCov, described in section 3.2, was assembled.

Once the dataset was available, we made a straight evaluation of the COVIDx trained network on the GranaCov dataset. Since the two datasets (i.e., COVIDx and GranaCov) are assumed to represent the same problem and to be samples of the same phenomenon, similar results, in terms of accuracy were expected. Surprisingly, the results on the GranaCov dataset were quite different to the ones obtained on COVIDx. This unexpected issue triggered the search for explanations. The following causes were identified:

• COVID-19 positive cases with no lung involvement. In this case, no information can be obtained from the CXR image. A review process conducted on our database has made this issue surface.

• Doubtful or interim cases in which even trained radiologists find it difficult to reach a diagnosis with no additional information (i.e., only using CXR information).

• Mislabeled images due to errors produced when the information was added to the database. The unprecedented admission rates, especially during the first months of the pandemic, increased the chance of human errors of this kind.

To address the first of the above points, a model capable of discerning between lung-involved and non-lung-involved images was generated, so that it could be used to filter out CXR images of COVID-19 patients with no lung involvement, that is, images that may be hindering the development of a model. The COVIDx dataset was used for this purpose. Instead of using the COVID/Non-COVID labeling of the data, we opted to label CXRs as normal-pneumonia/COVID, i.e., joining the last two classes into a single one. The generated model achieved a 91% test performance for the task of classifying between pulmonary versus non-pulmonary-involved CXR images. The application of this model to the local dataset produces some interesting outcomes: it establishes 3557 images labeled as Non-COVID as being pulmonary-involved cases, and 444 COVID-19 positive images as unaffected cases.

Analogously, a segmentation process was applied to optimize the training process as non-relevant parts of the images are eliminated and substituted with a solid black background color. For this task, U-net, which is a pre-existing CNN designed for biomedical data segmentation was used [35]. This model was trained using pre-segmented CXR images and achieved 98% accuracy on our images.

The output of the U-net is a binary mask of the same size as the input image. The application of this mask to the original image results in any pixels not associated with the lung area being transformed into black pixels (background). Therefore, U-net is only used during the pre-processing step to remove pixel data not belonging to lung areas and it is not used to obtain a COVID-19 diagnosis. Fig. 3 shows a sample of the image obtained from the DICOM file and the resulting segmented image using U-net.

Fig. 3.Example of CXR image segmentation using U-net.

The segmentation process avoids any learning effort to be wasted outside the area of interest and does not require any human intervention, except a small percentage of very low-quality images that cannot be repaired using the image preprocessing pipeline explained in this section.

All the adjustments performed on the processing pipeline were geared towards achieving competitive performance on the GranaCov dataset too. The steps composing the currently deployed processing pipeline are depicted in Fig. 4.

Fig. 4.Intelligent diagnosis system blocks. DICOM, Digital Imaging and Communication in Medicine.

3.2 Validation

The validation process of all the steps explained in previous sections is based on the use of independent training and test sets. The error or accuracy achieved in the training set is useful for understanding how the model is learning, but it is not a realistic estimate of how the model behaves in the case of new observations. To achieve a more accurate estimate, it is necessary to use a previously ‘unseen’ dataset, the test set. The model validation strategy adopted in this work consisted of dividing the dataset into training and test subsets, and additionally into a validation subset. The validation set was used to choose the best training model.

All the accuracy values included in this work refer to the behavior of the model on the test set, not on the training or validation set. The training and validation sets used during the learning process contain 70% and 10% of the images, respectively. The remaining ones are used in the test set to classify using the trained model and evaluate the model performance.

The accuracy of every trained network described above on the training set was over 99%, meaning perfect learning on the instances used for the network weights adaptation. However, the most relevant figure is the accuracy on unknown instances, totally independent of those used during training, namely, the test set. As previously indicated, the average performance of VGG16 models on both COVIDx 1 and COVIDx 2 was 91%, and the performance of the MobileNetV2 was 92% and 95%, respectively. These percentages are average values over several independent partitions of the dataset, so they are a valuable approximation of the real values. On the other hand, the performance on the GranaCov dataset in training was again over 98%. However, the performance observed on the test set was rather low. Thus, further research is necessary before an intelligent model reliable enough is available.

Once the models are trained and validated, they are tested for some time in medical practice. The intention of this is to validate the models in a real-world environment (Emergency Department) by medical professionals. The workflow and a web application have been developed to allow its implementation but have not been implemented in clinical practice yet.

3.3 Deployment of the final system

The AI-based diagnosis system was fully implemented using the Python programming language using Tensorflow with Keras for all the DL related tasks as stated previously.

In order to make the use of the developed tool easier for physicians, a user-friendly web application has been designed and implemented. The architecture of the tool is quite straightforward, including the image database, the intelligent system model and the user interface. This application has been developed using the JavaScript and PHP programming languages. The image database is stored in a Redundant Array of Independent Disks (RAID) storage device, with an ext4 file system. The database is managed by PostgreSQL. All the components of the application are deployed with containers—through docker services—scattered along with several servers with Linux based operating systems. Therefore, only open-source software is used.

The workflow supported by the application is quite straightforward: as new patients arrive at the Emergency Department, their data are captured, a CXR is obtained and, once it is available, the physician retrieves the patient’s data and requests a diagnosis from the tool. Once the image is analyzed and the result is available (it takes less than one second), the latter is displayed on the web page. This way, the application can be used from any device with an Internet connection, namely universally accessible. This workflow is illustrated in Fig. 5. As the initial motivation for the usage of these intelligent systems was a fast triaging system, measurements of running times are necessary. The most computationally intensive task is the image analysis step, which takes an average running time of 0.75 seconds per image. This is not the limiting time for the whole process, but the acquisition of the CXR, which is the longest step. A screenshot of the current version of the application as used during physician evaluation is shown in Fig. 6.

Fig. 5.Medical diagnosis application. DICOM, Digital Imaging and Communication in Medicine. App, application.

Fig. 6.Diagnosis application screenshot.