Machine learning techniques as an efficient alternative diagnostic tool for COVID-19 cases

1. Introduction

The Coronavirus (COVID-19) disease is caused by a novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and first began in the Wuhan City, Hubei Province, China, early in December 2019 [1, 2]. Although COVID-19 is a new virus, knowledge on this disease is daily generated. Several authors [1, 3] described SARS-CoV-2 as a type $\beta$-coronavirus cluster. COVID-19 has been defined as the third known zoonotic coronavirus, following the SARS and the Middle East respiratory syndrome (MERS) [3, 4, 5]. The COVID-19 affects the human’s lower respiratory tract, and it is often manifested as pneumonia [4]. Typical symptoms for COVID-19 positive patients include cough, fatigue, and fever, but other symptoms as diarrhea, digestive problems, dyspnoea, hemoptysis, headache, lymphopenia, and sand sputum production have been also identified [6, 7, 8, 9, 10]. Studies have shown that such symptoms likely appear after an incubation period of approximately 5.2 days [9, 11]. In contrast, the average time to death after symptoms appearance ranges from 6 to 41 days with a mean of 14 days [9, 10].

The mentioned symptoms are common to other respiratory syndromes. Hence, detection and confirmation of a patient having a COVID-19 viral infection are performed using polymerase chain reaction (PCR) tests, which, to date, remains the standard gold examination [12]. However, there are several considerations and limitations to this type of diagnostic test. For instance, authors have reported that: (i) there are more than seven different SARS-CoV-2 nucleic acid PCR tests [13, 14]; (ii) when considering the viral load in samples, it has been shown that upper respiratory tract samples have their peak viral loads three days after symptom onset [15]; and (iii) as the SARS-CoV-2 epidemic evolves globally, we need better diagnostic tests that are rapid, reliable, validated and widely available. Other authors further elaborated on the accessibility to tests [16], including that the supply of the reagents does not satisfy the demand. Also, the results of PCR tests take one day or longer to be obtained after sampling (in contexts that are not overcrowded and saturated by infected people). Considering such limitations, some investigations have called for an initial screening with an auxiliary examination based on computed tomography (CT) or chest X-rays [12, 13, 14, 16, 17, 18, 19]. Typical chest manifestations of COVID-19 diagnosed patients include consolidation, crazy paving pattern, ground-glass opacities, and reticular pattern, which are typical CT manifestations [19, 20, 21].

Artificial intelligence techniques have been proposed for screening, tracking, and predicting patient health outcomes and for developing drugs and vaccines through clinical trials [22, 23]. However, most of the research has been focused on the early detection and diagnosis of COVID-19 patients using deep learning (DL) techniques [18, 24, 25, 26, 27].

Table 1 (Ref. [18, 24, 25, 26, 28, 29, 30]) presents a summary of some recent approaches used for COVID-19 diagnosis with medical images and techniques. Convolutional neural networks (CNN) have been used in recent literature on the topic. For instance, CNN models utilizing CT images of COVID-19, pneumonia, and healthy patients were considered in [26]. Similar work was conducted in [24, 18]. More recently, authors have derived hybrid techniques based on DL (CNN features) and support vector machines (SVM) algorithms, named convolutional support vector machines (CSVM) [28]. Also, COVID-19 patients using X-ray images were predicted employing CSVM [31]. These authors identified that CSVM with a polynomial kernel provided the best result.

Similarly, CSVM models with additional feature ranking techniques improved the classification performance for COVID-19 [25]. As a different approach, a high accuracy in predicting COVID19 could be reached by using transfer learning [27]. Combining architectures such as dense convolutional network (DenseNet), residual neural network (ResNet), and very deep CNN for large-scale image recognition (VGG), a high-performance model was proposed in [27], which reached an accuracy level of 97.1%. Deep features from 13 different CNN models with SVM as the classification technique were used in [29]. These features differ from previously mentioned works as they are implemented in two steps: first decomposing images using CNN and then classifying data with SVM. The hybrid ResNet50 CNN and SVM model provided the best results based on accuracy, the area under the receiver operating characteristic curve (AUROC), F1-score, false-positive rate (FPR), false-negative rate (FNR), and sensitivity. It is also common to find other machine learning (ML) techniques without deep features [32]. These ML techniques are employed as a benchmark for more complex models, but their predictive power is not usually assessed due to their use as a benchmark and not as a classifier itself.

The studies mentioned above demonstrate that DL (and CNN) is (are) highly effective for discriminating COVID-19 images. Nevertheless, SVM has been used as an alternative to a DL-based classifier, as CNN needs large data sets for training, validating, and reducing the overfitting [29]. There are still other drawbacks to be addressed when CNN-based models are considered. Firstly, CNN models heavily rely on high-performance computing, where complex models can take several hours to days to be trained, developed, analyzed, and implemented in a clinical context [33]. This is a critical issue in rural and impoverished areas that lack access to a proper technological infrastructure [34]. Secondly, commonly used algorithms and network architectures are complex to design and explain, which contrasts with desired biomedical and health contexts [35]. These drawbacks take significant importance in the current context of a pandemic. Notably, in the present world situation, a rapid response is needed in communities where the health systems are saturated, and access to PCR tests (or other tests, such as rapid tests) is limited [14, 15, 16]. Furthermore, it is commonly found in the literature that impoverished, rural, and small communities lack infrastructure access [34]. Significant socioeconomic differences and disparities have been found in the context of the COVID-19 pandemic [36, 37, 38, 39]. Therefore, in such a context, simple, fast, and cheap tools, such as those provided by ML techniques, can be of great value.

The objective of this paper is to explore the capabilities of ML techniques, in comparison to DL, as valuable tools to diagnose coronavirus positive cases based on X-ray chest images, providing an accessible, affordable, and fast solution to the initial screening of COVID-19 patients. Note that our investigation takes a different approach to what has been studied in the literature. We develop a full factorial experiment to train a series of ML-based techniques, such as artificial neural networks (ANN), random forest (RF), and SVM. The design of experiment considers different data-balancing methods, cross-validation approaches, kernel functions, and hyper-parameter optimizations. For each technique studied here, the statistical experiment allows us to obtain predictive power metrics such as accuracy, sensitivity, and specificity. This experiment is crucial for re-sampling and data-balancing methods to accurately account for data unbalancing associated with low COVID-19 positive rates of detection.

Methods to handle unbalanced data can be classified into internal or external. On an internal level, one may tune the existing design to address the class imbalances, being its advantage that this level is specific to an algorithm. On the external level, different types of data sampling methods are used. The external approach is straightforward and can be applied to any algorithm, but it incurs overfitting or loss of information [32, 40].

Unlike the existing literature, we use a design of experiment to estimate the classification power of ML techniques, which are often not fully optimized and explored by previous research. We demonstrate that the ML techniques can provide stable predictions with low and suitable FPR and FNR. The ML techniques that utilize few resources can be used as COVID-19 diagnostic tools in contexts where access to health infrastructure is lacking, such as in impoverished and rural regions or communities.

The rest of the paper is organized as follows. Section 2 presents the methods used in this research as well as details of the data employed for training and testing the proposed ML models. Section 3 reports the results from the design of experiment, while Section 4 concludes and discusses the main findings of our research.

2. Methods

This section describes the process followed to create a usable data set based on the analysis of CT images. Then, we provide the details (training, testing, and tuning) of the ML techniques used.

2.1 Data set: X-ray image source and database creation

The data set used for training models was created from a series of steps, including data (image) pre-processing, object detection, and feature extraction [41, 42]. Fig. 1 depicts the overall process followed in this study. The image pre-processing step considers cutting, resizing, and formatting X-ray images. Object detection is focused on making a demarcation of the areas in the image where the objects of interest for the analysis are located. The extraction of characteristics considers two main features. First, we obtain textured X-ray images, which quantify the distribution of the intensities of the pixels within each object based on their average and standard deviation (SD). Among the most advanced texture feature extraction techniques, we consider pixel-pixel co-occurrence patterns [43] and granularity measures at different scales [44, 45]. The second class of features describes the X-ray image contour based on segmentation masks: the roughness or circularity of the contour [46]. More specifically, we create features that quantify the shape of the object, its image moments (which can be computed with or without reference intensities), and features that quantify the texture of the pixels. Then, the final data set considers the response variable (COVID-19 positive, healthy, or other pneumonia patients), 95 other covariates from X-ray image characteristics created from the feature extraction process, and patient’s data (age and sex). The feature extraction process was developed with the EBImage package of the R software [47, 48]. The X-ray images were obtained from public repositories of COVID-19 (based on assembling of medical images in existing publications) and from the public chest X-ray database [49, 50].

Fig. 1.Overall data set and model building process.

As a result of the analysis of images, and a merge of these data with patient’s characteristics, the data set finally contains: (i) three types of views related to posterior-anterior (PA), anterior-posterior (AP), and left (L) parts; (ii) sex, age, and diagnosis (COVID-19, healthy, pneumonia, SARS, streptococcus, and others) of patients; and (iii) metrics of the images.

2.2 Machine learning techniques: training and testing

Different ML techniques were used for classification in this study, including ANN, RF, and SVM. A CNN was also used as a benchmark for comparison with ML techniques. All models were trained to predict patients with positive COVID-19 against healthy patients or with pneumonia. Performance evaluation metrics were used to assess the capability of ML techniques for classification. The mean and its confidence interval for the metrics were obtained using a k-fold-repeated-cross-validation (KFRCV) with five and ten as the number of folds, separately, and with ten repeats each. The data cleaning, training, and testing processes are depicted in Fig. 2. We use the R software and its libraries caret (for creating models), tidyverse (for data managing), e1071 (for calculating metrics), and pROC (for creating curves).

As shown in Fig. 2, we apply data-balancing after the data cleaning step (filtering and pre-processing). This balancing includes down-sampling, up-sampling, synthetic minority oversampling technique (SMOTE) [51], and Rose. Note that, for the down-sampling method, we also use an external sampling approach to account for the reduced number of data points of the positive class (COVID-19). For each of the data-balancing methods, we perform a re-sampling algorithm based on KFRCV. Each model is trained with both five and ten folds. Additionally, the k-fold cross-validation approach is repeated ten times to estimate mean values and their confidence intervals for obtaining performance metrics accuracy, AUROC, sensitivity, and specificity. The training process is performed for different types of ANN, RF with varying numbers of trees, and SVM with different kernel functions. We perform a hyper-parameter optimization during re-sampling (tuning length = 15). We calculate the confusion matrix for each of the models to determine accuracy, AUROC, sensitivity, and specificity values. The details about each ML technique and its hyper-parameter optimization are discussed next.

Fig. 2. Algorithm followed for data analytics, model building, and sampling.

ML hyper-parameters were tuned using grid and random search. RF models were created using different numbers of trees (100, 300, 500, 1500, and 2000). For tuning parameters, the number of variables available for splitting varied randomly, at each tree node, ten times. For SVM, linear, polynomial, and radial kernel functions were compared. For SVM tuning parameters, different costs (for each kernel), sigma (for radial), degree, and scale (for polynomial) were implemented. These parameters change 15 times randomly. Table 2 presents the range and selection of the values of parameters in SVM and RF for better understanding. ANN models with both multi-layer and multi-decay search were developed. For tuning parameters, different numbers of layers, neurons, and decay were also considered. These numbers change throughout a grid search where, for multilayer, the number of layers (first layer) goes from 1 to 3, and the number of neurons from 1 to 10. For subsequent layers, we considered 0 to 10 for the number of neurons. For ANN with multi-decay search, we assumed one layer with neurons ranging from 1 to 10. We also used a grid search for the decay parameter of the ANN. In multi-layer ANN, decay takes five random numbers in the range 10${}^{-10}$ to 10. For the multi-decay ANN, 30 numbers were randomly chosen from the same range.

Table 2.Range and selected values for the indicated ML technique, model, and parameter.

ML technique	Model*	Parameter	Range	Selected
SVM	Radial	C	[0.01, 788]	11.55
Linear	Sigma	[5.87E–4, 0.1]	0.01
C	[0.05, 880]	3.20
Polynomial	C	[4.74E–2, 186]	0.99
Scale	[1.56E–5, 1.73]	1.73
Degree	{1, 2, 3}	1
RF	100	Mtry**	{1,…,97}	81
300	74
500	97
1500	97
2000	81
where Model*: refers to the kernel function or to the number of trees for the SVM or RF techniques, respectively; and Mtry**: is a parameter corresponding to the number of variables available for splitting at each tree node in RF.

The CNN was built based on conventional structures, including input, three convolution layers (16, 32, and 64 filters with 3 $\times$ 3 pixels), max-pooling (2 $\times$ 2 pixels), and ending with a fully connected layer. A rectified linear unit was considered as the activation function, used after each convolution layer and for the fully connected layer. The hyper-parameters selection considered different configurations that were tested in models of varying complexity. The model configuration that presented the best performance in terms of the loss and accuracy functions was chosen. The model was implemented using the Google cloud platform (https://cloud.google.com, accessed: 5 June 2021).

3. Results

This section presents the results of our study, which include an exploratory data analysis, ML techniques for COVID-19 versus pneumonia patients and COVID-19 versus healthy patients, as well as the hyper-parameters selection for COVID-19 versus healthy cases.

3.1 Exploratory data analysis and global performance of ML techniques

Table 3 reports the descriptive statistics for the data under study. There are 3046 cases (instances) of healthy subjects (44% females), 62 cases of pneumonia (53% females), and 174 COVID-19 positive patients, with 148 cases with known sex (38% females). COVID-19 patients have a mean age of 53.4 years (SD = 20.7 years), pneumonia patients have a mean age of 43.7 years (SD = 18.6 years), and healthy patients have a mean age of 45.3 years (SD = 16.5 years). From this table, observe that ages and proportions of females and males are, in general, relatively similar in statistical terms with non-significant p-values at 1%, except for proportions in healthy and COVID-19 groups. However, a much larger number of healthy cases (compared to COVID-19 and pneumonia) is detected. Since the instances of COVID-19 and pneumonia are less, four different data-balancing methods were used: down-sampling, up-sampling, SMOTE, and rose. The data-balancing methods were applied before and during the KFRCV re-sampling process. For tuning, threshold, and model selection, we considered those that maximize the AUROC.

Table 3.Summary statistics of the indicated data set.

	Female patients	Male patients	p-value	Total
Group	N (%)	Age mean (SD)	N (%)	Age mean (SD)	% (b/z)	Age	N (%)	Age mean (SD)
Healthy	1345 (44)	44.9 (16.6)	1701 (56)	45.7 (16.4)	$<$ 0.001${}^{(\ast \ast )}$	0.170${}^{n.s.}$	3046 (92.9)	45.3 (16.5)
Pneumonia	33 (53)	38.2 (19.1)	29 (47)	48.1 (18.0)	0.590${}^{(n.s.)}$	0.041${}^{(n.s.)}$	62 (1.9)	43.7 (18.6)
COVID-19	56 (38)	53.8 (20.9)	92 (62)	53.1 (20.5)	$<$ 0.001${}^{(\ast \ast )}$	0.842${}^{(n.s.)}$	148 (4.5)	53.4 (20.7)
Total	1434 (43.6)	45.1 (16.8)	1822 (55.5)	46.2 (16.7)	$<$ 0.001${}^{(\ast \ast )}$	0.105${}^{(n.s.)}$	3256 (100)	45.1 (16.7)
where ${}^{\left(b/z\right)}$: binomial or z-test, ${}^{\left(W/t\right)}$: Wilcoxon or t-test, ${}^{(n.s.)}$: non-significant, ${}^{(\ast \ast )}$: significant at 1%.

The classification performance of AUROC, sensitivity, and specificity metrics for each ML technique is shown in Fig. 3 through boxplots. The left panel depicts this performance to discriminate among COVID-19 and healthy patients, whereas the right panel displays the results for COVID-19 and pneumonia cases. The figure shows the statistical distribution of the results obtained based on all repeated cross-validation processes and all instances of hyper-parameter search (the total number of trained and tested models). Note that very similar distributions are detected in most cases, except for the RF model when comparing COVID-19 and pneumonia groups, regarding the sensitivity and specificity metrics, where a higher variability and asymmetry are assessed.

Fig. 3.Boxplots of the global performance of the indicated ML technique, metric, and group comparison.

3.4 Hyper-parameters selection for COVID-19 versus healthy cases

Table 6 reports the metric performance for different models defined by hyper-parameters of the ML techniques. Results are presented for the down-sampling method as it reported the best results. Focusing on AUROC values, most models perform similarly, with the best results being obtained by SVM, independently of the chosen kernel function used. Results are described as the mean value with its 95% confidence interval. Also, note that such an interval is narrow, in most cases, indicating a stable performance for most models. A second important observation is that all SVM models presented the best sensitivity-specificity balancing.

Table 6.Results (in %) for the mean and its 95% CI of the indicated ML technique, model, and metric.

ML technique	Model	k-folds	AUROC	Sensitivity	Specificity
SVM	Radial	10	0.90 (0.90, 0.90)	0.79 (0.77, 0.81)	0.83 (0.81, 0.85)
Polynomial	10	0.89 (0.88, 0.90)	0.81 (0.79, 0.83)	0.81 (0.79, 0.82)
Linear	10	0.90 (0.88, 0.90)	0.80 (0.78, 0.82)	0.81 (0.79, 0.82)
RF	1500	10	0.89 (0.87, 0.90)	0.76 (0.74, 0.78)	0.82 (0.80, 0.83)
2000	10	0.89 (0.87, 0.90)	0.76 (0.74, 0.77)	0.82 (0.80, 0.84)
300	10	0.88 (0.87, 0.90)	0.76 (0.74, 0.78)	0.82 (0.80, 0.84)
500	10	0.88 (0.87, 0.89)	0.76 (0.74, 0.79)	0.82 (0.80, 0.84)
100	10	0.88 (0.87, 0.89)	0.75 (0.74, 0.77)	0.81 (0.79, 0.83)
ANN	Multi-decay	5	0.89 (0.88, 0.90)	0.80 (0.77, 0.81)	0.83 (0.81, 0.85)
Multi-layer	5	0.86 (0.85, 0.88)	0.76 (0.73, 0.78)	0.81 (0.79, 0.83)
Convolutional	5	0.85 (0.82, 0.87)	0.76 (0.71, 0.80)	0.74 (0.68, 0.79)

Interestingly, both SVM and RF perform better than CNN (for example, AUROC value of 0.89 for SVM against 0.84 for CNN). ANN (multi-decay, multi-layer) and CNN are trained using 5-fold cross-validation due to computational burden. In contrast, SVM and RF are trained and evaluated using 10-fold cross-validation. The high AUCROC scores obtained by both RF and SVM are depicted in Fig. 4. Note that SVM models show a better balance between the true positive rate (TPR) and FPR, resulting in a higher AUROC score.

Fig. 4.ROC for RF (left) and SVM (right) with the indicated number of trees and kernel.

For a better model performance analysis, Table 7 presents accuracy and F1-score. Despite being well-known performance metrics, models are selected by maximizing the AUROC. Therefore, these values do not represent the best performance dimensions of the process.

Table 7.Results (in %) for the mean and its 95% CI of the indicated ML technique, model, and metric.

ML technique	Model	Accuracy	F1-score
SVM	Radial	81.20 (80.03, 82.38)	80.65 (79.35, 81.95)
Polynomial	80.71 (79.39, 82.02)	80.59 (79.20, 81.99)
Linear	80.31 (78.89, 81.73)	80.08 (78.57, 81.58)
RF	100	76.81 (75.26, 78.35)	76.06 (74.45, 77.67)
300	76.46 (75.02, 77.90)	75.69 (74.16, 77.23)
500	76.75 (75.31, 78.19)	76.00 (74.46, 77.55)
1500	76.67 (75.22, 78.11)	75.80 (74.24, 77.36)
2000	76.49 (75.07, 77.92)	75.73 (74.19, 77.27)

Computational times were extracted for the best models and portrayed in Table 6. Besides the computational burden mentioned, Table 8 also shows that ANN models use almost 500% higher computational time than the next most lasting model (RF-2000). Note that computational times for CNN are not reported. CNN models were trained using the Google cloud platform with significantly different computational technical characteristics.

Table 8.Runtimes (in seconds) of overall cross-validation for the indicated ML technique, model, and metric.

ML technique	Model	Runtime (in seconds)
SVM	Radial	103.11
Linear	876.14
Polynomial	87.83
RF	100	111.20
300	323.56
500	535.30
1500	1598.52
2000	2138.20
NN	Multi-decay	9537.21
Multi-layer	9496.19
Convolutional	-

4. Discussion and final remarks

Conventional chest radiography can be helpful to identify different patterns of lung alterations in COVID-19 patients, allowing us to grade disease severity based on total lung involvement. In addition, X-ray has a shallow radiation dose, and it is more accessible and cheaper than other diagnostic imaging methods, such as CT scans. Furthermore, trained professionals are needed to interpret the results. Thus, the support of tools from artificial intelligence for fast and accurate diagnosis is crucial, especially when resources are scarce, and response times must be immediate to avoid the spread of the disease. This research demonstrated the capabilities of ML techniques to identify COVID-19 cases from healthy or pneumonia patients using X-ray images. The flexibility of simple models and accessibility to X-ray images are needed, particularly for health systems of hospitals located in rural zones that lack infrastructure. This limitation is common in developing countries, as in Latin America, where the number of COVID-19 cases has reached dramatically high levels in countries like Argentina, Brazil, Chile, Colombia, and Peru. Besides, the lack of appropriate infrastructure generates an increasing need for diagnosis techniques that can be efficiently utilized and updated.

In this context, ML techniques offer the appropriate flexibility without a compromise in predictive accuracy. However, ML models have not been thoroughly studied in the existing literature, as they are generally used as benchmarks for more complex algorithms, such as CNN or other DL models. Nevertheless, often CNN models are an impractical solution due to their high computational requirements and lack of access to data and infrastructure in rural areas of developing countries. Thus, the results obtained in our study offer higher discriminatory capability in comparison to other standard clinical-based diagnosis techniques. For instance, scientists from Johns Hopkins Medicine [52] have shown that, over the four days of infection before the typical time of symptom onset, the mean FNR is 67% with a 95% confidence interval of 27% to 94%. Another recent systematic review reinforces the need for repeated testing in patients with suspicion of COVID-19 infection, given that up to 29% of patients could have an initial PCR false-negative result [53]. Furthermore, PCR’s limitations include the need for higher-level laboratory facilities, proper swab sample techniques, clinician experience, and error-free patient samples [54, 55]. Hence, if clinical suspicion is high, an infection should not be ruled out based on a PCR test only [52]. Therefore, as shown in this study, ML models are suitable tools that provide comparable or better results than clinical tests. Thus, ML models can be used as a support tool for fast initial screening of patients when diagnosing for COVID-19.

The full factorial design of experiment used in this paper provided a more stable discriminatory accuracy or classification power of ML techniques. Indeed, we accounted for design parameters, such as different data balancing methods, that must be studied in a context that arises in a COVID19 pandemic (a large number of PCR tests and a small number of positive cases). Among the evaluated models, SVM obtained the best results when correctly identifying infected patients. SVM with different kernel functions provided AUROC values as high as 90% with well-balanced sensitivity-specificity scores of approximately 80% with narrow confidence intervals.

It is important to note that the results presented in our study can be further improved. Our findings were obtained from a relatively small set of positive infected patients (174) with publicly available X-ray exams. Our findings must be validated and improved with access to both larger sets of images and exams with better image quality. Therefore, there seems to have great opportunities to further enhance the accuracy of ML techniques’ results. This is significantly important since ML is transforming the current medical practice and plays a critical role in the digital healthcare era [56]. These ML techniques help doctors to improve their diagnostic ability, impacting patients’ prognostic, quality of life, and effective treatment prescriptions. Moreover, the use of such techniques has also implications for the health systems, reducing exam time, saving costs for the medical practice, and improving daily productivity [57].

Author contributions

NB, MT, and FF designed the study, collected the data, and created the models and codes. NB, FF, and VL performed the data analysis. NB, MT, FF, and VL drafted the paper. GD and NG performed a clinical development and validated the analysis and results. NB, FF, and VL revised the paper critically for important intellectual contents. All authors approved the final version to be published.

Ethics approval and consent to participate

Not applicable.

Acknowledgment

The authors thank four Reviewers for their constructive comments on an earlier version of this paper.

Funding

This research was partially supported by grants FONDECYT 11181176 (F. Feijoo) and 1200525 (V. Leiva) from the National Agency for Research and Development (ANID) of the Chilean government under the Ministry of Science, Technology, Knowledge, and Innovation.

Conflict of interest

The authors declare no conflict of interest. Víctor Leiva is a member of the Editorial Board of this journal.