Deep learning algorithm performance compared to experts in visual evaluation of interior vena cava collapse on ultrasound to determine intravenous fluid need in dehydration management

1. Introduction

Volume depletion secondary to various underlying causes is a frequent scenario in pediatric emergency patients [1]. In fact, worldwide, diarrhea claims tens of thousands of children’s lives through dehydration [2]. However, determining whether a child has significant enough volume depletion to warrant intravenous fluid resuscitation or will simply recover with oral hydration can be surprisingly challenging for many providers [3]. Placing an intravenous line into a child can be difficult and traumatic for patient and parent alike, thus making unnecessarily placed lines even more significant in cases where oral hydration was safe to pursue and likely to work [4]. Signs which may point to volume depletion in adults can be absent in children until they are critically ill [5]. Scores and decision rules, designed to differentiate if young children are significantly dehydrated or not, are not uniformly used and have questionable clinical utility [6].

Ideally, all pediatric patients at risk for significant volume depletion could be noninvasively screened and the degree of hypovolemia identified or at least the need for intravenous fluids ruled out. Point-of-care ultrasound (POCUS) offers such a tool and the most common POCUS approach is interrogation of the inferior vena cava (IVC). Evaluation of the IVC has been shown to be a reliable indicator of volume status and results in good interrater reliability [7]. The largest body of IVC and volume status or fluid responsiveness literature has come from adult patients. In adults, measuring collapsibility of the inferior vena cava (cIVC) has been shown to predict fluid responsiveness fairly well, when performed in non-ventilated patients by expert sonologists (AUROC = 0.82) [8]. However, work by another group of authors raised concerns that the test characteristics were not so favorable when novice sonologists performed the POCUS examination (AUROC = 0.69) [9]. Other researchers have also reported, poor to moderate interrater agreement in cIVC measurement among non-expert sonologists [10, 11, 12]. Training clinicians with no ultrasound experience, to a level at which they attain competence with POCUS use and accurately measure cIVC to guide IV fluid resuscitation, is emerging as the most significant hurdle to broad adoption of this approach.

Prior work suggests that euvolemic children will not have complete IVC collapse on inspiration, thus on ultrasound the anterior and posterior walls never touch in a sagittal plane throughout the proximal IVC [13]. Such patients can be treated without intravenous fluid infusion and be discharged home. However, significant collapse is more frequently seen in children with dehydration. One practical limitation to POCUS evaluation of the IVC is the skill required to not only obtain the correct image but then rapidly interpret IVC behavior in real time at the patient’s bedside. This may be made more difficult in a busy environment or with a sick, poorly cooperative child. Published studies have shown that a good cutoff point for determining if a child requires intravenous fluid administration is the presence of complete IVC collapse on ultrasound —suggesting IVF is required [13]. While, on the surface, appearing simple by description, the procedure can be difficult for novice ultrasound users. A tool to automatically identify when the IVC collapses completely could prove useful in broader clinical practice. Adapting deep learning (DL) as a solution to overcome the lack of adequate training, thereby enabling widespread use of POCUS targeted resuscitation more feasible, has been proposed [14].

In this study, researchers explored the development of a Deep Learning (DL)/Artificial Intelligence (AI) algorithm which would identify occurrence of complete IVC collapse. Accurate and rapid identification of either the presence or lack of complete IVC collapse would allow novice users to identify children likely to be go home without bounce back for dehydration. A DL algorithm was created using ultrasound videos of the IVC in pediatric patients being evaluated for volume status, to identify those with complete IVC collapse among spontaneously breathing patients. Researchers then compared the performance of the DL algorithm against 2 POCUS experts on new (not previously part of the DL training dataset) IVC videos.

2. Methods

2.1 Study design

Researchers used de-identified ultrasound video data from a prior prospective observational study of patients ranging up to 21 years of age, who presented to the ED [15]. Videos of IVC scans were obtained for patients with a history of diarrhea with suspected dehydration and or a history of emesis being treated with ondanseton. Patients who were unstable were not included in this dataset. Patient were scanned after being triaged, but prior to fluid resuscitation or rehydration. Researchers performed an IVC ultrasound examination which included obtaining and recording a sagittal view of the proximal IVC during several respiratory cycles. This study was approved by the Icahn School of Medicine at Mount Sinai institutional review board, reference number 15-1599.

Providers used a Sonosite M-Turbo ultrasound system with a phased array transducer (P21 -5 to 1 MHz). The researchers used a cardiac preset for imaging the IVC and saved 6 second retrospective cine loops, Fig. 1. Complete IVC collapse was defined as opposing walls of the IVC coming into contact with each other at any point, Fig. 2. A total of 5 researchers enrolled patients and scanned IVCs, 4 pediatric emergency medicine (PEM) attendings and 1 PEM resident. The researchers scanned with the transducer in the subxiphoid location using the liver as an acoustic window. A sagittal plane over the mid proximal IVC was attained, visualizing the IVC entry into the right atrium, Fig. 1. Researchers focused their attention just caudal to the junction of the hepatic veins with the IVC.

Fig. 1.Shows sagittal proximal IVC (inferior vena cava) segment appearance on ultrasound at maximum expiration in one of the study patients.

Fig. 2.Shows a patient whose IVC collapses completely upon inspiration from left to right. Complete IVC collapse was defined as opposing walls of the IVC coming into contact with each other at any point.

2.2 Study data

A total of 166 individual patient videos of the IVC were available for algorithm training. A standard 80%/20% training and validation split was applied to the 166 patient videos for algorithm training. Video data was in MP4 format. Each video was reviewed by two PEM POCUS providers and rated as completely collapsing or not. Complete collapse was defined as the anterior and posterior walls of the proximal IVC touching at any point during inspiration. An additional new 18 videos, data not seen previously by the algorithm and not from the same training set, were used for algorithm testing as real world data, Fig. 3.

Fig. 3.The study process flow in graphical form.

2.3 Algorithm design

Researchers used the Anaconda package manager with Python programming language version 3.72, to manage packages and facilitate scripting. Based on prior experience, a decision was made to utilize a publicly available Keras-based (a python deep learning library or framework) VGG-16 bidirectional LSTM DL algorithm. The VGG-16 model can be accessed from several public sources, including github.com (an online scripting repository). An early version of a convolutional neural network (CNN), VGG-16 uses 16 layers and has been shown to be superior for ultrasound DL applications when compared to modern and more complex CNNs [9]. The LSTM architecture allows analysis of individual ultrasound frames by the VGG-16 CNN, while it tracks temporal changes and relationships in real-time video to create an algorithm for ultrasound video analysis, Fig. 4. An LSTM such as this one has additional layers in the network algorithm architecture which tracks temporal changes between images (changes in the ultrasound image in a cine loop such as a beating heart). In contrast to standard LSTM networks, which are unidirectional, the bidirectional aspect allows temporal related information to flow in both forward and reverse directions through the algorithm. This feature makes the architecture more sensitive and specific for detecting any changes from frame to frame within a video, thereby better identifying action. It further improves a network’s understanding of the context the motion occurs in. Investigators used weights trained on the UCF-101 Action Recognition Data Set (University of Central Florida) for the VGG-16 bidirectional LSTM. Weights are learnable parameters in neural networks and account for a CNN’s ability to interpret images.

Fig. 4.Shows the LSTM architecture developed and employed in the study.

This appears to be the first instance of LSTM DL architecture being applied to pediatric IVC measurement. The majority of prior LSTM work has focused on ECG and EEG analysis as well as computed tomography image analysis [16, 17, 18]. These imaging types pose different challenges to analysis than due to limited number of frames requiring analysis and change occurring over a short period of time. Assessing pediatric IVC behavior in acute settings results in considerable movement due to rapid breathing, heart rate and in younger patients poor cooperation. LSTM has been previously applied to adult IVC assessment using similar technique, but adults have differing physiology and larger anatomic size than children [19, 20]. LSTM architecture has been used for needle tip analysis and possible ultrasound guidance but only on non-human tissue in a laboratory setting [21]. The closest other LSTM work with ultrasound targeted lung ultrasound analysis in adults but utilized a mono-directional LSTM network with DenseNet 201 embedded in the architecture. However, the VGG 16 utilized in the current approach may be more efficacious when analyzing ultrasound images [22, 15].

The bidirectional LSTM algorithm was trained by incrementally manipulating learning rates, optimizers, batch size and learning rates (which are all adjustable settings which affect the performance of a CNNs during training) during training for optimal accuracies and training times. Simultaneously researchers worked to avoid exploding gradients (dramatic changes in learning parameters of the CNN during training) which can result in training failure. The total epochs used, defined as one round of training through all of the data, was adjusted as needed to optimize results while avoiding overfitting. Analyzing 90 frame runs proved to be the most efficacious approach.

3. Results

3.1 Outcomes

Table 1 provides reviewers’ interpretation of collapse and DL algorithm’s prediction of collapse completeness for all 18 videos. The DL algorithm and both reviewers were able to complete review and categorization of each test video. The POCUS reviewers felt all videos were adequate for collapse assessment. None of the patients who had partial IVC collapse required IVF resuscitation or admission solely for dehydration. All patients with incomplete IVC collapse (Fig. 5) were able to be discharged from the ED and did not return to the same medical system, based on chart review. This DL algorithm was able to complete analysis of each previously unseen real world test video. Algorithm agreement with the PEM POCUS attending was substantial with a Cohen’s kappa of 0.78 (95% CI 0.49 to 1.0). Algorithm agreement with PEM resident was substantial with Cohen’s kappa of 0.66 (95% CI 0.31 to 1.0). The PEM resident and PEM POCUS attending also had substantial agreement, yielding a Cohen’s kappa of 0.67 (95% CI 0.32 to 1.0).

Table 1.Comparison table of Deep Learning algorithm, PEM resident and PEM POCUS attending assessment of IVC collapse.

DL algorithm prediction	PEM resident	PEM POCUS attending
Incomplete collapse	Complete collapse	Incomplete collapse
Complete collapse	Complete collapse	Complete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Complete collapse	Complete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Complete collapse	Complete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Complete collapse	Complete collapse
Complete collapse	Complete collapse	Complete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Incomplete collapse	Incomplete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Incomplete collapse	Complete collapse
Complete collapse	Complete collapse	Complete collapse
Complete collapse	Complete collapse	Complete collapse
Complete collapse	Complete collapse	Complete collapse
Incomplete collapse	Incomplete collapse	Incomplete collapse
Complete collapse	Complete collapse	Incomplete collapse

Fig. 5.Shows a patient whose IVC collapsed incompletely upon inspiration.

The DL algorithm disagreed with the PEM POCUS attending in two patients. Both disagreements resulted from a prediction by the DL algorithm of complete IVC collapse, while the PEM POCUS attending felt the IVC did not collapse completely. The DL algorithm and PEM resident disagreed in three instances with the algorithm predicting complete IVC collapse twice while the resident rated the IVC as not collapsing completely. In the third disagreement the DL algorithm predicted incomplete collapse while the resident judged the IVC to collapse completely. In two out of the three disagreements between the DL algorithm and resident, the POCUS attending agreed with the algorithm. A review of the videos which caused disagreement did not reveal significant challenges with image artifact or shadowing involving the IVC when compared to the remaining test videos.

4. Discussion

In this study, researchers developed a novel AI algorithm for IVC ultrasound interpretation in pediatric emergency patients being evaluated for dehydration. Considerable prior work has appeared in the medical literature including comparison of the IVC diameter in maximum and minimum and its relationship with the aorta diameter [23]. This approach is quite laborious, requiring measurement of the aorta and IVC maximum and minimum diameters to establish IVC collapsibility and is rarely used in clinical practice, making it essentially obsolete [13]. However, even the process of IVC measurement for maximum and minimum diameters, to establish IVC collapsibility, takes time and requires additional ultrasound experience that some novices may not possess [24]. Prior work has also established that operators are able to visually categorize the degree of IVC collapse into discrete categories showing substantial interobserver agreement [25]. Additionally, investigators have previously shown that IVC collapse duration, when assessed by blinded reviewers, correlated well with PO versus IV rehydration in pediatric patients [13]. Further, results indicated that lack of IVC collapsibility correlated with successful PO rehydration and discharge from the ED without patient bounce back [13].

Recently, attention has been focused on the potential utility of AI in POCUS applications and the promise for decreased cognitive burden in obtaining focused findings from ultrasound examinations by novices. More complex applications have also been explored and some are available for advanced ultrasound imaging to improve workflow and decrease examination times [26]. Creating a DL application which would allow clinicians with less ultrasound training to employ focused ultrasound in an equivalent way to POCUS experts would greatly increase patient access to a useful, non-invasive, bedside diagnostic and monitoring tool. While expert POCUS users are be able to quickly estimate degree of IVC collapse in a child on ultrasound, novices and less trained POCUS users may struggle to simply obtain and keep a mid-sagittal view of the proximal IVC. An algorithm which automatically “visually” interprets the degree of IVC collapse, without the novice operator having to take more steps, is directly in line with the promise of AI impact in clinical practice, especially imaging interpretation.

Most prior work with IVC assessment automation in ultrasound focused on animal models, created complex offline interpretation approaches with multiple steps required or compared AI results with carefully obtained measurements by ultrasound technologists [27]. Chen et al. [27], attempted to automate IVC collapse analysis and constructed a DL algorithm capable of identifying and measuring the IVC diameter in a porcine model. The DL algorithm was trained on 48 data sets of IVC images which included both dynamic and static images. Investigators then compared the DL algorithms IVC diameter measurements to those of expert sonographers. The researchers reported the algorithm successfully identified the IVC 98% of the time and most IVC diameter measurements made were within 15% of an expert sonologist’s read. However, the researchers utilized a stepwise process that included significant captured video clip pre-processing, filters and image size modifications. Additionally, investigators relied on Color Doppler use to identify areas of interest based on blood flow presence alone, a technique typically unneeded in POCUS practice. Although promising, the multiple cumbersome steps in the described approach is not suitable for actual clinical situations. A more promising study was published by, Belmont et al. and used Kana-de-Lucas-Tomasi (KTL) feature tracking and pyramidal segmentation to build an artificial intelligence algorithm which was capable of identifying and measuring cIVC [28]. The somewhat complex algorithm analyzed 57 cine loops from 47 non-ventilated dialysis patients and showed good agreement with POCUS expert manual measurements. More than 95% of the DL measurements were within <10% compared to expert reads). Still, the algorithm’s clinical application would likely face challenges in this format due to a requirement for clear and consistent IVC edge depiction in order to enable tracking. A requirement which can be difficult to maintain in the clinical setting.

A more practical approach is required, one that would focus on bedside application and tailored to enabling rapid interpretation in real time when used by ultrasound novices. Despite mental imagery conjured up by the term Deep Learning, if properly structured, DL can be implemented as a simple to use application and obviate the need for multiple offline steps. Deep Learning is the cornerstone of medical imaging analysis with artificial intelligence. The vast majority of work, both research and commercial, has focused on high dollar imaging modalities such as MRI and CT [29]. Plane x-rays have also generated considerable work, but ultrasound in general, lags far behind. Even less attention has been focused on POCUS related applications.

While seemingly magical in its capability to the uninitiated, DL is simply a massive amount of computations performed on ever larger portions of an image and correlating prediction outcomes with ground truth results, which indicate if the image is a cat or dog or in the case of medical images, gallbladder or heart, among others. Currently, there are at least two commercial products available, on ultrasound machines marketed in the USA, which have automatic IVC evaluation delivering a collapsibility index [20]. One of these uses a Deep Learning algorithm, which the other one is older and hard coded. Unfortunately, these algorithms are available on a small number of the ultrasound machines sold to POCUS providers and will not work with other ultrasound machine types, if they were transferable. Both of these automatic software applications depend on identifying the IVC and then tasking the software with marking IVC borders and then creating measurements from these, using M-mode tracings, followed by simple calculations. Calculation for both depends on excellent image quality, which may be very challenging in some clinical settings and is less likely to occur when ultrasound examinations are being performed by novices. The algorithm employed in this study uses a different property of Deep Learning networks, one that is similar to what experts do when they “eye-ball” an ultrasound assessment such as ejection fraction for the left ventricle. This is often referred to as the regression approach to DL algorithm function. Thus, the highest quality images with clear edges are not important for function to the same degree as current commercial versions of auto IVC evaluation tools. Additionally, the algorithm allows novices to focus on locating the proximal IVC and holding the transducer aligned over it. Researchers were able to achieve these results through a combination of a previously studied optimized CNN choice for ultrasound analysis and a fine tuned Bidirectional LSTM architecture which allowed for analysis of actual 6 second video clips of the IVC, the same as those used in clinical patient management. Therefore, this is the closest step to actually implementing and testing such an approach in an actual clinical setting.

The clinical impact of a DL algorithm that visually estimates IVC collapse is potentially significant. With fewer computational steps required there is increased efficiency and the possibility for a DL algorithm to run on simpler and smaller computer chips. For instance, a common approach for an algorithm such as this would be to take a live video feed from the ultrasound device and have the DL application running on a separate laptop or tablet in real time to assess the degree of IVC collapse. This approach implies that individual medical centers could design their own algorithms, tailored to the ultrasound devices available in their department and even fine-tuned to their patient population, which can change around the world. When a child presents to an emergency department, regardless of where in the world it is located, a novice ultrasound provider, nurse or tech, can assess volume status easily as long as they can obtain a view of the proximal IVC.

This study has multiple limitations. First the training sample size is considered quite small for DL algorithm design. A much larger dataset, one that is over 1000 individual patients would make for a more robust algorithm. Second, the test dataset used, (n = 18 patients) is also relatively small, but importantly allowed for a prospective, real life evaluation of algorithm performance, something typically absent in AI and imaging studies [30]. Implementing an executable application version of this algorithm was not performed by us, due to funding limitations and IRB permission, but is rudimentary and could be accomplished with a separate laptop that would take a video feed from an ultrasound device and in real time analyze and overlay results onto the ultrasound image. This application has not yet been tested in daily ED practice with various providers, a step for another future study, ideally once funding was available.

5. Conclusions

Researchers demonstrated that an DL algorithm trained on a small number of patient ultrasound cine loops can have substantial agreement with two blinded expert POCUS reviewers in judging complete versus incomplete IVC collapse in pediatric patient. Lack of complete IVC collapse is inversely correlated with the need for admission and IV fluid hydration, making this a useful practical tool in volume status assessment. Further, these results suggest that DL visual estimation of IVC collapse can be accurate and could be implemented by just one emergency department for their own use, even when commercial products are unavailable or unaffordable. This could expand patient access to point of care technology by enabling novices with little training to use the diagnostic tool at bedside.

Author contributions

MB, LB and JT were involved in research study design. MB and LB performed algorithm creation, training, testing and final predictions. JT, MB and LB analyzed the data and wrote the manuscript. All authors contributed to the editorial changes in the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This study was approved by the Icahn School of Medicine at Mount Sinai institutional review board, reference number 15-1599.

Acknowledgment

Researchers would like to thank Xianshun Chen (chen0040) for posting example LSTM structure code on github: https://github.com/chen0040/keras-video-classifier.git.

Funding

This research received no external funding.

Conflict of interest

Michael Blaivas has consulted for the following commercial entities within the last 12 months: Ava AG, Anavasi Diagnostics, Sonosim Inc, EchoNous Inc, Ethos Medical. The remaining authors declare that there is no conflict of interest regarding the publication of this article.

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.