Derivation and validation of an automated electronic search algorithm to identify patients at risk for obstructive sleep apnea

Background. Automated extraction of data from electronic health records has allowed high-quality retrospective analyses of large cohorts. Objectives. To derive and validate an automated electronic search algorithm to identify surgical patients with a diagnosis of or at high risk for obstructive sleep apnea (OSA). Methods. From 558 adult patients who underwent surgery from January 1, 2011, through December 31, 2015, we constructed a derivation cohort of 100 subjects selected using the initial search algorithm to have equal numbers of patients with high and low likelihood of having OSA. This algorithm conducted a free-text electronic search of patient diagnoses and interrogated results of a preoperative checklist that specifically queried patients regarding OSA history and screened for OSA risk using Flemons criteria. The derivation cohort was then manually reviewed to identify patients with OSA risk and results were used to refine the algorithm. Second, the algorithm was validated with the other 458 patients (the validation cohort). The sensitivity and specificity were compared again with manual chart review of the respective group. Results. In the derivation cohort, the automated electronic algorithm achieved a sensitivity of 98.2% and a specificity of 100.0% compared with the manual review. In the validation cohort, sensitivity was 100.0% and specificity was 98.4% in this comparison. Conclusion. An automated electronic search algorithm was developed that interrogates electronic health records to identify, with a high degree of accuracy, surgical patients with a diagnosis of or at high risk for OSA.


INTRODUCTION Background and Importance
Automated extraction of data from electronic health records (EHRs) has become a sophisticated tool that has allowed investigators new avenues for conducting high-quality retrospective analysis of large patient cohorts.(1) These automated techniques have been used with a high degree of accuracy to determine preoperative prognosticators such as the Charlson Comorbidity Index (2) and identify postoperative complications, such as postoperative myocardial infarction, (3) in large cohorts of surgical patients.Surgical patients with a diagnosis of or at high risk for obstructive sleep apnea (OSA) have increased risk of postoperative pulmonary complications.(4,5) OSA is frequently undiagnosed, and a substantial proportion of adult surgical patients should be considered at high risk for undiagnosed OSA.(6,7) In response to the many undiagnosed cases, several assessment tools have been developed to screen surgical patients for OSA.(8)(9)(10) Investigators have proposed that all surgical patients be screened preoperatively for OSA risk and that this information be used in decision-making algorithms to tri-age surgical patients to appropriate levels of postoperative monitoring.(11) The current practice at our institution during the preoperative nursing check-in is to screen surgical patients for OSA diagnosis and to screen those without an OSA diagnosis using the Flemons criteria.(9) Reliable data identifying surgical patients with the diagnosis of or at high risk for OSA would be useful for investigators conducting outcomes research on large patient cohorts where manual data extraction is not practical.However, evidence is limited on the derivation and validation of an electronic search technique that identifies these patients and on its effectiveness compared with manual review of EHRs.Herein, we describe the derivation and validation of an automatic electronic search algorithm for identifying surgical patients with the diagnosis of or at high risk for OSA.

Objective
The study's primary objective was to derive and validate an automated electronic search algorithm that identifies which surgical patients should undergo preoperative screening for OSA because of either previous diagnosis or high risk, using an assessment tool.The secondary objective was to calculate sensitivity and specificity values of our electronic search algorithm compared with the reference standard of manual comprehensive EHR review.

Participants
The Mayo Clinic Institutional Review Board approved this study.Consistent with Minnesota Statute 144.295, the study included only patients who provided authorization for research use of their EHRs.The setting of this study was the Mayo Clinic Hospital -Rochester Campus, Minnesota.Participants in this report were 558 surgical patients who underwent general anesthesia at our institution from January 1, 2011, through December 31, 2015-a subset included in a separate and unrelated retrospective study designed to assess postoperative outcomes related to anesthetic management variables.For the present study, we randomly selected 100 patients to construct the derivation cohort and selected the other patients (n=458) for the validation cohort.

Manual Data Extraction Strategy
For the present study, manual review of the EHRs was considered the gold standard for identification of patients with or deemed to be at high risk for OSA.Surgical patients on arrival at our institution's presurgical area undergo a preoperative checklist by a registered nurse.This checklist includes a question of whether the patient has a history of OSA, with the answer marked yes or no.If the answer is negative, the patient is screened for OSA using the Flemons criteria, to categorize patients with high or low risk of OSA.(9) The results of this inquiry are recorded in the EHR.For this study, the EHRs of the patient cohort were manually reviewed by the lead author (O.O.O.).The review consisted of evaluating the EHR antecedent to the date of surgery, including past medical history and diagnosis sections of all clinical notes, as well as information obtained during the nursing preoperative checklist.

Automated Electronic Data Extraction Strategy
Our institution uses the United Data Platform, a clinical data warehouse that obtains, consolidates, and standardizes all clinical data collected within the institution (eg, demographic information, diagnoses, clinical notes, hospital flow sheets).The platform can be interrogated to extract clinical data through a Web-based querybuilding tool called Advanced Cohort Explorer (ACE).Using Boolean logic, researchers can use the ACE system to develop distinct text search strategies of the United Data Platform and identify pertinent clinical data, such as specific keywords.To develop the electronic search query for OSA, we entered synonyms, abbreviations, and medical acronyms associated with OSA into an ACE text query.Furthermore, a comprehensive list of terms to exclude patients who did not have OSA was developed to make the electronic search algorithm more specific.For instance, we excluded such phrases as "no history of, " "denies, " "rule out, " and "negative for" OSA.To establish a more uniform methodology, we restricted the application of the automated algorithm to note searches to the Diagnosis section of a patient's clinical notes.In addition, we interrogated the nurse-administered preoperative checklist obtained on the day of surgery.Patients who provided an affirmative response (ie, "yes") to the nurse's query regarding a history of OSA were coded by ACE as having OSA.Patients who denied a history of OSA (ie, a "no" response) subsequently underwent OSA screening with Flemons criteria.( 9) Patients whose Flemons scores were consistent with a high risk of OSA were coded by ACE as having OSA.The results of queries were combined, and when a patient had a diagnosis of OSA, affirmed a history of OSA during the checklist, or had a high risk of OSA, the patient was coded to "yes" for OSA.Subjects were divided into derivation and validation cohorts.The derivation cohort of 100 subjects was constructed using the initial OSA electronic search algorithm to consist of two equal samples of subjects at high or low likelihood for OSA.The derivation cohort underwent manual review of subject medical records to detect a true risk or presence of OSA.On the basis of obtained information from the manual review, the initial algorithm was refined to incorporate mismatches between automated search and manual review.Using this refined algorithm, we queried the remainder of the study patients as a validation cohort (figure 1).Disagreements between the automated searches and the manual searches were adjudicated by the senior author (T.N.W.), to whom the search results were masked.These 2 authors (O.O.O. and T.N.W.) were not involved in algorithm design and implementation.

Statistical Analyses
The study subjects were divided into derivation and validation cohorts as described above.For each sample, the sensitivity and specificity of the final automated electronic search algorithm for identifying OSA were calculated using the manual review of the EHRs as the gold standard.Findings are summarized using point-estimates and corresponding 95% exact binomial confidence intervals (C.I.).Statistical software (JMP version 10.0; SAS Institute Inc) was used to compare and validate the automated search vs the manual review.

RESULTS
From a pool of 558 adult surgical patients, a derivation cohort of 100 subjects was selected with an initial automated electronic search algorithm for equal numbers of patients with high or low likelihood for OSA.The validation cohort consisted of the remaining 458 subjects.A comparison between manual chart review for OSA and the final automated electronic search algorithm is summarized in the table 1.In the derivation cohort, the automated digital algorithm achieved a sensitivity of 98.2% (95% C.I., 90.3%, 99.9%) and a specificity of 100.0%(95% C.I., 92.0%, 100.0%) compared with the manual review.For the validation cohort, the sensitivity of the automated digital algorithm was 100.0%(95% C.I., 89.1%, 100.0%) and the specificity was 98.4% (95% C.I., 96.0, 99.3).

DISCUSSION
This study showed that an automated electronic search algorithm can be developed to interrogate the EHR to identify surgical patients with a diagnosis of or at high risk for OSA in a clinical practice where all surgical patients are screened for OSA.The comparison between the algorithm and the manual chart review found that this automated strategy performs favorably and with a high degree of accuracy.Our results add credence to the use of computerized searches of EHRs of large patient cohorts to extract clinical variables, processes, and outcomes of interest.(2,3,(12)(13)(14) The increased perioperative risk that OSA poses for surgical patients is well recognized, (4,5) and the condition has become the topic of clinical management guidelines that call for increased vigilance of affected patients.(11,15) Despite the risk, OSA may present with vague symptoms (eg, increased daytime sleepiness), and population studies have suggested that most (approximately 90%) of OSA cases are undiagnosed, (7) including OSA of surgical patients.(6) In response, clinicians have increasingly called for preoperative screening of all surgical patients for OSA.However, the gold standard to diagnose OSA-overnight polysomnography-is time consuming, expensive, and of limited access, and thus it is an impractical screening tool.Various simple assessment tools have been developed to screen patients for OSA.(810,15) These caveats have implications for the development of an automated search algorithm.Because reliance on billing codes or free-text searches of clinical notes and diagnoses are inadequate, OSA is usually undiagnosed.Although these OSA screening tools perform reasonably well, they are by no means completely accurate.( 16) Yet, patients identified through this automated algorithm should be considered at high risk for OSA.Further, although an automated search algorithm still requires data entry into the EHR, it can greatly facilitate the transparency of diagnosis, which although textually recorded, may be buried within redundant health records and be unnoticed by perioperative providers.

LIMITATIONS
This study has the inherent limitations of a retrospective study design.Several aspects of our clinical practice may limit the usefulness of an automated search strategy.Our practice has a registered nurse screen the surgical patients for OSA risk during the preoperative checklist.However, many practices have not adopted this practice of universal screening, (17,18) and in such a clinical setting our automated digital search algorithm would be less accurate.Undoubtedly, some patients bypass the nurse-administered checklist and therefore the OSA screen, but such cases typically occur only in emergencies or when a patient is already intubated and mechanically ventilated.In addition, our practice to assess sleep apnea uses the Flemons criteria.(9) This use raises a question of portability when the algorithm is applied in a practice that uses an alternative assessment tool, such as STOP-BANG.(8) Because our automated search algorithm relies on assessment tools that screen for OSA, its accuracy is limited by the performance of the screening tool.( 16) In addition, the data entries in EHRs represent a dynamic process, and in OSA, conditions may change with time (corrective oral surgery and weight loss).Therefore, these factors always need to be considered in final risk assessment.Incomplete data points or inconsistencies with the text search phrases can limit the applicability of this search algorithm.However, this latter limitation likely accounts for a small number of patients in the database.Lastly, the algorithm was designed for retrospective identification of surgical patients at risk for OSA.An area of future direction is to develop this algorithm for surveillance purposes.

CONCLUSION
The present study details the development of an automated digital search algorithm that interrogates EHRs to identify not only surgical patients with the diagnosis of OSA but also those at high risk for OSA.These results reflect our clinical practice, where all surgical patients are preoperatively screened for OSA using an assessment tool.

CLINICAL RELEVANCE STATEMENT
OSA is common, is frequently undiagnosed, and is associated with increased risk of postoperative complications in surgical patients.This study describes the development, derivation, and validation of an automated digital search algorithm that interrogates EHRs to identify with a high degree of accuracy the surgical patients who have or are at high risk for OSA.OSA, Obstructive Sleep Apnea.Sensitivity and Specificity are presented using point estimates and 95% exact binomial confidence intervals.

Table 1 .
Performance of an automated electronic search algorithm for the detection of obstructive sleep apnea by the interrogation of electronic health records of surgical patients.