The role of radiological imaging in the diagnosis and treatment of diseases has greatly expanded in the last decade [1 Sunshine JH, Hughes DR, Meghea C, Bhargavan M. What Factors Affect the Productivity and Efficiency of Physician Practices? Med Care 2010; 48(2 ): 110-7.-3 Sodickson A, Baeyens PF, Andriole KP, et al. Recurrent CT, cumulative radiation exposure, and associated radiation-induced cancer risks from CT of adults Radiology 2009; 251(1 ): 175-84.]. Images are interpreted by radiologists, and pertinent findings recorded in the form of narrative text. In practice, the radiologist should contact the referring clinician whenever a critical imaging result is present. Failure to promptly communicate critical imaging test results is not uncommon and such delays are a major source of malpractice claims in radiology [4 Gordon JR, Wahls T, Carlos RC, Pipinos II, Rosenthal GE, Cram P. Failure to recognize newly identified aortic dilations in a health care system with an advanced electronic medical record Ann Intern Med 2009; 151(1 ): 21-7.: W5.-8 Berlin L. Statute of limitations and the continuum of care doctrine AJR Am J Roentgenol 2001; 177(5 ): 1011-6.]. The Joint Commission emphasized a need for improved communication of critical results from diagnostic procedures between and among caregivers by making it a National Patient Safety Goal for 2011 [9The Joint Commission. 2011 National Patient Safety Goals. Available from: http://www.jointcommission.org/standards_inform ation/npsgs.aspx 2011 [Cited: 2012 Jan 11];].

Our institution established an enterprise-wide communi-cation of Critical Test Results (CCTR) policy for communi-cation of critical imaging results among over 600,000 imaging procedures performed annually [10 Hunsaker A, Khorasani R, Doubilet P. Policy for Communicating Critical and/or Discrepant Results Available from: http://www. brighamandwomens.org/cebi/files/Radiology%20Policy%20Critica l-Discrepant%20 Results.pdf. 3-10-2009 [Cited: 2012 Jan 11];, 11 Khorasani R. Optimizing communication of critical test results J Am Coll Radiol 2009; 6(10 ): 721-3.]. Full implementation of this policy is expected to promote significant safety improvements for a considerable patient population. However, manual analysis to evaluate the policy’s impact on the rate of communicating critical imaging results presented a resource-intensive, time-consuming challenge. A pilot test was conducted on a limited random sample of 12,193 reports from a three year period accounting for approximately 0.7% of the total reports during that time [11 Khorasani R. Optimizing communication of critical test results J Am Coll Radiol 2009; 6(10 ): 721-3.]. Although routine evaluation of adherence to the CCTR policy could provide invaluable feedback to end-users (which might be expected to enhance and sustain adherence), continued analysis of a greater number of samples was not sustainable due to the time burden it placed on Radiology division chiefs performing these audits manually.

To overcome this limitation, we used natural language processing and information retrieval/extraction (NLP-IR/E) applications that have been developed for the clinical domain [12 Friedman C, Alderson PO, Austin JH, Cimino JJ, Johnson SB. A general natural-language text processor for clinical radiology J Am Med Inform Assoc 1994; 1(2 ): 161-74.-15 Savova GK, Masanz JJ, Ogren PV, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications J Am Med Inform Assoc 2010; 17(5 ): 507-13.]. Several NLP applications focus specifically on recognizing clinical findings in radiology reports [16 Taira RK, Soderland SG, Jakobovits RM. Automatic structuring of radiology free-text reports Radiographics 2001; 21(1 ): 237-45.-19 Dreyer KJ, Kalra MK, Maher MM, et al. Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study Radiology 2005; 234(2 ): 323-9.]. Evaluation of applications that identify patient information in clinical reports has been done for such foci as smoking status, patient symptoms, and clinical diagnoses, such as psoriatic arthritis and status of tumors [14 Zeng QT, Goryachev S, Weiss S, Sordo M, Murphy SN, Lazarus R. Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system BMC Med Inform Decis Mak 2006; 6: 30., 20 Meystre S, Haug PJ. Natural language processing to extract medical problems from electronic clinical documents: performance evaluation J Biomed Inform 2006; 39(6 ): 589-99.]. However, these previous evaluations have been limited to single diseases and/or imaging examinations and have not focused on assessing the steps required for customizing or reconfiguring the application for various clinical conditions. Thus, although a system might successfully find cases with documented lung cancer, it may not necessarily be able to find cases with pneumothorax. In many instances, manually reconfiguring an application for a different clinical condition than that for which the tool was developed requires significant effort as well as injection of expert opinion [21 Dixon BE, Samarth A. Innovations in Using Health IT for Chronic Disease Management AHRQ National Resource Center for Health Information Technology AHRQ Publication No 09-0029-EF 2009.]. Describing these customization processes are essential steps for clinical application and performance evaluation [22 Reiner BI. Customization of medical report data J Digit Imaging 2010; 23(4 ): 363-73.].

The first objective of this project is to describe the process of customizing two different NLP-IR/E applications – an open-source Information Extraction toolkit utilized by the General Architecture for Text Engineering (GATE) research community called A Nearly New Information Extraction system (ANNIE);[23 Cunningham H, Maynard D, Bontcheva K, Tablan V. GATE: A framework and graphical development environment for robust NLP tools and applications In: Proceeding of the 40th Anniversary Meeting of the Association for Computations Lingusitics (ACL’02); July 2002; Philadelphia. ] and a toolkit developed in-house, named Information for Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) - to retrieve radiology reports with three distinct critical imaging findings (pulmonary nodule, pneumothorax, and pulmonary embolus), illustrating different levels of customization required to use these applications. The second objective is to evaluate application performance in retrieval of radiology reports with the specified critical imaging findings.

The level of customization required for each of the NLP-IR/E applications when applied to multiple diseases is presented in Table 6. Customization is categorized as “high,” when task completion requires performing more than one subtask, leading to greater time and/or effort, and “low,” when there is one subtask required to complete a task. Data entry for ANNIE and iSCOUT requires minimal effort since entire reports are processed in a single input file for both applications. Manual review was “low” for both applications since data files required a single review, leading to an appropriate file format (e.g. character-delimited text files) for data processing. For software development, ANNIE requires jape programming to enable use of ANNIE within GATE, and to run it over a desired corpus (e.g., radiology reports). Jape programming is also required to enable access to the Gazetteer. No further software development was required for iSCOUT. Enumerating query lists is likewise more labor and time-intensive for ANNIE, requiring three lists to be completed for each disease, compared to only a single list for iSCOUT. Terminology code identification, which can be utilized to further expand the list of query terms to enhance report retrieval, was not performed in this study. This task is optional for both applications, and would require further programming for ANNIE, but not for iSCOUT.

The precision and recall for each NLP-IR/E application are shown in Figs. (1-3). Comparison between the three applications is shown for three separate queries for pulmonary nodule, pulmonary embolus and pneumothorax.

Fig. (1) Precision and recall values for retrieving reports with pulmonary nodule and 95% confidence intervals.􀀁

Fig. (2) Precision and recall values for retrieving reports with pulmonary embolism and 95% confidence intervals.

Fig. (3) Precision and recall values for retrieving reports with pneumothorax and 95% confidence intervals.

This project describes two NLP-IR/E applications and compares their performance in retrieving radiology reports that describe three distinct critical findings. Unlike other previous evaluations, this project focuses on both customization as well as accuracy of these applications for disease-specific retrieval. Customization of an application involves restructuring a system’s functionality to accommodate specific goals. Customization requires close coordination with experts, both in clinical as well as technical matters. Thus, a project’s goals as well as available resources may dictate which application is most appropriate, assuming accuracy measures of the applications are comparable.

This evaluation focuses on three distinct clinical diseases. However, the applications are expected to be utilized for retrieving reports with many more critical imaging findings. Retrieving reports with pulmonary nodule can provide insight into the rate of follow-up for incidental pulmonary nodules and determine if it is consistent with the Fleischner Criteria, as recommended in a national clinical practice guideline [26 MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society Radiology 2005; 237(2 ): 395-400.]. Identifying and retrieving reports that describe findings consistent with pulmonary embolism is important for assessing quality of patient care [27 Duriseti RS, Brandeau ML. Cost-effectiveness of strategies for diagnosing pulmonary embolism among emergency department patients presenting with undifferentiated symptoms Ann Emerg Med 2010; 56(4 ): 321-.]. Quality assessment of CT orders and ordering patterns for suspected pulmonary embolism is vital to reduce unnecessary radiation exposure [3 Sodickson A, Baeyens PF, Andriole KP, et al. Recurrent CT, cumulative radiation exposure, and associated radiation-induced cancer risks from CT of adults Radiology 2009; 251(1 ): 175-84.]. For pneumothorax, the ability to retrieve specific reports can provide insight into an investigation of adverse events related to central line placements, as pneumothorax is a known complication of this procedure [28 Harley KT, Wang MD, Amin A. Common procedures in internal medicine: improving knowledge and minimizing complications Hosp Pract (Minneap) 2009; 37(1 ): 121-7., 29 Ayas NT, Norena M, Wong H, Chittock D, Dodek PM. Pneumothorax after insertion of central venous catheters in the intensive care unit: association with month of year and week of month Qual Saf Health Care 2007; 16(4 ): 252-5.]. More importantly, the ability to automatically identify reports with any of these three critical findings can provide a reminder for radiologists to alleviate delays in communicating critical results. In addition, it can be used to routinely select cases in order to determine prevalence rates of timely communication of critical results, measuring adherence to a CCTR policy and allowing for continuous quality assurance and process improvement initiatives.

With regard to the precision and recall of each application, ANNIE and iSCOUT both consistently displayed similar excellent precision and recall values [19 Dreyer KJ, Kalra MK, Maher MM, et al. Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study Radiology 2005; 234(2 ): 323-9., 20 Meystre S, Haug PJ. Natural language processing to extract medical problems from electronic clinical documents: performance evaluation J Biomed Inform 2006; 39(6 ): 589-99., 30 Cheng LT, Zheng J, Savova GK, Erickson BJ. Discerning tumor status from unstructured MRI reports--completeness of information in existing reports and utility of automated natural language processing J Digit Imaging 2010; 23(2 ): 119-32.]. Overlap of 95% confidence intervals for each application indicates no significant differences in precision and recall between the two. These results are not unexpected given the similarity in NLP-IR/E algorithms underlying these applications.

ANNIE attained higher precision values for all three queries. The ability to reduce false positives is due largely to its ability to maintain unique negation lists for specific searches. The negation lists, in particular, helps refine searches to only specific instances where pulmonary nodules are seen. For example, we would want to negate a report that indicates an attenuating nodule that “likely represents a sebaceous cyst”. The ability to include the word “cyst” in the negation list mitigates this problem. We were able to do this with a number of other terms, including “liver” and “kidney”to focus the query only to the lungs. This increased precision, however, come at a cost. ANNIE needs to be manually programmed to the end user’s specifications in order to accommodate and access specific lists. In addition, selecting terms that are included in each of the four potential lists is not an easy task, requiring careful thought from one or more experts. Reviewing local term usage from sample reports a priori becomes a critical component for identifying appropriate terms that will be included in each list.

iSCOUT demonstrated high precision and recall, similar to ANNIE. Utilizing the Header Extractor significantly increased precision, since the search was limited to certain sections of the report. For example, the phrase, “Multiple axial CT images of the chest were obtained following IV contrast administration using a pulmonary embolism protocol,” contains the term “pulmonary embolism” and would have caused the tools to erroneously retrieve reports with this protocol stated in the Technique section of the report. The ease of performing a query utilizing iSCOUT was significant, given that no additional programming is required. In addition, populating a single term list was conveniently simple. For pneumothorax, for instance, the term list only had four terms (see Table 5).

This retrospective study compared the use of two applications for retrieving reports with critical imaging findings from a single institution. Further testing in multiple settings would document generalizability of both applications for various clinical domains. Moreover, retrieval was based on documented presence of the findings within textual reports. Thus, documentation of findings is essential in evaluating information retrieval.

In addition, both applications were utilized to maximize the following metrics for information retrieval - precision and recall. Feature selection (e.g. enumerating query lists) was performed empirically, and not exhaustively. Utilizing controlled terminology could have further increased recall, as previously demonstrated in retrieving liver cysts using iSCOUT [24 Lacson R, Andriole KP, Prevedello LM, Khorasani R. Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) J Digit Imaging 2012; 25(4 ): 512-9.]. Finally, supervised classification algorithms, previously implemented for information retrieval, were not available in either application [31 D'Avolio LW, Nguyen TM, Farwell WR, et al. Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC) J Am Med Inform Assoc 2010; 17(4 ): 375-82.]. Incorporating these algorithms into information retrieval applications could further enhance precision and recall of these tools.

Both applications can be used for information retrieval, comparing favorably with several information retrieval tools currently in use [19 Dreyer KJ, Kalra MK, Maher MM, et al. Application of recently developed computer algorithm for automatic classification of unstructured radiology reports: validation study Radiology 2005; 234(2 ): 323-9., 30 Cheng LT, Zheng J, Savova GK, Erickson BJ. Discerning tumor status from unstructured MRI reports--completeness of information in existing reports and utility of automated natural language processing J Digit Imaging 2010; 23(2 ): 119-32.-32 Savova GK, Fan J, Ye Z, et al. Discovering peripheral arterial disease cases from radiology notes using natural language processing AMIA Annu Symp Proc 2010 2010; 722-6.]. ANNIE had consistently high precision values, although using it required the most customization. iSCOUT performed equally well with less customization. Appropriate utilization of these applications for retrieving reports with critical findings is important in ensuring adherence to practice guidelines. The American College of Radiology (ACR) in its 2005 Practice Guideline for Communication of Diagnostic Imaging Findings strongly advised radiologists to expedite notification of imaging results to referring physicians “in a manner that reasonably ensures timely receipt of the findings” [33 Kushner DC, Lucey LL. Diagnostic radiology reporting and communication: the ACR guideline J Am Coll Radiol 2005; 2(1 ): 15-21.]. Utilizing these applications in retrieving reports with critical findings and verifying whether timely and appropriate communication was initiated can greatly expedite the evaluation process and provide invaluable feedback to caregivers.