The role of radiological imaging in the diagnosis and treatment of diseases has greatly expanded in the last decade [1-3]. Images are interpreted by radiologists, and pertinent findings recorded in the form of narrative text. In practice, the radiologist should contact the referring clinician whenever a critical imaging result is present. Failure to promptly communicate critical imaging test results is not uncommon and such delays are a major source of malpractice claims in radiology [4-8]. The Joint Commission emphasized a need for improved communication of critical results from diagnostic procedures between and among caregivers by making it a National Patient Safety Goal for 2011 [9].

Our institution established an enterprise-wide communi-cation of Critical Test Results (CCTR) policy for communi-cation of critical imaging results among over 600,000 imaging procedures performed annually [10, 11]. Full implementation of this policy is expected to promote significant safety improvements for a considerable patient population. However, manual analysis to evaluate the policy’s impact on the rate of communicating critical imaging results presented a resource-intensive, time-consuming challenge. A pilot test was conducted on a limited random sample of 12,193 reports from a three year period accounting for approximately 0.7% of the total reports during that time [11]. Although routine evaluation of adherence to the CCTR policy could provide invaluable feedback to end-users (which might be expected to enhance and sustain adherence), continued analysis of a greater number of samples was not sustainable due to the time burden it placed on Radiology division chiefs performing these audits manually.

To overcome this limitation, we used natural language processing and information retrieval/extraction (NLP-IR/E) applications that have been developed for the clinical domain [12-15]. Several NLP applications focus specifically on recognizing clinical findings in radiology reports [16-19]. Evaluation of applications that identify patient information in clinical reports has been done for such foci as smoking status, patient symptoms, and clinical diagnoses, such as psoriatic arthritis and status of tumors [14, 20]. However, these previous evaluations have been limited to single diseases and/or imaging examinations and have not focused on assessing the steps required for customizing or reconfiguring the application for various clinical conditions. Thus, although a system might successfully find cases with documented lung cancer, it may not necessarily be able to find cases with pneumothorax. In many instances, manually reconfiguring an application for a different clinical condition than that for which the tool was developed requires significant effort as well as injection of expert opinion [21]. Describing these customization processes are essential steps for clinical application and performance evaluation [22].

The first objective of this project is to describe the process of customizing two different NLP-IR/E applications – an open-source Information Extraction toolkit utilized by the General Architecture for Text Engineering (GATE) research community called A Nearly New Information Extraction system (ANNIE);[23] and a toolkit developed in-house, named Information for Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) - to retrieve radiology reports with three distinct critical imaging findings (pulmonary nodule, pneumothorax, and pulmonary embolus), illustrating different levels of customization required to use these applications. The second objective is to evaluate application performance in retrieval of radiology reports with the specified critical imaging findings.

The level of customization required for each of the NLP-IR/E applications when applied to multiple diseases is presented in Table 6. Customization is categorized as “high,” when task completion requires performing more than one subtask, leading to greater time and/or effort, and “low,” when there is one subtask required to complete a task. Data entry for ANNIE and iSCOUT requires minimal effort since entire reports are processed in a single input file for both applications. Manual review was “low” for both applications since data files required a single review, leading to an appropriate file format (e.g. character-delimited text files) for data processing. For software development, ANNIE requires jape programming to enable use of ANNIE within GATE, and to run it over a desired corpus (e.g., radiology reports). Jape programming is also required to enable access to the Gazetteer. No further software development was required for iSCOUT. Enumerating query lists is likewise more labor and time-intensive for ANNIE, requiring three lists to be completed for each disease, compared to only a single list for iSCOUT. Terminology code identification, which can be utilized to further expand the list of query terms to enhance report retrieval, was not performed in this study. This task is optional for both applications, and would require further programming for ANNIE, but not for iSCOUT.

The precision and recall for each NLP-IR/E application are shown in Figs. (1-3). Comparison between the three applications is shown for three separate queries for pulmonary nodule, pulmonary embolus and pneumothorax.

Fig. (1). Precision and recall values for retrieving reports with pulmonary nodule and 95% confidence intervals.􀀁

Fig. (2). Precision and recall values for retrieving reports with pulmonary embolism and 95% confidence intervals.

Fig. (3). Precision and recall values for retrieving reports with pneumothorax and 95% confidence intervals.

This project describes two NLP-IR/E applications and compares their performance in retrieving radiology reports that describe three distinct critical findings. Unlike other previous evaluations, this project focuses on both customization as well as accuracy of these applications for disease-specific retrieval. Customization of an application involves restructuring a system’s functionality to accommodate specific goals. Customization requires close coordination with experts, both in clinical as well as technical matters. Thus, a project’s goals as well as available resources may dictate which application is most appropriate, assuming accuracy measures of the applications are comparable.

This evaluation focuses on three distinct clinical diseases. However, the applications are expected to be utilized for retrieving reports with many more critical imaging findings. Retrieving reports with pulmonary nodule can provide insight into the rate of follow-up for incidental pulmonary nodules and determine if it is consistent with the Fleischner Criteria, as recommended in a national clinical practice guideline [26]. Identifying and retrieving reports that describe findings consistent with pulmonary embolism is important for assessing quality of patient care [27]. Quality assessment of CT orders and ordering patterns for suspected pulmonary embolism is vital to reduce unnecessary radiation exposure [3]. For pneumothorax, the ability to retrieve specific reports can provide insight into an investigation of adverse events related to central line placements, as pneumothorax is a known complication of this procedure [28, 29]. More importantly, the ability to automatically identify reports with any of these three critical findings can provide a reminder for radiologists to alleviate delays in communicating critical results. In addition, it can be used to routinely select cases in order to determine prevalence rates of timely communication of critical results, measuring adherence to a CCTR policy and allowing for continuous quality assurance and process improvement initiatives.

With regard to the precision and recall of each application, ANNIE and iSCOUT both consistently displayed similar excellent precision and recall values [19, 20, 30]. Overlap of 95% confidence intervals for each application indicates no significant differences in precision and recall between the two. These results are not unexpected given the similarity in NLP-IR/E algorithms underlying these applications.

ANNIE attained higher precision values for all three queries. The ability to reduce false positives is due largely to its ability to maintain unique negation lists for specific searches. The negation lists, in particular, helps refine searches to only specific instances where pulmonary nodules are seen. For example, we would want to negate a report that indicates an attenuating nodule that “likely represents a sebaceous cyst”. The ability to include the word “cyst” in the negation list mitigates this problem. We were able to do this with a number of other terms, including “liver” and “kidney”to focus the query only to the lungs. This increased precision, however, come at a cost. ANNIE needs to be manually programmed to the end user’s specifications in order to accommodate and access specific lists. In addition, selecting terms that are included in each of the four potential lists is not an easy task, requiring careful thought from one or more experts. Reviewing local term usage from sample reports a priori becomes a critical component for identifying appropriate terms that will be included in each list.

iSCOUT demonstrated high precision and recall, similar to ANNIE. Utilizing the Header Extractor significantly increased precision, since the search was limited to certain sections of the report. For example, the phrase, “Multiple axial CT images of the chest were obtained following IV contrast administration using a pulmonary embolism protocol,” contains the term “pulmonary embolism” and would have caused the tools to erroneously retrieve reports with this protocol stated in the Technique section of the report. The ease of performing a query utilizing iSCOUT was significant, given that no additional programming is required. In addition, populating a single term list was conveniently simple. For pneumothorax, for instance, the term list only had four terms (see Table 5).

This retrospective study compared the use of two applications for retrieving reports with critical imaging findings from a single institution. Further testing in multiple settings would document generalizability of both applications for various clinical domains. Moreover, retrieval was based on documented presence of the findings within textual reports. Thus, documentation of findings is essential in evaluating information retrieval.

In addition, both applications were utilized to maximize the following metrics for information retrieval - precision and recall. Feature selection (e.g. enumerating query lists) was performed empirically, and not exhaustively. Utilizing controlled terminology could have further increased recall, as previously demonstrated in retrieving liver cysts using iSCOUT [24]. Finally, supervised classification algorithms, previously implemented for information retrieval, were not available in either application [31]. Incorporating these algorithms into information retrieval applications could further enhance precision and recall of these tools.

Both applications can be used for information retrieval, comparing favorably with several information retrieval tools currently in use [19, 30-32]. ANNIE had consistently high precision values, although using it required the most customization. iSCOUT performed equally well with less customization. Appropriate utilization of these applications for retrieving reports with critical findings is important in ensuring adherence to practice guidelines. The American College of Radiology (ACR) in its 2005 Practice Guideline for Communication of Diagnostic Imaging Findings strongly advised radiologists to expedite notification of imaging results to referring physicians “in a manner that reasonably ensures timely receipt of the findings” [33]. Utilizing these applications in retrieving reports with critical findings and verifying whether timely and appropriate communication was initiated can greatly expedite the evaluation process and provide invaluable feedback to caregivers.