This retrospective, Health Insurance Portability and Accountability Act-compliant study was approved by the Institutional Review Board, and performed at an urban, quaternary care, academic medical center. All abdominal X-ray, Computed Tomography (CT) scan and Magnetic Resonance Imaging (MRI) radiology reports generated by the Abdominal Radiology Division in the third quarter of 2016 (July-September 2016) were included in the analysis. Exclusion criteria included reports from other modalities (e.g., Upper Gastro-Intestinal series, Hysterosalpingogram) and abdominal interventional procedures (e.g., paracentesis). The radiology department has 13 abdominal radiology subspecialty-trained attending physicians, 5 abdominal radiology subspecialty fellows and 39 diagnostic radiology residents. Every finalized radiology report is signed by one of the attending physicians, whether or not it was first reviewed by a trainee.

We assessed radiology reports for the use of 16 previously-identified phrases that convey diagnostic certainty [3]. In particular, we evaluated the use of three phrases for which radiologists had “good agreement” with referring providers in conveying diagnostic certainty - “diagnostic of/for”, “represents” and “unlikely” - as well as 13 phrases for which there was variable agreement among radiologists and non-radiologists about conveying diagnostic certainty [3].

We utilized information retrieval and query matching to extract phrases conveying diagnostic certainty from within the impression sections of radiology reports. We utilized a publicly available toolkit, Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) for this task [21]. The Header Extractor component in iSCOUT enables selective retrieval of sections of reports by restricting a search to content within various headers of a report (e.g., Impression, Findings). From the Impression section, we subsequently performed query matching for each individual diagnostic certainty phrase using Microsoft Access (Microsoft Corporation, Redmond, Washington). Reports that included at least one of the 16 phrases conveying diagnostic certainty were retrieved.

We assessed the proportion of reports with phrases conveying diagnostic certainty, as well as the frequency and distribution of those phrases. In addition, we calculated the proportion of reports with “good agreement” phrases out of all reports with phrases that convey diagnostic certainty. Each report that contains one or more phrases that convey diagnostic certainty is counted once in the denominator. Each report is counted once in the numerator (i.e. reports with “good agreement” phrases) if it contains at least one “good agreement” phrase. We assessed the impact of imaging modality, trainee contribution to report generation and individual radiologists in the usage of “good agreement” phrases. Chi-square test was performed to evaluate the difference between the proportion of reports with “good agreement” phrases and reports from X-ray, CT scan and MRI examinations, between attending-only reports or joint attending and trainee reports, and between the individual attendings. For comparing individual attendings, those who contributed fewer than 100 reports were excluded from statistical analysis. All statistical analyses were performed using commercially available software (R 3.4.1 software, Vienna, Austria). A p-value of <0.05 was considered statistically significant.

During the study period, a total of 5,598 unique radiology reports were generated by the Abdominal Radiology Section; 1,226 (22%) abdominal X-ray reports, 3,010 (54%) CT scan reports and 1,362 (24%) MRI reports. There were 1,734 attending-only reports and 3,864 joint attending and trainee reports. The 13 radiologist attendings accounted for an average of 430 reports each. Among 5,598 reports, 2,071 (37%) contained at least one diagnostic certainty phrase.

Fig. (1) demonstrates the proportion of reports containing phrases used to convey diagnostic certainty out of all reports, by imaging modality (examples of phrase usage are provided in Table 1. Only 119 (6%) of phrases were “good agreement” phrases. The most commonly used diagnostic certainty phrases in abdominal imaging were “likely” (including with modifiers), and “may represent”, accounting for 15% and 5%, respectively, of all reports with phrases that convey diagnostic certainty. Specifically, in abdominal x-ray reports, the three most common phrases were “likely”, “may represent”, and “consistent with”; in CT scans, the most common words were “likely”, “may represent” and “may be”; in MRI imaging, the most frequently used phrases were “likely”, “likely (with modifiers)”, and “consistent with”.

Fig. (1). Proportion of reports containing phrases (n=16) used to convey diagnostic certainty out of all reports, by modality.

Table 2 shows the results of bivariate analyses comparing use of phrases with “good agreement” between imaging modalities. There was significant difference in how frequently phrases with “good agreement” were used as compared with phrases with variable agreement between imaging modalities. Reports from MRI and CT exams had a higher usage of “good agreement” phrases compared to X-ray reports (p=0.0003; chi-square analysis).

Table 3 shows the results of bivariate analyses comparing use of phrases with “good agreement” between reports generated by attending physicians, and trainees with attending physicians. No significant difference was noted in usage of phrases with “good agreement” between these two groups (p<0.5169). Fig. (2) shows the proportion of reports containing phrases used to convey diagnostic certainty out of all reports generated by attending-only and trainees with attending physicians. Reports that were generated by attending physicians had “likely”, “may be” and “may represent” as most commonly used phrases, whereas joint reports from both attending and trainee physicians most commonly contained “likely”, “may represent” and “consistent with”.

Fig. (2). Proportion of reports containing phrases (n=16) used to convey diagnostic certainty in attending-only and trainees with attending physician reports.

Individual attending physicians used diagnostic certainty phrases with “good agreement” in 0-14% of their reports that contain diagnostic certainty phrases. Table 4 demonstrates a significant difference in use of phrases with “good agreement” in reports generated between individual attending physicians (p<0.0019).

This study is limited to phrases that were commonly encountered in our institution and have been previously published. These phrases may vary by geographic location, practice, and institution. We only analyzed reports from a single academic medical center, and our findings may not reflect terminology usage that conveys diagnostic certainty in other institutions. We expect that there is greater variability in terminology usage than identified within one institution alone, highlighting the importance of assessing terminology usage. In addition, the use of diagnostic certainty phrases was not assessed in the context of the findings being described. For example, physicians may be more likely to use phrases with “good agreement” for more benign findings as opposed to findings suggesting malignancy. However, the use of uncertain phrases has been previously shown to lead to ambiguous reports [12].

Further studies to enhance communication between providers by providing terminology to express their degree of certainty for findings are needed. In some cases, developing more standard terminology based on diagnostic confidence in specific imaging findings or patient outcomes may be useful. Standardizing phrases that convey certainty, and limiting the use of phrases with variable agreement among physicians may improve communication of radiology results.