RESEARCH ARTICLE


Factors Impacting the Use of Terminology to Convey Diagnostic Certainty in Radiology Reports



Ronilda Lacson*, Victor Babatunde, Atul Shinagare, Ramin Khorasani
Department of Radiology, Brigham and Women's Hospital/Harvard Medical School, Boston, USA


Article Metrics

CrossRef Citations:
0
Total Statistics:

Full-Text HTML Views: 119
Abstract HTML Views: 88
PDF Downloads: 47
ePub Downloads: 31
Total Views/Downloads: 285
Unique Statistics:

Full-Text HTML Views: 91
Abstract HTML Views: 70
PDF Downloads: 41
ePub Downloads: 29
Total Views/Downloads: 231



© 2018 Lacson et al.

open-access license: This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: https://creativecommons.org/licenses/by/4.0/legalcode. This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

* Address correspondence to this author at the Department of Radiology, Brigham and Women's Hospital/Harvard Medical School, Boston, USA, Tel: (617)5259712, Fax: 617515-7575; E-mail: rlacson@rics.bwh.harvard.edu


Abstract

Background:

Variable use of phrases expressing diagnostic uncertainty can lead to ambiguous radiology reports, a concern for information processing.

Objective:

This study aimed to quantify the usage of phrases conveying diagnostic certainty for abdominal imaging findings and assess factors that impact use of phrases with “good agreement” between radiologists and referring providers.

Methods:

This retrospective, Institutional Review Board-Approved study included all diagnostic reports generated by the Abdominal Radiology Division at an academic medical center July-September 2016. We assessed the use of 16 diagnostic certainty phrases using information retrieval from the Impression section of radiology reports. Phrases with good provider agreement for conveying the level of certainty are defined as “good agreement” phrases - including “diagnostic of”, “represents” and “unlikely.” We assessed the impact of imaging modality, trainee contribution to report generation, and individual radiologists.

Results:

In 5,598 radiology reports, 2,071 (37%) contained diagnostic certainty phrases, 119 (6%) of which were “good agreement” phrases. There was a significant difference between how frequently “good agreement” phrases were used in Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and X-ray reports (p=0.0003). There was a significant variation among attending radiologists on the use of “good agreement” phrases (p<0.0019). There was no difference in the use of “good agreement” phrases in reports generated by attending radiologists alone compared to reports with trainees.

Conclusion:

Although phrases to convey diagnostic certainty were commonly used in radiology reports, the use of phrases with a good agreement was uncommon. Standardizing terminology to convey diagnostic certainty may reduce ambiguity in radiology reports and generate more accurate information processing tools.

Keywords: Radiology, Terminology, Uncertainty, MRI, CT, X-ray.



1. INTRODUCTION

Radiology reports serve as the primary method of communication between radiologists and other healthcare providers [1-3]. Many studies and interventions have been implemented to increase timely report generation, decrease error rates in reporting, and provide comprehensive documentation of findings [4-10]. Information extraction and natural language processing tools have integrated certainty modifiers when extracting disease concepts [11]. However, despite multiple studies reporting radiologists' usage of vague and uncertain phrases in their reports [3, 12-14], there has been little effort to evaluate the use of terminology conveying diagnostic certainty in radiology reports, and whether providers agree on the level of certainty they convey.

While ideally, radiology reports are concise and unambiguous [15, 16], in practice, there is wide variability in the terminology used by radiologists in reporting findings, depending on their degree of certainty and judgment [3, 9]. This variability in expressing certainty has been significantly associated with reports that are more ambiguous [12]. Standardized reporting has been shown to decrease ambiguity for particular radiology procedures [17-20]. Breast Imaging Reporting and Data System [BI-RADS] for breast imaging reporting has been utilized to help understand and communicate the probability of a pathologic finding [19]. Similarly, Prostate Imaging Reporting and Data System (PI-RADS) and Lung Reporting and Data System (Lung-RADS) have been used to communicate the probability of prostate and lung cancer, respectively [17, 18]. However, expressing the probability of other imaging findings, other than these few malignancies, is a difficult task. Moreover, the standardized terminology for conveying diagnostic certainty in abdominal radiology reports is not yet established.

A previous study identified 15 commonly-used phrases utilized by radiologists for expressing certainty [3]. In that study, the two phrases which had the highest agreement among radiologists in conveying the level certainty were “diagnostic of” and “unlikely”, based on a questionnaire wherein radiologists voluntarily ranked 15 phrases in order of diagnostic certainty. The phrase “diagnostic for” and “represents” were ranked with the highest likelihood in conveying malignancy by both radiologists and primary care physicians in a separate study [13]. The purpose of this study was to assess the usage of 16 phrases to convey diagnostic certainty for imaging findings in the Abdominal Division of an academic radiology practice [3, 12], specifically to quantify the usage of phrases with “good agreement” between radiologists and referring providers that are expected to express less ambiguity. Understanding the level of variability and factors that impacted the usage of phrases with “good agreement” will help in developing standard terminology to convey diagnostic certainty.

2. MATERIALS AND METHODS

2.1. Study Setting and Cohort

This retrospective, Health Insurance Portability and Accountability Act-compliant study was approved by the Institutional Review Board, and performed at an urban, quaternary care, academic medical center. All abdominal X-ray, Computed Tomography (CT) scan and Magnetic Resonance Imaging (MRI) radiology reports generated by the Abdominal Radiology Division in the third quarter of 2016 (July-September 2016) were included in the analysis. Exclusion criteria included reports from other modalities (e.g., Upper Gastro-Intestinal series, Hysterosalpingogram) and abdominal interventional procedures (e.g., paracentesis). The radiology department has 13 abdominal radiology subspecialty-trained attending physicians, 5 abdominal radiology subspecialty fellows and 39 diagnostic radiology residents. Every finalized radiology report is signed by one of the attending physicians, whether or not it was first reviewed by a trainee.

2.2. Phrases Conveying Diagnostic Certainty

We assessed radiology reports for the use of 16 previously-identified phrases that convey diagnostic certainty [3]. In particular, we evaluated the use of three phrases for which radiologists had “good agreement” with referring providers in conveying diagnostic certainty - “diagnostic of/for”, “represents” and “unlikely” - as well as 13 phrases for which there was variable agreement among radiologists and non-radiologists about conveying diagnostic certainty [3].

2.3. Data Collection

We utilized information retrieval and query matching to extract phrases conveying diagnostic certainty from within the impression sections of radiology reports. We utilized a publicly available toolkit, Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT) for this task [21]. The Header Extractor component in iSCOUT enables selective retrieval of sections of reports by restricting a search to content within various headers of a report (e.g., Impression, Findings). From the Impression section, we subsequently performed query matching for each individual diagnostic certainty phrase using Microsoft Access (Microsoft Corporation, Redmond, Washington). Reports that included at least one of the 16 phrases conveying diagnostic certainty were retrieved.

2.4. Outcomes and Statistical Analysis

We assessed the proportion of reports with phrases conveying diagnostic certainty, as well as the frequency and distribution of those phrases. In addition, we calculated the proportion of reports with “good agreement” phrases out of all reports with phrases that convey diagnostic certainty. Each report that contains one or more phrases that convey diagnostic certainty is counted once in the denominator. Each report is counted once in the numerator (i.e. reports with “good agreement” phrases) if it contains at least one “good agreement” phrase. We assessed the impact of imaging modality, trainee contribution to report generation and individual radiologists in the usage of “good agreement” phrases. Chi-square test was performed to evaluate the difference between the proportion of reports with “good agreement” phrases and reports from X-ray, CT scan and MRI examinations, between attending-only reports or joint attending and trainee reports, and between the individual attendings. For comparing individual attendings, those who contributed fewer than 100 reports were excluded from statistical analysis. All statistical analyses were performed using commercially available software (R 3.4.1 software, Vienna, Austria). A p-value of <0.05 was considered statistically significant.

3. RESULTS

3.1. Cohort Characteristics

During the study period, a total of 5,598 unique radiology reports were generated by the Abdominal Radiology Section; 1,226 (22%) abdominal X-ray reports, 3,010 (54%) CT scan reports and 1,362 (24%) MRI reports. There were 1,734 attending-only reports and 3,864 joint attending and trainee reports. The 13 radiologist attendings accounted for an average of 430 reports each. Among 5,598 reports, 2,071 (37%) contained at least one diagnostic certainty phrase.

3.2. Frequency and Distribution of Diagnostic Certainty Phrases

Fig. (1) demonstrates the proportion of reports containing phrases used to convey diagnostic certainty out of all reports, by imaging modality (examples of phrase usage are provided in Table 1. Only 119 (6%) of phrases were “good agreement” phrases. The most commonly used diagnostic certainty phrases in abdominal imaging were “likely” (including with modifiers), and “may represent”, accounting for 15% and 5%, respectively, of all reports with phrases that convey diagnostic certainty. Specifically, in abdominal x-ray reports, the three most common phrases were “likely”, “may represent”, and “consistent with”; in CT scans, the most common words were “likely”, “may represent” and “may be”; in MRI imaging, the most frequently used phrases were “likely”, “likely (with modifiers)”, and “consistent with”.

Table 1. Examples of diagnostic certainty phrase usage in the Impression section of the radiology report.
Diagnostic Certainty Phrase Examples
Likely New mild compression of superior endplates of T12 and L3 likely degenerative
Probably Two tiny cystic lesions in segment 6 are probably benign
Suspicious of PI-RADS* 4 lesion within the left mid gland posterior lateral peripheral zone, with contour suspicious of left extracapsular extension
Diagnostic of Multiple hepatic masses measuring up to 2.5 cm most with signal characteristics diagnostic of hemangiomas
Unlikely Increase in the size of a sclerotic lesion in L1. In the absence of the known malignancy this is unlikely to be metastasis
Most likely Bilateral hypodense renal lesions, most likely cysts
*PI-RADS: Prostate Imaging Reporting and Data System
Fig. (1). Proportion of reports containing phrases (n=16) used to convey diagnostic certainty out of all reports, by modality.

3.3. Diagnostic Certainty Phrases by Imaging Modality

Table 2 shows the results of bivariate analyses comparing use of phrases with “good agreement” between imaging modalities. There was significant difference in how frequently phrases with “good agreement” were used as compared with phrases with variable agreement between imaging modalities. Reports from MRI and CT exams had a higher usage of “good agreement” phrases compared to X-ray reports (p=0.0003; chi-square analysis).

Table 2. Bivariate analysis comparing frequency of diagnostic certainty phrases by imaging modality in reports that contain diagnostic certainty phrases (n=2,071/5,598).
Imaging Modality Reports that Contain Phrases with
Good Agreement/ Reports that Contain Diagnostic Certainty Phrases
Percent
X-ray (n=1,226) 6/331 2%
CT (n=3,010) 67/1,159 6%
MRI (n=1,362) 48/581 8%
RESULTS
Chi-square Statistic (comparing X-ray with CT+MRI) 16.00
P-value 0.0003

3.4. Diagnostic Certainty Phrases by Generating Physician

Table 3 shows the results of bivariate analyses comparing use of phrases with “good agreement” between reports generated by attending physicians, and trainees with attending physicians. No significant difference was noted in usage of phrases with “good agreement” between these two groups (p<0.5169). Fig. (2) shows the proportion of reports containing phrases used to convey diagnostic certainty out of all reports generated by attending-only and trainees with attending physicians. Reports that were generated by attending physicians had “likely”, “may be” and “may represent” as most commonly used phrases, whereas joint reports from both attending and trainee physicians most commonly contained “likely”, “may represent” and “consistent with”.

Table 3. Bivariate analysis comparing frequency of diagnostic certainty phrases in reports generated by attending physicians, and trainees with attending physicians that contain diagnostic certainty phrases.
Reports Reports that Contain Phrases With
Good Agreement/ Reports that Contain Diagnostic Certainty Phrases
Percent
Attending-only reports (n=1,734) 29/559 5%
Trainee & Attending joint reports (n=3,864) 90/1,512 6%
TOTAL (n=5,598) 119/2,071 6%
RESULTS
Chi-square Statistic 0.44
P-value <0.5169*
* statistically significant
Fig. (2). Proportion of reports containing phrases (n=16) used to convey diagnostic certainty in attending-only and trainees with attending physician reports.

3.5. Diagnostic Certainty Phrases by Attending Physician

Individual attending physicians used diagnostic certainty phrases with “good agreement” in 0-14% of their reports that contain diagnostic certainty phrases. Table 4 demonstrates a significant difference in use of phrases with “good agreement” in reports generated between individual attending physicians (p<0.0019).

Table 4. Bivariate analysis comparing frequency of diagnostic certainty phrases in reports generated by individual attending physicians that contain diagnostic certainty phrases.
Attendings Reports that Contain Phrases With
“Good Agreement”/
Reports that Contain Diagnostic Certainty Phrases
Percent
Attending 1 13/156 8%
Attending 2 11/221 5%
Attending 3 18/133 14%
Attending 4 19/378 5%
Attending 5** 4/73 5%
Attending 6** 5/74 7%
Attending 7 3/133 2%
Attending 8 8/161 5%
Attending 9** 0/40 0%
Attending 10 16/256 6%
Attending 11 19/284 7%
Attending 12** 1/36 3%
Attending 13 2/126 2%
TOTALS 119/2,071 6%
RESULTS
Chi-square Statistic 24.52
P-value <0.0019*
* statistically significant **excluded from statistical analysis (phrase usage<100)

4. DISCUSSION

In a previous study, surveyed radiologists and non-radiologists ranked the phrases “diagnostic of” and “represents” as the phrases which conveyed the highest degree of certainty for expressing diagnostic findings in radiology reports [3]. Phrases such as “likely” and “consistent with” were generally considered to have a variable agreement in conveying diagnostic certainty [3]. However, although radiologists perceive that there is a variable agreement in conveying certainty in using these phrases, in this study, these were frequently encountered phrases in radiology reports.

These results confirm previous findings that radiologists frequently utilize phrases which convey diagnostic uncertainty. Multiple publications express the need for minimizing uncertainty and ambiguity when reporting findings [12, 22]; nevertheless, there continues to be the widespread utilization of these phrases. Uncertainty in diagnostic testing is unavoidable because the sensitivity and specificity of tests is rarely 100%. MRI and CT, for instance, are more accurate for diagnosing metastatic tumors in the peritoneum compared to US. MRI is more accurate for detection of lymph node metastases from ovarian cancer than CT [23]. Thus, rather than ignoring uncertainty, it is critical to focus on factors that impact the use of diagnostic certainty phrases, and potentially standardize how certainty is conveyed in reports.

Factors that were significantly associated with utilizing diagnostic certainty phrases with a variable agreement for conveying diagnostic certainty included imaging modality (i.e., X-ray), generating reports without trainees, and individual attending physicians. As expected, there is more frequent use of “good agreement” phrases in MRI and CT reports, compared to X-ray, as these tests typically provide more diagnostic detail than X-rays. There may also be the concurrent use of “good” and variable agreement terms in the same report. However, we focused on comparing usage of “good agreement” terms in each report. Reports generated with trainees compared to those generated by attendings alone did not result in more frequent use of phrases with good agreement. In addition, there was variation in use of “good agreement” phrases between attending physicians.

Communication of uncertainty has been referred to as hedging, with unnecessarily ambiguous expressions [24]. Communicating ambiguity, however, has more often resulted in mismanagement and delayed care [16]. Thus, radiologists have recognized that this practice does not protect from litigation [24]. Rather, it is driven by the uncertainty that clinicians face [25].

It appears that individual attending physicians need education in order to minimize the use of phrases that do not have “good agreement”. However, educating physicians alone may not be an optimal approach for changing behavior without adequately described standards for communicating diagnostic certainty [26, 27]. As stated previously, uncertainty is unavoidable. Thus, having the means to convey uncertainty in a way that expresses the degree of certainty, like diagnostic test statistics (i.e. 99% sensitivity is greater than 80%), is necessary. Standardizing phrases for conveying certainty might aid in enhancing communication and minimizing ambiguity [22, 28]. This is especially critical for natural language processing and information extraction tools so that findings and disease concepts are described with appropriate certainty modifiers.

4.1. Limitations

This study is limited to phrases that were commonly encountered in our institution and have been previously published. These phrases may vary by geographic location, practice, and institution. We only analyzed reports from a single academic medical center, and our findings may not reflect terminology usage that conveys diagnostic certainty in other institutions. We expect that there is greater variability in terminology usage than identified within one institution alone, highlighting the importance of assessing terminology usage. In addition, the use of diagnostic certainty phrases was not assessed in the context of the findings being described. For example, physicians may be more likely to use phrases with “good agreement” for more benign findings as opposed to findings suggesting malignancy. However, the use of uncertain phrases has been previously shown to lead to ambiguous reports [12].

Further studies to enhance communication between providers by providing terminology to express their degree of certainty for findings are needed. In some cases, developing more standard terminology based on diagnostic confidence in specific imaging findings or patient outcomes may be useful. Standardizing phrases that convey certainty, and limiting the use of phrases with variable agreement among physicians may improve communication of radiology results.

CONCLUSION

In abdominal imaging reports, more than a third of reports contained at least one diagnostic certainty phrase, however, phrases with good agreement between physicians were infrequently used. Standardization of terminology used to convey diagnostic certainly may help create less ambiguous reports and improve quality of care. Furthermore, it can generate more accurate and reliable information processing tools for research and quality initiatives.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

Health Insurance Portability and Accountability Act-compliant study was approved by the Institutional Review Board.

HUMAN AND ANIMAL RIGHTS

No Animals were used in this research. All human research procedures followed were in accordance with the ethical standards of the committee responsible for human experimentation (institutional and national), and with the Helsinki Declaration of 1975, as revised in 2013.

CONSENT FOR PUBLICATION

Not applicable.

CONFLICT OF INTEREST

The authors declare no conflict of interest, financial or otherwise.

ACKNOWLEDGEMENTS

The authors would like to thank Laura Peterson for reviewing the manuscript.

REFERENCES

[1] Friedman PJ. Radiologic reporting: Structure. AJR Am J Roentgenol 1983; 140(1): 171-2.
[2] Marcovici PA, Taylor GA. Journal club: Structured radiology reports are more complete and more effective than unstructured reports. AJR Am J Roentgenol 2014; 203(6): 1265-71.
[3] Khorasani R, Bates DW, Teeger S, Rothschild JM, Adams DF, Seltzer SE. Is terminology used effectively to convey diagnostic certainty in radiology reports? Acad Radiol 2003; 10(6): 685-8.
[4] Augmenting the impact of technology adoption with financial incentive to improve radiology report signature times. Journal of the American College of Radiology: JACR 2010; 7(3): 198-204.
[5] Prevedello LM, Ledbetter S, Farkas C, Khorasani R. Implementation of speech recognition in a community-based radiology practice: effect on report turnaround times. J Am Coll Radiol 2014; 11(4): 402-6.
[6] Gollub MJ, Panicek DM, Bach AM, Penalver A, Castellino RA. Clinical importance of reinterpretation of body CT scans obtained elsewhere in patients referred for care at a tertiary cancer center. Radiology 1999; 210(1): 109-12.
[7] Hatzoglou V, Omuro AM, Haque S, et al. Second-opinion interpretations of neuroimaging studies by oncologic neuroradiologists can help reduce errors in cancer care. Cancer 2016; 122(17): 2708-14.
[8] Leung JW, Margolin FR, Dee KE, Jacobs RP, Denny SR, Schrumpf JD. Performance parameters for screening and diagnostic mammography in a community practice: are there differences between specialists and general radiologists? AJR Am J Roentgenol 2007; 188(1): 236-41.
[9] Langlotz CP, Meininger L. Enhancing the expressiveness and usability of structured image reporting systems. 2000; In: Enhancing the expressiveness and usability of structured image reporting systems. Proceedings / AMIA Annual Symposium AMIA Symposium.; 2000; pp. 467-71.
[10] Prevedello LM, Farkas C, Ip IK, et al. Large-scale automated assessment of radiologist adherence to the Physician Quality Reporting System for stroke. J Am Coll Radiol 2012; 9(6): 414-20.
[11] Krauthammer M, Hripcsak G. knowledge model for the interpretation and visualization of NLP-parsed discharged summaries. 2001; In: Proceedings / AMIA Annual Symposium AMIA Symposium; 2001; pp. 339-43.
[12] Chapman WW, Fiszman M, Frederick PR, Chapman BE, Haug PJ. Quantifying the characteristics of unambiguous chest radiography reports in the context of pneumonia. Acad Radiol 2001; 8(1): 57-66.
[13] Gunn AJ, Tuttle MC, Flores EJ, et al. Differing interpretations of report terminology between primary care physicians and radiologists. J Amer College Radiol: JACR 2016; 13(12 Pt A): 1525-9.
[14] Hobby JL, Tom BD, Todd C, Bearcroft PW, Dixon AK. Communication of doubt and certainty in radiological reports. Br J Radiol 2000; 73(873): 999-1001.
[15] Berlin L. Communicating findings of radiologic examinations: Whither goest the radiologist’s duty? AJR Am J Roentgenol 2002; 178(4): 809-15.
[16] Berlin L. Pitfalls of the vague radiology report. AJR Am J Roentgenol 2000; 174(6): 1511-8.
[17] van Riel SJ, Ciompi F, Jacobs C, et al. Malignancy risk estimation of screen-detected nodules at baseline CT: Comparison of the PanCan model, Lung-RADS and NCCN guidelines. Eur Radiol 2017; 27(10): 4019-29.
[18] Rosenkrantz AB, Oto A, Turkbey B, Westphalen AC. Prostate imaging reporting and data system (PI-RADS), version 2: A critical look. AJR Am J Roentgenol 2016; 206(6): 1179-83.
[19] Müller-Schimpfle M, Wersebe A, Xydeas T, et al. Microcalcifications of the breast: How does radiologic classification correlate with histology? Acta Radiol 2005; 46(8): 774-81.
[20] Feig SA. Role and evaluation of mammography and other imaging methods for breast cancer detection, diagnosis, and staging. Semin Nucl Med 1999; 29(1): 3-15.
[21] Lacson R, Andriole KP, Prevedello LM, Khorasani R. Information from Searching Content with an Ontology-Utilizing Toolkit (iSCOUT). J Digit Imaging 2012; 25(4): 512-9.
[22] Panicek DM, Hricak H. How sure are you, doctor? A standardized lexicon to describe the radiologist’s level of certainty. AJR Am J Roentgenol 2016; 207(1): 2-3.
[23] Tempany CM, Zou KH, Silverman SG, Brown DL, Kurtz AB, McNeil BJ. Staging of advanced ovarian cancer: comparison of imaging modalities--report from the radiological fiagnostic oncology group. Radiology 2000; 215(3): 761-7.
[24] Waite S, Scott JM, Drexler I, et al. Communication errors in radiology - Pitfalls and how to avoid them. Clin Imaging 2018; 51: 266-72.
[25] Bruno MA, Petscavage-Thomas J, Abujudeh HH. Communicating uncertainty in the radiology report. AJR Am J Roentgenol 2017; 209(5): 1006-8.
[26] Soumerai SB, McLaughlin TJ, Avorn J. Improving drug prescribing in primary care: A critical analysis of the experimental literature. Milbank Q 1989; 67(2): 268-317.
[27] Lesuis N, den Broeder N, Boers N, et al. The effects of an educational meeting and subsequent computer reminders on the ordering of laboratory tests by rheumatologists: an interrupted time series analysis. Clin Exp Rheumatol 2017; 35(3): 379-83.
[28] Schwartz LH, Panicek DM, Berk AR, Li Y, Hricak H. Improving communication of diagnostic radiology findings through structured reporting. Radiology 2011; 260(1): 174-81.