Introduction

Electronic health records (EHRs) serve as a unifying source of data for clinical care, research, administrative work, billing, and compliance. Clinical documentation in the EHR has become increasingly complex and challenging to read. Recognizing this problem, in 2015 the American College of Physicians convened their Medical Informatics Committee to publish a position paper on the goals of electronic health record (EHR) generated documentation. The committee stated, “The primary goal of EHR-generated documentation should be concise, history-rich notes that reflect the information gathered and are used to develop an impression, a diagnostic and/or treatment plan, and recommended follow-up.”¹ However, despite efforts to reinvent clinical documentation for the modern era, much remains unchanged from the time of paper charts.² Additionally, the digitization of health records has created a tension between structured data formats, which are more easily interpreted by computers, and flexible documentation formats, which may increase readability.³ In this way, digital health records can be more difficult for a human to read than the paper charts they replaced. Integrating artificial intelligence to augment and support clinical documentation processes may relieve some of this burden, but this goal has not yet been fully achieved.⁴ The competing priorities of the EHR lead to clinical documentation that is extensive and challenging for human end users to understand, a phenomenon known as “note bloat.”⁵ In attempting to serve many roles, clinical documentation often ends up serving none optimally.

For physicians, the record must document the patient’s presenting signs and symptoms along with diagnostic test results and clinical reasoning, ultimately leading to a diagnosis. For administrators, the clinical record must adequately document the level of patient care provided, to allow for appropriate financial reimbursement and compliance. As patient access to EHR portals has become more common (and even, in some cases, mandated by legislation), it has become increasingly important that personal medical records become more accessible, comprehensible, and portable, so that patients can effectively understand and coordinate their healthcare across multiple providers.⁶ Metrics for evaluating how well clinical documentation can be understood by these stakeholders are lacking. Often, note length is used as a surrogate to detect “note bloat,” but the limitations of this as a single measure are self-evident.

An additional method of comprehension evaluation is a readability score; these scores evaluate texts to estimate the ease with which they can be understood by a reader. Readability can be assessed using multiple scales. A popular scale is the Flesch-Kincaid (FK) grade level, which was developed for the United States Armed Forces in the 1970s and has been used to judge the readability of many documents.^7,8 The FK grade level is calculated based on and is directly proportional to the ratio of total words to total sentences and the ratio of total syllables to total words; the result is a number indicating the grade level that would be required for a reader to understand the content. Therefore, a document with longer sentences comprised of words with more syllables will have a higher grade level score compared with a document with short sentences of monosyllabic words. A FK grade level of 8 or below is generally considered to be best practice for communication to the general public. In medicine, the FK grade level has been used to grade patient educational handouts and websites, dismissal summaries, questionnaires, and patient consent forms.^9–16 However, there has been little research into the readability of clinical documentation.

While not sparking a revolution in concise, easily readable, clinical documentation, the EHR can easily facilitate evaluation of clinical notes and related quality improvement initiatives. For example, in one study, clinical alerts were shown to improve compliance of anesthesia documentation.¹⁷ Another study evaluated the “attractiveness” of clinical notes in an EHR.¹⁸ Additionally, quality of clinical documentation was evaluated in one study using a standardized scale, the Physician Documentation Quality Instrument (PDQI).¹⁹ Previous assessments of note quality largely required human adjudication. In this study, we evaluated the feasibility and utility of using the FK score as an automated tool for assessing clinical documentation quality and its association with the underlying electronic health record (EHR) software.

Methods

All clinical notes recorded between 2016 and 2019 at a large healthcare organization were evaluated across specialties. In May 2018, the organization transitioned from a highly-customized, “homegrown” EHR to a widely-used, commercial EHR. Notes were classified as either being created in the old or the new EHR. To limit variability, only labeled clinical consultation notes were included in the analysis; this note type was consistent between the two EHR systems. A custom model was used to tokenize and perform sentence splitting on each input document, yielding number of sentences and words per document.²⁰ Syllables in each word were calculated using a custom Python script that splits along vowels with special handling for multi-vowel syllables.²¹ FK grade level was calculated⁷ for every clinical note and summarized by author specialty. Summary statistics including minimum, maximum, 25^th percentile, median, and 75^th percentile were calculated. A paired T-test was performed to assess the relationship of each specialty’s notes before and after the EHR transition. A two-way analysis of variance (ANOVA) statistical test was performed to assess the relationship between author specialty and timing of authorship with median FK grade level. Two additional readability scores were calculated for comparison (Gunning Fog Index,²² Dale-Chall readability²³) and reported in the supplementary materials.

Results

A total of 630,246 clinical notes were evaluated across 10 medical specialties including anesthesiology, cardiology, family medicine, gastroenterology, general internal medicine, neurology, obstetrics-gynecology, pediatrics, psychiatry, and pulmonology (Table 1). The evaluated notes mostly came from cardiology (177,256; 28.1%), neurology (159,702; 25.3%), and gastroenterology practices (96,448; 15.3%). Anesthesia had the fewest notes evaluated (1,588; 0.3%). Note volume was relatively evenly split between the old EHR (294,625; 46.7%) and the new EHR (335,621; 53.2%).

Overall, general internal medicine consultation notes had the highest median FK grade level (8.62) over the four-year study period. This was followed by the other specialties with median FK grade levels varying from 8.55 (psychiatry) to 7.72 (family medicine). Several specialties had a decrease in median FK grade levels following the EHR transition (anesthesiology, obstetrics and gynecology, pediatrics, pulmonology), though overall readability remained fairly stable. Figure 1 illustrates the changes in median, 25^th, and 75^th percentile FK grade level by specialty over the study period stratified by EHR.

In matched analysis by specialty, notes written in the new EHR had FK grade levels that were not statistically different those in the previous EHR (p=0.93). Matched by specialty, the standard deviation of the FK grade levels of the notes significantly decreased from the previous EHR to the new EHR (mean difference in standard deviation of 0.29, p<0.01). All specialties except family medicine had a decrease in the standard deviation of their FK grade level note scores.

Discussion

In this study, the readability of notes written by physicians performing consultations before and after an EHR transition were not significantly affected by the underlying EHR system. There was similar distribution of FK grade level readability scores for clinical notes across specialties on both the old and new EHRs. This finding should be reassuring and suggests that physicians will continue to write similarly readable notes regardless of the tool used for writing.

While the FK scores of consultation notes remained stable during the transition time, the variability of the scores decreased overall. This suggests that notes in the new EHR were generally more uniform in readability, possibly related to new note composition tools, as the new EHR included improved templating tools and provided easier access to copying components of previously written notes. Both features could have contributed to the decrease in variability of FK scores following the EHR transition.

Some worry that increased use of note templating tools worsens the readability of clinical documents.²⁴ Our results showed that despite significant use of these features in the new EHR, readability scores overall were stable. While the stability of results can be considered comforting, a more desirable outcome would have been improved readability scores with the new EHR.

There are many potential strategies to improve readability of health documentation. Readability may be improved by decreasing stringent requirements, particularly for including structured data that is readily accessible in other areas of the EHR. Documenting vital signs, past medical history, medications administered makes it more challenging for a reader to find key information. Templates should be evaluated for readability principles before deployment. Adopting formal structures and vocabularies focused on describing the rationale behind a patient’s care improves note quality and discourages the use of copy and paste.^25,26 Some have proposed allowing physicians to simply annotate the record rather than repeatedly documenting the same findings.²⁷

Limitations

This study had several limitations. It did not include human assessment of FK score readability of clinical documentation. There is also some missing data in early 2016 due to a data platform change. Only clinical notes for consultations were considered in this study, which may not have provided a complete evaluation of the medical record. Additionally, the entire text of the clinical notes was evaluated. Limiting the evaluation to just the non-collapsed sections of the notes may have improved readability scores and more accurately assessed the most-read sections of the note.

Conclusions

Results from this study of clinical note readability scores pre- and post-EHR transition showed that clinical documentation readability remained stable and therefore may not be strongly associated with underlying EHR. Migrating to a new EHR alone should not be expected to improve readability of clinical notes. Our study results also suggested that FK grade level or other readability scores may be reliably used in the future evaluations of clinical documentation. The scores can easily be applied in an automated way across an entire health system’s EHR. These evaluations could be used to identify individuals or specialties whose clinical notes are significantly different than others. Further work is needed to clarify the relationship between readability and documentation quality.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no funding for this research.