Identifying the Severity of Heart Valve Stenosis and Regurgitation Among a Diverse Population Within an Integrated Health Care System: Natural Language Processing Approach

Introduction

Valvular heart disease (VHD) is a leading cause of cardiovascular morbidity and mortality worldwide [1-3] and poses a substantial health care and economic burden on health care systems [4,5]. The prevalence of VHD, especially aortic stenosis, is expected to rapidly increase in the United States and Europe due to population aging [4,5]. Accurate assessments of the burden of VHD are increasingly relevant as the treatment options for these patients continue to expand. VHD research based on administrative diagnostic codes shows incomplete identification [6] or inaccuracy of coding [7]. Accurate and complete identification of VHD based on information from echocardiography reports other than diagnosis codes has the potential to facilitate patient care and VHD-related cardiovascular research.

Advances in diagnostic imaging technologies have greatly improved the precision and efficiency of assessing heart valve disorders [8,9]. Echocardiography is the primary imaging modality for evaluating valve structure and function and assessing the severity and hemodynamic consequences of VHDs. Transthoracic echocardiography (TTE) provides key insights into the mechanisms of VHDs [8]. The wealth of data and information generated by the interpretations of echocardiographic studies significantly aids clinical management and research. Although the format of echocardiography reports is often templated, the content in each section remains as free text. This presents a challenge for systematic analysis, necessitating advanced natural language processing (NLP) techniques to transform from unstructured into structured and analyzable data [10].

Over the past years, applications of NLP algorithms or systems have been developed to automatically extract clinical information from free-text clinical notes [11-13]. Rule-based or machine learning–based NLP studies [6,14-22] have attempted to extract information about valve severity and related measurements from echocardiography reports. Most of these studies have concentrated on extracting some specific conditions and measurements, such as aortic stenosis and peak velocity. Two exceptions are Nath et al [18], who created EchoInfer, a system capable of extracting a set of data elements (~80) reported in echocardiography reports, and Dong et al [19], who developed an NLP system that extracts ~43 data elements described in echocardiography reports. Although both systems extracted elements relevant to VHD, the performance was based on the overall data elements rather than the clinically relevant feature of the severity of individual VHD. Additionally, the small training and validation samples in both studies limited the capabilities to accurately assess performance for less common VHDs, such as mitral valve, tricuspid valve, and pulmonic valve stenosis. The purpose of this study was twofold: (1) to develop and validate a computerized algorithm for extracting the severity of stenosis and regurgitation of the 4 heart valves (aortic valve, mitral valve, tricuspid valve, and pulmonic valve) and (2) to apply the validated algorithm to all TTE reports within the large integrated Kaiser Permanente Southern California (KPSC) health care system to estimate the frequencies of VHD across a diverse population.

Methods

Study Setting and Population

The study subjects were health plan enrollees of the KPSC, an integrated health care system providing comprehensive medical services to 4.8 million members across 15 large medical centers and more than 250 medical offices throughout Southern California. The demographic characteristics of KPSC members are diverse and largely representative of the residents in Southern California [23], with health insurance through group plans, individual plans, Medicare, and Medicaid. Patients aged 18 years or older who underwent at least 1 TTE within the KPSC system between January 1, 2011, and December 31, 2022, were included in this study.

Ethical Considerations

The KPSC Institutional Review Board reviewed and approved the study protocol, with a waiver of the requirement for informed consent (approval number 13490). The study complied with the Health Insurance Portability and Accountability Act. Only authorized persons were provided access permission to perform all analyses.

NLP Algorithm and Process

Figure 1 outlines the steps for identifying valvular stenosis and regurgitation, and detailed descriptions follow later.

‎

Echocardiography Report Extraction and Annotation

The TTE reports during the study period were extracted from the KPSC’s electronic health record (EHR) system. These reports were written by physicians and were generally structured in a templated format. Most reports contain the following sections: (1) title, patient demographics, procedure performed, performing provider, and procedure indication; (2) exam quality; (3) dimensions/measurements; (4) findings/results; (5) impression; (6) miscellaneous; (7) summary/conclusion; and (8) physician signature. Despite the templated structure, the content within each section is in free-text format, and the report can have a varying order of or incomplete sections. Examples of deidentified TTE reports are included in Table S1 in Multimedia Appendix 1.

An initial list of phrases and terms related to capturing stenosis, regurgitation, and severity of the 4 heart valves was compiled based on the input of the study cardiologist, published case definitions, and ontologies [6,18,19,24] and enriched by the training data set to capture additional linguistic variations, such as abbreviations and misspellings. The collected terms are listed in Table S2 in Multimedia Appendix 1.

To effectively capture the severity of the rare heart valve stenosis described in the TTE reports, 2 sets of TTE reports were prepared for annotation and algorithm training. The first data set contained a total of 800 TTE reports, of which 200 (25%) were randomly selected from each of the 4 aortic valve peak velocity groups (≤2.5, 2.6-2.9, 3.0-4.0, ≥4.0 m/s) instead of simple random selection from the extracted entire TTE reports (data set 1). The second data set contained another sample with 400 TTE reports based on diagnosis codes (data set 2): 134 (33.5%) reports randomly selected for patients with a mitral stenosis diagnosis (International Classification of Diseases 10th Revision [ICD-10] code I05.0 or I05.2), 133 (33.3%) reports randomly selected for patients with a tricuspid stenosis diagnosis (ICD-10 code I07.0, I07.2, I36.0, or I36.2), and 133 (33.3%) reports randomly selected for patients with a pulmonic stenosis diagnosis (ICD-10 code I37.0 or I37.2). Both data sets were manually reviewed by an experienced board-certified cardiologist and a medical student to record the presence/absence (Table S2 in Multimedia Appendix 1) and severity (Table S3 in Multimedia Appendix 1) of stenosis and regurgitation of the 4 heart valves. These annotated TTE reports were split into 6 batches, each containing 200 reports. The first batch was reviewed by both annotators to ensure quality and consistency. The rest were equally divided between the 2 annotators.

NLP Algorithm Development

We first divided the reports in the annotated data sets into the sections of echocardiography report extraction and annotation described before and then subsectioned them based on titles and subtitles. Each subsection uniquely captured information about a specific valve (aortic, mitral, tricuspid, or pulmonic). The selected sections and subsections were then preprocessed through the letter lowercase conversion, misspelled word correction (as shown in Table S4 in Multimedia Appendix 1), and tokenization (ie, segmenting text into linguistic units, such as words and punctuations) [25] for further NLP processing. The study terms/phrases and their abbreviations and acronyms were collected by the cardiologist before NLP development. For each of the study terms/phrases, misspelled word correction was performed by manually examining the top 100 similar words derived from a trained deep learning word2vec model [26,27] based on the study corpus; 100% of data set 1 and 50% of data set 2 were used for training, and 50% of data set 2 was used for validation.

We used the annotated reports to develop a rule-based computerized algorithm via an iterative process to determine the presence/absence and severity status of stenosis and regurgitation in the 4 heart valves (aortic, mitral, tricuspid, and pulmonic). Table S5 in Multimedia Appendix 1 summarizes the included sections and subsections using which the following search steps were applied. The process was applied to each sentence within the included sections or subsections:

• Search for terms associated with stenosis and regurgitation. The status was labeled as “no evidence” if no relevant term was found (Table S1 in Multimedia Appendix 1).
• If a relevant term is found, search for the negated terms associated with the identified stenosis and regurgitation terms. If a negation was found (eg, no aortic stenosis, without evidence of aortic stenosis), the identified stenosis or regurgitation term was ignored.
• Search for history terms (eg, a prior study showed trace mitral regurgitation) associated with the identified stenosis and regurgitation terms. If an associated history term was detected, the detected stenosis and regurgitation term was also ignored (Table S6 in Multimedia Appendix 1).
• Search for severity terms. If no severity term was found, the sentence was labeled “unknown severity.” If multiple severity terms were detected, the severity of the report was assigned based on the following priority: prosthetic, very severe, severe, moderate to severe, moderate, mild to moderate, mild, trace to mild, trace, and sclerosis. Trace to mild and trace were only applied for regurgitation, while sclerosis was only applied for aortic stenosis (Table S2 in Multimedia Appendix 1).

Discordant cases between the computerized algorithm and manually annotated labels were reviewed and adjudicated by the cardiologist. If the adjudicated results differed from the computerized results within each round, they were used to refine the algorithm and process.

NLP Algorithm Validation

The results from the final computerized algorithm were compared with the manually annotated results in the validation data set. The proportions of true-positive (TP), false-positive (FP), and false-negative (FN) cases were used to estimate sensitivity, the positive predicted value (PPV), and the overall F₁-score (a measure of the overall model fit). Sensitivity was defined as the proportion of reports correctly labeled by the computerized algorithm (TP) among all reports (TP+FN) ascertained by chart review. The PPV was defined as the proportion of reports correctly labeled (TP) among all those labeled by the computerized algorithm (TP+FP). The overall accuracy of the F₁-score for each comparison was calculated via the standard formula 2 × PPV × sensitivity/(PPV + sensitivity).

Estimating the Severity of Stenosis and Regurgitation at the Report Level

The finalized computerized algorithm was implemented via Python 3.10 to process the entire study set of TTE reports. The status and severity level of stenosis and regurgitation for each of the 4 heart valves (aortic, mitral, tricuspid, and pulmonic) were reported for all TTE reports during the study period. In TTE reports with VHDs detected, the severity levels of the diseases at the report level were summarized by age group (18-49, 50-64, 65-79, and ≥80 years), sex, and race/ethnicity (non-Hispanic White, non-Hispanic Black, non-Hispanic Asian/Pacific Islander, non-Hispanic Native American, Hispanic, multiple races, other/unknown).

Results

Performance Assessment of the NLP Algorithm

The performance of the computerized algorithm against the manually annotated results based on the validation data set is summarized in Table 1 for stenosis and Table 2 for regurgitation. The PPV, sensitivity, and F₁-score of having positive stenosis and regurgitation were 100%, 100%, and 1 for aortic, mitral, and tricuspid valves; 96.2%, 96.2%, and 0.96 for pulmonic stenosis, respectively; and 97.0%, 98.5%, and 0.98 for pulmonic regurgitation, respectively. The PPV, sensitivity, and F₁-score of prosthetic valves were also 100%, 100%, and 1 for aortic, mitral, and tricuspid valves and 92.3%, 92.3%, and 0.92 for pulmonic valves, respectively. For TTE reports with specific severity detected, the PPV was 100% for most of the severe categories, with several exceptions (eg, 80% for severe mitral stenosis and 50% for unknown severity pulmonic stenosis; Table 1). Sensitivity was also 100% for most of the severe categories, with several exceptions (eg, 87.5% for moderate-to-severe mitral stenosis; Table 1).

Estimating the Severity of Stenosis and Regurgitation at the Report Level

A total of 1,225,270 TTE reports among 677,106 patients were extracted from the KPSC EHR system during the study period. Slightly more than half (n=621,237, 50.7%, data not shown) of them were for male patients. The median age at the time of the echocardiogram was 67 years (IQR 55-77). The mean number of TTEs performed per patient was 1.8 (SD 1.6) during the study period (data not shown). The distributions of the stenosis and regurgitation severity across the TTE reports identified by the NLP algorithm and process are summarized in Figure 2. Of the 1,225,270 TTE reports, 111,300 (9.08%), 20,246 (1.65%), 397 (0.03%), 2585 (0.21%), 345,115 (28.17%), 802,103 (65.46%), 903,965 (73.78%), and 286,903 (23.42%) reports had evidence of aortic stenosis, mitral stenosis, tricuspid stenosis, pulmonic stenosis, aortic regurgitation, mitral regurgitation, tricuspid regurgitation, and pulmonic regurgitation, respectively. In addition, 50,507 (4.12%), 22,656 (1.85%), 1685 (0.14%), and 1767 (0.14%) of the heart valves were identified as prosthetic aortic, mitral, tricuspid, and pulmonic valves, respectively. The distribution of severity levels among each identified VHD is shown in Figure 3. Mild and moderate were the most common severity levels of heart valve stenosis, while trace and mild were the most common ones for regurgitation. More details can be found in Table S7 in Multimedia Appendix 1.

In TTE reports with VHDs detected, the severity level of the diseases stratified by sex, race/ethnicity, and age group at the time of the TTE are presented in Tables 3-6 for stenosis and Tables 7-10 for regurgitation. Males had a higher frequency of aortic stenosis and all 4 valvular regurgitations, while females had more mitral, tricuspid, and pulmonic stenosis. Non-Hispanic Whites had the highest frequency of all 4 valvular stenosis and regurgitations. The distribution of stenosis and regurgitation severity was similar across race/ethnicity groups. The frequencies of aortic and mitral stenosis increased with age, whereas the frequencies of tricuspid and pulmonic stenosis decreased with age. The frequency of valvular regurgitation increased with age for all 4 heart valves. Among the TTE reports with stenosis detected, younger patients were more likely to have mild aortic stenosis, while older patients were more likely to have severe aortic stenosis. However, the frequencies of mitral stenosis were opposite (milder mitral stenosis in older patients and more severe mitral stenosis in younger patients). In contrast, for TTE reports with regurgitation detected, younger patients had a higher frequency of severe/very severe aortic regurgitation, while older patients had higher frequencies of mild aortic regurgitation and severe/very severe mitral/tricuspid regurgitation. The distribution of severity can be found in Table S8 in Multimedia Appendix 1.

‎

‎

Discussion

Principal Findings

In this study, we developed a computerized algorithm to identify the presence/absence and the severity of stenosis and regurgitation of the 4 heart valves (aortic, mitral, tricuspid, and pulmonic) from reports of routinely performed TTEs. This algorithm yielded high accuracy in extracting information, except for a few severity groups due to their small numbers. This process was successfully implemented in a large integrated health care system to estimate the frequencies of VHD described in the TTE reports among a demographically diverse population.

Comparison With Prior Work

Echocardiography is the primary imaging technique for evaluating the severity of VHD. Incorporating an NLP algorithm developed to extract information about valvular lesion severity from unstructured echocardiogram reports allows the identification of patients with VHD across a large population. This is useful because the frequency of surveillance imaging is dependent on the severity of the valvular lesion. Specifically, patients with mild valvular lesions typically require imaging every 3-5 years, those with moderate lesions need evaluations every 1-2 years, and those with severe lesions need evaluations every 6-12 months [28]. Identifying and categorizing patients with VHD at a population level ensures that all patients receive timely and adequate follow-up.

The performance of the algorithms reported in this study was comparable with those reported in previous studies [6,18,19]. In line with findings from previous studies [6,18,19], we observed that the percentage of VHD increases with patient age [9]. VHD affected both sexes, although certain conditions showed sex-specific patterns. Aortic regurgitation was more commonly observed in males, a finding that aligns with other studies, indicating a male predominance of aortic regurgitation [4]. Conversely, tricuspid regurgitation was more commonly observed in females in this population. Further research into the incidence, prevalence, and associated risk factors of these valvular lesions will enhance our understanding of the causes behind the observed sex differences [29].

Recent studies have attempted to extract stenosis and regurgitation from echocardiography reports [6,17-19]. Solomon et al [6] focused primarily on the extraction of aortic stenosis and a few continuous measurements. Although Nath et al [18] and Dong et al [19] attempted to retrieve stenosis and regurgitation of heart valves, their performances were not assessed for each condition independently, and neither of the authors evaluated performance by severity of illness. Even for the combined evaluation, Dong et al [19] reported low performance for both the precision and recall of identifying stenosis of the 4 heart valves. The approach taken by this study has several advantages. First, part of our training and validation samples included TTE reports of potential patients diagnosed with mitral, tricuspid, and pulmonic stenosis. Therefore, the samples included a fair number of patients with these relatively rare conditions, which allowed the computerized algorithm to train/recognize corresponding potential patterns. Second, our study evaluated each case of stenosis or regurgitation independently. However, the performance of some severity levels needs cautious interpretation due to few cases and small validation samples. A larger sample and additional validations based on external data sets in future work can yield more robust performance and strengthen the evidence for the algorithm’s utility and robustness in real-world clinical settings.

Strengths and Limitations

A key feature of our algorithm is its ability to extract important elements from written echocardiogram reports and convert them into structured data elements. This capability enables a health care system to provide true population care by tracking the number of patients with varying degrees of valvular lesions. By doing so, the health care system can ensure that surveillance monitoring and follow-up appointments are appropriately scheduled. This integration was a crucial consideration for this study. It will be a focus of future work, as it enhances the practical utility and adoption of the algorithm in clinical settings. In addition to extracting severity, future work will also enhance the computerized algorithm to retrieve other VHD-related measurements [19] to facilitate patient care and management.

Research of heart valve conditions based on diagnosis codes only may be impacted by the inaccuracy of coding, especially for minority populations. Crousillat et al [7] showed that diagnosis codes for aortic stenosis are less accurate for racial and ethnic minorities and less severe stages of the disease and, therefore, cannot be used to evaluate observed care disparities. This issue is likely to be alleviated by the application of NLP to TTE reports. Solomon et al [6] demonstrated that NLP application captures 35.4% more aortic stenosis compared to diagnosis code identification. Future studies are needed to understand and mitigate recently identified VHD care disparities and improve outcomes for patients [28].

This study primarily used a rule-based approach for NLP. Transformer-based models, such as bidirectional encoder representations from transformers (BERT) [30], have gained popularity in recent years in clinical research involving NLP. These large language models can effectively capture the text’s intricate relationships via word embedding representation and attention mechanism and, therefore, are capable of analyzing information from unstructured notes in the health care domain more accurately [31-34]. Future research may integrate these sophisticated machine learning or deep learning language models into NLP algorithms to further boost performance and to handle the complexity of medical language more effectively.

Our study acknowledges several limitations. First, the completeness and accuracy of the extracted information were dependent on the information documented in the TTE reports. Incomplete or inaccurate documentation could lead to misclassification. Despite our efforts to correct misspelled words, there could be additional unidentified errors. Second, the templated formats of echocardiography reports could limit the diversity and specificity of language used, potentially affecting the algorithm’s sensitivity. Third, although our training process was quite comprehensive and included a relatively large number of notes, the rules and lexicons developed from our training data sets were still not highly comprehensive. For example, the severity of mitral stenosis in “moderate to borderline severe calcific mitral stenosis” should be “moderate to severe.” However, the algorithm identified it as “severe” because of the additional word “borderline” prior to “severe.” Therefore, more samples could be used to enhance the rules and lexicons in the future, especially for rare conditions (mitral, tricuspid, and pulmonic stenosis). Fourth, numerous abbreviations in the study used terms with multiple meanings, which complicated the identification process. For instance, the word “as” could mean “aortic stenosis”; “ms” could mean “mitral stenosis” or “millisecond,” a time unit used for velocity measurements; and “tr” could mean “trace” or “tricuspid regurgitation.” Although we applied a set of rules to determine the exclusion of used abbreviated terms, the algorithm could still potentially misuse the meaning of the abbreviation of these terms. Fifth, if a severity term was found to appear prior to a set of stenosis or regurgitation terms listed together (eg, mild as/ai/mr), our algorithm only assigned the severity to the first term, leading to incomplete labeling of the severity of other terms. Sixth, the TTE reports of patients with congenital valvular conditions frequently used the terms “systemic AV” and “subpulmonic AV,” which represented the morphologic tricuspid valve and morphologic mitral valve, respectively. However, the meanings of these 2 terms are different in patients with noncongenital disease. Our algorithm did not search for these 2 terms for patients with congenital valvular conditions, which could lead to potential misclassification of congenital valvular conditions. Although the population of patients with congenital VHDs is very small, future work can modify the algorithm to improve the performance for congenital valvular conditions. Lastly, although the NLP algorithm in this study was trained by a large number of annotated echocardiogram reports from the KPSC, algorithm may need to be adjusted before it is implemented in other health care organizations, due to variations in report formatting and reporting requirements. Typically, reformatting echocardiogram reports before implementing the algorithm can enhance its adaptability. When feasible, users may also consider retraining the algorithm using notes specific to the organization for improved performance. However, the complexity of our NLP algorithm and process might limit its adoption in settings without specialized expertise in NLP or access to similar resources for algorithm development and validation [35].

Conclusion

The computerized algorithm developed can effectively identify heart valve stenosis and regurgitation and the severity of valvular involvement. This algorithm has potential applications in clinical research and patient cardiovascular care management. The computerized algorithm needs further adjustments to accommodate variations in the format and presentation of TTE reports when it is implemented in other health care organizations.