Informatics and Data Science

Application of Large Language Models to Mammography Rad-Path Correlation for Automated Data Extraction and Concordance Prediction

Location: Concourse Foyer Kiosk

First Author(s)

SS

Shawn H. Sun, MD, MS

Resident Physician
UCI Medical Center
Anaheim, California, United States

Co-Author(s)

KC

Kellie Cho, BS

Medical Student
UCI Medical Center, United States
DS

Dillon Sommer, BS

Research Fellow
UCI Medical Center, United States
JZ

Jasmine Zhao, MD

Resident Physician
UCI Medical Center, United States
RH

Roozbeh Houshyar, MD

Physician
UCI Medical Center, United States
JY

Jennifer Young, MD

Physician
UCI Medical Center, United States
PC

Peter Chang, MD

Physician
UCI Medical Center, United States
IT

Irene Tsai, MD

Physician
UCI Medical Center, United States

Purpose: To evaluate the performance of an open-source large language model (LLM) in automatically extracting key findings from mammography and pathology reports and predicting radiology-pathology (rad-path) concordance.

Methods/Materials: A total of 211 patients (215 biopsy targets) who underwent stereotactic biopsy in 2023 at a single academic institution were included. Mammography and pathology reports were collected and de-identified. To address class imbalance, 38 augmented discordant cases were generated. The gpt-oss model was used to extract mammographic findings associated with biopsy targets and their corresponding pathology results from the raw reports. Both a base and a prompt-engineered version of the model were evaluated for concordance prediction. A manual review was performed to assess extraction accuracy and predictive performance. Model comparisons were conducted using the McNemar test.

Results: Among the original cohort, BI-RADS categories 4 and 5 were assigned to 98.6% (208/211), and 2.4% (5/211) of cases, respectively. There were 4 discordant and 207 concordant cases. Extraction accuracy for salient findings was 99.5% (214/215) for mammography reports and 100% (215/215) for pathology reports. For concordance prediction, the base model achieved 64.2% accuracy (sensitivity 85.7%, specificity 59.9%), while the prompt-engineered model achieved 97.2% accuracy (sensitivity 90.5%, specificity 98.6%; p < 0.001).

Conclusions: LLMs can extract key mammography and pathology findings with near-perfect accuracy. Prompt engineering substantially improves rad-path concordance prediction performance without model fine-tuning. Manual rad-path correlation is labor-intensive and time-consuming. LLMs offer a scalable approach with high potential to assist this process, enhancing workflow efficiency for breast imagers and enabling large-scale retrospective analysis of historical data that was previously infeasible due to resource constraints.