Informatics and Data Science

Development and Evaluation of a Locally-Hosted Large Language Model with Retrieval-Augmented Generation for CT Decision Support

Location: Concourse Foyer Kiosk

First Author(s)

GL

Gabriel K. Lau, BA

Medical Student
University of Virginia Department of Radiology & Medical Imaging, Virginia, United States

Co-Author(s)

TC

Tien Comlekoglu, PhD

Medical Student
University of Virginia Department of Radiology & Medical Imaging, United States
AD

Ayush Doshi, MD

Resident Physician
University of Virginia Department of Radiology & Medical Imaging, United States
AK

Arun Krishnaraj, MD, MPH, FACR

Professor, ACR Chair of Commission on Patient- and Family-Centered Care
University of Virginia Department of Radiology & Medical Imaging, United States

Purpose: Choosing the right imaging protocol in a timely manner is a crucial part of patient care and involves collaboration between radiologists and referring providers. This consultation may be time-consuming and can contribute to delays in patient care. To address this, we developed and evaluated a locally-hosted Large Language Model (LLM) utilizing Retrieval-Augmented Generation (RAG) as a clinician support tool. The system was created to deliver quick imaging recommendations based on hospital-specific documentation. We aim for our system to enhance workflow efficiency, comply with internal documentation, and offer decision support to referring providers.

Methods/Materials: We provided a lecture presentation to non-radiology residents, covering proper CT orders for various indications, to serve as a knowledge base for an LLM. We implemented a RAG pipeline to provide this context to a locally hosted LLM. To assess the impact of model size on performance, we evaluated six open-source LLMs with parameter counts ranging from 3-billion to 8-billion parameters from the Llama, Qwen, Microsoft Phi, and Mistral model families. LLMs were evaluated using a standard set of 26 questions derived from lecture materials, and outputs were scored for semantic similarity to the expected answer using the f1 BERTscore. Local LLM outputs were additionally benchmarked against the commercial Gemini 2.5 Flash LLM. All outputs were evaluated for accuracy by radiologists.

Results: Mean f1 BERTscores ranged from 0.34 (Qwen2.5 3B) to 0.67 (Ministral 8B). The Ministral model matched the commercial Gemini 2.5 Flash (f1 score 0.66) given an equivalent system prompt, RAG pipeline, and questions. The Ministral model answered 21/26 questions correctly.

Conclusions: An open-source locally-hosted LLM utilizing a RAG framework may be used to provide decision support for imaging orders. This approach represents a promising and secure solution to augment clinical workflows and assist in timely and effective delivery of patient care.