Clinical Acceptability of a Generative AI System for Automated Chest X-ray Reporting: A Multi-Institutional Non-inferiority Study

Purpose: To evaluate whether draft chest X-ray reports generated by a generative artificial intelligence (AI) system are non-inferior in diagnostic acceptability compared with reports written by board-certified radiologists in a multi-institutional setting.

Methods/Materials: This retrospective, multi-institutional, blinded crossover study included 1,455 patients who underwent chest radiography. Draft reports generated by a generative AI system (test arm) were compared with concise reports written by board-certified radiologists (control arm). Four radiologists independently evaluated all reports using the RADPEER scoring system; scores of 1–2 were classified as acceptable and score 3 as unacceptable. The primary endpoint was the difference in unacceptable diagnosis rates between arms. Non-inferiority was defined as the upper bound of the one-sided 97.5% confidence interval (CI) of the difference (test minus control) being less than a prespecified margin of 3.3%. Inter-rater agreement was assessed using Gwet’s AC1 and Fleiss’ kappa.

Results: The unacceptable diagnosis rate was 1.581% (23/1,455) in the AI-generated reports and 1.718% (25/1,455) in the radiologist-generated reports. The difference in unacceptable diagnosis rates between the two arms was −0.138% (95% CI, −1.100 to 0.818), with the upper bound of the CI well within the non-inferiority margin, thereby confirming non-inferiority of the AI-generated reports. High inter-rater agreement was observed for binary classification of report acceptability, with a Gwet’s AC1 of 0.928 (95% CI, 0.920–0.935). Fleiss’ kappa was low at 0.093 (95% CI, 0.069–0.118), reflecting the known kappa paradox due to the predominance of acceptable diagnoses. Subgroup analyses by institution demonstrated consistent non-inferiority results.

Conclusions: Draft chest X-ray reports generated by a generative AI system demonstrated non-inferior diagnostic acceptability compared with reports written by board-certified radiologists across multiple institutions. These findings support the potential role of generative AI as a reliable assistive tool for automated chest X-ray reporting in clinical workflow.