Evaluating Open-Source AI Models for Bone Fracture Detection Using the MURA (Musculoskeletal Radiographs) Dataset: A Comparative Validation Study for Low-Resource Clinical Settings

Purpose: Accurate, timely fracture detection remains a major challenge, especially in low-resource healthcare settings with limited radiology expertise. Deep learning has made automated detection more accessible through open-source models, but their reliability and clinical readiness remain uncertain. This study evaluates two models—YOLOv7-BoneFractureDetection and MohtashamMurshid-BoneFractureClassification—using the MURA musculoskeletal radiograph dataset to assess performance and clinical potential in resource-limited environments.

Methods/Materials: Fifty musculoskeletal radiographs (25 fracture and 25 non-fracture) were randomly sampled from the MURA dataset (Stanford University Public Medical Library). Models were deployed via Cursor IDE and tested interactively using Streamlit.io. The results from each model produced binary predictions (fracture / no fracture) and corresponding confidence scores.
Using Pandas and Scikit-learn within Python, we determined the sensitivity, specificity, precision, F1 score, accuracy, average confidence, and AUC of both language models.

Results: YOLOv7 performed best overall (82% accuracy) with high precision (90%) and specificity (92%), meaning fewer false positives. It also showed strong consistency (F1 80%, AUC 0.89) and true-positive confidence (78.4%), making it ideal for confirmatory support.
BoneFractureClassification had higher sensitivity (88%) but lower precision (71%) and specificity (64%), causing more false positives. With an F1 of 78.6% and AUC of 0.85, it may be better suited for triage screening.

Conclusions: Results show a clear trade off between accuracy and sensitivity: YOLOv7 delivers more precise, reliable detection, while BoneFractureClassification prioritizes recall. YOLOv7’s conservative precision fits confirmatory workflows, whereas BoneFractureClassification may be better for initial triage. Both models show promise for AI-assisted fracture detection but need larger, multi-site validation against radiologist consensus. Open-source tools like these could reduce delays and clinician workload in low-resource settings, improving global access to orthopedic diagnostics. Further work on validation, explainability, and usability is key.