Multimodal AI chatbot delivers different results in ophthalmological image analysis

Rajeev H. Muni, MD, M.S.

Credit: Unity Health Toronto

The latest version of the artificial intelligence chatbot ChatGPT-4 correctly answered approximately two-thirds of image-based multiple-choice questions in a publicly available dataset, according to a new cross-sectional study. ophthalmological case.¹

However, the large-scale language model (LLM) answered correctly more often for questions that did not rely on ophthalmological image interpretation (82%) than for image-based questions (65%). The chatbot was stratified by specialty, with the best performance for retinal cases and the worst performance for neuro-ophthalmology cases.

“As multimodal LLM becomes increasingly popular, it remains essential to continue to emphasize its appropriate use in medicine and to highlight concerns surrounding confidentiality and bioethics,” said Rajeev of the Department of Ophthalmology at St. Michael's Hospital. -Written by a research team led by H. Muni, MD, M.S.A. Unity Health Toronto.

Recent evidence points to the potentially transformative nature of AI chatbots in healthcare, particularly ophthalmology, reducing the burden on healthcare workers, from patient education to remote monitoring of eye diseases.² However, like any new technology, regulatory compliance, privacy, and the integration of AI into healthcare systems must be addressed before the mold is cast.

Previous research by Muni et al. found that an earlier version of ChatGPT-4, which was limited to text-based prompts, improved performance at an impressive rate in medical and ophthalmology settings.³ Since ophthalmology relies on multimodal image interpretation to confirm diagnostic accuracy, the team believes that this new ability for chatbots to interpret ophthalmological images could be important to reach the next stage. I pointed out that there is.¹

“New releases of chatbots have great potential to increase the efficiency of ophthalmic image interpretation, reducing clinician workload, reducing interpretation variability and errors, and ultimately improving patient outcomes. ,” the researchers wrote.

The cross-sectional analysis used publicly available data from the OCTCases medical education platform based at the Canadian Center of Excellence. Each case is divided into retina, neuro-ophthalmology, uveitis, glaucoma, ocular oncology, and pediatric ophthalmology. All multiple-choice questions across all ophthalmology cases available on the platform were examined for analysis.

Muni and colleagues created a new ChatGPT Plus account to ensure they had no previous conversation history with LLMs before the study began. The LLM account was given multimodal functionality by the chatbot's parent organization, OpenAI, and all relevant cases and images were populated from his October 16, 2023 to his October 2023. The accuracy of the chatbot, measured as the percentage of correct answers for image recognition, was utilized as follows. Primary endpoint of analysis.

Overall, the analysis consisted of 136 cases with 448 images on OCTCase. Of these cases, 429 cases (82%) were formalized as multiple-choice questions and subjected to statistical analysis. Of these cases, 125 were accompanied by optical coherence tomography (OCT) scans (92%) and 82 by fundus imaging (60%).

In their analysis, Muni et al. found that ChatGPT-4 correctly answered 299 (70%) of the multiple-choice questions in all ophthalmology cases. LLM performance was best on questions related to the retina (77%) but worst on neuro-ophthalmology categories (58%) (18% difference) [95% CI, 7.5–29.4]; P <.001).

Questions from other ophthalmology specialties showed moderate performance, including ocular oncology (72%), pediatric ophthalmology (68%), uveitis (67%), and glaucoma (61%).

Across 303 multiple-choice questions requiring image interpretation, ChatGPT-4 answered 196 questions (65%) correctly. The 126 non-image-based questions had high scores, with 103 (82%) answered correctly. Overall, chatbots performed better on non-image-based questions (difference, 17%) [95% CI, 7.8 – 25.1]; P <.001), especially in pediatric ophthalmology (difference, 47%) [95% CI, 8.5 - 69.0]; P = .02).

Future analyzes should focus on the chatbot's ability to interpret different ophthalmic imaging modalities and learn when to be as accurate as certain machine learning systems in ophthalmology, Muni et al. did.

“As the accuracy of chatbots increases over time, they may have the potential to inform clinical decision-making in ophthalmology through real-time analysis of ophthalmological cases,” Muni and colleagues wrote.

References

1. _{Mihalachi A, Huang RS, Popovic MM, et al. Accuracy of interpretation of ophthalmological clinical images by artificial intelligence chatbots. JAMA Ophthalmology. Published online on February 29, 2024. doi:10.1001/jamaophysicalmol.2024.0017}
2. _{Tan TF, Thirunavukaras AJ, Jin L, Lim J, Poe S, Teo ZL, Ang M, Chan RVP, Ong J, Turner A, Karlstrom J, Wong TY, Stern J, Ting DS. Artificial intelligence and digital health in global eye health: opportunities and challenges. Lancet Globe Health. 2023 Sep;11(9):e1432-e1443. doi: 10.1016/S2214-109X(23)00323-6. PMID: 37591589.}
3. _{Iapoce C. Artificial intelligence chatbot appears to improve ophthalmology knowledge assessment. HCP live. July 18, 2023. Accessed March 1, 2024. https://www.hcplive.com/view/artificial-intelligence-chatbot-Appears-improve-ophysical-knowledge-assessment.}