ChatGPT to unseat physicians?

On November 30th, 2022, OpenAI released a GPT-3.5-based chatbot called “ChatGPT”, leaving the world in absolute awe. This chatbot is a general Large Language Model (LLM) which means it can understand language if trained on a proper amount of text data. Thus, it also has the capacity to generate -from scratch- meaningful sequences of words and full sentences (1). The success of ChatGPT was such huge that in only five days, it had already got over a million users worldwide (2).

Ever since, ChatGPT has drawn significant attention in the medical field, to the point that it has inconspicuously made its way on the authors list in a number of scientific papers and has also been challenged to take medical exams and provide medical advice. This has resulted in a massive debate which has gained both opponents and supporters. The major concern among the debaters is whether ChatGPT will eventually manage to take over what physicians, medical professionals and researchers have been doing up until now.

In a paper published in late 2022 (3), ChatGPT was challenged with 305 questions from all the three steps (1, 2CK & 3) of the United States Medical Licensing Examination (USMLE) exam. Overall, the results showed a moderate accuracy and a high internal concordance which put the ChatGPT in the passing range (60%). It is noteworthy that the scores are to improve as the LLM is further trained.

On a separate note, the use of ChatGPT as an author or co-author in several papers has had a greater impact on the debate, and this is while OpenAI website clearly suggests that ChatGPT, and I quote, “sometimes writes plausible-sounding but incorrect or nonsensical answers” (1). Only a few weeks into 2023, an interesting article was published in the Science journal which made notable points. It was mentioned that ChatGPT still has limitations in the case of academic use, for example it has, in several occasions, attempted to cite non-existing papers as references. The aforementioned article also pointed out that scientific articles are ethically not allowed to use ChatGPT in the writing process, since it is actually an act of plagiarism and thus, the paper can no more be considered as an original piece of work (4). Furthermore, in other recent papers (5, 6), authors stated that, since ChatGPT is trained on text data from all over the internet, it is very likely to generate repetitive sentences and even possibly overuse specific phrases, which give rise to issues of both plagiarism and killing of creativity, a skill in which the human mind is definitely superior. Having said all the above, if an author insists on ignoring the new law of keeping ChatGPT out of the academic writing, it has been proven that it is a truly tough task to distinguish AI-generated texts from original. In a study by Gao et al., human reviewers were given a series of scientific abstracts with the task of separating the original from the fake AI-generated. The results were shocking as the reviewers were only able to tell 68% of the fake abstracts apart; along with 14% of the original abstracts which they falsely flagged as fake (7).

Moreover, the matter of accountability is to be considered when discussing medical decisions made merely by ChatGPT with no human supervision; who is in charge when an error occurs? This issue also remains up for ethical debate as ChatGPT keeps integrating with the clinic in the very near future (6).

In conclusion, although ChatGPT has taken us a step closer to a revolution in academia, it still lacks series of specific features to be given more serious credit. Therefore, at least for now, it seems that clinicians are keeping their seats, since the more reliable computers are located in the very human brain.

Sadra Behrouzieh

1. OpenAI. ChatGPT: Optimizing Language Models for Dialogue 2022 [cited 2023. Available from: https://openai.com/blog/chatgpt/.
2. Gupta P. Internet Sensation ChatGPT Crosses 1 Million Users In 5 Days 2022 [Available from: https://entm.ag/fJbqOi.
3. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. medRxiv. 2022:2022.12.19.22283643.
4. Thorp HH. ChatGPT is fun, but not an author. Science. 2023;379(6630):313.
5. Biswas S. ChatGPT and the Future of Medical Writing. Radiology. 2023:223312.
6. Kitamura FC. ChatGPT Is Shaping the Future of Medical Writing but Still Requires Human Judgment. Radiology. 2023:230171.
7. Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv. 2022:2022.12.23.521610.