August 09, 2023 – A recent study conducted by the University of California, Riverside (UCR) shed light on the strengths and limitations of Google and ChatGPT in the realm of healthcare. While these platforms offer valuable resources, particularly in the field of Alzheimer’s and dementia, there are areas that require improvement to fully realize their potential.
The researchers at UCR acknowledged the positive aspects of both Google and ChatGPT, including their ability to provide up-to-date and unbiased information. As Vagelis Hristidis, PhD, a professor of computer science and engineering at UCR’s Bourns College of Engineering, explained, combining the best features of these platforms could result in a more effective system for treating various conditions.
The study focused on dementia and Alzheimer’s, given their growing prevalence. The Centers for Disease Control and Prevention (CDC) estimated that the number of people aged 65 and older with dementia would increase from 5 million in 2014 to 14 million by 2060. To evaluate the interaction of Google and ChatGPT with these conditions, the researchers posed 60 queries related to disease processes and services for patients and their families.
The findings revealed a mixed picture. While Google provided current information, it often displayed biased results, favoring service providers seeking customers. On the other hand, ChatGPT offered impartial information, but it was frequently outdated with limited sources. Hristidis highlighted the advantage of Google’s comprehensiveness and up-to-dateness, while acknowledging ChatGPT’s reliability and accuracy.
However, both platforms scored low in readability, indicating potential challenges for individuals with lower education levels and limited health literacy skills. Hristidis expressed optimism that improvements in readability were achievable through existing AI methods, whereas enhancing reliability and accuracy would be more difficult and require significant advancements in AI research.
This study builds on previous research that has also pointed out limitations in ChatGPT’s performance. For example, a study in gastroenterology education showed that ChatGPT-3 and ChatGPT-4 failed to achieve passing scores in multiple-choice self-assessment tests. Consequently, the researchers recommended against the use of ChatGPT for medical education in gastroenterology at present.
While Google and ChatGPT hold promise as valuable resources for healthcare, it is crucial to validate their effectiveness. These research findings underscore the need for further validation and improvement to ensure their optimal utilization in the healthcare field.
How are Google and ChatGPT useful in healthcare?
Google and ChatGPT offer up-to-date and nonbiased information that can serve as valuable resources for various healthcare conditions, including Alzheimer’s and dementia.
What are the limitations of Google and ChatGPT in healthcare?
Google’s results often display bias, favoring service providers, while ChatGPT’s information can be outdated and sourced from a limited number of places. Additionally, both platforms score low in readability, potentially limiting their accessibility for individuals with lower education levels and health literacy skills.
What improvements are necessary for Google and ChatGPT in healthcare?
Enhancing readability can be achieved through existing AI methods, but improving reliability and accuracy requires significant advancements in AI research. Validation studies are crucial to ensure their effectiveness in healthcare settings.
Are there any concerns regarding ChatGPT’s performance?
A study in gastroenterology education found that ChatGPT-3 and ChatGPT-4 failed to achieve passing scores in multiple-choice self-assessment tests. As a result, the researchers advised against using ChatGPT for medical education in gastroenterology at this time.