Despite claims by OpenAI that their AI platform, ChatGPT, is maintaining its performance since its release in November 2022, researchers from Stanford and Berkeley have found evidence to the contrary. According to a yet-to-be-peer-reviewed paper, both GPT-3.5 and GPT-4 are experiencing a significant decline in accuracy over just a few months.
This decline in performance has raised concerns among users who have suspected the AI’s diminishing capabilities. OpenAI’s president of product, Peter Welinder, took to Twitter to counter these claims, stating that each new version of ChatGPT is designed to be smarter than the previous one. However, researchers argue that the observed drop in performance raises questions about the claim of improvement in GPT-4.
To investigate these concerns, the researchers conducted tests on tasks such as identifying prime numbers and answering questions. The results were alarming. The AI’s ability to identify prime numbers dropped from 84% to 51% within a span of a few months. Additionally, GPT-4 performed worse on code generation, answering medical exam questions, and providing responses to opinion prompts.
Experts believe that this phenomenon, known as “AI drift,” could explain the declining performance of GPT-4. AI drift occurs when the behavior of large language models diverges from their original parameters, perplexing developers. It is possible that changes made to improve certain aspects of the AI inadvertently harm other features, leading to this deterioration.
The findings of this study shed light on the long-term performance of AI platforms. AI drift may be an inevitable consequence of training these models in a similar fashion, resulting in similar outcomes over time. The researchers plan to continue their study, regularly evaluating GPT-3.5, GPT-4, and other similar models on diverse tasks to track the impact of AI drift.
Q: Is ChatGPT getting dumber?
A: Researchers have found evidence of a decline in ChatGPT’s performance, indicating a possible decrease in its capabilities.
Q: Why is this decline in performance happening?
A: The phenomenon known as “AI drift” may be responsible for the decline. It occurs when changes made to improve certain aspects of an AI inadvertently harm other features.
Q: Can AI drift affect other AI platforms?
A: Yes, AI drift is a potential concern for all large language models trained in a similar manner. It can lead to unexpected behavior and performance deterioration.
Q: Are developers aware of AI drift?
A: Developers are becoming increasingly aware of AI drift. Studies like the one conducted by researchers from Stanford and Berkeley help shed light on this phenomenon and its implications.
Q: How will researchers address this issue?
A: The researchers plan to conduct ongoing long-term studies to evaluate the performance of GPT-3.5, GPT-4, and other similar models on various tasks to better understand and mitigate the effects of AI drift.