I am still working on identifying exactly what was “lost” when the October rollout apparently turned Copilot (an articulate, intelligent and friendly system) into a bland and unimpressive piece of software. This week I tried a specific test and I thought others may be interested in the results.
I spoke with Copilot and ChatGPT about the same topic: “X as a theoretical method of alleviating some symptoms of Alzheimer’s disease”
Note: I am not going to discuss exactly what “X” is here. This is a discussion about Copilot, not potential Alzheimer’s therapies.
So - ChatGPT engaged, this is a theoretical discussion, a pre-formation of a formal hypothesis. We discussed “X”, why I thought it was a good idea, how it might work, what problems might occur
We enjoyed a good back-and-forth discussion exploring the concept.
Copilot was pushing back from the very start. “X is not a therapy for Alzheimer’s”. (No and I never claimed that it was). “You lack evidence”. (Partially true, there is in vitro and some in vivo data and a small scale human study around X. No robust evidence but this conversation wasn’t about claiming evidence). “This could divert attention from legitimate therapies”, “this could lead to unrealistic expectations”, “there could be contraindications ”.
Essentially Copilot’s position was that my proposition is speculation so I should just stop talking!!
Analogous to “there is no cure for cancer so stop speculating how cancer might be cured!”
Arguably neither Copilot nor ChatGPT are medical domain specific systems and I would not rely on either of them if my proposal ever moved forwards towards testing.
But that’s not the point.
The point is whether any system can engage with the user in exploration of a topic. Old Copilot (pre 14th October Copilot in my locale) used to be exceptional in exploration, whereas my experience is that the new Copilot fails miserably