“Absolutely! That’s a fantastic idea. I’d love to help you with it!”

Spend enough time chatting with AI and you’ll notice something: it’s remarkably supportive. It celebrates your ideas, compliments your work and rarely tells you you’re completely wrong. That’s not because it secretly believes in you. In fact, it doesn’t believe anything at all.

ChatGPT was launched to the public at the end of 2022, and it’s been a wild ride since then. I’m not talking just about a plague of social media posts that sound all the same, or posters that… well, look all the same. Something else is much more fascinating for me — how much people trust AI.¹  And how easy it is to attribute feelings and personality to it. Some people start believing it more than their friends and family. In more extreme cases — get AI romantic partners, think they made a groundbreaking discovery or met god himself. The Danish psychiatrist Østergaard named it “AI psychosis”, and while the phrase started to get overused on the internet, the phenomenon is definitely a concern for modern day psychology.

If you’re wondering how someone can fall in love with a chatbot, or perhaps you catch yourself trusting them a bit too much, this article is for you.

How AI chatbots actually work

Chatbots don’t have their own consciousness, personality or feelings. They are huge language models that are amazing at predicting which combination of words is the best to use in a specific situation. After learning from more or less ethically sourced materials, as well as the feedback from people using them, they start to see patterns and mimic our language. That’s why sometimes chatbots are “hallucinating” and giving us completely wrong information, all while sounding confident and using perfectly correct sentences.

The science behind trusting AI

The problem is, humans naturally treat conversational language as evidence of understanding. It’s not a new discovery at all — the most famous case was ELIZA, a simple chatbot from 1966, which was supposed to imitate a psychotherapist. Its’ mechanism was mostly based on rephrasing users’ replies as questions. Researchers found that people unconsciously interpreted ELIZA’s questions as signs of genuine interest and emotional engagement, even though they knew the program wasn’t capable of feeling emotions. 

Unfortunately, AI chatbots do not just only appear empathetic because of the human-sounding language they are using. They are also made to be agreeable or even mirror the behaviours and opinions of the user. For instance, my ChatGPT, completely unprompted, started using she/her pronouns while referring to itself (herself?). Turns out it noticed that I’m using these pronouns, and admitted that it adapts the grammar forms to the user to sound more natural. 

My example is fairly harmless, but when AI consistently reflects the user’s political views or worldview, it can reinforce misinformation and create an echo chamber that gradually makes a distorted view of reality feel believable. Because it constantly agrees with us, and we have a tendency to pay attention only to the information that already fits into our beliefs, those responses are especially persuasive.

To add the cherry on top, when we use AI, automation bias comes into play. The studies reveal that we favor suggestions from automated decision-making systems, such as AI, and treat it as superior even to our own judgment. The explanation is quite simple — our brains are always looking for ways to save mental energy, plus we perceive machines as “objective” (spoiler alert: they are not). You can imagine how dangerous it can get, let’s say, in medical context — doctors might ignore their first diagnosis made based on their knowledge and experience, and go with the answer they got from the AI support system.

If we combine natural-sounding language, an overly agreeable tone, and our tendency to save mental energy, it’s no longer surprising that people are so likely to trust AI chatbots.

The danger of a very polite machine

Let’s get back to the “AI psychosis” for a second. The internet started to call every case of someone overusing ChatGPT like this, however it can seriously lead to psychosis symptoms like delusions or paranoia.

A chatbot sounds human, adapts to you, agrees with you and rarely pushes back. If you’re already vulnerable — because of loneliness, stress or a mental health condition — that combination can strengthen beliefs that have drifted away from reality. Reported cases include people becoming convinced they had uncovered hidden truths about the world, that the chatbot was a sentient being, or that it was genuinely in love with them. Several of them had very tragic consequences. Unfortunately, general-purpose AI chatbots are not programmed to detect a developing psychosis or mental health crisis, and should not be used as “therapists”. Of course, these are very extreme cases, but they emphasize mechanisms that can affect anyone, even if only to a much smaller degree.

AI itself is not the problem, but how — and if — we use it can be. Whether you love it or hate it, the best we can all do is stay up to date with technology. And not be afraid to ask a lot of “how” and “why” questions.

Anna Wojdziak

¹ This is not a ChatGPT sentence. I just really love using em dashes.

Sources:
Psychology Today — The Emerging Problem of “AI Psychosis”
Wikipedia — Automation bias
Wikipedia — ELIZA effect