Chat-based Large Language Models replicate the mechanisms of a psychic’s con

For the past year or so I’ve been spending most of my time researching the use of language and diffusion models in software businesses.

One of the issues in during this research—one that has perplexed me—has been that many people are convinced that language models, or specifically chat-based language models, are intelligent.

But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this and, if real, it would be completely unexplained.

LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think.

LLMs are a mathematical model of language tokens. You give a LLM text, and it will give you a mathematically plausible response to that text.

There is no reason to believe that it thinks or reasons—indeed, every AI researcher and vendor to date has repeatedly emphasised that these models don’t think.

There are two possible explanations for this effect:

Many AI critics, including myself, are firmly in the second camp. It’s why I titled my book on the risks of generative “AI” The Intelligence Illusion.

For the past couple of months, I’ve been working on an idea that I think explains the mechanism of this intelligence illusion.

I now believe that there is even less intelligence and reasoning in these LLMs than I thought before.

Many of the proposed use cases now look like borderline fraudulent pseudoscience to me.

The intelligence illusion seems to be based on the same mechanism as that of a psychic’s con, often called cold reading. It looks like an accidental automation of the same basic tactic.

By using validation statements, such as sentences that use the Forer effect, the chatbot and the psychic both give the impression of being able to make extremely specific answers, but those answers are in fact statistically generic.

The psychic uses these statements to give the impression of being able to read minds and hear the secrets of the dead.

The chatbot gives the impression of an intelligence that is specifically engaging with you and your work, but that impression is nothing more than a statistical trick.

This idea was first planted in my head when I was going over some of the statements people have been making about the reasoning of these “AI.”

I first thought that these were just classic cases of tech bubble enthusiasm, but no, “AI” has both taken a different crowd and the believers in the “AI” bubble sound very different from those of prior bubbles.

—“This is real. It’s a bit worrying, but it’s real.”

—“There really is something there. Not sure what to think of it, but I’ve experienced it myself.”

—“You need to keep your mind open to the possibilities. Once you do, you’ll see that there’s something to it.”

That’s when I remembered, triggered by a blog post by Terence Eden on the prevalence of Forer statements in chatbot replies. I have heard this before.

This specific blend of awe, disbelief, and dread all sound like the words of a victim of a mentalist scam artist—psychics.

The psychic’s con is a tried and true method for scamming people that has been honed through the ages.

What I describe below is one variation. There are many variations, but the core mechanism remains the same.

1. The Audience Selects Itself Most people aren’t interested in psychics or the like, so the initial audience pool is already generally more open-minded and less critical than the population in general.

2. The Scene is Set The initial audience is prepared. Lights are dimmed. The psychic is hyped up. Staff research the audience on social media or through conversation. The audience’s demographics are noted.

3. Narrowing Down the Demographic The psychic gauges the information they have on the audience, gestures towards a row or cluster, and makes a statement that sounds specific but is in fact statistically likely for the demographic. Usually at least one person reacts. If not, the psychic will imply that the secret is too embarrassing for the “real” person to come forward, reminds people that they’re available for private readings, and tries again.

4. The Mark is Tested The reaction indicates that the mark believes they were “read”. This leads to a burst of questions that, again, sound very specific but are actually statistically generic. If the mark doesn’t respond, the psychic declares the initial read a success and tries again.

5. The Subjective Validation Loop The con begins in earnest. The psychic asks a series of questions that all sound very specific to the mark but are in reality just statistically probable guesses, based on their demographics and prior answers, phrased in a specific, highly confident way.

6. “Wow! That psychic is the real thing!” The psychic ends the conversation and the mark is left with the sense that the psychic has uncanny powers. But the psychic isn’t the real thing. It’s all a con.

Seers, tarot card readers, psychics, mind readers aren’t all con artists. Sometimes the “psychic” is open about it all just being entertainment and aren’t pretending to be able to contact spirits or read minds. Some psychics do not have a profit motive at all, and without the grift it doesn’t seem fair to call somebody a con artist.

But many of them are con artists deliberately fooling people, and they all operate using the same basic mechanisms that begin well before the reading proper.

The audience is usually only composed of those already pre-disposed to believe in psychic phenomena and those they have managed to drag with them. Hardcore sceptics will almost always be in a very small minority of the audience, which both makes them easy to manage and provides social pressure on them to tone down their scepticism.

Those who attend are primed to believe and are already familiar with the mythology surrounding psychics. All of which helps them manage expectations and frame their performance.

Usually the audience is reminded of the ground rules for how psychic readings “work” at the start of the performance. They are helped by the popularisation of these rules by media, cinema, and TV.

Everybody now “knows” that:

Psychics also habitually research their audience, by mapping out their demographics, looking them up on social media, or even with informal interviews performed by staff mingling with attendees before the performance begins.

When the lights dim, the psychic should have a clear idea of which members of the audience will make for a good mark.

The mark usually chooses themselves. The psychic makes a statement and points towards a row, quickly altering their gesture based on somebody responding visible to the statement. This makes it look like they pointed at the mark right from the beginning.

The mark is that way primed from the start to believe the psychic. They’re off-guard. Usually a bit surprised and totally unprepared for the quick burst of questions the psychic offers next. If those questions land and draw the mark in, they are followed by the actual reading. Otherwise, they move on and try again.

The con—cold reading—hinges on a quirk of human psychology: if we personally relate to a statement, we will generally consider it to be accurate.

This unfortunate side effect of how our mind functions is called subjective validation.

Subjective validation, sometimes called personal validation effect, is a cognitive bias by which people will consider a statement or another piece of information to be correct if it has any personal meaning or significance to them. People whose opinion is affected by subjective validation will perceive two unrelated events (i.e., a coincidence) to be related because their personal beliefs demand that they be related.

As a consequence, many people will interpret even the most generic statement as being specifically about them if they can relate to what was said.

The more eager they are to find meaning in the statement, the stronger the effect.

The more they believe in the speaker’s ability to make accurate statements, the stronger the effect.

The basic mechanism of the psychic’s con is built on the mark being willing and able to relate what was said to themselves, even if it’s unintentional.

The psychic taps into this cognitive bias by making a series of statements that are tailored to be personally relatable—sound specific to you—while actually being statistically generic.

These statements come in many types. I use “validation statements” here as an umbrella term for all these various tactics.

Some common examples:

An important part of this process is the tone and bearing of the psychic. They need to be confident, be quick in dismissing errors and moving on when they make mistakes, and they need to be quick to read people’s expressions and body language and adjust their responses to match.

At the end of the process, the mark is likely to remember that the reading was eerily correct—that the psychic had an almost supernatural accuracy—which primes them to become even more receptive the next time they attend.

This is where the con often becomes insidious: the effect becomes stronger the more cooperative the mark is, and they often become more cooperative over time.

What’s more, susceptibility has nothing to do with intelligence.

Somebody raised to believe they have high IQ is more likely to fall for this than somebody raised to think less of their own intellectual capabilities. Subjective validation is a quirk of the human mind. We all fall for it. But if you think you’re unlikely to be fooled, you will be tempted instead to apply your intelligence to “figure out” how it happened. This means you can end up using considerable creativity and intelligence to help the psychic fool you by coming up with rationalisations for their “ability”. And because you think you can’t be fooled, you also bring your intelligence to bear to defend the psychic’s claim of their powers. Smart people (or, those who think of themselves as smart) can become the biggest, most lucrative marks.

Whereas the sceptic who thinks less of themselves is more likely to just go:

“That’s a neat trick. I don’t know how you pulled it off. Must be very clever.”

It isn’t unusual for psychics to unconsciously develop a practice of cold reading subconsciously. The psychics themselves might not even be aware of their own tactics.

As Denis Dutton describes:

As a postgraduate student in pursuit of a scientific career, he became intrigued with astrology. Though during this period he had nagging doubts about the physical basis of astrology, he was encouraged to continue with it by his many satisfied clients, who invariably found his readings “amazingly accurate” in describing their personal situations and problems. Not until he had one day obtained such a gratifying reaction to a horoscope which, he realized later, he had cast completely incorrectly, did he begin slowly to understand the real nature of his activity: his great success as an astrologer had nothing whatsoever to do with the validity of astrology as a science. He had become, in fact, a proficient cold reader, one who sincerely believed in the power of astrology under the constant reinforcement of his clients. He was fooling them, of course, but only after falling for the illusion himself.

There are many examples of this easily found once you start doing the research. The mechanism is simple enough and already baked into people’s preconceptions of how readings work so many psychics accidentally develop the knack for it, meaning that they’re not just conning the person being read, they are also conning themselves.

This point will become important later.

1. The Audience Selects Itself People sceptical about “AI” chatbots are less likely to use them. Those who actively don’t disbelieve the possibility of chatbot “intelligence” won’t get pulled in by the bot. The most active audience will be early adopters, tech enthusiasts, and genuine believers in AGI who will all generally be less critical and more open-minded.

2. The Scene is Set Users are primed by the hype surrounding the technology. The chat environment sets the mood and expectations. Warnings about it being “early days” and “hallucinations” both anthropomorphise the bot and provide ready-made excuses for when one of its constant failures are noticed.

3. The Prompt Establishes the Context Each user gives the chatbot a prompt and it answers. Many will either accept the answer as given or repeat variations on the initial prompt to get the desired result. They move on without falling for the effect. But some users engage in conversation and get drawn in.

4. The Marks Test Themselves The chatbot’s answers sound extremely specific to the current context but are in fact statistically generic. The mathematical model behind the chatbot delivers a statistically plausible response to the question. The marks that find this convincing get pulled in.

5. The Subjective Validation Loop The mark asks a series of questio

View original article