Buddhist Catholic Contemplative Evangelical General Christian Hindu Jewish Latter-day Saint Muslim New Visions Progressive Christian

Beliefs

Topics

Columnists

Resource Library

Research Tools

Video

Podcasts

Thought Readers

Faith In Media Tracker

Newsletters

Ad-Free & Subscriber Log-In Subscription Support

Follow Patheos

Home
Pagan
Technology and Religion
Chess Grifters, Chinese Rooms, and AI

Chess Grifters, Chinese Rooms, and AI

Last updated on: October 3, 2026 at 11:45 am by Tom Swiss

0 Comments

Chess Grifters, Chinese Rooms, and AI 2026-10-03T11:45:29-04:00 Tom Swiss

LOGO Patheos Explore the world's faith through different perspectives on religion and spirituality! Patheos has the views of the prevalent religions and spiritualities of the world.

Harry Anderson is best known for playing Judge Harry Stone on TV’s Night Court in the 1980s and 90s. But before that, he was a street magician with a fascination for the intersection of the con artist and illusionist — a fascination born from his own experiences running the “shell game” on the streets of LA in the 1960s, and then turning that into a street magic act after getting beaten up by a disgruntled mark.

Raised by a single mother who worked as a casino dealer in Las Vegas and then as a sex worker in Chicago, the young Anderson was sometimes looked after by the men who played cards in the lobby of Chicago’s Ambassador East Hotel. They taught him the ways of the gambler and grifter, and inspired his “Harry the Hat” persona, a fast-talking con man who was “a bit of a nincompoop” even as he bewildered his marks and his audience.

As writer David Hill put it, “Most magicians told the audience they were doing something magical, and the audience didn’t want to believe it. Harry the Hat told the audience he was lying to them, but the audience wanted to believe it was real.”[Hill]

In his fun little book Games You Can’t Lose, Anderson, writing in his “Harry the Hat” persona, describes a fictional(?) con in which The Hat, who knows nothing about chess, bets that he can play the mark’s entire chess club and win at least as many games as he loses.[Anderson] He does this by playing half the boards as black and half as white, acting as a relay between one member of the club playing white and one playing black — essentially taking on the role of the mailman in several games of postal chess.

Now, based solely on his behavior at the boards without a broader context, an observer might judge that Harry the Hat “understands” chess at about the median level of the chess club. But he does not. He is merely acting as a conduit for the understanding possessed by the club members.

I bring this up because AI.

Because LLMs are often able to respond sensibly to questions put to them, users are prone to believe that the machine “understands” the topic at hand; rather than seeing that the machine is acting like The Hat in the chess con, relaying the understanding of others.

We live in a world where AI psychosis is already killing people — the website LLMDeathCount.com currently lists 198 deaths linked to AI — and the software-driven extinction of humanity is discussed as a real possibility. It is therefore essential that we not conflate machines, which we can and should turn off when they are a threat, with people, whom we cannot.

Believing that a machine “understands” some topic in the same sense a human does is a step towards such conflation. It is therefore important that we dispel this misunderstanding about understanding.

Chinese Rooms and Relays

John Searle’s famous “Chinese Room” thought experiment*[Searle]* asks us to imagine a man with no knowledge of the Chinese language, locked in a room with a large batch of writing in Chinese, and a set of rules — based entirely on the shape of the characters, with no references to their semantics — for how to give back certain sequences of symbols in response to sequences of symbols presented to him.

The rules are in English, and … [t]hey enable me to correlate one set of formal symbols with another set of formal symbols….Unknown to me, the people who are giving me all of these symbols…call the symbols I give them back in response to the third batch “answers to the questions,” and the set of rules in English that they gave me, they call the “program.”…Suppose also that after a while I get so good at following the instructions for manipulating the Chinese symbols and the programmers get so good at writing the programs that from the external point of view—that is, from the point of view of somebody outside the room in which I am locked—my answers to the questions are absolutely indistinguishable from those of native Chinese speakers. Nobody just looking at my answers can tell that I don’t speak a word of Chinese.

John Searle, Matthew Breindel uploader Matro at en.wikipedia, CC BY-SA 3.0, via Wikimedia Commons

But in this scenario, although from outside there seems to be understanding, the man shuffling the symbols does not understand them any more than Anderson’s grifter understands chess. If there is any understanding at all here, the man is merely passing along the understanding that the programmers have encoded into the rules — just as Anderson’s grifter acts as a conduit for the understanding of the chess club members.

Searle argues that there is not even understanding encoded in the program, that such formal systems can never “understand”. We are getting into matter of semantics here, but this brushes up against an issue in the the history of AI.

The dominant approach in AI from the field’s start in the 1950 through the late 1980s rested on symbolic approaches which suggested that well-planned, complex sets of formal rules (like the “programs” in the Chinese room example) could lead to “intelligent” systems. This stands opposed (or at least complementary) to statistical, “connectionist” approaches, based on interconnected networks of simple units — such as modern machine learning systems.

I don’t want to get bogged down in the interesting questions Searle addresses of systems, embodiedness, symbol manipulation versus synapse simulation, and what exactly “strong AI” means, or the responses of his critics. It’s good, heady stuff, and I recommend reading the article and the opposing “Reflections” that accompany it in Douglas Hofstadter and Daniel Dennett’s book The Minds I.[Hofstadter and Dennett]

I just want to point out that LLMs are still language models, resting on the manipulation of linguistic symbols rather than anything resembling the biological processing of sense experience and interaction with the world. And I want to use Searle’s thought experiment to build a bridge between Anderson’s chess grifter and LLMs.

In the case of LLMs, rather than writing formal programs like those of the Chinese Room, programmers have used enormous computational power to distill the understanding of the world represented in human-authored texts, into a “connectionist” statistical model.

But this is just a difference in how the “program” is created. Just like the man in the Chinese Room, the software that references these models and uses them to produce responses to input, does not understand a darn thing.

If you ask an LLM “what does dawn in the mountains smell like?”, you may get an answer including phrases like “a crisp clean blend of earth, vegetation, and atmosopheric chemistry” and “profoundly clean, bracingly cold, and green”; at least, that’s what Google’s Gemini just kicked back to me.

But the machine has no understanding, no experience, of “dawn” or “mountain” or “crisp” or “cold” or “profound”. It is just slicing and dicing and statistically mixing the words of human beings that do understand these things through direct experience. The only understanding here is the understanding that the token “mountain” appears near the tokens “cold” and “green” in the training data more often than it does near the tokens “calculus” or “videotape”.

A Good Trick, A Bad Metaphysics

Now, this statistical legerdemain is a good trick! It enables LLMs to provide a pretty good precis of a body of text, to take an existing bit of text and transform it to be more like another bit of text (“take my notes and make them a business letter”), and to give software a natural language interface.

Used sensibly and skeptically, this is useful. But at it heart it’s the same trick as Anderson’s chess grifter: represent someone else’s understanding as your own.

And mistaking it for intelligence tends to re-enforce the horrible Platonic metaphysics that are at the heart of so many of our problems: the idea that the real, sensory and experiential world, is less important than some abstract world of forms and words.

The AI techbro vision would have us ruin the actual physical world, cut down forests to build out the power grid, accelerate climate change or poison the environment with nuclear waste, in order to build and power hyperscale data centers, all so that the human experience can sink into abstractions presented by machines owned and operated by multinational gigacorporations, which are in turn controlled by owning class psychopaths.

This is not a vision of “machines of loving grace”. This is a vision of the Torment Nexus. AI doesn’t have to be this way; and the correction starts with seeing through the machine’s imitation of a person.

Imitation Games

Anyone even casually interested in AI has heard of the “Turing Test”. It was introduced by mathematician Alan Turning in a 1950 paper, “Computing Machinery and Intelligence.”[Turing] Turing proposes to consider the question, “Can machines think?”, and notes that this should begin by defining the terms “machine” and “think.” But he immediately deflects from this, complaining that relying on the “normal use of the words” for such a question is “dangerous”, making the answer rely on public opinion about what words mean. He therefore must, he argues, replace the question with a more objective test:

The new form of the problem can be described in terms of a game which we call the ‘imitation game.” It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either “X is A and Y is B” or “X is B and Y is A.” The interrogator is allowed to put questions to A and B…

…

…It is A’s object in the game to try and cause C to make the wrong identification….

In order that tones of voice may not help the interrogator the answers should be written… The ideal arrangement is to have a teleprinter communicating
between the two rooms….The object of the game for the third player (B) is to help the interrogator.

Alan Turing

He then proposes replacing the man in this scenario — the man whose goal is to fraudulently convince the interrogator that he is in fact the woman — with a computer. This is the famed “Turing Test.” (Though it’s usually ignored that Turing specifically has the computer trying to imitate a woman rather than a man. Decide for yourself whether to read gender or queer politics into that or not. It’s also usually ignored that he’s considering it a success to teach a machine to lie, or that he takes the existence of ESP seriously, and speculates about the effect the “psychokinetic powers of the interrogator” might have on a computer equipped with a random number generator — points which might give us some pause.)

But let’s look at this original “imitation game” a moment.

For simplicity, we are going to decline to open the can of worms of gender transition, or the distinction between sex and gender. We’re going to put ourselves in the world Turing inhabited when he wrote this, where men were men and women were women — and non-conformity to sex roles could be a crime — without endorsing that world.

In that context, if I (manly man that I am) were a contestant in this imitation game, how might I successfully pretend that I were a woman?

One way to foil the interrogator would be to have a woman in the room with me, pass the questions on to her, and use her answers as a basis for my own. Or perhaps a committee of women could advise me. In this case I would be using exactly the strategy of Anderson’s chess grifter: acting as a go-between for the interrogator and my lady co-conspirator(s).

If I could not convince the women in my life to help me, I could access female insight secondhand. I could pile up books by female authors, and base my responses on relevant passages from their work. This would be more subtle than asking a woman to respond directly to the questions, and would require some interpolation on my part, but it is fundamentally the same strategy: acting as a conduit for the understanding of others.

Even if I memorized much of the works of, say Emily Dickenson and Edna St. Vincent Milay and Ursula Le Guin, rather than having their books open in front of me, and used my knowledge of their work to compose my replies, it does not mean that I “understand” what it is to be a woman, to live as a woman in society, to have a female body. Merely that I “understand” the reports of others of what it is to be a woman.

What’s the Difference?

Does that matter? Is it enough to be able connect the querent to relevant information in the reports of others? It depends on the context.

If the question is, can my book learning enable me to help a querent locate relevant information about some aspect of living as a woman, the answer is likely yes.

If the question is whether such book learning means I deserve status in society as a woman, likely no.

I am not saying that it is impossible for a person born male to live as a woman and accumulate enough experience to warrant being granted the social status of a woman. I am speaking here of men like me, who live entirely as men other than participation in this odd little game of Turing’s, whose experience of womanhood is restricted to book learning.

Just so, the ability of an LLM to process texts written by humans in a way that distills the human understanding of the authors is a good and useful trick in helping us locate relevant information in human reports about some aspect of human living.

But this machine ability to summarize human authors’ understanding of the world, does not mean that these machines deserve status in society as persons.

And likewise, I am not saying that it is impossible for a machine to relate to the world in a more animal-like manner, and accumulate enough experience to warrant being granted the social status of a “person”.

But this would be an entirely different sort of machine from overgrown auto-complete, these un-embodied large language models.

So don’t be a sucker. As Harry the Hat might tell you, never draw to an inside straight, never play poker with a guy who does card tricks, and never mistake an LLM for a person.

References:

Anderson, Harry, and Turk Pipkin. Harry Anderson’s Games You Can’t Lose: A Guide for Suckers. Pocket Books, 1989. Available on-line at https://archive.org/details/isbn_9780671745523/

Hill, David. “Remembering Harry the Hat: A Magician Hiding in Plain Sight.” The Ringer. 17 Apr 2018. https://www.theringer.com/2018/04/17/tv/harry-anderson-obituary-night-court-magic

Hofstadter, Douglas R., and Daniel C. Dennett, eds. The Mind’s I: Fantasies and Reflections on Self and Soul. Bantam Books, 1982.

Searle, John R. “Minds, Brains, and Programs.” The Behavioral and Brain Sciences, vol. 3. Cambridge University Press, 1980. Available on-line at https://web.archive.org/web/20071210043312/http://members.aol.com/NeoNoetics/MindsBrainsPrograms.html as of October 2026; also collected in The Mind’s I.

Turing, A. M. “Computing Machinery and Intelligence.” Mind 49: 433-460. Available on-line at https://courses.cs.umbc.edu/471/papers/turing.pdf as of October 2026; also collected in The Mind’s I.

New Visions

Technology and Religion

Discordianism

Ethics, Virtue, and Right Living

Previous Post

September 14, 2026

Jason Arday, Con Man