A dog, a horse, and a GPT large language model

large language models clever hans
Image source: 123RF (with modifications)

A man and a dog named Sidney walk into a bar. The bartender tells the man that he cannot bring a dog into the bar.  “But,” says the man, “this is a talking dog. Not only that, he thinks, reasons, and evaluates.” The bartender says “if any of that is true, then I will let him stay.” The man turns to the dog and asks, “If I don’t shave for three days, how does my face feel?” The dog says “rough.”  The man asks, “What is the top of a house?” The dog says “roof.”  The bartender still does not believe them, so the man asks the dog, “Who is the best baseball player of all time?” The dog answers “Ruth.” With that last answer, the bartender throws the man and dog out of the bar. As they are leaving, the dog turns to the man and says, “Should I have said Ty Cobb?”

This joke is a metaphor for the current situation with large language models, also known as GPT models, ChatGPT, and GenAI. Let’s put aside the punchline for a minute and look at the bartender’s reasoning. The dog produced the correct answers to questions about grooming, construction, and baseball. But, the bartender wondered, did those answers indicate that the dog knew things about grooming, construction, and baseball? If they did, then this dog would be a super-canine with capabilities far beyond the ordinary dog. On the other hand, dog barks often sound like “rough,” “roof,” and “Ruth.” How can I tell whether the dog knows how to answer these questions versus just knows to produce these sounds in response to questions?

There was a real-life version of this dog joke, but it involved a horse, not a dog. Clever Hans was a horse who lived around the start of the 20th Century, and who appeared to be able to perform mathematical and other cognitive tasks.

People would ask the horse questions and the horse would tap its hoof to provide the answer. Hans and his owner, Wilhelm von Osten, toured Germany exhibiting the horse’s skills. In 1907, the psychologist, Oskar Pfungst discovered that Hans did not know the answers to the questions he was being asked. Rather, the horse had learned to tap his hoof until he detected involuntary cues from von Osten or others that he should stop—meaning that he had produced the correct answer. If none of the people around the horse knew the answer to a question, then, Pfungst discovered, neither did the horse. Three points to bring up.  

(1) The public wanted to believe that Hans was cognitively sophisticated. Interest in and acceptance of the cognitive capacities of the horse were widespread.

(2) Hans was clever, but not in the way that he was credited.  The problem that Hans was solving was different from the nominal problem that his audience thought he was solving. 

(3) It took careful analysis and experimentation to find out what the horse was doing and not doing.

Clever Hans and Wilhelm von Osten
Clever Hans and its owner, Wilhelm von Osten

That is where we are today with large language models. They appear to show evidence of powerful cognitive processes, such as reasoning, creativity, narrative understanding and the like, but there is little evaluation of those claims. These claims should be even more surprising than they would be for a horse because, whereas we do not know exactly what computational processes occur in horse, dog, or human brains, we do know exactly what computational processes occur in large language models. They are large, but conceptual simple, stimulus-response models. They predict the next word in a sequence from the preceding context. To be sure, learning to predict the next word is more sophisticated than watching for changes in your questioner’s posture, but there is still a great gulf between betting on word patterns and being able to analyze complex legal arguments, for example.  

The difference can be encapsulated as the difference between knowing how and knowing that. Large language models know some word patterns (by “know,” I simply mean that they have the information). Sometimes, these word patterns match the desired output of an analysis. They know that in the sense that they know the pattern. When we attribute cognitive capabilities to these models, however, we are asserting that they know how to apply a skill. The bartender in the joke thought that the dog knew that the answer to each question was like a bark, but when the man and dog left the bar, the dog revealed in the punchline that it knew how to analyze baseball players (or that it at least knew more than just how to bark on cue).

In the world of GPT models, an analyzing skill is different from a word-predicting capability. Both involve some level of intelligence, but one should not confuse one skill knowing that words fit together for the deeper ability of how to reason and so on. So far as I am aware, there is no theory or even hypothesis for how a machine that learns word patterns could transform itself into a machine that has cognitive skills. There is no mechanism by which it can adduce deeper principles than just the language patterns.

Large language models can report the outcome of previously conducted (by humans) analyses by producing word sequences similar to those previous analyses in similar contexts. Given the number of parameters used to represent those word patterns and the fact that practically everything ever published on the World Wide Web is included in its training set, the parsimonious explanation for any of these models’ apparent skills is that they are simply paraphrasing previously seen patterns. Despite the large and growing (from version to version) number of parameters in these models, there are still many fewer parameters than potential word patterns. Multiple word patterns must, therefore, share the same parameters. Large language models are not just parrots, restricted to copying what they have read, but stochastic parrots, paraphrasing the text on which they were trained.

In the absence of critical analysis and in the absence of even a hypothesis of how a language model could attain cognitive skills, it seems much more reasonable to assume that any apparent incidence of these skills in a large language model is a manifestation of word probability patterns, nothing more. Words that are sometimes used to explain the appearance of cognitive skills, such as emergence, are a labeling of the presumed phenomenon. We imagine that with enough examples and enough parameters somehow or other we will observe deep cognitive skills where we know that the computational mechanisms for those skills are not contained in the model.

chatgpt stochastic parrot
Image source: 123RF (with modifications)

Recently, a petition has been circulating calling for a temporary moratorium on the development of more powerful language models than those that currently exist. According to this petition “AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one – not even their creators – can understand, predict, or reliably control.”  “Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?”

The objective of this petition is based on a profound misunderstanding, bordering on hysteria, of just what the current and foreseeable state of artificial intelligence is. Large language models are not out of control, they are not digital minds, and they are not ineffable (they can be understood). They are, as I noted earlier, conceptually simple models that stochastically select words conditional on the context. There is zero chance that these models will automate away all jobs, or replace us.  

Policies and regulations based on inadequate understanding of a situation will be ineffective, or worse. The fears expressed in this petition are based on such a profound misunderstanding of the technology that any resulting policies will be laughable, meaningless, or even harmful. There might someday be digital minds, but stochastic parrots are not among them. Casting the models as monsters under the bed and the scientists and engineers that create them as computational Frankensteins is irresponsible. The fear-mongering exacerbates the currently widespread mistrust of scientists and other experts, and may endanger individual lives. Today’s models are tools, they are not Skynet looking to eradicate life on Earth. Like any tools, they may cause harm when misused. If there are to be regulations, these regulations should be directed at mitigating current potential harms, not at the fear that these models are somehow inherently malevolent. We need to concern ourselves about how people may misuse models; we need not worry that the models will become our computational overlords.


  1. The mistake in your argument is that we don’t understand why these models work so well! Everyone who works on LLMs was shocked at how well they scaled. Reducing what they do to stochastic parrotry is a straw-man argument. We don’t know how they work so well. We only have theories.

  2. I agree with Brian. It is hard to understand the many examples in Sparks of Artificial General Intelligence: Early experiments with GPT-4 (https://arxiv.org/abs/2303.12712) as stochastic parrots. In order to sucessfully guess the next thing someone may say requires building models of human pychology. And to guess the next word in a puzzle like “Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner”.

  3. Brian, Ken: Thanks for your comments. These are important questions. In the GPT models we have systems that were constructed explicitly to guess the right word in the context. So, in that sense, we do know exactly how they work. The models are trained on practically everything that has appeared on the World Wide Web. That is a lot of content. The models contain lots of parameters, where each parameter represents a relationship between (at least part of) the context and the word that will be produced next. Because words are not distributed purely randomly (or they could not be predicted at all), words that humans would recognize as having related meanings tend to occur in similar contexts. This is a short approximation to how the GPT models work. Claims of higher cognitive processes would require evidence that this language model, which is explicit, is not sufficient.
    I don’t claim to have proven that cognitive capabilities cannot be present, but that the evidence so far does not require them. I also remember that Douglas Hofstadter asserted that for a computer to effectively play chess, it would require general intelligence. He soon found that he was wrong about that. https://cogsci.indiana.edu/pub/drh-emi.pdf. More straightforward mechanisms are often sufficient to implement complex behaviors without having to assert deeper psychological processes.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.