According to an article in Scientific American, written by George Musser, no one yet knows how ChatGPT and its artificial intelligence cousins will transform the world, and one reason is that no one really knows what goes on inside them. “Some of these systems’ abilities go far beyond what they were trained to do – and even their inventors are baffled as to why. A growing number of tests suggest these AI systems develop internal models of the real world, much as our own brain does, though the machines’ technique is different.”

“Everything we want to do with them in order to make them better or safer or anything like that seems to me like a ridiculous thing to ask ourselves to do if we don’t understand how they work,” says Ellie Pavlick of Brown University, one of the researchers working to fill that explanatory void.

At one level, says Musser, Pavlick and her colleagues understand GPT (short for generative pretrained transformer) and other large language models, or LLMs, perfectly well. The models rely on a machine-learning system called a neural network. Such networks have a structure modeled loosely after the connected neurons of the human brain. The code for these programs is relatively simple and fills just a few screens. It sets up an autocorrection algorithm, which chooses the most likely word to complete a passage based on laborious statistical analysis of hundreds of gigabytes of Internet text. Additional training ensures the system will present its results in the form of dialogue. “In this sense, all it does is regurgitate what it learned – it is a “stochastic parrot,” in the words of Emily Bender, a linguist at the University of Washington.

But LLMs have also managed to ace the bar exam, explain the Higgs boson in iambic pentameter and make an attempt to break up their users’ marriage. “Few had expected a fairly straightforward autocorrection algorithm to acquire such broad abilities” Musser writes.

That GPT and other AI systems perform tasks they were not trained to do, giving them “emergent abilities,” has surprised even researchers who have been generally skeptical about the hype over LLMs. “I don’t know how they’re doing it or if they could do it more generally the way humans do – but they’ve challenged my views,” says Melanie Mitchell, an AI researcher at the Santa Fe Institute.

“It is certainly much more than a stochastic parrot, and it certainly builds some representation of the world – although I do not think that it is quite like how humans build an internal world model,” adds Yoshua Bengio, an AI researcher at the University of Montreal.

At a conference at New York University in March, philosopher Raphaël Millière of Columbia University offered yet another jaw-dropping example of what LLMs can do. “The models had already demonstrated the ability to write computer code, which is impressive but not too surprising because there is so much code out there on the Internet to mimic,” Musser says. He adds that Millière went a step further and showed that GPT can, however, execute code, too.

The philosopher typed in a program to calculate the 83rd number in the Fibonacci sequence. “It’s multistep reasoning of a very high degree,” he says. And the bot nailed it. When Millière asked directly for the 83rd Fibonacci number, however, GPT got it wrong: this suggests the system wasn’t just parroting the Internet. Rather it was performing its own calculations to reach the correct answer.

Although an LLM runs on a computer, it is not itself a computer. It lacks essential computational elements, such as working memory. In a tacit acknowledgement that GPT on its own should not be able to run code, its inventor, the tech company OpenAI, has since introduced a specialized plug-in a tool ChatGPT can use when answering a query – that allows it to do so. But that plug-in was not used in Millière’s demonstration. Instead, he hypothesizes that the machine improvised a memory by harnessing its mechanisms for interpreting words according to their context – a situation similar to how nature repurposes existing capacities for new functions.

According to Musser, “this impromptu ability demonstrates that LLMs develop an internal complexity that goes well beyond a shallow statistical analysis. Researchers are finding that these systems seem to achieve genuine understanding of what they have learned.”

For considerably more, read How AI Knows Things No One Told It - Scientific American.