ChatGPT is an engine of cultural transmission

LLMs and Gopnikism

There’s a very creepy moment in Philip K. Dick’s classic science fiction, novel Time Out of Joint. Dick’s book is set in an idealized 1950s small town, which turns out to be an illusion, built to lull one man into predicting missile strikes in an unending war. As this man’s illusory world begins to break down, a soft drink stand suddenly disappears. Left in its stead is a sheet with the words ‘soft drink stand’ printed on it in block letters. Like many startling moments in Philip K. Dick novels, this doesn’t quite make narrative sense - but it doesn’t have to. It gives you a brief glimpse into the flimsy foundations of a universe where things are not quite as they seem.

Dick’s jarring image is a backdoor into Alison Gopnik’s account of Large Language Models (LLMs) as “cultural technologies.” If we want to understand how LLMs might have important implications and uses that are quite different from the standard stories and controversies, this account is a great place to start.

As discussed in the first post in this series, Gopnik explains the limits of LLMs - why they are wildly unlikely to become intelligent in the ways that people like Sam Altman talk about. But she and her colleagues do much more than this. They provide a foundation for thinking about LLMs’ cultural consequences. LLMs may not be culturally creative in their own right. But they can shape - and demonstrably are shaping - the conditions for human creativity and innovation.

Gopnik’s main point, made in this Wall Street Journal article, and at greater length in an academic article co-authored with Eunice Yiu and Eliza Kosoy, is that LLMs operate in a space of information that is disconnected from base reality. An LLM, doesn’t ‘know’ (to the extent that it ‘knows’ anything) that the phrase ‘soft drink stand’ refers to something that exists in the physical universe. For it, soft, drink, and stand is a series of “tokens,” individual strings of letters that don’t refer to, or have relationships with anything except other tokenized strings of letters. What the LLM ‘knows’ is the statistical weights associated with each of these tokens, which summarize its relationship with other tokens, rather than the world we live in.

“Gopnikism,” as Cosma has dubbed this understanding of LLMs, implies that in Yiu, Kosoy and Gopnik’s description, these models are incapable of distinguishing between “veridical and nonveridical representations in the first place.” Or as Cosma puts it more bluntly, “an LLM isn't doing anything differently when it ‘hallucinates’ as opposed to when it gets things right.” Our capacity as humans to get things right or wrong depends on our relationship to base reality, and our ability to try to solve the “inverse problem” of mapping how this reality works. LLMs don’t have that opportunity to explore and try to figure out what causes what.

That is why there are sharp limits on the ability of LLMs to map and understand the world. LLMs are perfect Derridaeians - “il n'y pas de hors texte” is the most profound rule conditioning their existence. If a piece of information isn’t available, at least by implication, somewhere in the corpus of text and content that they have been trained upon, then they are incapable of discovering it. And some things that are very obvious to humans are very hard for them to discover.

You can see some of the consequences if you provide LLMs and humans with descriptions of real world physical problems, and ask them to describe how these problems might be solved without the usual tools. For example, Gopnik and her co-authors have investigated what happens when you ask LLMs and kids to draw a circle without a compass. You could ask both whether they would be better off using a ruler or a teapot to solve this problem LLMs tend to suggest rulers - in their maps of statistical associations between tokens, ‘rulers’ are a lot closer to ‘compasses’ than ‘teapots.’ Kids instead opt for the teapot - living in the physical universe, they know that teapots are round.

This nicely deflates a lot of the more exuberant public OMG-AGI rhetoric. Of course, LLMs can improve their ability to answer this kind of question. Sometimes this is because non-obvious causal relationships are turned into text it can assimilate as, e.g. people start to write on social media about the ruler-teapot example. Sometimes it is because the text contains more latent information about such problems than you might think, and LLMs are getting better at uncovering that information. But - and this is Gopnik’s point - they can’t ever discover these relationships in the ways that human beings can - by trying things out in the real world, and seeing what works. Again - all they can see are the tokens for ‘soft’ and ‘drink’ and ‘stand’ - not the soft drink stand itself (other forms of ML - especially combined with robotics - are not subject to the same fundamental limitations).

But this has much broader and more interesting implications than making Sam Altman sad. Human beings learn in two kinds of ways - by imitating and by innovating. Gopnikism argues that LLMs are incapable of innovating, but they are good at imitating, and for some purposes at least, they are much better at it than human beings.

That is why we can think of LLMs as a cultural technology. A lot of human culture involves imitation rather than innovation, as cognitive scientists and anthropologists like Robert Boyd and Peter Richerson have emphasized. Brian Christian’s recent book, The Alignment Problem, has a short accessible discussion of how human beings are more likely than other primates to over-imitate - to do things that they see other humans doing, but that aren’t actually necessary to achieving a particular goal.

But imitation usually involves some kind of cultural transmission. Culture - under this particular account - is collective human knowledge, which is preserved, communicated and organized through a variety of means. It is passed on most straightforwardly when humans can directly observe each other, but over the millennia, we have also come up with more complex technologies of transmission. Languages, stories, libraries and such all allow information to be transmitted and organized.

Now, we have a new technology for cultural transmission - LLMs. The vast corpus of text and data that they ingest is a series of imperfect snapshots of human culture. Gopnikism emphasizes that we ought pay attention to how LLMs are likely to transmit, recombine and re-organize this cultural information, and what consequences this will have for human society.

At the simplest level, this might push us to ask how “lossy” LLMs are. Boyd and Richerson describe how cultures may lose information because they only have imperfect means for transmitting it. In the past, small cultures have sometimes lost access to complex and difficult-to-make tools, because they have gradually or suddenly lost the information on how to make them. Errors crept in, or the wrong person died at the wrong time. Thinking about such errors suggests that we should think about “hallucinations” as one kind of transcription error that may arise as LLMs transmit human culture from some humans to others. The LLM isn’t itself making a mistake itself - any more than an old analogue phone line is making a mistake when static creeps in - but it is introducing cultural noise in ways that may have consequences. LLMs might also involve other kinds of information problems - e.g. through their capacity to generate bullshit at scale. But to really develop how this works, we would need to understand not just how culture is transmitted, but how it is received (there are fairly sharp controversies around this topic, to be explored in later posts).

LLMs might have more complex cultural consequences than that. While Gopnikism suggests that LLMs are incapable of true innovation, it happily acknowledges that they may be the occasion of innovation in others. Printing presses don’t think - but printing has radically reshaped human culture, permitting flows of knowledge, collaboration and debating that would otherwise have been unthinkable. Gopnik and her co-authors speculate that LLMs might similarly make the transmission of information more “efficient,” greatly enhancing discovery. Equally, jaded academics might look to the undergraduate papers they are grading, and worry that our intellectual culture risks degenerating into a monochromatic sludge of cheery mediocrity, as LLMs inculcate a kind of lowest common denominator of what has already been said, carefully filtered through reinforcement learning to remove the merest hint of diverse or controversial opinion. That would not be a world of high innovation.

Presumably, neither outcome is inevitable. Figuring out the relationship between AI and innovation is one of the major research agendas of James Evans. He suggests that we should not think of AI as an “imitation game,” using it not simply to organize and sift through scientific knowledge, but to discover the holes in n-dimensional science space, showing us the places where there really ought be connections between researchers, and there are not. Under this definition, we might view AIs as “friendly aliens that facilitate our ability to collaborate with each other” (this phrase anthromorphizes - xenomorphizes? - the AIs, but the underlying point is clear).

It might also be useful to look outside academic research, to other people who have asked when cultural technologies are more likely to spur creativity, and when they dull it. The cultural project of Oulipo was, as much as anything else, to use deterministic rules and systems to create different kinds of literature, art and poetry. What kinds of tennis could you play as you built increasingly weird and complicated nets? So too, Brian Eno’s personal adaptation of cybernetics, using generated variation as a tool of discovery.

The great intellectual advantage of Gopnikism, as I see it, is to make us ask these questions. You can envisage a future in which the consequences of LLMs (and perhaps other kinds of ML too) are largely negative, creating conformity, or (though this is not a major focus right now) sterile dissension. Or you could see a future in which they’re used to engender human creativity and problem solving, even if they cannot replace it. Understanding LLMs as cultural technologies presses us to think about which future is more likely, and, perhaps, how best to reach the better futures and avoid the worse ones.

One final caveat, which applies not just to this post, but to the others I’ll be writing. I’m not a major figure in these debates myself, but I’m not an outside observer either. I've talked at length about these questions with Gopnik and Evans (two people who I make immediate beelines towards whenever I see them at an academic gathering), and have co-authored with Cosma and other people who I’ll be talking about in future posts. So this is a partial map of these debates, in both senses of that word. Still, since I haven’t seen anyone else providing such a map, I figured I might as well do it myself, not least to get my own head in order. Equally, I haven’t shared advance drafts of these posts with the people who I’m talking about, figuring that if I make blunders, they possibly be helpful blunders, making obvious whichever sloughs of misunderstanding I’ve inadvertently waded into. Caveat lector.