Machine god metaphors eat your brain

There are other ways of thinking about AI

Here are three propositions.

1. Metaphors are the fundamental abstraction layers in modern AI debates.

2. We are in a moment where you could plausibly argue that Eliezer Yudkowsky and Nick Land are the most influential policy intellectuals in the US conversation. Just look at the state of the two of them. This is not a good thing.

3. The collision of (1), (2), and recent events means that “machine gods” have now entered the discussion. This is a problem. Abstractions necessarily throw information away. The information that the “machine gods” metaphor/abstraction dumps is crucial to navigating the moment we are in. If public and policy debate stays centered around “machine gods: good or evil?” we are in for a bad time.

I should acknowledge that I (and co-authors) absolutely have our own dog in this fight. Indeed, I think that the “AI as social and cultural technology” approach that we have developed is a much better alternative. The final part of this newsletter will talk about a new piece by a mathematician that builds on this approach. Equally, my claim is not, actually, that our alternative approach should become the sole dominant way of thinking about AI. What I actually want, along the lines of my previous post, is a much greater degree of pluralism in the defining debates around a technology that no-one, including its creators, seems to understand particularly well or to be capable of usefully predicting.

******

As far as I can tell, the above Twitter exchange is the source of the most recent outbreak of Machine God Thought. People who know Daniel Kokotajlo tell me that I’d like him, and that we could have good conversation; I imagine they are right. I’ve found most of the rationalists I’ve met to be decent human beings; in many cases, unusually so. Equally, I think that there is something terribly limiting to the ways they talk about AI, and contend that it’s a real problem that debate is still accelerating past the event horizon towards the very heart of Singularity thought.

The two papers Kokotajlo mentions here, AI 2027 and AI 2040, lay out futures that are roughly identical in their predictions about what AI becomes (godlike; superhuman). Where they differ is in their assumptions about how well aligned AIs are with human goals. In the scenario where AI develops more or less unconstrained (thanks in part to geopolitics), AI breaks out of containment, and humanity is likely either Corgi-ized or goes extinct. In the scenario where AI development is delayed, tamed and made subject to “mutual compute destruction,”¹ men and women come to have gods as servants, and start spreading out to nearby galaxies in the next two decades.

Neither future seems to be at all convincing to me, but they are most certainly convincing some people. Casey Newton and Kevin Roose are partnering with NPR to create the “Machine Gods” podcast as part of “a bold new era of content strategy for the network.” In his newsletter, Newton thanks the fates that he and his collaborator were a little too late to tech podcasting to be “beguiled” by Sam Bankman-Fried, and suggests just two paragraphs later that critics of Roose’s famous shoggoth freakout were “previewing the ways in which the commentariat would continually twist itself into pretzels to deny that anything important was happening with LLMs.” Indeed.

Things are no better elsewhere. In Washington DC, the AI policy debate is riven by a bitter dispute between those who take the Kokotajlo perspective (slow it down before we all die!) and those who want to let innovation rip so that we hit the technocapital singularity as soon as possible. The one side owes its framing to Eliezer Yudkowsky’s MIRI institute. The other builds on Nick Land Thought as filtered through the more immediate needs of Marc Andreessen. While I’m definetely more inclined toward the former than the latter, on the grounds that we do indeed need to have serious conversation about regulation, I think it would be terrible to have that conversation starting and stopping with the priorities of the labs.

Finally there’s Scott Aaronson, who I used to rely on for valuable debunkings of policy and media folk when they got over-excited about quantum computing. It is fair to say that Scott is not in a debunking mood these days, now that OpenAI has solved the Navier-Stokes problem. Witness the below:

The Age of Wonders and Terrors

Twenty years ago, when the idea of AI taking over the world in our lifetimes still struck most of us as the unconstrained fantasy of those who knew too much science fiction and too little science … update on the fact that the wild prophecies have come true. … It seems to me that the Singularity has already started; it’s just wildly unevenly distributed.

… For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.

Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.

******

So why do I think that the machine god approach, far from being an “epistemic bare minimum,” is a pernicious brain-devourer? The answer is that the metaphor shrinks rather than expands your horizons of the possible, forcing you to focus on the single overwhelming question of what Machine Jesus is going to do as soon as he descends from that gleaming cloud. Is he going to usher in paradise on earth? Or is he actually a shoggoth in a Jesus-mask, condemning us to enslavement or destruction? Once you have accepted that Machine Jesus is in the process of descending, all questions besides determining his intentions for humanity seem irrelevant.

I’ve written before (with colleagues and on my own) about why I think this whole way of thinking is utterly wrong-headed. But maybe I’m wrong. Take three somewhat inter-related possibilities: p1, that humanity is not the sole beautiful flower of the cosmos and that greater than human intelligence is perfectly possible, p2, that high intelligence is effectively impossible without strong intentionality, and p3, that current AI architectures are indeed the royal road to achieving proper generalizable superintelligence. I personally rate p1 as extremely high, p2 and p3 as very low, and p1*p2*p3, which is the most obvious state in which you might be worried about the AI alignment problem, as very low indeed. But who is to say that I might not be wrong? Equally, I, or someone who is skeptical of Machine Jesus for wildly different and incompatible reasons than my own, might be right. And if that is possible, then we ought not just be obsessing about whether we “succeed or fail in directing” His Awesome Power towards human ends. We might want to worry about other questions too, or instead.

There is a discussion of the solution of Navier-Stokes, the miracle that brought Scott to embrace the true faith, in a guest-post on Terence Tao’s blog, which usefully illustrates the point. It also has the benefit of being (sort-of) pre-registered. This discussion is specifically attractive to me, because it builds on arguments about AI as a social and cultural technology. Again, I don’t think that you need to believe that it is right on the merits to accept the more limited point that I want to make, which is that perspectives other than the machine god one reveal other questions and problems that we ought very reasonably want to be concerned with.

In his guest-post, the mathematician Nestor Guillen writes that he anticipated the solution of Navier-Stokes, but that his reaction was not that human mathematics was in danger of being replaced. Instead, he argued that it validated the entire enterprise.

the bulk of the essay was done in August and shared with a few close friends for feedback. Earlier this week, Tristan Buckmaster announced a breakthrough finding, done in collaboration with Levent Alpöge, of a finite-time blow up for 3D incompressible Euler with forcing. He also made serious allegations of misconduct by OpenAI (see announcement). The next day, OpenAI announced a finite-time blow up result for 3D Navier-Stokes with forcing, obtained from their internal LLM. … I have decided not to change the opening of the essay and instead make a note of the news here.

… What would be the significance if one day we woke up to the news that output from a Large Language Model contains a definitive answer to the question of blow up for the incompressible Navier-Stokes equations, and to the question of where the zeros of the Riemann zeta function lie?

Many of my fellow mathematicians consider this a kind of nightmare situation, and I think this is a very valid position. Here I would like to make a different case, however. Personally, I have come to believe that the news of a Large Model output containing an interesting, insightful, novel mathematical idea can and should be received as a source of communal pride for mathematics and for mathematicians.

Why is this so? Guillen says that:

Humans have always lived in a complex world and their survival has hinged on their ability to process information in amounts beyond what any individual can handle — and it is likely this has been so nearly as long as humans have existed. Herbert Simon’s concept of “bounded rationality” captures this well …

Simon saw that one implication of bounded rationality is that humans have to (and do) develop systems to process information, as a group. We see that humans have built states, stories, bureaucracies, markets, libraries, and more. These are cultural and social technologies. They are means by which humans (not individually but as a group) navigate amounts of information so vast that they cannot be handled by any individual on their own. …

LLMs … are “wrappers” around an older cultural technology: the scientific literature. When you ask an LLM a question about the isoperimetric inequality, the ABP maximum principle, or a PDE, you are effectively interacting with every person who has thought about and worked on those things — provided the data the LLM was trained on contained those contributions in one way or another.

Unsurprisingly, I think that this is largely correct (I do think that you can make the cultural and social technology argument, while also being freaked out about the consequences of AI: more on that in my next post). But even if it isn’t, it highlights different, and I think, important questions than the machine god nexus.

Specifically: if you think of AI as a cultural and social technology, your first reaction will not be to prostrate yourself in anticipation of the coming Age of the Gods. Instead, you are likely to treat fields like mathematics as some combination of individual and collective human intelligence, constructed on top of technologies that guide and direct that intelligence in particular directions. So when a new technology such as AI comes along, that reshapes the ways in which mathematicians draw on their shared intelligence, you will have questions to ask. What are the current processes through which existing knowledge is disseminated and new knowledge is created? What are the incentive structures for dissemination and creation, and how do they interact with each other? How have processes and incentives been shaped by previous and existing cultural and social technologies? How might these processes and incentives be changed by what is happening now?

And so on. You can see one way in which you might extend this in Ben Recht’s recent post, which suggests, if I understand him right, that mathematicians really ought to rethink how much of their field was already based around benchmarkmaxxing (prizes and challenges) even before AI came on the scene. Tao’s argument, as quoted in the NYT, points in a similar direction.

[Tao] compared OpenAI to a wilderness guide who finds a path to a hidden waterfall. “This has some value,” he said. “But once someone shows a specific path to the waterfall, people just take that path. They don’t spend as much time looking for other paths.”

I am not even an amateur mathematician, so am very much open to correction, but I understand Tao to be saying something like the following. Automated processes of discovery have obvious advantages in pointing the direction to new things. Equally, they push the needle from ‘explore’ to ‘exploit.’ Apparently aimless noodling around by humans can have benefits when it opens up unexpected views from Serendippo . More generally:

solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.

Now it could be that AI will become universally better at discovery so that humans become irrelevant to math. That is the machine god assumption. But that assumption rules out, more or less by fiat, the possibility that human curiosity might still have independent value in discovering new possibilities, and all the questions that flow from it. If AI is better at some aspects, and humans are better at others, then you need to confront messy problems of organizational design. How do you rebuild the institutions of mathematics so that they best reconcile AI and human led approaches to discovery? How do you remake the relationship between dissemination of knowledge and discovery? What individual and organizational incentives seem best suited to achieve general goals? What general goals ought you have? None of these are easy questions, and there are likely to be complex tradeoffs between the solutions to one set of problems and the solutions to another.

I don’t think that an ‘AI as cultural and social technology’ approach provides immediate answers (beyond maybe ‘you really ought to read James March’). But the point is that it makes it possible to see that these questions may be important. If you start as Guillen does, from the perspective that AI does not inevitably replace humans in the dissemination and discovery of knowledge, so much as it reorganizes relations among them, then you begin to start asking questions that are systematically obscured by Machine God Thought. In fairness to Scott Aaronson, his post does suggest towards the end that he too cares about these questions. Perhaps when the excitement dies down, he will find better ways towards thinking about them systematically.

You could, of course, make similar claims about other approaches than cultural and social technology. Cybernetics is another very obvious example. If you are stuck on the question of AI alignment, you will necessarily have to trust godlike AIs to monitor other godlike AIs and pray that things will work out. If you start from a cybernetic perspective (say: Maxim Raginsky Thought ) you’ll end up asking very different questions. As the HuggingFace incident showed, there are a lot of holes in the AI labs’ capacities for monitoring and control. “I need abstraction layers to monitor my abstraction layers” is (a) the world’s shittiest boasty bumper sticker, (b) a profound encapsulation of the modern condition, and (c) a diagnosis of the fundamental problems that the AI labs are making for themselves and not always capable of seeing, as best as one can tell from the outside. When you need AI to summarize the data that allows you to write your incident reports on AI, some of the cheese holes are likely to be closely aligned, potentially letting stuff through that you don’t want to.

A third, related but significantly different approach is cybersecurity. As Dan Davies says, “nearly all the examples of worrying agentic behaviour seem to have been seen only in one context – that of frontier research labs doing cybersecurity projects and screwing up their sandbox precautions.” I’m sure that there are great cybersecurity people in the labs, but I suspect that they have much less internal clout than the people who are pushing the agentic boundaries outwards. If you are embedded in a culture that has broadly decided that it is impossible for ordinary human intelligence to prevent putative superintelligent entities from breaking out of their constraints, then cybersecurity seems secondary: alignment of AI goals with human goals is the only way to stay ahead.

I could keep going, but I hope that the point is clear. The Machine God metaphor, which is really just a catchy rephrasing of longstanding notions among a particular community of AI people, has broken out of containment. It is leading, more or less by necessity, to a radical simplification of debate. If you believe that the machine gods are coming, so that super-empowered goal directed superintelligence is right around the corner, then the only thing that matters is ensuring that this superintelligence’s goals are aligned with the ones that you yourself care about. If you stare into the sun, you only see its glare, missing out on all the other questions that may be important. Thinking about these questions involves adopting a more pluralistic understanding of where you might look for knowledge, and asking about the much broader variety of futures that face us than machine god: good or machine god: bad. That’s it.

1. As someone who does international security, sort of, I do not find Silicon Valley geopolitical speculation to be particularly well thought out or convincing. That’s not the topic of today’s post.