The Engineer as Magus

From Cornelius Agrippa to Dario Amodei?

It may be a cliche to use metaphors of demons, angels, and magical rites of summoning and control to talk about AI, but it’s a very convenient cliche for me. I have an amateur interest in the history of Renaissance magic, having been exposed at an impressionable age to the fiction of John Crowley and the non-fiction of Dame Frances Yates. Hence this post, which draws on the writings and ideas of Cornelius Agrippa, the notorious Renaissance era magus, engineer, self-glorifier, and scoundrel, to explain a new essay by Dario Amodei (the founder of Anthropic), which has been making the rounds.

It might appear as though there is not much in common between these texts. Certainly, there is a remarkable difference between Amodei’s and Agrippa’s personal characters and motivations. As best as I can tell as an outsider, Amodei is as honest and well intentioned as anyone in his position can reasonably be. He and his cadre famously left OpenAI because they didn’t like where Sam Altman (a chancer if ever there was one) was taking it. But the comparison is useful. Bear with me.

Agrippa’s history is related in Anthony Grafton’s recent and excellent book on the history of the magus from the late-mediaeval era through the Renaissance. Grafton quotes Agrippa as arguing that:

The mathematical disciplines are so necessary and so organically connected to magic that one who practices magic without them will go completely astray and his work will be in vain …. But even without the aid of natural powers, works like natural ones can be produced from the mathematical disciplines alone as Plato says: not things that take part in truth and divinity, but certain simulacra related to them, like bodies that walk and speak even though they lack any animal virtue, like the statues and automata of Daedalus among the ancients, which Aristotle mentions, the self-moving tripods of Vulcan and Daedalus, about which we read in Homer, which went into battle of their own will … the statue of Mercury that spoke … To this category belong all the miracles of simulacra that are produced by geometry and optics … Hence the magus, who is a master of natural philosophy and mathematics … can also, not surprisingly, work many wonders, which may astonish even the most prudent of men.

The point - which is of course not original to Grafton - is that the origins of science and engineering are hopelessly commingled with the history of magic. Agrippa and his peers, contemporaries, and near-contemporaries, saw the art of the magus and the mathematical and mechanical arts as so intertwined that they were effectively identical. Alchemy; Newton’s obsession with astrology; statues and brazen heads that seemed to speak; mechanical crucifixes such as the Rood of Boxley; all these commingled mathematics and engineering with belief in a world that was suffused by invisible connections and intelligence.

The magus could use mathematics and relations of sympathy and similarity to compel these arcane forces to do what they wanted. The ‘magic squares’ that delight amateur mathematicians today were quite literally considered to be magical. The one depicted in the Durer engraving above is supposed to dispel Saturn’s melancholic grip through numerical patterns that capture Jupiter’s attention.

But there seemed to be something particularly magical about automata and simulacra - machines that simulated conscious action and volition. As Grafton summarizes the intellectual tangle of the period, "automata were invested with special powers - powers that could somehow be seen as both mechanical and spiritual.”

Over the centuries, the fascination with automata and simulacra faded as mathematicians and engineers discovered more practical applications for their discoveries. Now, they are back. The mathematical disciplines are producing simulacra that actually can speak: Large Language Models. Hence, the connections between Agrippa’s celebration of the mathematical disciplines, and Amodei’s vision of a world that will again be suffused with invisible intelligence, so that those who master the arts of creation and instruction will have countless automata at their command.

I again want to stress that Amodei’s vision of a world transformed by AI stems from conviction rather than hucksterism. There is a lot that is attractive about his essay, and where it comes from, even if you don’t buy it. Equally, his evident sincerity makes it easier to explain what I think is mistaken about his understanding of the world. Like the magi of the Renaissance, he confuses artifice with intelligence, so that engineering ends in a kind of magic.

My particular spin on the underlying idea of his essay is as follows. Thanks to Amazon Web Services and the like, people these days talk less about computers than “compute” - a flow that you can turn on and off when you need it, as simply as turning a faucet. What if the same becomes true for intelligence? That is - what if you can use AI to automate 100% genuine creative intelligence, creating genius-as-a-service?

Amodei makes it clear that this is not a thought experiment. He thinks it quite possible that we will have the equivalent of a “country of geniuses in a datacenter” by 2026: general purpose AI that you can apply to an enormous variety of scientific, economic and political problems. This would involve:

an AI model—likely similar to today’s LLM’s in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties: In terms of pure intelligence it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc. … it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access … it does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary … The resources used to train the model can be repurposed to run millions of instances of it … Each of these million copies can act independently on unrelated tasks, or if needed can all work together in the same way humans would collaborate.”

This is the world that Amodei thinks awaits us, possibly very soon, though he acknowledges it might perhaps take much longer. And it strongly resembles the world of wonders that Agrippa anticipated. “[P]owerful AI could [give] … us the next 50-100 years of biological progress in 5-10 years.” This is possible because Amodei is “not talking about AI as merely a tool to analyze data” (bold and italics in original), but “a virtual biologist who performs all the tasks biologists do,” designing and running experiments via lab robots. Very soon, the mathematical disciplines will be able to produce and manage “bodies that walk and speak even though they lack any animal virtue.” It will all be quite wonderful, so long as we can get there.

There are many different ways you could criticize this vision. One is to build on Ramani and Wang’s argument that no matter how awesome bottled intelligence might be, there will be bottlenecks. Amodei is confident that these bottlenecks can be circumvented - Ramani and Wang provide reasons to worry they cannot.

But there is a simpler criticism. Agrippa - and the others of his kind - seemed to think that magic and mechanics would at some point coincide, so that a sufficiently sophisticated simulacrum might tap into the forces of the cosmos. As above, so below. Might there be a similar error here?

If we replace 'forces of the cosmos’ with ‘true intelligence,’ my personal answer is yes. The key paragraph in Amodei’s vision statement is where he explains his dissatisfaction with neuroscientific understandings of intelligence:

When I was working in neuroscience, a lot of people focused on what I would now consider the wrong questions about learning, because the concept of the scaling hypothesis / bitter lesson didn’t exist yet. The idea that a simple objective function plus a lot of data can drive incredibly complex behaviors makes it more interesting to understand the objective functions and architectural biases and less interesting to understand the details of the emergent computations. I have not followed the field closely in recent years, but I have a vague sense that computational neuroscientists have still not fully absorbed the lesson. My attitude to the scaling hypothesis has always been “aha – this is an explanation, at a high level, of how intelligence works and how it so easily evolved”, but I don’t think that’s the average neuroscientist’s view, in part because the scaling hypothesis as “the secret to intelligence” isn’t fully accepted even within AI.

This passage clarifies the bet that Anthropic and its rivals are making. If you believe that intelligence is some combination of (a) an objective function, (b) some workaday architecture (since intelligence evolves easily), and (c) shitloads of data, then indeed, scale is the means through which the ghost is compelled into the machine. All you have to do is build bigger, with ever more data, until true intelligence is achieved. Once that is done - and Amodei seemingly anticipates it will be accomplished sooner rather than later - you will have automated intelligence that can be delivered at scale, potentially transforming the human condition in some quite radical ways.

Indeed, we see regular claims from AI companies that LLMs are coming to reason as well as human beings or better. The problem is that we also see results telling us that LLMs are better described as engaging in ‘probabilistic pattern-matching rather than formal reasoning.’ The prospect that objective-function-plus-transformerlike-system-produced-pattern-matching-architecture will begin to really think, if only enough data is thrown into the pot is, contrary, to Amodei, a heroic bet. There are reasons why neuroscientists have not converted en masse to the True Faith of Easy Intelligence Via Scaling, which have to do with the scientific knowledge they have accumulated about how existing brains actually seem to work.

More generally, what AI companies call reasoning, isn’t, really. It is brittle in ways that actual human reasoning, with all its flaws, is not. We might all be less confused if some vast mechanical process were to scour and rewrite ArXiv’s hoard of papers about LLMs, replacing every instance of the term “reasoning” with the phrase “I Can’t Believe It’s Not Reasoning (tm),” and a snark emoji with a URL resolving to a picture of Fabio.

Maybe that is too harsh, and you may want to take my opinions (as you always should) with a healthy dollop of salt. After all, there are billions of dollars, and the hard work and careers of thousands of intelligent people riding on the opposite side of the bet. But I’m happy with my side of the wager, which, I should make clear, is based on the arguments of other people who are much more intelligent than I am. They present evidence, which I find compelling, that we are unlikely to scale our way into the “geniuses in a datacenter” model on the basis of existing technologies, or anything that we’re likely to develop soon. You would need magic to turn an LLM-type model into actual intelligence (though there are other ways in which you could get to machine intelligence - that humans exist implies that it is certainly not impossible).

The people whom I find more credible suggest that LLMs are simulacra, and will never be more than that, unbreathing statues that seem to speak, but that will never be inspired. If you prefer Herbert Simon’s language, they are “[a]rtificial things [that] may imitate appearances in natural things while lacking, in one or many respects, the reality of the latter.” Artificial things, as a general class, channel and process information. That they might occasionally be mistaken for human intelligences is not a hint as to their ultimate uses, but a distraction from it.

Put differently: Agrippa and his brethren were wrong to think that they could use magic to manipulate the intelligences they believed to suffuse the cosmos. But they were right to see the beginnings of a vast transformation in the ingenuity of the artificers. The mathematical disciplines did indeed give rise to wonders to astonish the most prudent of men. But these wonders were ordinary rather than magical. The “occult automata” of the Renaissance were no more than curiosities, but the hydraulic, clockwork and pneumatic systems that powered them were ancestors of the banausic steam engines and geared machineries that came to drain mines, power factories and enable new forms of transport in later centuries.

So too, I suspect Amodei and others are wrong to believe that they can bottle genius and transform the world. But that doesn’t mean that LLMs and their cousins are useless. If they were useless, they would be far less interesting. The question - as with other monstrous technologies such as markets and bureaucracies - is how they will be deployed, and whether their deployment will be more likely to impose hurts than to generate general benefits.

If my bet is right, we’re going to see a crash - sooner rather than later - in the perceived value of LLM-type technologies. Investors will realize that they will not, in fact, be able to profit from billions of disembodied geniuses laying the foundations for the transhumanist libertarian utopia, or, for that matter, the Iain M. Banks’ Culture model that Amodei rightly prefers.* That may open up the way to a more serious evaluation of the political, economic, social and moral consequences of a family of technologies that do not promise to transform the human condition, but will plausibly change how organizations work, and have a whole variety of other applications.

I think Amodei’s essay is valuable as a thoughtful - and well thought out - development of a point of view that I believe to be fundamentally wrong. I imagine his thinking would be more valuable still after the magic rubs away, and we’re able to start thinking seriously about what these technologies are doing.

* Which is probably a good thing. Banks - a cheerful Scots social democrat with a keen eye for oppression and a real understanding of the tragedy of the human condition - would have been horrified by the prospective enrichment of the few, and appalled by some of his latter day fans. Not Amodei, I don’t imagine - but what would he have made of latter day Elon Musk?**

** I suspect that the actual answer to the rhetorical question is “mincemeat.” Banks was not notably shy in his opinions.

Update: Misspelling of Amodei’s name spotted approximately 1 minute after the email went out. My friend and collaborator, Hugo Mercier, has a cognitive science explanation of why it is that you are more likely to spot errors immediately after the piece has gone out into the world and not before …