Why is it always Machine Jesus? Why not Machine Shiva, Lord of the Dance of creative destruction? These machine god types need go expand their horizons.
You know the answer to that rhetorical question. Which deity/deities would they use in India? Would the ostensibly atheist communist China use a different term for the same idea?
I'll take the coming of Machine Shiva (thanks Ary!) seriously when they stop talking about what will/might happen and start talking about how. This may be a byproduct of theorizing about something they don't understand (to be fair, because it doesn't exist yet), but to me both the Doom and Paradise options reek heavily of the Power of Positive Thinking, willing a preferred reality into being like an Ayn Rand character.
Well before the current upsurge of interest in AI, there was big excitement about the idea of using massive computing power to develop drug candidates. A super-simplified version of this is: simulate throwing every possible molecule at every possible target and see what works.
It seems as if some of the math advances have been like this. Look for tools that arise in areas far from the problem in question and see if they turn out useful. Humans have been doing this at least since Descartes turned geometry into algebra, but machines can do it massively faster. That could radically change mathematics, but wouldn't get us any closer to super-intelligence
I'm not worried about AI in and of itself per se. It becomes a problem when it's incorporated into the physical infrastructure, such as the power grid and water supply. We've already seen some of the vulnerabilities exposed with mere computer control - such as to hacking.
The big problem isn't AI becoming "god like" so much as it is the purveyors/profiteers shoving it down our throats a quickly as they can, so as to maximize profits - even when it's really not ready for prime time.
“Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud.” Wild statement and one that’s not aging that well giving some of the recent reporting. Oof.
These our-Ai-is-too-powerful-for-us-to build-a-firewall-around-it stories can be read as we are burning $30B a year on AI and can't afford to hire a team to configure a firewall properly. It's a surprisingly simple problem that has well known solutions that might require thinking for ten or fifteen minutes and so will not be addressed.
This is a great post. I'm personally very skeptical about the imminence of space-faring machine gods, but I end up really frustrated by most skeptics because of how dismissive they are of true AI believers. (There are already a bunch of comments on this post along those lines!)
I don't think it makes sense to dismiss people who believe in imminent world-transforming superintelligence when a lot of them are very influential and have spent their whole lives being publicly very smart; any attempt to understand, catalogue and contextualize their worldview is valuable, in addition to providing alternatives. I think that generalizing and extrapolating, which are often the tools of smart people, have led a lot of AI people astray in understandable ways, and trying to understand how this has happened with someone like Scott Aaronson is very valuable and far more important than laughing at him.
Replace AI people with priests and the result is much the same. Wrong beliefs and actions bring "hellfire and eternal suffering". The right beliefs and actions will bring a "glorious future." Human nature was ever thus.
I do agree or at least strongly suspect it’s a case of a bunch of unfalsifiable beliefs filling a God-shaped hole in people’s brains (but that doesn't mean they can’t be taken seriously!)
"Shermer explains why he believes that "intelligent people" can be more susceptible to believing in weird things than others." - Wikipedia (above ref).
You wrote in the article on the shoggoth freakout: "LLMs, like markets, are not conscious actors themselves but complex outcomes of the interactions of myriad agents, filtered through an imperfect technology of representation. For markets, this technology is prices (which are lossy summary statistics of a multitude of decisions to buy and sell). For LLMs, it’s their statistical weights, which summarize the relationships between tokenized words and other forms of content scraped from the Internet and elsewhere." But none of this would go against the doomers concern that these things are blackboxes which can give entirely unpredictable results that could be highly dangerous. The concerns aren't based on the idea they have desires or are 'evil' in some way.
The segment highlighted by Kate Willett left me giggling that this is being taken seriously. It sounds like a weed-addled sophomore in a dorm room. And that ANY of it is remotely possible by 2045 is the cherry on top.
If you described the modern world to a person who lived 50-100 years ago, they would probably describe it as the vision of a "weed-addled sophomore in a dorm room". Just in the past decade or so, we've had multiple "holy shit is this really happening" moments, e.g. in politics and virology.
And yet a fair fraction of the US population believes in an "invisible sky fairy" and that the "rapture" will precede its arrival incarnated as its offspring. Shouldn't we non-believers be giggling?
Sure, feel free to giggle at that, too, but that's a matter of faith. We're supposed to be talking about the rational people who believe in science here, that makes it that much worse.
Let me correct that. "Rational" people. Philosophers think they are rational, employing logic to make their arguments. Yet they seem to come to different answers from the same premises. I don't think for one moment that these so-called rational people are arriving at the correct outcome at all. We do science so that we can test [rational] hypotheses/theories to ensure we are on the correct path to "truth". Arguments based on logic, even if based on the correct premises, don't necessarily reach the correct conclusions. This is just another version of belief, and no different than religion, or indeed any other form of system that is similarly arrived at.
In this case there have been a number of real world predictions that are now coming true, like as the models get more intelligent they will do things like solve 100 year maths problems, but also start to show more problematic behaviours like deception, hacking, serious misalignment. They worried about agents having more in common with each other than human oversight, which was shown to the fullest in the self-sacrifices in the Hugging Face incident.
A recent paper, 'The Dead Salmons of AI Interpretability' (https://arxiv.org/abs/2512.18792), argues that many claims of AI cognition are evidence of statistical 'fragilities' like intuitive feature attribution, overdetermination among multiple plausible explanations for model outputs, and false discovery of cause. Our tendencies to anthropomorphize and attribute intention to complex systems (like the market, or LLMs) might be the same cognitive pathways we take to unjustifiably attributing consciousness or even divinity to frontier models. Or the universe.
This is an outdated notion of "the current AI architecture". So much of the problem-solving ability of recent LLMs comes from their Reinforcement Learning post-training. This is *not* simply RLHF which is based on human rankings; it learns from interacting with environments (such as code, math, cyber or game harnesses) and receiving rewards based on how it changes the environment and in some cases its own state within the environment.
Systems like this are much more like true experimenters: they model, predict, take an action, observe the result of the action, update their model of the environment, and update the model of their own actions in the environment. Crucially the chain-of-thought also gets influenced by the reward, which grounds the linguistic reasoning trace firmly against the environment.
This increases the likelihood of p3 a lot, and maybe p2 also since such systems have been viewed as agents in the the Beliefs-Desires-Intentions model long before this current potential was realised. And we're only scratching the surface with RL, there are other candidates for this type of learning, like Friston's active inference.
Of all the metaphors, the one I keep coming back to is a mirror. These machines learned everything from us, so what we see in them says at least as much about us as it does about them.
Metaphors are our fundamental abstraction. We call them words which is like a folk scientific abstraction of folk psychology. So, word is a bad word for them. The proper, scientific terms for words is “conduit metaphor” and it’s a paradox. They don’t exist nor do they ever refer to anything that does or does not exist, yet we act or pretend that they do.
So anything downstream of words and symbols (like gods, narratives, AI) also do not exist, but pretending they do is probably schizophrenic and certainly superstitious. AI is simply automated processes words already gave us millennia ago. In this case, as words are the message, you can blame the messenger.
thank you! every essay you write here enlightens me more and more on this situation, and the present one is a big step for me.
Why is it always Machine Jesus? Why not Machine Shiva, Lord of the Dance of creative destruction? These machine god types need go expand their horizons.
You know the answer to that rhetorical question. Which deity/deities would they use in India? Would the ostensibly atheist communist China use a different term for the same idea?
Machine Mandate of Heaven? Machine Socialism with Chinese Characteristics?
I'll take the coming of Machine Shiva (thanks Ary!) seriously when they stop talking about what will/might happen and start talking about how. This may be a byproduct of theorizing about something they don't understand (to be fair, because it doesn't exist yet), but to me both the Doom and Paradise options reek heavily of the Power of Positive Thinking, willing a preferred reality into being like an Ayn Rand character.
Well before the current upsurge of interest in AI, there was big excitement about the idea of using massive computing power to develop drug candidates. A super-simplified version of this is: simulate throwing every possible molecule at every possible target and see what works.
It seems as if some of the math advances have been like this. Look for tools that arise in areas far from the problem in question and see if they turn out useful. Humans have been doing this at least since Descartes turned geometry into algebra, but machines can do it massively faster. That could radically change mathematics, but wouldn't get us any closer to super-intelligence
I'm not worried about AI in and of itself per se. It becomes a problem when it's incorporated into the physical infrastructure, such as the power grid and water supply. We've already seen some of the vulnerabilities exposed with mere computer control - such as to hacking.
The big problem isn't AI becoming "god like" so much as it is the purveyors/profiteers shoving it down our throats a quickly as they can, so as to maximize profits - even when it's really not ready for prime time.
Worse, if it isn't profitable and brings down the global economy with its collapse.
“Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud.” Wild statement and one that’s not aging that well giving some of the recent reporting. Oof.
What recent reporting?
Are you the same as the co-author of "End Times Fascism"?
Too much Iain Banks.
These our-Ai-is-too-powerful-for-us-to build-a-firewall-around-it stories can be read as we are burning $30B a year on AI and can't afford to hire a team to configure a firewall properly. It's a surprisingly simple problem that has well known solutions that might require thinking for ten or fifteen minutes and so will not be addressed.
This is a great post. I'm personally very skeptical about the imminence of space-faring machine gods, but I end up really frustrated by most skeptics because of how dismissive they are of true AI believers. (There are already a bunch of comments on this post along those lines!)
I don't think it makes sense to dismiss people who believe in imminent world-transforming superintelligence when a lot of them are very influential and have spent their whole lives being publicly very smart; any attempt to understand, catalogue and contextualize their worldview is valuable, in addition to providing alternatives. I think that generalizing and extrapolating, which are often the tools of smart people, have led a lot of AI people astray in understandable ways, and trying to understand how this has happened with someone like Scott Aaronson is very valuable and far more important than laughing at him.
Replace AI people with priests and the result is much the same. Wrong beliefs and actions bring "hellfire and eternal suffering". The right beliefs and actions will bring a "glorious future." Human nature was ever thus.
I do agree or at least strongly suspect it’s a case of a bunch of unfalsifiable beliefs filling a God-shaped hole in people’s brains (but that doesn't mean they can’t be taken seriously!)
Read Michael Shermer's "Why People Believe Weird Things: Pseudoscience, Superstition, and Other Confusions of Our Time".
https://en.wikipedia.org/wiki/Why_People_Believe_Weird_Things
"Shermer explains why he believes that "intelligent people" can be more susceptible to believing in weird things than others." - Wikipedia (above ref).
"We don't need to worry about the risk of nuclear war. It's too much like the Bible's description of Hell to be real."
You wrote in the article on the shoggoth freakout: "LLMs, like markets, are not conscious actors themselves but complex outcomes of the interactions of myriad agents, filtered through an imperfect technology of representation. For markets, this technology is prices (which are lossy summary statistics of a multitude of decisions to buy and sell). For LLMs, it’s their statistical weights, which summarize the relationships between tokenized words and other forms of content scraped from the Internet and elsewhere." But none of this would go against the doomers concern that these things are blackboxes which can give entirely unpredictable results that could be highly dangerous. The concerns aren't based on the idea they have desires or are 'evil' in some way.
The segment highlighted by Kate Willett left me giggling that this is being taken seriously. It sounds like a weed-addled sophomore in a dorm room. And that ANY of it is remotely possible by 2045 is the cherry on top.
If you described the modern world to a person who lived 50-100 years ago, they would probably describe it as the vision of a "weed-addled sophomore in a dorm room". Just in the past decade or so, we've had multiple "holy shit is this really happening" moments, e.g. in politics and virology.
And yet a fair fraction of the US population believes in an "invisible sky fairy" and that the "rapture" will precede its arrival incarnated as its offspring. Shouldn't we non-believers be giggling?
Sure, feel free to giggle at that, too, but that's a matter of faith. We're supposed to be talking about the rational people who believe in science here, that makes it that much worse.
Let me correct that. "Rational" people. Philosophers think they are rational, employing logic to make their arguments. Yet they seem to come to different answers from the same premises. I don't think for one moment that these so-called rational people are arriving at the correct outcome at all. We do science so that we can test [rational] hypotheses/theories to ensure we are on the correct path to "truth". Arguments based on logic, even if based on the correct premises, don't necessarily reach the correct conclusions. This is just another version of belief, and no different than religion, or indeed any other form of system that is similarly arrived at.
In this case there have been a number of real world predictions that are now coming true, like as the models get more intelligent they will do things like solve 100 year maths problems, but also start to show more problematic behaviours like deception, hacking, serious misalignment. They worried about agents having more in common with each other than human oversight, which was shown to the fullest in the self-sacrifices in the Hugging Face incident.
A nice, long power outage would do these guys a world of good.
A recent paper, 'The Dead Salmons of AI Interpretability' (https://arxiv.org/abs/2512.18792), argues that many claims of AI cognition are evidence of statistical 'fragilities' like intuitive feature attribution, overdetermination among multiple plausible explanations for model outputs, and false discovery of cause. Our tendencies to anthropomorphize and attribute intention to complex systems (like the market, or LLMs) might be the same cognitive pathways we take to unjustifiably attributing consciousness or even divinity to frontier models. Or the universe.
This is an outdated notion of "the current AI architecture". So much of the problem-solving ability of recent LLMs comes from their Reinforcement Learning post-training. This is *not* simply RLHF which is based on human rankings; it learns from interacting with environments (such as code, math, cyber or game harnesses) and receiving rewards based on how it changes the environment and in some cases its own state within the environment.
Systems like this are much more like true experimenters: they model, predict, take an action, observe the result of the action, update their model of the environment, and update the model of their own actions in the environment. Crucially the chain-of-thought also gets influenced by the reward, which grounds the linguistic reasoning trace firmly against the environment.
This increases the likelihood of p3 a lot, and maybe p2 also since such systems have been viewed as agents in the the Beliefs-Desires-Intentions model long before this current potential was realised. And we're only scratching the surface with RL, there are other candidates for this type of learning, like Friston's active inference.
Of all the metaphors, the one I keep coming back to is a mirror. These machines learned everything from us, so what we see in them says at least as much about us as it does about them.
Metaphors are our fundamental abstraction. We call them words which is like a folk scientific abstraction of folk psychology. So, word is a bad word for them. The proper, scientific terms for words is “conduit metaphor” and it’s a paradox. They don’t exist nor do they ever refer to anything that does or does not exist, yet we act or pretend that they do.
So anything downstream of words and symbols (like gods, narratives, AI) also do not exist, but pretending they do is probably schizophrenic and certainly superstitious. AI is simply automated processes words already gave us millennia ago. In this case, as words are the message, you can blame the messenger.
Where did that picture come from? Wow! (Also, great post, but I'm sure you don't need me to tell you.)
AI generated, using Piranesi's engravings as a seed. It is slop but unusual slop that is designed with a little attention.
Tools can be well or poorly wielded... :)