Great piece. I think so much tech determinism is a rehash of eschatology with a thermodynamics branding - which they then dispense with when it suits. The retrocausality thing is case in point.
I get the annoyance at the hype and hubris. But this seems like a bunch of hand waving to me. You should always frame things probabilistically with humility. But if you understand what's going on technically it starts to look very very unlikely that we don't end up with superhuman intelligence of some sort within 20-30 years, with shorter timelines possible. There's lots of assumptions and over confident predictions people pile on top of that. But that doesn't change the fact that it's coming and it's going to be a huge deal.
thanks but there are lots of people who (a) "understand what's going on technically" and (b) think that this degree of confidence is utterly misplaced. Including the aforementioned Cosma https://bactra.org/notebooks/nn-attention-and-transformers.html#attention , Ben Recht https://www.argmin.net/ and others. That these counterarguments are not as visible in the discourse is a direct reflection of the problem that the post describes.
If Cosma has a strong counterargument, I haven't read it. When I first read that post, his points around "It's Just X" vs. "You Can Do That with Just X!?!!?!" strike me as perfectly aligned with the inductive argument in favor of the "superintelligence is overdetermined" case. Over and over, we bump into a limitation, skeptics say "aha, see, this wall cannot be breached"... and then we either:
* Brute-force through it with straightforward approaches. "You can do that with just X?" Yes! moar parameters! moar tokens!
* Find an elegant way to climb it from first principles, my favorite -- eg: building AI around generative world models rather than language models, as in active inference
* Or simply hack our way around it. LLMs intrinsically can't do X? Let's just give them a tool that does X, and/or let them write code that does X! Tada, ersatz neurosymbolic AI! <-- this one really caught me by surprise in 2024, but somehow the damn thing works!
Then, there's the positive feedback: more people are around trying to solve these problems, more open research and code gets published that solves problems and introduces new building blocks, solutions then recombine and solve new, bigger problems. Increasingly this process itself is LLM-aided. Yes, lots of self-deluding crap and lots of wheel-reinvention. But that doesn't invalidate the overall pace of progress.
So: I'm certainly not with those who believe that "scale is all you need", or that existing LLMs "think" in the same sense that we do, but IMO the burden of proof is increasingly on the skeptics to show reason to believe there is a wall at all. And yes, I know that Ben's writing at least gestures at it, by highlight a lot of intrinsic assumptions about how we even think about intelligence, decision-making etc. But I haven't seen it coalesce into any kind of strong argument about impossibility to bridge from where we are to something that we could unarguably recognize as intelligence. And, given the amount of smart people on the skeptic camp, I increasingly infer that no such argument is viable.
Couldn't have said it better. The existence of clearly bad guys (Musk, Altman) with huge financial stakes in hyping the hype to the hypiest possible stratosphere is a poor reason to ignore the evidence mounting day after day after month after month. How humans choose to define consciousness or intelligence will, in the end, be entirely beside the point.
This reminds me of nothing as much as the folks who were telling us that the world was going to end in 2012 (because Mayan Calendar). Of course, when the world did NOT end, they told us that actually it had ended in a sense.
Have either of you used Claude Code or Codex? It's already superhuman in many dimensions, though for sure very spiky and missing some key elements. We went from backpropagation to systems that are superhuman in some dimensions on verifiable tasks in 13-14 years. It's incredibly unserious to pretend it's meaningful to compare a fad among a bunch of new age burner types to misinterpret old Mayan fairy tales to a new technology on that trajectory.
To Henry’s reply, I love Ben’s substack. I've never seen him argue really anything like what you are. Yes he's pushed back against dumb hype. But as far as I've seen he's not pretending that what we can see with our own eyes isn't real.
Maybe I misunderstood. Were you not trying to argue that the arguments that the chance of transformational AI in the next few decades are high are wildly overblown and not to be taken all that seriously?
I have a good friend who has been in tech since the late 1990s. He uses Claude Code daily and is verdict is 'meh'. It is useful under careful supervision, but 'superhuman' is not how he describes it.
And my point is not that the new age types misinterpreted Mayan fairy tales. It is that when the fairy tale did not come true, they did not waver in their story.
2027 will come around, and 2028 too, without any Singularity, and without self-sustained AI take-off. But I am sure the True Believers will still Believe.
I just got around to reading this, and it really threw me for a loop. Correct me if I'm wrong, but isn't this backward teleology or movie in reverse or whatever simply a variation on the familiar revolutionary trope? That is, suppose you have a theory of history that says a certain type of revolution is predetermined. (I won't speculate on what theory this might be.) Then everything happening now is interpreted backwards from that future event. To put it differently, the present and even the past are not entirely known. There's so much information, so many perspectives. But if the future is perfectly known, that becomes the fixed point from which you can work backwards to understand what's happening now.
It sounds so weird to write this, but I've spent time in the company of people who actually thought this way, and there was no getting through to them. I suppose there are end times religious people who are also isomorphic, but I've never encountered one.
So much of what Altman and his friends say is just bug eyed nonsense that uses circular logic, probabilities that come from determining how many angels can dance on the head of a pin.
I'm remembering the video of Sam Bankman-Fried baked off his mind playing video games or playing frauds. You could tell he hadn't slept for at least two days.
It's really unfortunate these people have so much money they can p make their bad science fiction plots reality.
Any time when an author tells a story about people acting some way because it was predicted it's usually some weak sauce.
The only time I can think of it being done well is Dune. But the prophecy there was fake.
The thing about the Nick Land quote—"what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy’s resources."—is that it's lousy analysis and an even worse basis for real-life action, but that it's an *awesome* SF story in less than 30 words. It sticks in the mind, and influences you against your will, since it is *such* a good, distilled story. Indeed, you might *almost* say that "what appears to some thinkers as an analysis of our social situation is an invasion from the mind of Nick Land that influences by means of an SF story social actions and beliefs which are inimical to its hosts".
Don't know about AI in particular but computer technology in general has radically changed the actuality of human life and the possibilities for the future of civilization during my lifetime. Either we master it or it will master us indeed!
Asserting that there is one future that is drawing us towards itself is a totally religious reaction to the observation that we are living in uncontrollable chaos beset on every side. We are saved from destruction because this entity-like construct is guiding us in ways that we are incapable of understanding towards an optimally beneficial future. Evidently the present scares people out of their atheism and who can blame them.
if the machines manage to eliminate the entirety of humanity, then what? kill off all flora & fauna? and after singular event? implosion or go where no man has gone before?
At least Chigurh commits to his world view by acting in it and making it partly so. Altman doesn't make anything other than lies and irresponsible statements with no accountability for their harmful outcome.
There is a podcast called "Philosophy Bites", squib interviews of professional philosophers that last 15 or 20 minutes, which I sometimes use to fill little interstices in my day. Two of these are on the subject of thought experiments.
One, with Julian Baggini, offers the familiar framing of Dennett's "intuition pumps": thought experiments don't prove anything in the way that a non-thought experiment does, but they do help to draw out what our intuitions on a subject are.
But of course, this framing assumes that "we" have intuitions; intuitions in common. Another interview is with Edouard Machery; he has examined this assumption empirically across cultures. In some cases, the assumptions holds, and people in very different cultures share the same intuitions about a thought experiment. But in others it does not, and intuitions prove to be quite sensitive to cultural context. I suspect the latter is closer to the universal situation; after all, we are quite limited in the spectrum of cultures that we can access today, compared to all the cultures that have ever been.
Anyway, the presumed universality of thought experiments seems particularly perverse to me when it is applied to *a singularity*, which by definition is a place where mathematical continuity stops and you don't know what happens next. How are you supposed to perform backward induction from that? You don't know what the playoffs in the game even are!
Great piece. I think so much tech determinism is a rehash of eschatology with a thermodynamics branding - which they then dispense with when it suits. The retrocausality thing is case in point.
Tiny correction: “Sailing TO Byzantium”.
of course it is - thanks! I have the poem memorized from my youth, but clearly not the title.
Did you mention the future being determined by a past event? If so, prepare for a world of bare buns and feet.
An NYT writer asked a chatbot to plan a X trip. The bot forgot to pack socks and underwear).
https://www.nytimes.com/2026/04/30/travel/ai-travel-plan-gemini-vacation.html?unlocked_article_code=1.e1A.SNYM.-L2T5zDA_vFe&smid=url-share
I get the annoyance at the hype and hubris. But this seems like a bunch of hand waving to me. You should always frame things probabilistically with humility. But if you understand what's going on technically it starts to look very very unlikely that we don't end up with superhuman intelligence of some sort within 20-30 years, with shorter timelines possible. There's lots of assumptions and over confident predictions people pile on top of that. But that doesn't change the fact that it's coming and it's going to be a huge deal.
thanks but there are lots of people who (a) "understand what's going on technically" and (b) think that this degree of confidence is utterly misplaced. Including the aforementioned Cosma https://bactra.org/notebooks/nn-attention-and-transformers.html#attention , Ben Recht https://www.argmin.net/ and others. That these counterarguments are not as visible in the discourse is a direct reflection of the problem that the post describes.
If Cosma has a strong counterargument, I haven't read it. When I first read that post, his points around "It's Just X" vs. "You Can Do That with Just X!?!!?!" strike me as perfectly aligned with the inductive argument in favor of the "superintelligence is overdetermined" case. Over and over, we bump into a limitation, skeptics say "aha, see, this wall cannot be breached"... and then we either:
* Brute-force through it with straightforward approaches. "You can do that with just X?" Yes! moar parameters! moar tokens!
* Find an elegant way to climb it from first principles, my favorite -- eg: building AI around generative world models rather than language models, as in active inference
* Or simply hack our way around it. LLMs intrinsically can't do X? Let's just give them a tool that does X, and/or let them write code that does X! Tada, ersatz neurosymbolic AI! <-- this one really caught me by surprise in 2024, but somehow the damn thing works!
Then, there's the positive feedback: more people are around trying to solve these problems, more open research and code gets published that solves problems and introduces new building blocks, solutions then recombine and solve new, bigger problems. Increasingly this process itself is LLM-aided. Yes, lots of self-deluding crap and lots of wheel-reinvention. But that doesn't invalidate the overall pace of progress.
So: I'm certainly not with those who believe that "scale is all you need", or that existing LLMs "think" in the same sense that we do, but IMO the burden of proof is increasingly on the skeptics to show reason to believe there is a wall at all. And yes, I know that Ben's writing at least gestures at it, by highlight a lot of intrinsic assumptions about how we even think about intelligence, decision-making etc. But I haven't seen it coalesce into any kind of strong argument about impossibility to bridge from where we are to something that we could unarguably recognize as intelligence. And, given the amount of smart people on the skeptic camp, I increasingly infer that no such argument is viable.
Couldn't have said it better. The existence of clearly bad guys (Musk, Altman) with huge financial stakes in hyping the hype to the hypiest possible stratosphere is a poor reason to ignore the evidence mounting day after day after month after month. How humans choose to define consciousness or intelligence will, in the end, be entirely beside the point.
This reminds me of nothing as much as the folks who were telling us that the world was going to end in 2012 (because Mayan Calendar). Of course, when the world did NOT end, they told us that actually it had ended in a sense.
Have either of you used Claude Code or Codex? It's already superhuman in many dimensions, though for sure very spiky and missing some key elements. We went from backpropagation to systems that are superhuman in some dimensions on verifiable tasks in 13-14 years. It's incredibly unserious to pretend it's meaningful to compare a fad among a bunch of new age burner types to misinterpret old Mayan fairy tales to a new technology on that trajectory.
To Henry’s reply, I love Ben’s substack. I've never seen him argue really anything like what you are. Yes he's pushed back against dumb hype. But as far as I've seen he's not pretending that what we can see with our own eyes isn't real.
I suspect you are arguing with a voice in your head here, rather than what I am writing, so I'll leave you to it.
Maybe I misunderstood. Were you not trying to argue that the arguments that the chance of transformational AI in the next few decades are high are wildly overblown and not to be taken all that seriously?
I have a good friend who has been in tech since the late 1990s. He uses Claude Code daily and is verdict is 'meh'. It is useful under careful supervision, but 'superhuman' is not how he describes it.
And my point is not that the new age types misinterpreted Mayan fairy tales. It is that when the fairy tale did not come true, they did not waver in their story.
2027 will come around, and 2028 too, without any Singularity, and without self-sustained AI take-off. But I am sure the True Believers will still Believe.
Another prediction if there aren't enough already https://0utcast.substack.com/p/the-emergence-and-the-end-of-bullshit
Subscribed! Curious how futures eat pasts in our collective narrative soup.
The Civilization games have a lot to answer for as a framework for analyzing history
I just got around to reading this, and it really threw me for a loop. Correct me if I'm wrong, but isn't this backward teleology or movie in reverse or whatever simply a variation on the familiar revolutionary trope? That is, suppose you have a theory of history that says a certain type of revolution is predetermined. (I won't speculate on what theory this might be.) Then everything happening now is interpreted backwards from that future event. To put it differently, the present and even the past are not entirely known. There's so much information, so many perspectives. But if the future is perfectly known, that becomes the fixed point from which you can work backwards to understand what's happening now.
It sounds so weird to write this, but I've spent time in the company of people who actually thought this way, and there was no getting through to them. I suppose there are end times religious people who are also isomorphic, but I've never encountered one.
Fantastic piece
So much of what Altman and his friends say is just bug eyed nonsense that uses circular logic, probabilities that come from determining how many angels can dance on the head of a pin.
I'm remembering the video of Sam Bankman-Fried baked off his mind playing video games or playing frauds. You could tell he hadn't slept for at least two days.
It's really unfortunate these people have so much money they can p make their bad science fiction plots reality.
Any time when an author tells a story about people acting some way because it was predicted it's usually some weak sauce.
The only time I can think of it being done well is Dune. But the prophecy there was fake.
The thing about the Nick Land quote—"what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy’s resources."—is that it's lousy analysis and an even worse basis for real-life action, but that it's an *awesome* SF story in less than 30 words. It sticks in the mind, and influences you against your will, since it is *such* a good, distilled story. Indeed, you might *almost* say that "what appears to some thinkers as an analysis of our social situation is an invasion from the mind of Nick Land that influences by means of an SF story social actions and beliefs which are inimical to its hosts".
Don't know about AI in particular but computer technology in general has radically changed the actuality of human life and the possibilities for the future of civilization during my lifetime. Either we master it or it will master us indeed!
Asserting that there is one future that is drawing us towards itself is a totally religious reaction to the observation that we are living in uncontrollable chaos beset on every side. We are saved from destruction because this entity-like construct is guiding us in ways that we are incapable of understanding towards an optimally beneficial future. Evidently the present scares people out of their atheism and who can blame them.
if the machines manage to eliminate the entirety of humanity, then what? kill off all flora & fauna? and after singular event? implosion or go where no man has gone before?
At least Chigurh commits to his world view by acting in it and making it partly so. Altman doesn't make anything other than lies and irresponsible statements with no accountability for their harmful outcome.
This strikes me as an engineer’s view of history
There is a podcast called "Philosophy Bites", squib interviews of professional philosophers that last 15 or 20 minutes, which I sometimes use to fill little interstices in my day. Two of these are on the subject of thought experiments.
One, with Julian Baggini, offers the familiar framing of Dennett's "intuition pumps": thought experiments don't prove anything in the way that a non-thought experiment does, but they do help to draw out what our intuitions on a subject are.
But of course, this framing assumes that "we" have intuitions; intuitions in common. Another interview is with Edouard Machery; he has examined this assumption empirically across cultures. In some cases, the assumptions holds, and people in very different cultures share the same intuitions about a thought experiment. But in others it does not, and intuitions prove to be quite sensitive to cultural context. I suspect the latter is closer to the universal situation; after all, we are quite limited in the spectrum of cultures that we can access today, compared to all the cultures that have ever been.
Anyway, the presumed universality of thought experiments seems particularly perverse to me when it is applied to *a singularity*, which by definition is a place where mathematical continuity stops and you don't know what happens next. How are you supposed to perform backward induction from that? You don't know what the playoffs in the game even are!