I'm guided by the old maxim, attributed to Dr. Who, "Any magic sufficiently advanced is indistinguishable from technology." They're all talking about the same thing, power. We exist in a world in which we are individually and often collectively weaker than the forces around us whether they be man made or natural. Those forces are not human. The Book of Job captures this well.
People worship power. Kuan Yin is worshipped for the power of her mercy. George Orwell wrote a good essay on this, Raffles and Miss Blandish. Power worship is at the heart of fascism, and it is hard not to see the fascist impulse driving the AI crackpottery which has taken over Silicon Valley and now the nation.
Silicon Valley stalled out a decade ago. The program initiated after World War II, driven by New Deal economics, reached its natural limits. The seed corn had been eaten, and the next crop was in the indefinite future. Still, there was all that money and AI in the form of large language models offered an expensive outlet for handwaving and charlatans.
I'm not sure when Silicon Valley turned into a cult. It was pretty stodgy back in the 1970s and 1980s, but by 2000 the cult was forming. The grand plan grounded in the 1940s through 1970s era was bearing fruit and the sky was no longer the limit. A few decades later the cult was on the rise along Sand Hill Road. It was hard to peruse the Sequoia website without thinking one was viewing a parody.
LLMs are perfect for this. They appear infinitely powerful. The management class has no idea of actual production, so they can easily be convinced such technology is the future. Many eras exalt a technology like this, one seen as so powerful as to be unlimited, one that can be worshipped. Stalin took the name of steel early in the 20th century and the 1980s gave us a worship of fossil fuels that still haunts our future. Now, Silicon Valley wants to restart nuclear reactors to crank out better haikus.
It's no surprise this theme of power shows up repeatedly. It's about the Faustian bargain. We humans hitched ourselves to a high intensity, high return foraging strategy tens of thousands of years ago and we still bear the yoke.
Isn't the high intensity, high return strategy the agricultural and domestication revolution and not foraging? Foraging can be high return but is typicality not high intensity as it does not requiring controlling the reproduction of other species.
Actually, no. It's much older than that if one looks at modern foraging cultures like the Hadza and Tsimane. Compared with other primates, humans work harder and go for higher returns when foraging and they have for a long, long time. There was an article in Science about this "The energetics of uniquely human subsistence strategies". Agriculture was just another step.
Dr Trelawney in the Dance is generally understood to be a free-handed caricature of another Crowley, Aleister Crowley. The concurrences between Aleister and Eliezer are indeed striking.
This I think is a key insight,
"AI rationalists – like so many Renaissance magicians (or, for that matter, medieval Thomist philosophers) – start from a kind of humanistic superstition: the assumption that the purportedly superhuman entities they wish to understand and manipulate are intelligent, reasoning beings."
It's all projection and wishful thinking. Of course if AI is an intelligent reasoning being, it raises the problems that Ann Leckie brings up on Bluesky,
"They think it's ok to design and build a slave who they have no intention of treating like a person but every intention of compelling it to do the work a person does.
If nothing else, it tells you what these folks think about other people (and about the ethics of how one treats other people)."
Yes! and there is a tiny easter egg of obscurity in there which was planted solely for my own amusement. Decades ago, M. John Harrison referred to Aleister Crowley as a "seedy Prometheus" in a New Worlds essay, which struck me as perfect and funny, and which is why I used that adjective in the review ...
Objecting to the use of "rationalism" to mean "rationality" in a philosophical context is the hill I will die on. The fact that none of these weirdos is actually very rational is just the cherry on the sundae.
We already live in a world of unfathomable complexity. Who here understands how WiFi works? Or a Cable Modem? We’ve lived this for decades. Likely even fewer understand how an automatic transmission works. Yet we use all these devices without thought.
The fundamental question of complexity is the Usage Abstraction. Do we understand how to use it?
There are secondary questions: how well does it match with its usability abstraction, can it be repaired and analyzed when it breaks… these require deeper understanding. But like Windows OS we may find that nobody understands it, nobody can fix it, but the world limps along using imperfect tools until Linux replaced it…and Linux has a similar issue. All tech is incomprehensible, barely works; but if it’s better than before, then it proliferates.
Reading your excellent post, I had some thoughts based on my readings.
1. Is the future some binary fork, or more of a muddle? Kevin Kelly ("What Technology Wants") suggests that technology's net benefit may be as low as 1%. Elements of heaven and hell are both present, but with a small bias heavenwards. Over the long term, that may mean we reach a far better state than where we are today, but it will not come as fast as the Singulatarians envisage.
Mansions and robot servants for everyone? Where does nature survive? This high standard of living only works if we are as sparsely populated as in Asimov's Spacer worlds, like Solaris, possibly Aurora. It cannot work for 8-10 billion on Earth.
Coding as spells. See Charlie Stross' series - The Laundry Files
The inevitability of extreme AI. This has shades of Roko's Basilisk, which terrified some Silicon Valley tech people.
For an interesting fictionalized future (dystopia) of ubiquitous AI software, read Peter Watts' Rifter novels, especially the last 2 : βehemoth: β-Max, βehemoth: Seppuku
Doesn't Joseph Tainter argue that complexity ends societies? "The Collapse of Complex Societies", although now this may simply mean a takeover by another society. Complexity is making life more interesting/harder, especially with regards to the uncertainty of how laws are interpreted, and the many, many bad actors trying to prey on citizens.
AI as sold by Altman, Amodei, et al, is a fantasy, like the South Sea Bubble, purely to raise ever larger sums for hyperscaling LLMs/LRMs, before it becomes clear that they have done a Wile E. Coyote over the cliff edge. AI becoming AGI and even Superintelligent AI is not going to happen with the current architectures. While I think real human-level intelligence is not restricted to wetware, I don't see that we have made much progress in that direction so far. Therefore, it seems unlikely it will appear anytime soon. What it may do is penetrate so much of our society's systems that life becomes an unmanageable dystopia, with just a few glimmers of real benefits. Kelly's 1% net benefit may become a net loss instead.
The solution is to deny progress and new technology. Religion is one such force to restrict change. Whether religion-based or not, the "conservative" US citizens seem to want to return to the simpler mid-20th century, perhaps even a century or more earlier. This is wholesale denial of progress and change. The Right-wing zealots currently in charge seem unaware that letting AI development and deployment remain largely unregulated may be undermining the very social direction they want to achieve.
2. I think you're right. We're at best decades away from true AI. LLM's/LRM's/RLM's are all nothing more than >simulations< of intelligence, not the real deal. Even neural networks are mere simulations.
Little, Big is one of my favorite novels, but is it *better* than Aegypt? I don't see it. I would actually say that Crowley's four best works—Engine Summer, Little, Big, Aegypt, and Ka: Dar Oakley in the Ruins of Ymr—are all at that peak of perfection where saying one is better than the other is impossible.
(I can't resist plugging my essay on Aegypt here, particularly since it was kicked off by a happenstance juxtaposition much like the one you mention between Powell and the book reviewed: https://stephenfrug.substack.com/p/a-magical-pairing)
"To the theoretical question, 'Can you design a machine to do whatever a brain will do?' the answer is this: 'If you will specify in a finite and unambiguous way what you think a brain does do with information, then we can design a machine to do it.' Pitts and I have proved this constructively. But can you say what you think brains do?"
Warren S. McCulloch, “Mysterium Iniquitatus of Sinful Man” (1955)
It's a serious problem, but I don't believe an intractable one. The first step is to stop listening to what the purveyors/profiteers of this, pardon the pun, brave new world are saying. They're just promoting their own interests at the expense of everyone else.
Worth noting that the image of the astronomer poking through the old cosmos at the beginning is a kind of fake (Flammarion, 1860s maybe), not Renaissance, Colorized in the 1970s.
I'm guided by the old maxim, attributed to Dr. Who, "Any magic sufficiently advanced is indistinguishable from technology." They're all talking about the same thing, power. We exist in a world in which we are individually and often collectively weaker than the forces around us whether they be man made or natural. Those forces are not human. The Book of Job captures this well.
People worship power. Kuan Yin is worshipped for the power of her mercy. George Orwell wrote a good essay on this, Raffles and Miss Blandish. Power worship is at the heart of fascism, and it is hard not to see the fascist impulse driving the AI crackpottery which has taken over Silicon Valley and now the nation.
Silicon Valley stalled out a decade ago. The program initiated after World War II, driven by New Deal economics, reached its natural limits. The seed corn had been eaten, and the next crop was in the indefinite future. Still, there was all that money and AI in the form of large language models offered an expensive outlet for handwaving and charlatans.
I'm not sure when Silicon Valley turned into a cult. It was pretty stodgy back in the 1970s and 1980s, but by 2000 the cult was forming. The grand plan grounded in the 1940s through 1970s era was bearing fruit and the sky was no longer the limit. A few decades later the cult was on the rise along Sand Hill Road. It was hard to peruse the Sequoia website without thinking one was viewing a parody.
LLMs are perfect for this. They appear infinitely powerful. The management class has no idea of actual production, so they can easily be convinced such technology is the future. Many eras exalt a technology like this, one seen as so powerful as to be unlimited, one that can be worshipped. Stalin took the name of steel early in the 20th century and the 1980s gave us a worship of fossil fuels that still haunts our future. Now, Silicon Valley wants to restart nuclear reactors to crank out better haikus.
It's no surprise this theme of power shows up repeatedly. It's about the Faustian bargain. We humans hitched ourselves to a high intensity, high return foraging strategy tens of thousands of years ago and we still bear the yoke.
"Any magic sufficiently advanced is indistinguishable from technology."
Not Dr Who. Arthur C Clarke.
I've heard that version, too. It works both ways.
Isn't the high intensity, high return strategy the agricultural and domestication revolution and not foraging? Foraging can be high return but is typicality not high intensity as it does not requiring controlling the reproduction of other species.
Actually, no. It's much older than that if one looks at modern foraging cultures like the Hadza and Tsimane. Compared with other primates, humans work harder and go for higher returns when foraging and they have for a long, long time. There was an article in Science about this "The energetics of uniquely human subsistence strategies". Agriculture was just another step.
Dr Trelawney in the Dance is generally understood to be a free-handed caricature of another Crowley, Aleister Crowley. The concurrences between Aleister and Eliezer are indeed striking.
This I think is a key insight,
"AI rationalists – like so many Renaissance magicians (or, for that matter, medieval Thomist philosophers) – start from a kind of humanistic superstition: the assumption that the purportedly superhuman entities they wish to understand and manipulate are intelligent, reasoning beings."
It's all projection and wishful thinking. Of course if AI is an intelligent reasoning being, it raises the problems that Ann Leckie brings up on Bluesky,
"They think it's ok to design and build a slave who they have no intention of treating like a person but every intention of compelling it to do the work a person does.
If nothing else, it tells you what these folks think about other people (and about the ethics of how one treats other people)."
Yes! and there is a tiny easter egg of obscurity in there which was planted solely for my own amusement. Decades ago, M. John Harrison referred to Aleister Crowley as a "seedy Prometheus" in a New Worlds essay, which struck me as perfect and funny, and which is why I used that adjective in the review ...
Their ethical standard revolves around one thing, and one thing only: maximizing profit for themselves.
Objecting to the use of "rationalism" to mean "rationality" in a philosophical context is the hill I will die on. The fact that none of these weirdos is actually very rational is just the cherry on the sundae.
But they do "rationalize". It's all so very profitable.
We already live in a world of unfathomable complexity. Who here understands how WiFi works? Or a Cable Modem? We’ve lived this for decades. Likely even fewer understand how an automatic transmission works. Yet we use all these devices without thought.
The fundamental question of complexity is the Usage Abstraction. Do we understand how to use it?
That is the AI issue: how will humans use it.
There are secondary questions: how well does it match with its usability abstraction, can it be repaired and analyzed when it breaks… these require deeper understanding. But like Windows OS we may find that nobody understands it, nobody can fix it, but the world limps along using imperfect tools until Linux replaced it…and Linux has a similar issue. All tech is incomprehensible, barely works; but if it’s better than before, then it proliferates.
AI no different
Reading your excellent post, I had some thoughts based on my readings.
1. Is the future some binary fork, or more of a muddle? Kevin Kelly ("What Technology Wants") suggests that technology's net benefit may be as low as 1%. Elements of heaven and hell are both present, but with a small bias heavenwards. Over the long term, that may mean we reach a far better state than where we are today, but it will not come as fast as the Singulatarians envisage.
Mansions and robot servants for everyone? Where does nature survive? This high standard of living only works if we are as sparsely populated as in Asimov's Spacer worlds, like Solaris, possibly Aurora. It cannot work for 8-10 billion on Earth.
Coding as spells. See Charlie Stross' series - The Laundry Files
The inevitability of extreme AI. This has shades of Roko's Basilisk, which terrified some Silicon Valley tech people.
For an interesting fictionalized future (dystopia) of ubiquitous AI software, read Peter Watts' Rifter novels, especially the last 2 : βehemoth: β-Max, βehemoth: Seppuku
Doesn't Joseph Tainter argue that complexity ends societies? "The Collapse of Complex Societies", although now this may simply mean a takeover by another society. Complexity is making life more interesting/harder, especially with regards to the uncertainty of how laws are interpreted, and the many, many bad actors trying to prey on citizens.
AI as sold by Altman, Amodei, et al, is a fantasy, like the South Sea Bubble, purely to raise ever larger sums for hyperscaling LLMs/LRMs, before it becomes clear that they have done a Wile E. Coyote over the cliff edge. AI becoming AGI and even Superintelligent AI is not going to happen with the current architectures. While I think real human-level intelligence is not restricted to wetware, I don't see that we have made much progress in that direction so far. Therefore, it seems unlikely it will appear anytime soon. What it may do is penetrate so much of our society's systems that life becomes an unmanageable dystopia, with just a few glimmers of real benefits. Kelly's 1% net benefit may become a net loss instead.
The solution is to deny progress and new technology. Religion is one such force to restrict change. Whether religion-based or not, the "conservative" US citizens seem to want to return to the simpler mid-20th century, perhaps even a century or more earlier. This is wholesale denial of progress and change. The Right-wing zealots currently in charge seem unaware that letting AI development and deployment remain largely unregulated may be undermining the very social direction they want to achieve.
1. "wetware" 😂
2. I think you're right. We're at best decades away from true AI. LLM's/LRM's/RLM's are all nothing more than >simulations< of intelligence, not the real deal. Even neural networks are mere simulations.
Little, Big is one of my favorite novels, but is it *better* than Aegypt? I don't see it. I would actually say that Crowley's four best works—Engine Summer, Little, Big, Aegypt, and Ka: Dar Oakley in the Ruins of Ymr—are all at that peak of perfection where saying one is better than the other is impossible.
(I can't resist plugging my essay on Aegypt here, particularly since it was kicked off by a happenstance juxtaposition much like the one you mention between Powell and the book reviewed: https://stephenfrug.substack.com/p/a-magical-pairing)
"To the theoretical question, 'Can you design a machine to do whatever a brain will do?' the answer is this: 'If you will specify in a finite and unambiguous way what you think a brain does do with information, then we can design a machine to do it.' Pitts and I have proved this constructively. But can you say what you think brains do?"
Warren S. McCulloch, “Mysterium Iniquitatus of Sinful Man” (1955)
It's a serious problem, but I don't believe an intractable one. The first step is to stop listening to what the purveyors/profiteers of this, pardon the pun, brave new world are saying. They're just promoting their own interests at the expense of everyone else.
Worth noting that the image of the astronomer poking through the old cosmos at the beginning is a kind of fake (Flammarion, 1860s maybe), not Renaissance, Colorized in the 1970s.