"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
LLMs, as I assume that's what you mean by AI (Machine Learning is something different), are typically deployed with a harness that provides supporting information/abilities. But that harness is not smart (it's just normal code).
The code I've seen from LLMs ranges from mediocre to okay. It does best at small scale code. Large scale code created by LLMs is insanely bad.
I take your point, I recently wrote a harness and it was surprisingly unspecial; my real point is that these models are multi-modal, usually trained on lots of different kinds of data, mostly images and text but increasingly audio and video as well.
That said, would you agree that even if LLMs are only mediocre programmers in isolation, they will eventually become superhuman? I dislike AI code quite a lot but it can write as well as me much more quickly nowadays.
Training a model that will be used for programming on images, video, or non-related textual stuff - makes the model larger, more expensive and slower. It also results in a model that is less effective than a smaller model which was constructed for a single purpose. Multi-modal stuff is a deadend, and it's driven by either the ludicrous AGI dream, or the belief that if you make the model big enough then you will no longer need expensive training.
FWIW, we reached the limits on size about 1-1.5 years ago. The improvements from increasing scale are at best mediocre, and the costs of doing so are astronomical. That's why AI companies have pivoted to non-'AI' stuff as a means to improve the outputs of their models. Any improvements from large models will be incremental at best in the future.
I find LLMs a very useful code searching technology, and pretty good at generating (small) solutions to solved problems. They appear to be good at generating boiler plate code, though I've never understood why web coders require so much boiler plate in the first case. I've noticed that people who use them heavily seem to believe that they're faster - but I've seen little evidence that this is true. The quality of PRs has declined precipitously from what I've seen, even while the number has gone up. Also nobody seems to understand the code that their LLMs are generating, which doesn't seem great.
LLMs are hopeless at architecture, and they can't solve business problems. If you need to write glue code between two incompatible data systems -> good luck. Whether the immense cost of them justifies these advantages remains to be seen (nobody is paying close to the true cost currently). Managers were excited by AI because they believed they could fire most of their programmers. Will they be so excited when they discover they still need their programmers, and AI tools cost 1/3-> 1/2 of a typical salary? Who can say.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.
A dialogue with AI working through the psychological mechanisms underneath institutional collapse — ironic identification, the Powell project as completed, the ruling class control problem. latestwriting.substack.com
The danger is not that AI can optimize everything. It’s that institutions may increasingly restructure reality so more domains become optimizable.
"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Second book suggestion that turned into an order. You're a valuable resource... :-)
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
This reminds me of Rittel and Webber's "Dilemmas in a General Theory of Planning"(https://urbanpolicy.net/wp-content/uploads/2015/06/Rittel-Webber_1973_DilemmasInAGeneralTheoryOfPlanning.pdf) and also to VO Key's "The Lack of a Budgetary Theory" (https://www.jstor.org/stable/1948194). This is not at all to diminish Recht's contribution, but we have known for a long time that many decisions are not amenable to optimization.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
1. AI systems haven't been pure language models for a long time
2. What about normal programming? It's very good at programming, approaching superhuman performance even
LLMs, as I assume that's what you mean by AI (Machine Learning is something different), are typically deployed with a harness that provides supporting information/abilities. But that harness is not smart (it's just normal code).
The code I've seen from LLMs ranges from mediocre to okay. It does best at small scale code. Large scale code created by LLMs is insanely bad.
I take your point, I recently wrote a harness and it was surprisingly unspecial; my real point is that these models are multi-modal, usually trained on lots of different kinds of data, mostly images and text but increasingly audio and video as well.
That said, would you agree that even if LLMs are only mediocre programmers in isolation, they will eventually become superhuman? I dislike AI code quite a lot but it can write as well as me much more quickly nowadays.
Training a model that will be used for programming on images, video, or non-related textual stuff - makes the model larger, more expensive and slower. It also results in a model that is less effective than a smaller model which was constructed for a single purpose. Multi-modal stuff is a deadend, and it's driven by either the ludicrous AGI dream, or the belief that if you make the model big enough then you will no longer need expensive training.
FWIW, we reached the limits on size about 1-1.5 years ago. The improvements from increasing scale are at best mediocre, and the costs of doing so are astronomical. That's why AI companies have pivoted to non-'AI' stuff as a means to improve the outputs of their models. Any improvements from large models will be incremental at best in the future.
I find LLMs a very useful code searching technology, and pretty good at generating (small) solutions to solved problems. They appear to be good at generating boiler plate code, though I've never understood why web coders require so much boiler plate in the first case. I've noticed that people who use them heavily seem to believe that they're faster - but I've seen little evidence that this is true. The quality of PRs has declined precipitously from what I've seen, even while the number has gone up. Also nobody seems to understand the code that their LLMs are generating, which doesn't seem great.
LLMs are hopeless at architecture, and they can't solve business problems. If you need to write glue code between two incompatible data systems -> good luck. Whether the immense cost of them justifies these advantages remains to be seen (nobody is paying close to the true cost currently). Managers were excited by AI because they believed they could fire most of their programmers. Will they be so excited when they discover they still need their programmers, and AI tools cost 1/3-> 1/2 of a typical salary? Who can say.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
You had me at Dan Davies.
Thank you for this post. I will buy and read the book. And maybe share with my Rationalist friends. Maybe.
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I'll definitely add this one to my reading list.
Spot-on.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Basically like no interesting system in reality.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.
A dialogue with AI working through the psychological mechanisms underneath institutional collapse — ironic identification, the Powell project as completed, the ruling class control problem. latestwriting.substack.com