"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
LLMs, as I assume that's what you mean by AI (Machine Learning is something different), are typically deployed with a harness that provides supporting information/abilities. But that harness is not smart (it's just normal code).
The code I've seen from LLMs ranges from mediocre to okay. It does best at small scale code. Large scale code created by LLMs is insanely bad.
I take your point, I recently wrote a harness and it was surprisingly unspecial; my real point is that these models are multi-modal, usually trained on lots of different kinds of data, mostly images and text but increasingly audio and video as well.
That said, would you agree that even if LLMs are only mediocre programmers in isolation, they will eventually become superhuman? I dislike AI code quite a lot but it can write as well as me much more quickly nowadays.
Training a model that will be used for programming on images, video, or non-related textual stuff - makes the model larger, more expensive and slower. It also results in a model that is less effective than a smaller model which was constructed for a single purpose. Multi-modal stuff is a deadend, and it's driven by either the ludicrous AGI dream, or the belief that if you make the model big enough then you will no longer need expensive training.
FWIW, we reached the limits on size about 1-1.5 years ago. The improvements from increasing scale are at best mediocre, and the costs of doing so are astronomical. That's why AI companies have pivoted to non-'AI' stuff as a means to improve the outputs of their models. Any improvements from large models will be incremental at best in the future.
I find LLMs a very useful code searching technology, and pretty good at generating (small) solutions to solved problems. They appear to be good at generating boiler plate code, though I've never understood why web coders require so much boiler plate in the first case. I've noticed that people who use them heavily seem to believe that they're faster - but I've seen little evidence that this is true. The quality of PRs has declined precipitously from what I've seen, even while the number has gone up. Also nobody seems to understand the code that their LLMs are generating, which doesn't seem great.
LLMs are hopeless at architecture, and they can't solve business problems. If you need to write glue code between two incompatible data systems -> good luck. Whether the immense cost of them justifies these advantages remains to be seen (nobody is paying close to the true cost currently). Managers were excited by AI because they believed they could fire most of their programmers. Will they be so excited when they discover they still need their programmers, and AI tools cost 1/3-> 1/2 of a typical salary? Who can say.
Sorry for not replying your comment, I appreciate it even though I completely disagree haha.
First of all, I'm kind of shocked to hear you discuss the multi-modality this way.
(1) It has been repeatedly demonstrated that language models get back at language tasks when trained with other data modalities
(mPLUG-Owl)
To my knowledge the leading hypothesis is that the networks encode internal world models during training which is why they can infer/reinforce from many images/videos of cats fighting dogs that dogs fight cats linguistically.
I think it's true that the breadth of AI scaling strategies has widened but I would question whether this is really a sign of weakness in older approaches, where, empirically, do you see this plateau hitting or coming? The DCs are still being built, the parameter counts keep getting bigger and the models are still performing better.
Personally I do think coding agents are surprisingly bad at architecture for now but I also used to believe they would bad at functional decomposition and abstraction; if you can think fast enough, the innate weaknesses of models do kind of evaporate.
The model cost declines at the same level performance are the truly astounding thing; on at least 9x annual price declines for the same performance. Within a few years, everyone will be able to use today's frontier very cheaply.
> To my knowledge the leading hypothesis is that the networks encode internal world models during training which is why they can infer/reinforce from many images/videos of cats fighting dogs that dogs fight cats linguistically.
That's not what's happening, and anyone who claim's otherwise is either ignorant or being deliberately deceptive.
The images/videos are loaded with descriptive text/tokens. So if an LLM gets a lot of descriptive text that says cats fight dogs, then it will be biased s.t. it if cats come up, it is more likely to mention them fighting dogs. One of the reasons I hold the AI industry in contempt, is that they repeatedly go in for this kind of deliberately deceptive mystification.
Also, how is a model that knows cats can fight dogs useful for programming. But the cost of using that model is going to be far higher, just because it's bigger. It's not a smart approach.
> I think it's true that the breadth of AI scaling strategies has widened
I have no idea what this means. To get improvement in technologies that contain LLMs, AI companies have been wrapping them with more and 'ordinary' code. There was a significant improvement in the performance of LLM models that ended (according to AI researchers) in 2025. Using LLMs in this way strikes me as perfectly reasonable (using a giant computer to do so... less smart), but this is not 'intelligence'. It's just software.
> The DCs are still being built, the parameter counts keep getting bigger and the models are still performing better.
Performing better by what measurement. The metrics out there are constructed by AI enthusiasts and are unsurprisingly bullshit. If humans look for confirmation of their beliefs, they will generally find them. The reason the scientific method exists, is to counteract this human bias. AI evaluators are not using the scientific method. Instead what we're seeing is the kind of evaluations that would make paranormal investigators blush. It's bad.
> Personally I do think coding agents are surprisingly bad at architecture for now
I mean it's only surprising if you think they're intelligent. If you don't think they're intelligent, but are simply good at providing common answers to common questions, then this is utterly predictable.
> but I also used to believe they would bad at functional decomposition and abstraction;
I have no idea as to what you mean by this (I know what the words mean - I just don't understand what you mean by this)?
> if you can think fast enough, the innate weaknesses of models do kind of evaporate.
Again, this doesn't mean anything to me.
> The model cost declines at the same level performance are the truly astounding thing;
1. We do not have meaningful metrics for AI performance (or even any criteria for what that might look like).
2. Costs for the commercial models do not exist. For open models costs are pretty difficult to come by because these things have stochastical performance. Similar tasks can have different costs.
> on at least 9x annual price declines for the same performance. Within a few years, everyone will be able to use today's frontier very cheaply.
I have no idea where the dataset for such a claim would even exist. But the page you link to is an example of bad evaluations (epoch AI are one of the worst offenders). They tested it on data that is in the training data. I mean. Cool. That's not a useful metric, and we don't know if this tells us something about changes in training data, training methodologies, training time, size of the model or just that AI companies have got better at gaming the common metrics. It's garbage.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.
The danger is not that AI can optimize everything. It’s that institutions may increasingly restructure reality so more domains become optimizable.
"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Second book suggestion that turned into an order. You're a valuable resource... :-)
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
This reminds me of Rittel and Webber's "Dilemmas in a General Theory of Planning"(https://urbanpolicy.net/wp-content/uploads/2015/06/Rittel-Webber_1973_DilemmasInAGeneralTheoryOfPlanning.pdf) and also to VO Key's "The Lack of a Budgetary Theory" (https://www.jstor.org/stable/1948194). This is not at all to diminish Recht's contribution, but we have known for a long time that many decisions are not amenable to optimization.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
1. AI systems haven't been pure language models for a long time
2. What about normal programming? It's very good at programming, approaching superhuman performance even
LLMs, as I assume that's what you mean by AI (Machine Learning is something different), are typically deployed with a harness that provides supporting information/abilities. But that harness is not smart (it's just normal code).
The code I've seen from LLMs ranges from mediocre to okay. It does best at small scale code. Large scale code created by LLMs is insanely bad.
I take your point, I recently wrote a harness and it was surprisingly unspecial; my real point is that these models are multi-modal, usually trained on lots of different kinds of data, mostly images and text but increasingly audio and video as well.
That said, would you agree that even if LLMs are only mediocre programmers in isolation, they will eventually become superhuman? I dislike AI code quite a lot but it can write as well as me much more quickly nowadays.
Training a model that will be used for programming on images, video, or non-related textual stuff - makes the model larger, more expensive and slower. It also results in a model that is less effective than a smaller model which was constructed for a single purpose. Multi-modal stuff is a deadend, and it's driven by either the ludicrous AGI dream, or the belief that if you make the model big enough then you will no longer need expensive training.
FWIW, we reached the limits on size about 1-1.5 years ago. The improvements from increasing scale are at best mediocre, and the costs of doing so are astronomical. That's why AI companies have pivoted to non-'AI' stuff as a means to improve the outputs of their models. Any improvements from large models will be incremental at best in the future.
I find LLMs a very useful code searching technology, and pretty good at generating (small) solutions to solved problems. They appear to be good at generating boiler plate code, though I've never understood why web coders require so much boiler plate in the first case. I've noticed that people who use them heavily seem to believe that they're faster - but I've seen little evidence that this is true. The quality of PRs has declined precipitously from what I've seen, even while the number has gone up. Also nobody seems to understand the code that their LLMs are generating, which doesn't seem great.
LLMs are hopeless at architecture, and they can't solve business problems. If you need to write glue code between two incompatible data systems -> good luck. Whether the immense cost of them justifies these advantages remains to be seen (nobody is paying close to the true cost currently). Managers were excited by AI because they believed they could fire most of their programmers. Will they be so excited when they discover they still need their programmers, and AI tools cost 1/3-> 1/2 of a typical salary? Who can say.
Sorry for not replying your comment, I appreciate it even though I completely disagree haha.
First of all, I'm kind of shocked to hear you discuss the multi-modality this way.
(1) It has been repeatedly demonstrated that language models get back at language tasks when trained with other data modalities
(mPLUG-Owl)
To my knowledge the leading hypothesis is that the networks encode internal world models during training which is why they can infer/reinforce from many images/videos of cats fighting dogs that dogs fight cats linguistically.
I think it's true that the breadth of AI scaling strategies has widened but I would question whether this is really a sign of weakness in older approaches, where, empirically, do you see this plateau hitting or coming? The DCs are still being built, the parameter counts keep getting bigger and the models are still performing better.
Personally I do think coding agents are surprisingly bad at architecture for now but I also used to believe they would bad at functional decomposition and abstraction; if you can think fast enough, the innate weaknesses of models do kind of evaporate.
The model cost declines at the same level performance are the truly astounding thing; on at least 9x annual price declines for the same performance. Within a few years, everyone will be able to use today's frontier very cheaply.
https://epoch.ai/data-insights/llm-inference-price-trends
Curious what you have to say :)
> To my knowledge the leading hypothesis is that the networks encode internal world models during training which is why they can infer/reinforce from many images/videos of cats fighting dogs that dogs fight cats linguistically.
That's not what's happening, and anyone who claim's otherwise is either ignorant or being deliberately deceptive.
The images/videos are loaded with descriptive text/tokens. So if an LLM gets a lot of descriptive text that says cats fight dogs, then it will be biased s.t. it if cats come up, it is more likely to mention them fighting dogs. One of the reasons I hold the AI industry in contempt, is that they repeatedly go in for this kind of deliberately deceptive mystification.
Also, how is a model that knows cats can fight dogs useful for programming. But the cost of using that model is going to be far higher, just because it's bigger. It's not a smart approach.
> I think it's true that the breadth of AI scaling strategies has widened
I have no idea what this means. To get improvement in technologies that contain LLMs, AI companies have been wrapping them with more and 'ordinary' code. There was a significant improvement in the performance of LLM models that ended (according to AI researchers) in 2025. Using LLMs in this way strikes me as perfectly reasonable (using a giant computer to do so... less smart), but this is not 'intelligence'. It's just software.
> The DCs are still being built, the parameter counts keep getting bigger and the models are still performing better.
Performing better by what measurement. The metrics out there are constructed by AI enthusiasts and are unsurprisingly bullshit. If humans look for confirmation of their beliefs, they will generally find them. The reason the scientific method exists, is to counteract this human bias. AI evaluators are not using the scientific method. Instead what we're seeing is the kind of evaluations that would make paranormal investigators blush. It's bad.
> Personally I do think coding agents are surprisingly bad at architecture for now
I mean it's only surprising if you think they're intelligent. If you don't think they're intelligent, but are simply good at providing common answers to common questions, then this is utterly predictable.
> but I also used to believe they would bad at functional decomposition and abstraction;
I have no idea as to what you mean by this (I know what the words mean - I just don't understand what you mean by this)?
> if you can think fast enough, the innate weaknesses of models do kind of evaporate.
Again, this doesn't mean anything to me.
> The model cost declines at the same level performance are the truly astounding thing;
1. We do not have meaningful metrics for AI performance (or even any criteria for what that might look like).
2. Costs for the commercial models do not exist. For open models costs are pretty difficult to come by because these things have stochastical performance. Similar tasks can have different costs.
> on at least 9x annual price declines for the same performance. Within a few years, everyone will be able to use today's frontier very cheaply.
I have no idea where the dataset for such a claim would even exist. But the page you link to is an example of bad evaluations (epoch AI are one of the worst offenders). They tested it on data that is in the training data. I mean. Cool. That's not a useful metric, and we don't know if this tells us something about changes in training data, training methodologies, training time, size of the model or just that AI companies have got better at gaming the common metrics. It's garbage.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
You had me at Dan Davies.
Thank you for this post. I will buy and read the book. And maybe share with my Rationalist friends. Maybe.
AI isn’t simply limited, if it’s targeting arbitrary output (metaphors not tumors which are spoecific) then it’s false, illusory technology.
Turing was dead wrong, LLMs demonstrate how wrong his theory is.
https://substack.com/@eventperception/p-182707220
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I'll definitely add this one to my reading list.
Spot-on.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Basically like no interesting system in reality.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.