"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.
The danger is not that AI can optimize everything. It’s that institutions may increasingly restructure reality so more domains become optimizable.
"...or that excitable claim that humanity is doomed to be superseded because of the performance of AI on this or that test". We're doomed to be superseded because that's what the purveyors and profiteers are pushing for.
Human judgement can't really be measured, much less optimized. Finding those areas outside the "sweet spot" of AI optimization is where the remaining (hopefully gainful) employment will be found.
Second book suggestion that turned into an order. You're a valuable resource... :-)
Recht points to the limits of statistical rationality.
The real issue is that once governance relies on model‑driven logic, the system drifts; because models cannot carry human values or political judgment.
This reminds me of Rittel and Webber's "Dilemmas in a General Theory of Planning"(https://urbanpolicy.net/wp-content/uploads/2015/06/Rittel-Webber_1973_DilemmasInAGeneralTheoryOfPlanning.pdf) and also to VO Key's "The Lack of a Budgetary Theory" (https://www.jstor.org/stable/1948194). This is not at all to diminish Recht's contribution, but we have known for a long time that many decisions are not amenable to optimization.
Great read, Henry. I really agree with the point that neither the rationalists nor the denialists are helpful here. I've always thought of AI as a massive data warehouse. Garbage in, garbage out. Also, agreed on the measurement points: The tasks that can be clearly measured are the same tasks computers will beat humans at. So the framework evaluating AI's limits is the same one hiding them. That's a loop that's hard to escape once you see it.
I doubt we'd be having any of these discussions if these technologies had a more accurate names - such as stochastic processing.
Most of the debates about 'AI' have been driven by the marketing needs of LLM companies, and their desperate need to pretend that this technology is in someway human so as to justify their spending.
ML is cool (if often misapplied to areas where human variation makes them a poor fit). LLMs seem less useful, unless you're a hack programmer, or a fraudster. Maybe we'll finally find a use for them when the technology companies stop exploiting the Eliza effect, and focus on more limited problems.
The IBM 360 is a great choice of image. IBM’s 1979 internal memo, later leaked, stated that a computer “can never be held accountable, and therefore must never make a management decision.” That rule was not abolished. It was engineered around. The org structures we built to diffuse accountability are a very large part of why the limits question keeps coming back.
Great review! but what happens to what falls outside the sweet spot and doesn't disappear? Does the model at least register the existence of what it can't capture? It’s a strange pattern nowadays, everyone cites cybernetics, but no one talks about displacement. The framework describes the boundary but says nothing about entropy (which is ironic, given its own origins).
One might say that the following claim is in the spirit of Edmund Husserl……. “mathematical rationality is limited in what kinds of problems it is best placed to solve but has sweet spots that have yielded remarkable technological advances.”
You had me at Dan Davies.
Thank you for this post. I will buy and read the book. And maybe share with my Rationalist friends. Maybe.
AI isn’t simply limited, if it’s targeting arbitrary output (metaphors not tumors which are spoecific) then it’s false, illusory technology.
Turing was dead wrong, LLMs demonstrate how wrong his theory is.
https://substack.com/@eventperception/p-182707220
Just read this and it confused me. It seems to conflate the existence of conflicting value judgements which obviously can't be optimised away - with domains of high statistical uncertainty and variance where it rejects use of statistical methods which seems unjustified - the uncertainty is in the data not the methods - it doesn't make the methods invalid, they simply reflect the inherent uncertainty in the domain itself so I'm not sure there is really this *sweet spot* - and finally I don't see what the proposed alternative is for these domains if mathematical modelling is rejected.
I'll definitely add this one to my reading list.
Spot-on.
I like the term “finite games” for the types of systems that Nate Silver and his ilk cotton to. The rules are highly stable, the moves are constrained, and the outcomes are calculable.
Basically like no interesting system in reality.
Great post. Thought-provoking! An aspect I'd add is the often overlooked aspect of 'legitimacy' of choices/decisions. The (in some debatable sense) optimal solution isn't always legitimate, and the legitimate solution isn't neccesarilly optimal in some clearly defined sense.