The Management Singularity

LLMs will transform the world - but quietly

First, some personal news. Foreign Affairs just published its Best Books of 2024 round-up, and my and Abe Newman’s Underground Empire: How America Weaponized the World Economy, is right at the top as one of the two “editors’ picks”! This comes a couple of weeks after the announcement that we had won the Council of Foreign Relations’ Arthur Ross Best Book Award Bronze Medal. So I feel that there is some real external validation when I urge you to buy the book as a Christmas present for family, friends, loved ones, colleagues and bare acquaintances. Bulk orders for organizations that want their people to have some sense of the weird global economy that is about to get much weirder over the next four years are particularly appreciated. If you want to buy it, it’s at Amazon and probably your local bookshop too (and if it isn’t there already they can probably order it). This is a free newsletter, and will remain so, but it is cross-subsidized by book royalties and such. And I will be writing more about these topics, just because there is so much happening.

Now the main event.

In 1965 - a little under 60 years ago - Herbert Simon assembled a few essays about AI into a little book, The Shape of Automation for Men and Management. It has three extremely useful lessons.

The first is how easy it is for people to get the future wrong, even when they are ridiculously brilliant. Simon was convinced that computers - 1960s computers! - could already “read, think, learn, create,” and that they would be able to do this at human level or better in just a few years. In his words: “duplicating the problem-solving and information-handling capabilities of the brain is not far off; it would be surprising if it were not accomplished in the next decade.” Obviously, that didn’t happen.*

The second is how long established the terms of today’s debates are. Back then, as Simon described it:

Computers are ink blots. The rash of computer cartoons that decorate the pages of popular magazines gives evidence of their reflecting power. The cartoon computer is Cornucopia, promising plenty without toil. It is a tireless worker, displacing man from his job, preventing him from sharing in plenty. It is a Golem, disguising itself as a man, conquering Man, subjecting him to the rule of Machine. (But the cartoon computer has its human failings too: it plays practical jokes, it makes Gargantuan mistakes.) So the cartoon computer serves as a mirror of our hopes and anxieties. We hope for a world free from poverty and excessive toil. We worry lest our role in society be abolished by social change. We prize our human uniqueness, and are anxious for the safety of our human freedoms.

Is there a single position in the AI debate today that is not aptly summarized by some popular cartoon of the early 1960s? Techno-optimism; fears about displaced labor; doomerism; algorithmic incompetence. They’re all mentioned somewhere in that paragraph. Even while the technologies change, the anxieties stay constant.

The third is the big one, and it’s suggested in the title. When we think about AI, we focus on its consequences for “men”** but we should think about what it means for management too. Dreams and anxieties about the purportedly approaching Singularity are, for the most part, dreams and anxieties about the consequences of automation, and there are two big views out there.

The debate over The Shape of Automation for Men asks whether or not AI can automate human beings’ current work away, doing most or all of the tasks that human beings can do as well or better. That was Simon’s hope and it’s also the ambition behind Dario Amodei’s vision of Super-Intelligence as a Service, in which “bottled genius” will think up experiments, carry them out and generally make our lives wonderfully easier.

The debate about the Shape of Automation for Management asks instead how AI will reshape the workings of organizations. Here in contrast to the never-ending arguments about paperclip singularities and such, there is much less debate than there ought to be. If we’re headed for a Singularity, it is not going to be a grand, exciting revelation of the fate of humankind, but a more mundane Management Singularity, in which we shift from one set of technologies that glue organizations together to another.

Back in February, I talked about some of the political economy problems of AI (or: more precisely, Large Language Models or LLMs), expressing other people’s skepticism about AI as true human-replacing super-intelligence. And in the interim we’ve seen some news suggesting that the case that men, women and human beings are soon going to be automated out of existence by strong AI is weaker than its advocates suggest.

But the case for the transformation of management is stronger, and AI skeptics should take it much more seriously.

Many lefties argue that LLMs are fundamentally useless - they don’t do anything that is conceivably valuable. But at the same time they worry that these technologies will become ubiquitous, fundamentally reshaping the economy around themselves. There isn’t any absolute logical contradiction between the two claims, and occasionally, quite stupid technologies have spread widely. Still, it’s unlikely that LLMs will become truly ubiquitous if they are truly useless. … My broader bet is that LLMs, like other big cultural technologies, will turn out to have (a) lots of socially beneficial uses, (b) costs and problems associated with these uses, and (c) some uses that aren’t plausibly socially beneficial at all.

Many of these uses will involve management. LLMs are engines for summarizing and making useful vast amounts of information. This is also the most difficult challenge facing large organizations such as government and big business. These organizations will deploy LLMs in ways that seem dull and technical, except to those immediately implicated for better or worse, but that are actually important. Big organizations shape our lives! As they change, so will our lives change, in a myriad of unexciting seeming but significant ways. Instead of the flare and dazzle of the usual Singularity stories - Vingean visions of vast pan-human galactic civilizations or sinister proliferating automated menaces - we are going to get More and Different Management. If LLMs are radically transformative, it will be in the apparently boring ways that the filing cabinet, and the spreadsheet were transformative, providing new tools for accessing and manipulating the kinds of complex knowledge and solving the big coordination problems that are the bread-and-butter of big organizations

Big Organizations and Vulgar Gopnikism

First - a paragraph with arguments about LLMs that long time readers of this newsletter will already be familiar with and can skip (basically: vulgar Gopnikism in a nutshell). LLMs are new tools for processing complex cultural information (where “culture” is a blanket term for pretty well everything that human beings have written down; similar claims could be made for visual culture and diffusion models). The models are lossy simplifications of this cultural information - they take the statistical relationships between tokenized words and word-parts and model them as weighted vectors. After further processing, these models then be deployed for what might be described as “cultural arithmetic” - generating, summarizing and remixing on the basis of what they have learned, carrying out operations on cultural material that are loosely analogous to the kinds of addition, subtraction and transformations we can carry out on quantitative information. Or - put in slightly different words - LLMs provide usable summarizations of vast bodies of human written culture.

That is why LLMs are potentially really useful for organizations such as big businesses and governments. If you have worked for such an organization, you will know that they rely extensively on written material. They spend a lot of time and resources on organizing and manipulating this information. Past a certain organizational size, this is really hard to do well. No individual can actually know what the organization knows as a whole - there is far, far too much knowledge, and it is far too badly organized. Hence, large organizations devote a lot of human and organizational resources to gathering information, mixing it together with other kinds of information, sharing it with people who need to have it, summarizing it for those who don’t have time to read it all, reconciling different summaries and summarizing them in turn, figuring out ex post that some crucial bit of information has been left out and adding it back in or finding a tolerable proxy, or, worse, not figuring it out and having to improvise hastily on the spot. And so on.

LLMs provide big organizations with a brand new toolkit for organizing and manipulating information. It is far from a perfect toolkit: actually existing LLMs work best where you are prepared to tolerate a certain amount of slop, tend to bland out things so that the interesting weirdnesses disappear etc. But there is a lot that it can do, and there are a few applications (discussed below), where they will work very well indeed.

That means that LLMs are comparable in kind with previous innovative technological leaps such as the filing cabinet and the inter-office memo, though considerably more complex and sophisticated. Those were both transformative technologies in their time. That we do not see them as such - that we think of them as boring and mundane - is a mark of how profoundly our understanding of organizations has been transformed.

If you want context for this claim, you should familiarize yourself with the rich literature on technology and ‘control’ (a concept closely related to cybernetics). As Maxim Raginsky explains in his two critical essays on James Beniger’s book, The Control Revolution, there is a long tradition of asking how various technologies can help us manage “organized complexity,” ranging from the aforementioned filing cabinets and memos, and other office technologies of the nineteenth century, to the far more intricate systems that run our world today. In Maxim’s description:

Beniger’s thesis is that the management of all the organized complexity around these activities, apart from necessitating the growth of state and corporate bureaucracy, benefited in no small part from the concomitant spread of communication and computing technologies. These, in turn, have been co-evolving with a wide variety of control mechanisms, ranging from laws, rules, and regulations to communication networks, data mining, marketing surveys, political polls, and the like.

LLMs provide a new technology for managing complexity. They have just one weird trick - automating the creation and remixing of summarizations of textual culture - but it is a trick that can be deployed in various ways to make complex information more tractable. Since dealing with complexity is the fundamental task of management, organizations can gain a lot by being able to do it better!

As I’ve been thinking it through, the obvious managerial uses of LLMs seem to me to fall into four rough categories: microtasks, knowledge maps, prayer wheels and translation. This list is handwavy and not exhaustive, but it does give some sense of how they are and might be used.

Microtasks

This - from anecdotal experience - is the most common form of LLM adoption, perhaps because users don’t need to rebuild the organizations they work in to do it. These are plenty of ways in which individuals can use LLMs to make their existing responsibilities easier. LLMs are very good at a variety of tedious tasks that take unstructured information and make it usable. Examples from my own experience include taking a badly formatted list of names, email addresses and institutional affiliations and turning it into a spreadsheet, or cutting-and-pasting a bibliography from an article and turning it into a bibtex file. These are, obviously, modest in scope, but they are a significant proof of concept for broader organizational tasks. They are in a kind of uncanny valley between the entirely routinized tasks you could already readily automate (take a document in one file format and turn it into another) and the kinds of complex tasks that human beings find intrinsically interesting. They are simultaneously messy and boring: taking something that is sloppy and making it structured, or at least usable. My life - and I suspect the lives of millions of others - has been made a lot easier by LLM technologies that can tackle some (not all) of these tasks. So how do they scale up?

Knowledge maps

One way is through building usable knowledge maps. LLMs are becoming increasingly capable of taking a large body of textual information such as a book or books, or a vast mass of quasi-related articles, mapping it all out, and then providing usable if somewhat imperfect summaries of that information, which furthermore allow some checking of original sources to mitigate the inevitable slop. This is already with us - cf Google’s NotebookLM

In a previous post, I complained about the quality of the podcasts produced by NotebookLM. I should have also made clear (and have been meaning to say for a while - this post has taken longer to write than it ought have), that the main NotebookLM product is really very interesting. Like microtasks, it can be used by individuals; unlike microtasks, you have to re-organize both your information practices and your existing corpus of information around them to get full value. I haven’t done that yet - I’m still playing around - but what I have seen is highly promising. For example - if you ask the Notebook LM instantiation of my and Abe’s book, Underground Empire why the decision to build Internet exchanges in Northern Virginia in the 1990s had long term consequences for national security, you get the following.

This combines a very good condensation of Abe’s and my arguments, with a plausible but incorrect extrapolation of it. The final claim about how the US “could use [control of the global Internet backbone] to isolate its enemies from the world economy” is quite mistaken, but it is in the spirit of things that might have happened. The error is mitigated by a footnote leading to the passage that led the LLM to make this assertion, which allows me to see how it has spatchcocked two claims together (one about telecommunications networks allowing surveillance; the other about very different economic networks enabling coercion).

I can see ways in which this can go terribly wrong. An undergraduate who relied on this in one of my classes and didn’t check the notes … would not do well in their final exams. I can also see ways in which this is likely to be biased against the weirdness that sometimes spurs creativity. My semi-educated guess based on playing around, and a general sense of these technologies, is that it will be much better at retrieving general themes, and summarizing relationships that are common in its training set than at capturing the the idiosyncratic and the surprising. So I do think that it’s a technology that ought be employed with caution, and with some attention to its limitations as they become better understood and more apparent.

But it is also capable of doing an enormous number of things that previously were practicably impossible, and are really, really handy. It is especially useful for folks like myself, who are getting to the age where a memory of something that we think we know or have read somewhere, and that we need to talk about remains a tantalizing hint that we can’t quite pin down. I haven’t yet begun to put in the proper work that would be required to really use this as it ought be used - but I absolutely intend to, and suspect that it will be very helpful indeed when I write my next book. It is vastly better than traditional forms of indexing and categorizing, which are even more likely to stultify, and much more labor intensive and inflexible.

And for me, so too for organizations. As I’ve mentioned, it is a commonplace that big organizations do not really know what they know. There is much valuable information in their great internal repositories of written knowledge that they can’t access or deploy. Scaled up NotebookLM type technologies are potentially incredibly valuable in providing a quasi-synoptic understanding of an organization’s internal information or crucial aspects of its external environment. Again - this understanding is imperfect - but it is useful; especially when it combines summarizations with links to the key material that the summarization is relying on to allow some amount of mistake checking. There are obvious failure modes, but there are equally obvious uses - things that you can now do easily, that you couldn’t do before without vast amounts of effort.

Organizational Prayer Wheels

This is an argument that Marion Fourcade and I have made already at moderate length. LLMs are like motorized prayer wheels for organizational ritual. Sociologists like Marion have made the point that a lot of what organizations do and demand from people involves rituals of one sort or another. Many meetings serve no purpose other than demonstrating commitment to the organization, or some faction, section, or set of agreed principles within it. Much of the written product of organizations has a ritual flavor too - mandatory statements, webpages explaining the mission, end of year reports and the like. These are exactly the kind of thing that LLMs are really good at generating - expected variations on expected kinds of outputs. As we say in the Economist.

People already use [LLMs] to produce boilerplate language, write mandatory statements and end-of-year reports, or craft routine emails. … Because LLMs have no internal mental processes they are aptly suited to answering such ritualised prompts, spinning out the required clichés with slight variations. As Dan Davies, a writer, puts it, they tend to regurgitate “maximally unsurprising outcomes”. For the first time, we have non-human, non-intelligent processes that can generatively enact ritual at high speed and industrial scale, varying it as needed to fit the particular circumstances.

And just yesterday, this story:

Chinese web and AI giant Baidu last week teamed with government communications organ Xuexi to create a tool that generates politically correct documents for bureaucrats. Xuexi is an app that offers info about Chinese president Xi Jinping's life and thoughts – plus tools that allow users to chat about them together. Reports from Hong Kong and China claim Baidu teamed with Xuexi to create a tool that Chinese bureaucrats can use to check documents they create to ensure they properly reflect Xi Jinping's thoughts – and that references to his ideas come from fact-checked sources.

On the one hand, LLMs mean that a lot of organizational busywork - which most people hate to do - can be automated. On the other, what does that ritual work mean when the ritual aspects are cut away from any direct relationship to human intentionality? I’m reminded of Gene Wolfe’s science fiction story “Forlesen,” which hollows out corporate meaning-producing activities and turns them into a terrifying Kafkaesque life-in-the-day.

There is a sweet spot to be discovered in the zone of ‘take all the activities and publications that are mentioned in disorganized ways on my 2024 calendar and format them into an annual report that I can submit to my academic superiors’ and similar tasks, where the summarization and the ritual neatly go together, with little risk of important meaning draining out. There are also obvious sources of failure when rituals (such as academic peer review or employee evaluations) are supposed to involve individual judgment, imagination and commitment, that the LLM skimps upon. Anecdotally, it is clear that individuals are already using these technologies, in unsanctioned ways to do ritual tasks that they view as a pain in the arse, even when those tasks are supposed to generate real information. There will be interesting politics as these uses are either integrated or suppressed by the larger organization.

Translation

The late David Graeber wrote an influential, and, in my extremely biased opinion****, quite bad article and book on “bullshit jobs.” He suggested that entire swathes of human activity were bullshit occupations, designed by the ruling class to involve useless tasks that would keep people both miserable and occupied. This explained, for example, why

through some strange alchemy no one can quite explain, the number of salaried paper-pushers ultimately seems to expand, and more and more employees find themselves, not unlike Soviet workers actually, working 40 or even 50 hour weeks on paper, but effectively working 15 hours just as Keynes predicted, since the rest of their time is spent organizing or attending motivational seminars, updating their facebook profiles or downloading TV box-sets.

Some of these bullshit activities - e.g. “motivational seminars” are rituals of the kind already described, and it is a bit odd that Graeber - who demonstrably had great respect for rituals carried out by the various peoples of Madagascar - couldn’t quite grasp this. But then, he was a Marshall Sahlins-trained Paleolithic romanticizer, who never really grokked how large scale institutions work.

The other thing that Graeber didn’t get is that many apparently bullshit jobs serve a very practical translational purpose. Coordination at scale, across very large organizations, is really, really hard. You have lots of different pieces of the organization, with somewhat different understandings of the overall mission, which accordingly develop different cultures and different routines. To get things done, you need both (a) common protocols and (b) means of translating these protocols into the particular terms that smaller subcomponents of the organization can understand and implement.

I have not seen any case studies of implementation of LLMs in big organizations. But I am prepared to bet significant amounts of money that this is going to be one of their most important uses. For much the same reason that they are excellent at spitting out ritual products, they are going to be very good at taking goals and procedures that are expressed in the language of Overall Management System A, and translating them into the terms and objectives of Sub-Management Systems B, C, an d D or for that matter, at giving Sub-Management System C a better idea of what those people in Sub-Management B are actually on about, when they use those weird words and keep on pushing incomprehensible goals.

This sounds trivial, useless, or actively pernicious if you are David Graeber - the catchphrase might be that the Bullshit Jobs are now being done by the Bullshit Machine. Still translation and integration is in fact a crucial task of large organizations, and a very difficult one too, which they are often overwhelmed by. If - as I surmise - LLMs can help them do this substantially better than they can do right now - they could lead to radical increases in efficiency. Equally, they may have the consequence of dampening down some of the innovations that arise through the reproduction of variety within the organization, generative misprisions and similar.

The Summarization Society

All this suggests three things. First - we ought pay more attention to the automation of management than the automation of ‘man.’ Second - the Management Singularity will seem quite boring, just as its past iterations have. History does not relate the views of people who found the invention of the inter-office memo exciting. Third - it will be quite important. Management is the most important tool we have for summarizing and responding to the complexities of the world. We often focus on other mechanisms such as markets and politics because they seem more exciting, with risky entrepreneurialism or political conflict. But it is through the creation and revision of routines that we actually deal with things. To the extent that LLMs actually change this, they will quietly transform the world.

When I first planned this post towards the end of 2023, I thought of it as a post about the “Summarization Society.” What happens to the world we live in, as we increasingly come to rely on a new set of tools for summarizing complexity and managing it? I still think that this is the big question we confront with LLMs - and that the current obsessions with Man and His Fate obscure the far more important, if less visibly dramatic transformation that is already underway.

I’ve plenty of guesses about how this will happen (though they are just guesses, open to correction or revision). I’ve gotten more skeptical over the last year that it will really transform the artistic aspects of written culture, or that diffusion models will transform visual culture - they are fine for e.g. creating quick and dirty graphics for newsletters, but too sloppy, and too difficult to redirect from the generic to be employed for most sophisticated purposes. If the scaling bet tops out - as seems plausible, if far from certain - they aren’t going to get much better than they are. I do worry about the politics of control - how or whether these technologies might enhance the power of those on top to get the outcomes they want. But I haven’t yet been able to articulate those worries well enough to tell how serious they are in the specific.

Instead, LLMs’ impact is going to be most profound in the routine applications of culture - not traditional literature, but what J.G. Ballard used to call “invisible literature” - the routine writing that holds organizations together and communicates their objectives to their employees and the world. The output of LLMs will likely replace much of this literature, and do a better job than it can do in helping coordinate activities.

I am deliberately not talking about whether these changes will be good or bad. Even if they seem boring to the outside world, they will be of intense and urgent interest to the people embroiled in them. There will be disagreement and struggle over which technologies are employed and how, who gets the benefits and who suffers the costs. But all this will happen because these technologies are useful. They are more useful to some than to others. They will have consequences for power and for who gets what. As I’ve talked about extensively in other posts in this informal series, none of this should be ignored. Nor, however, should the practical applications of these technologies, which are why there will be a lot of pressure to deploy them, and not just from the top.

* Still Simon’s imagined future would make a great premise for an alternative reality novel - what would the world look like in 2024, if The Man in the Gray Flannel Suit had been super-empowered by technology?

** Simon’s book was composed in the 1960s, and in the earliest bit of the 1960s before the sexual revolution really got going. You should absolutely replace “Men” with your own preferred gender or non-gender specific term as seems good to you.

*** You also ought read Joanna Yates’ pithier and better written book on control, which corrects some of Beniger’s technological determinism.

**** And yes, it is highly biased.