I've said it once, and I'll say it again and again like a character in a Kauffman movie: It's deeply depressing that the killer app of contemporary machine learning is bullshit generation.
Along those lines:
"like many articles that take off in the discourse, I suspect that it owed its success more to its cultural resonances than its scientific results."
Any article about machine learning or AI in Nature is highly suspect and questionably reviewed. Best to treat those articles as blog posts, which thrive on cultural resonances by their very nature.
To be fair, the bullshit generation is mainly a party trick meant to dazzle investors and amuse the hoi polloi while they work on the Real Problem, "true" AI. The internal discussions on how they think they'll pull that rabbit out of the hat would be hi-larious if we could hear them. In the meantime, they need to figure out how to map the concept of reality into their "neural network" of word associations if their party trick is supposed to be used for anything involving lawyers, like life or money.
“Their” counterargument would be something like: “the best way to generate the next token is to have a model of the world/reality, and the better next-token generator, the better the world-model.”
Before I gave them up entirely, my attempts to get LLMs to write fiction from prompts had endless commonness and sameness, to the point of aping the same phrases again and again. Malkovich indeed.
We think about this problem very directly and narrowly in the language sciences, in particular because of the "empirical rejection" of basic ~Chomskian ideas LLMs appear to provide their boosters.
I believe their limitations are precisely captured by what their advocates seem to reify. That is, that their language capabilities would suggest that nature is statistically structured.
My contention is, following scientist Elliot Murphy, that "language processing is about the use and deployment of statistical information not as an end in itself, but in order to reach an underlying state of sensitivity to structure. Statistics seeking structure, not statistics driving structure."
Adapted to the question of cultural reproduction, it may be true that to some extent, cultural structures can be described statistically, but human culture itself is not driven by statistics, whereas LLMs’ representations of cultural structures *are* driven by statistics (made obvious by generalization error, or what we interpret as vacuous text).
If I can find some time in the next few months I would like to write a primer on “the language question” currently debated by linguistic theorists/computational scientists/neuroscientists. The major arguments and their implications speak to pretty much every argument about the vacant stare LLMs convey when you ask for something interesting.
I use Rewind AI to summarise Zoom meetings. It's not always accurate, but better than having minutes taken by someone who knows nothing about the topic under discussion. And it's much more reliable than me attempting, next day or next week, to remember what we talked about.
As I've said before, and as the OP implies, AI makes it easy to achieve mediocrity. In a world where huge amounts of work have always been spent on reaching this goal, there have to be some pretty big savings there.
I think it is useful for summarizing - the final post in this half-cocked series is going to be called "The Summarization Society." If you wanted to be paranoid, you could still make the It's a Cultural Trap Babel 17 argument - its convenience going together with its limitations.
I've found myself also wondering if the commercial willingness to inject "AI" into everyone's jobs, even where it's not particularly efficient or useful, is going to lead us to a "City of Ember" kind of scenario. We already live in a "fake it 'til you make it" culture where a lot of people get by in their work on unfounded confidence, rather than necessarily competence.
What happens when you live in a world where the original sources of truth & facts fall away into obscurity, drown out by the repeating waves of hallucinations and misinterpretations? Too many iterations of "bullshit in, bullshit out" between humans and machines across too many places, across too many years, means no one still has easy access to the untarnished information.
How long can your society function when the AI is telling your scientists and engineers convenient, but very wrong nonsense? Who fixes the machines, mechanisms, and systems of society once we've lost all the manuals and tools and know how?
I like the Malkovich metaphor (I still remember that scene 25 years later). Perhaps another term is “beige-ification”). Machine learning is inherently conservative as they are built on both the majority and the past.
At the moment, many are concerned with removing “hallucination”. But for creative endeavors, it strikes me that you want the opposite.
Removal of hallucination is more related to not filling in lacunae which should be left empty, than tightly following the distribution. Sometimes there is a ravine, even if the statistical model strongly suggests a bridge across would surely have been built by now, and it is prudent not to take a confident step over the cliff before checking for the presence of a bridge.
I think what we are seeing is the AI version of Human centipede. Models trained on shitposts shitting out "synthetic data" (aka overdigested info goo aka shit) that will be swallowed by another data-hungry/shiteating AI model. This metaphor doesn't quite work because it's more like an ouroboros of shiteating than an AI centipede shit chain.
You may have changed your mind about entropy as a *metaphor*, but to "increase conformity" implies increasing *literal* (information) entropy. Recall that for a countable space:
H(X) = - sum_x p(x) ln(p(x))
Where X is the space of measurable events and x is an element of X. When rare events are removed from X, this shifts weight to the more common events because the p(x) must be renormalized (something has to happen: sum_x p(x) = 1.) *Maximum* entropy is achieved when p(x) = k for all x, but even just compressing the range of p(x) still *increases* entropy.
I thought about the sloppy way I was using the term entropy when I was writing! But then remembered M. John Harrison's grumpiness with scientists who objected to his deliberately non-scientific employment of the term in Running Down and decided what the hell ...
This seems like the opposite? I am not objecting to your sloppy use of the term, but to your repudiation of your own accurate use of the term?
We readers are so perverse! You can't please all of the readers all of the time! But we ain't got a patch on writers. I have to say that I am unmoved by Harrison's grumpiness. Entropy is a subtle but important concept - more important than any concept of Harrison's - and I'd happily consign his book to oblivion if that was the price of learning what it means.
It's not surprising that a language engine trained probabilistically to seek the predictable should tend to make its mistakes in the direction of the mediocre and bland.
This looks like the mechanisation and intermediation of a process long established in mass culture.
You can see this happening right now without the benefit of an LLM in the television adaptation of "slow horses", where everything is pulled in the direction of mass market cliché.
I was thinking about a particular aspect of that last night. Frank matters a lot. So he naturally gets Name casting. And having been thus cast, his role in the narrative has to be increased — can’t have Hugo Weaving simply turn up near the end in the rain on the embankment.
But as that example may show it’s not just reversion to the mean. The show’s an adaptation and thus to some extent a translation, and there’s stuff that works in print better than it would on screen
Someone (I can't remember who) on Bluesky pointed out the brief bit where Cartwright pere is wearing thick spectacles as a plausible nod to Smiley, and how much more shambolic things are in the days of government senility than they were even in Le Carre's depressing original description. At least then they had something that was _worth_ bureaucratic plot and counterplot! Also, the plot engine is a presumably intended ironic rewrite of Karla's vulnerability and downfall.
I agree; the pull in the direction of a dumbed-down mass market dominance (in these days, 'going viral') is practically a defining characteristic of the culture of mass society in the twentieth century. Certainly a defining characteristic for critics of mass culture.
Interesting points about mass culture / mass market exhibiting same dynamics as language models. I had the same thought - that culture has an overall centripetal tendency. Thus, the cultural mainstream. I'm wondering thought what's the real expectation we'd have of a language model in this regard? Sounds like we'd like him to demonstrate that the various fringe views are there, even if not activated.
The stranger question concerns a possible mechanism that an LLM would use to select between mainstream and fringe view at a "mix" of the two that would ensure the right cultural balance to be achieved. Which begs the question, what's the right cultural balance?
A comparison with the internet is interesting as well. Seemed that for a long time it seemed to be a centrifugal, decentralising force, with the famous e2e principle, with edges connected by dumb pipes, with an emptiness in the middle of the network, instead of an intelligence. And there's anectodal evidence that this tendency continues, despite the centralizing mechanisms of platformisation: micro genres, and flows of information that according to some shift us back to oral-like communication.
Why is "The Machine Stops" still so damn relevant? Well, because endtropy still exists, and we still turn our backs on facts to listen to ourselves talk.
"Your descendants will be even in a better position than you, for they will learn what you think I think, and yet another intermediate will be added to the chain. And in time"—his voice rose—"there will come a generation that has got beyond facts, beyond impressions, a generation absolutely colourless, a generation 'seraphically free/From taint of personality,' which will see the French Revolution not as it happened, nor as they would like it to have happened, but as it would have happened, had it taken place in the days of the Machine."
Re: your term "endtropy" I thought it just a typo but found a piece with that name by the Frantics Dance Company, Iserlohn, Germany: ENDTROPY deals with the human striving for perfection and the paradox of its unattainability. The progressive optimization of one's own ego demands a high level of energy expenditure.
It comes down to incentives. If there is an incentive to mine from the edges, which there always is, then base models or indeed the application layer can do so. Its an easy fix. Using NotebookLM as the test case won't give this result but that doesn't mean it isn;t already abundantly possible.
I completely agree with the central point here, that idea of being centripetal is really powerful. It accords with my simplified metnal model of machine learning, which is essentially that it's just running an enormous, complicated regression - drawing a neat 'line of best fit' through the centre of a much messier, more dispersed reality.
I can see a couple of reasons for optimism, though. In the specific case of peer-review and originality, surely an ability to identify conformity across a field could, in principle, be the basis for assessing how far a given claim departs from that conformity. The AI doesn't need to generate something original, it just needs to identify how far soemthing departs from its (unoriginal) prediction.
The fact that the boundaries of LLMs seem to be moving away from building ever-bigger regression machines and towards chaining a bunch of them together also strikes me as a hopeful sign. Even if the LLMs themselves tend towards pablum, the possiblity of combining them in different ways might point to new locations for creativity and originality.
I've said it once, and I'll say it again and again like a character in a Kauffman movie: It's deeply depressing that the killer app of contemporary machine learning is bullshit generation.
Along those lines:
"like many articles that take off in the discourse, I suspect that it owed its success more to its cultural resonances than its scientific results."
Any article about machine learning or AI in Nature is highly suspect and questionably reviewed. Best to treat those articles as blog posts, which thrive on cultural resonances by their very nature.
I do think that the summaries it generates can be useful - but that they have obvious broader tradeoffs as discussed. The academic review article of my shtick is at https://www.dropbox.com/scl/fi/yaad3w5mofjubkpmt8qe0/AI-as-Governance-revised.docx?rlkey=26k6tujlnlmlfrrijvwwz13n6&dl=0
To be fair, the bullshit generation is mainly a party trick meant to dazzle investors and amuse the hoi polloi while they work on the Real Problem, "true" AI. The internal discussions on how they think they'll pull that rabbit out of the hat would be hi-larious if we could hear them. In the meantime, they need to figure out how to map the concept of reality into their "neural network" of word associations if their party trick is supposed to be used for anything involving lawyers, like life or money.
“Their” counterargument would be something like: “the best way to generate the next token is to have a model of the world/reality, and the better next-token generator, the better the world-model.”
Before I gave them up entirely, my attempts to get LLMs to write fiction from prompts had endless commonness and sameness, to the point of aping the same phrases again and again. Malkovich indeed.
Given the quality of the average peer review in our field, I shudder to think about the implications of training an LLM for that purpose.
We think about this problem very directly and narrowly in the language sciences, in particular because of the "empirical rejection" of basic ~Chomskian ideas LLMs appear to provide their boosters.
I believe their limitations are precisely captured by what their advocates seem to reify. That is, that their language capabilities would suggest that nature is statistically structured.
My contention is, following scientist Elliot Murphy, that "language processing is about the use and deployment of statistical information not as an end in itself, but in order to reach an underlying state of sensitivity to structure. Statistics seeking structure, not statistics driving structure."
Adapted to the question of cultural reproduction, it may be true that to some extent, cultural structures can be described statistically, but human culture itself is not driven by statistics, whereas LLMs’ representations of cultural structures *are* driven by statistics (made obvious by generalization error, or what we interpret as vacuous text).
This is interesting though I don't have nearly the grounding in language sciences I would need to respond intelligently!
If I can find some time in the next few months I would like to write a primer on “the language question” currently debated by linguistic theorists/computational scientists/neuroscientists. The major arguments and their implications speak to pretty much every argument about the vacant stare LLMs convey when you ask for something interesting.
I use Rewind AI to summarise Zoom meetings. It's not always accurate, but better than having minutes taken by someone who knows nothing about the topic under discussion. And it's much more reliable than me attempting, next day or next week, to remember what we talked about.
As I've said before, and as the OP implies, AI makes it easy to achieve mediocrity. In a world where huge amounts of work have always been spent on reaching this goal, there have to be some pretty big savings there.
I think it is useful for summarizing - the final post in this half-cocked series is going to be called "The Summarization Society." If you wanted to be paranoid, you could still make the It's a Cultural Trap Babel 17 argument - its convenience going together with its limitations.
I've found myself also wondering if the commercial willingness to inject "AI" into everyone's jobs, even where it's not particularly efficient or useful, is going to lead us to a "City of Ember" kind of scenario. We already live in a "fake it 'til you make it" culture where a lot of people get by in their work on unfounded confidence, rather than necessarily competence.
What happens when you live in a world where the original sources of truth & facts fall away into obscurity, drown out by the repeating waves of hallucinations and misinterpretations? Too many iterations of "bullshit in, bullshit out" between humans and machines across too many places, across too many years, means no one still has easy access to the untarnished information.
How long can your society function when the AI is telling your scientists and engineers convenient, but very wrong nonsense? Who fixes the machines, mechanisms, and systems of society once we've lost all the manuals and tools and know how?
I like the Malkovich metaphor (I still remember that scene 25 years later). Perhaps another term is “beige-ification”). Machine learning is inherently conservative as they are built on both the majority and the past.
At the moment, many are concerned with removing “hallucination”. But for creative endeavors, it strikes me that you want the opposite.
Removal of hallucination is more related to not filling in lacunae which should be left empty, than tightly following the distribution. Sometimes there is a ravine, even if the statistical model strongly suggests a bridge across would surely have been built by now, and it is prudent not to take a confident step over the cliff before checking for the presence of a bridge.
Here's a very gross metaphor:
I think what we are seeing is the AI version of Human centipede. Models trained on shitposts shitting out "synthetic data" (aka overdigested info goo aka shit) that will be swallowed by another data-hungry/shiteating AI model. This metaphor doesn't quite work because it's more like an ouroboros of shiteating than an AI centipede shit chain.
Garbage in, garbage out.
You may have changed your mind about entropy as a *metaphor*, but to "increase conformity" implies increasing *literal* (information) entropy. Recall that for a countable space:
H(X) = - sum_x p(x) ln(p(x))
Where X is the space of measurable events and x is an element of X. When rare events are removed from X, this shifts weight to the more common events because the p(x) must be renormalized (something has to happen: sum_x p(x) = 1.) *Maximum* entropy is achieved when p(x) = k for all x, but even just compressing the range of p(x) still *increases* entropy.
I thought about the sloppy way I was using the term entropy when I was writing! But then remembered M. John Harrison's grumpiness with scientists who objected to his deliberately non-scientific employment of the term in Running Down and decided what the hell ...
This seems like the opposite? I am not objecting to your sloppy use of the term, but to your repudiation of your own accurate use of the term?
We readers are so perverse! You can't please all of the readers all of the time! But we ain't got a patch on writers. I have to say that I am unmoved by Harrison's grumpiness. Entropy is a subtle but important concept - more important than any concept of Harrison's - and I'd happily consign his book to oblivion if that was the price of learning what it means.
It's not surprising that a language engine trained probabilistically to seek the predictable should tend to make its mistakes in the direction of the mediocre and bland.
I saw the movie years ago but somehow forgot that specific scene. Need to rewatch.
A really useful post that gets at some of what I've been seeing as well but wasn't able to explain to myself. Great stuff, thanks.
This looks like the mechanisation and intermediation of a process long established in mass culture.
You can see this happening right now without the benefit of an LLM in the television adaptation of "slow horses", where everything is pulled in the direction of mass market cliché.
And before that there was Disneyland.
I was thinking about a particular aspect of that last night. Frank matters a lot. So he naturally gets Name casting. And having been thus cast, his role in the narrative has to be increased — can’t have Hugo Weaving simply turn up near the end in the rain on the embankment.
But as that example may show it’s not just reversion to the mean. The show’s an adaptation and thus to some extent a translation, and there’s stuff that works in print better than it would on screen
Someone (I can't remember who) on Bluesky pointed out the brief bit where Cartwright pere is wearing thick spectacles as a plausible nod to Smiley, and how much more shambolic things are in the days of government senility than they were even in Le Carre's depressing original description. At least then they had something that was _worth_ bureaucratic plot and counterplot! Also, the plot engine is a presumably intended ironic rewrite of Karla's vulnerability and downfall.
I agree; the pull in the direction of a dumbed-down mass market dominance (in these days, 'going viral') is practically a defining characteristic of the culture of mass society in the twentieth century. Certainly a defining characteristic for critics of mass culture.
Interesting points about mass culture / mass market exhibiting same dynamics as language models. I had the same thought - that culture has an overall centripetal tendency. Thus, the cultural mainstream. I'm wondering thought what's the real expectation we'd have of a language model in this regard? Sounds like we'd like him to demonstrate that the various fringe views are there, even if not activated.
The stranger question concerns a possible mechanism that an LLM would use to select between mainstream and fringe view at a "mix" of the two that would ensure the right cultural balance to be achieved. Which begs the question, what's the right cultural balance?
A comparison with the internet is interesting as well. Seemed that for a long time it seemed to be a centrifugal, decentralising force, with the famous e2e principle, with edges connected by dumb pipes, with an emptiness in the middle of the network, instead of an intelligence. And there's anectodal evidence that this tendency continues, despite the centralizing mechanisms of platformisation: micro genres, and flows of information that according to some shift us back to oral-like communication.
"It was remarkable to see how many errors could be stuffed into 5 minutes of vacuous conversation."
Why is "The Machine Stops" still so damn relevant? Well, because endtropy still exists, and we still turn our backs on facts to listen to ourselves talk.
"Your descendants will be even in a better position than you, for they will learn what you think I think, and yet another intermediate will be added to the chain. And in time"—his voice rose—"there will come a generation that has got beyond facts, beyond impressions, a generation absolutely colourless, a generation 'seraphically free/From taint of personality,' which will see the French Revolution not as it happened, nor as they would like it to have happened, but as it would have happened, had it taken place in the days of the Machine."
Re: your term "endtropy" I thought it just a typo but found a piece with that name by the Frantics Dance Company, Iserlohn, Germany: ENDTROPY deals with the human striving for perfection and the paradox of its unattainability. The progressive optimization of one's own ego demands a high level of energy expenditure.
https://www.instagram.com/franticsdancecompany/p/Cy5Xoj_svZv/?locale=my&img_index=1
That's an interesting link. I wish I had done that on purpose now. : )
It comes down to incentives. If there is an incentive to mine from the edges, which there always is, then base models or indeed the application layer can do so. Its an easy fix. Using NotebookLM as the test case won't give this result but that doesn't mean it isn;t already abundantly possible.
I completely agree with the central point here, that idea of being centripetal is really powerful. It accords with my simplified metnal model of machine learning, which is essentially that it's just running an enormous, complicated regression - drawing a neat 'line of best fit' through the centre of a much messier, more dispersed reality.
I can see a couple of reasons for optimism, though. In the specific case of peer-review and originality, surely an ability to identify conformity across a field could, in principle, be the basis for assessing how far a given claim departs from that conformity. The AI doesn't need to generate something original, it just needs to identify how far soemthing departs from its (unoriginal) prediction.
The fact that the boundaries of LLMs seem to be moving away from building ever-bigger regression machines and towards chaining a bunch of them together also strikes me as a hopeful sign. Even if the LLMs themselves tend towards pablum, the possiblity of combining them in different ways might point to new locations for creativity and originality.
I like this view, and I think you've inspired me to write a post in response, so well done. Subscribed!