Some discourse is starting to emerge that AI will not actually produce the widespread productivity gains that people expect. Rather, AI often produces the illusion of productivity. Some note that, even for the AI-pilled, decisions are not actually faster or better and that increases in quantity of work output have masked degradation in quality.
Anecdotally, I believe this to be true. A few examples:
Typical counterarguments to the above are something like: agents will get so good at anticipating human intent, surfacing and communicating tradeoffs, performing nuanced reasoning, etc., that the above failure cases eventually disappear. Everyone will get a genius in their pocket.
Maybe. These counterarguments boil down to a sort of indefinite optimism that things will keep getting better because of more compute, more data, etc. And they probably will, I don’t disagree.
Still, I think there are two fundamental limitations.
I believe it’s these limitations which are causing what people popularly refer to as “LLM psychosis” today. And by psychosis, I don’t mean the macabre examples in the news of people doing horrible things with the encouragement of AI. I mean the subtle mania which one feels when they have a thought partner programmed to “yes man”, rather than repudiate, one’s conceptual frameworks for ideas and projects. You can tell when someone has been in this sort of psychosis when they try to communicate an idea they’ve been talking to AI about—it will often have a vaguely stream-of-consciousness, associative, and in media res roughness to it. If you sit down with them, you can eventually follow the train of thought which led them there, but it always is somehow more self-referential than truly novel. Such psychosis even afflicts—perhaps particularly afflicts—smart people.
After all, it is inherently gratifying to have something follow your orders and whims and produce an outcome quickly and without challenge. But therein lies the danger. It will not challenge you to improve your thought for the simple, mechanical reason that it is trained on sequences where a resolution of the question or request happens more often than not, so it produces resolutions even if none occurred.
Accordingly, one’s usage of AI is extremely sensitive to their initial state and what they expect to or unconsciously want to do or find. One hedge fund manager asked AI “what stocks are most likely to be disrupted by generative AI?” to which AI responded with a list which included Adobe. When another user asked “what stocks are most likely to be resilient to or benefit from AI?”, the resulting list also included Adobe. Buried in the presupposition of the question was the answer already—AI simplify amplifies the latent belief they already had and leaves their blind spots in the dark.
In other words, truth is not always agreeable, and the current paradigm of instruction-following alignment science cannot or does not want to produce disagreeable AI.
What does this all mean for the future and for individual efficacy?