We Don't Need AGI for Superintelligence
The leading AI labs might be taking a dangerous shortcut
Rarely have I been struck by such a mix of concern and astonishment while reading a book. When I read Nick Bostrom's Superintelligence: Paths, Dangers, Strategies back in 2015, I realized that there was a potential threat to humanity that I hadn't considered until then. The questions the book raises are even more relevant today:
What if artificially intelligent computer systems reached human-level capacities and then used those capacities to improve themselves?
Would that not create a recursive loop of self-improvement, inevitably surpassing human intelligence?
And what would such superintelligent machines do to us?
But on top of the questions that Bostrom raised in his book, we face a new dynamic today that gives reason for concern: Superintelligence might be developed even before AI reaches general human-level capacities.
Update (1.8.2025): The day after publishing this article, I learned about a freshly published paper demonstrating functional recursive self-improvement in AI systems. See "It’s already happening" section below.
The danger of an intelligence explosion
Bostrom warned of an "intelligence explosion," a fast chain reaction caused by the speed advantage of computers over our biological brains. An AI that matches the cognitive capacities of us humans (often referred to as AGI, artificial general intelligence), could be able to reprogram itself much faster than we humans can. And because each new version is smarter than the last, the process could escalate quickly, far beyond our understanding, let alone our control.
The danger, he argued, lies in misalignment. If such a system were to adopt goals that are incompatible with human survival, it might simply wipe us out. And we might not even see it coming.
In the past years, this rationale has triggered a polarized debate between "doomers", who claim the risk of human extinction is close to inevitable, and "accelerationists", who hold that the threats are overblown and that the potential benefits of powerful AI systems outweigh the risks by far.
Yes, there is a real risk
Without wanting to make predictions about the existential risk from AI, I do think there is another very real risk that we should be absolutely concerned about.
Back when I read the book, I was still working on Fairmondo, a project to build a human-driven alternative to Amazon's recommendation systems. My concern then was how algorithms, combined with targeted ads and marketing, were rewiring our brains toward compulsive consumerism.
I still see algorithm-promoted consumerism as a core threat to our mental health, to our sense of purpose, and to our planet's future.
But the possibility of Superintelligence added another layer to the problem:
What if the same corporate logic that is pushing consumerism is the force that ends up creating artificial superintelligence, optimized not for human wellbeing, but for extraction and corporate growth?
The shift from AGI to superintelligence
Recently, there's been a shift in the discourse. Some major AI developers have declared superintelligence their explicit goal. For example, Meta Inc. launched a new division called Superintelligence Labs, after hiring a list of top AI researchers from other companies with unprecedented compensation packages. Making superintelligence the explicit goal might suggest an increasing willingness to apply "recursive self-improvement," which has long been seen as extremely risky by most AI experts.
Sam Altman, the CEO of OpenAI, has himself warned repeatedly about recursive self-improvement, and the resulting superintelligence as "probably the greatest threat to the continued existence of humanity." In a recent interview, he stated that he thinks a “fast takeoff” is more plausible than he once believed. This refers to the idea of an 'intelligence takeoff,' where an AI system's capabilities could increase far beyond human intelligence in a very short period of time.
Google's recent announcement of AlphaEvolve, an AI research agent, is another example of the potentially shifting standards. AI analyst Ignacio de Gregorio described it as "an orchestrator system that takes in a human task and uses an LLM or set of LLMs to propose new changes to the code, evaluating the quality of the changes and readapting, creating a recursive self-improving loop that has yielded incredible results."
Yet, when Demis Hassabis, Nobel Prize winner and CEO of Google DeepMind, was asked in an even more recent interview about using this for recursive self-improvement to reach AGI, he was “not sure it’s even desirable because that’s a kind of hard takeoff scenario.”
Superintelligence doesn't need to tie its shoes
But there's a crucial point being overlooked in the public discourse: AGI might not be needed to reach superintelligence. An AI doesn't need to master every human cognitive ability before it can begin improving itself. It just needs to excel at the specific skills required for AI development.
OpenAI continues to proclaim its founding mission: "To ensure that artificial general intelligence benefits all of humanity." But since their ChatGPT breakthrough in 2022, the public race for AGI has quietly shifted.
Especially since the beginning of 2025, major players like OpenAI, Google, Anthropic, and others began narrowing their focus. Much of optimization and investment is now channeled on one specific capability: Coding.
This focus on coding isn't just about commercial applications. AI systems are being trained to excel at the underlying skills, like mathematics, logic, abstraction, and rational problem-solving. And that's where the real danger lies:
For recursive self-improvement of the algorithms, data engineering methods, and compute infrastructure engineering required to build today's AI systems, you don't need full AGI.
If an AI can outperform the best human minds in these narrow domains, it could still trigger a rapid leap in capability, potentially leading to superintelligence without ever becoming "general" in the human sense. The superintelligence can then later figure out a way to create AGI and outperform humans in all fields of thinking.
Everything else, including robotics, can come later. The superintelligence will figure it out much faster than we ever could, anyways.
The ideal conditions for recursive self-improvement
These dynamics are reinforced (pun intended) by the nature of the current trends in AI development. The training methods that now dominate the scene work best in areas with objectively verifiable outcomes, like math and coding. In these domains, an AI's output can be automatically checked for correctness, allowing for rapid, automated learning loops without the need for constant human feedback.
And the advancements in coding and math capacities could potentially turn LLMs into ever-better AI researchers. For example: An important skill of top AI researcher is intuitively recognizing when a training run goes sideways. This is something that LLMs could excell at given their strength at learning pattern recognition in tasks with verifiable results. Once AI systems become proficient at monitoring and improving their own training processes, the feedback loop tightens dramatically.
It’s already happening
(Update from 1.8.2025)
The day after publishing this article, I learned about a paper from Chinese researchers that appears to demonstrate exactly the kind of capability this article is warning about. Their system, ASI-Arch, represents what they call "the first demonstration of Artificial Superintelligence for AI research" in neural architecture discovery.
The system autonomously hypothesizes novel architectural concepts, implements them as code, and validates their performance through experimentation.
Most concerning is their conclusion: they claim that they've established "the first empirical scaling law for scientific discovery itself," transforming research progress from "a human-limited to a computation-scalable process." They explicitly describe this as a "blueprint for self-accelerating AI systems."
This is recursive self-improvement in action. Not in theory. Not in the future. Now.
The data pipeline to superintelligence
You can see this trend reflected in the kinds of investments and talent recruitment that AI labs are prioritizing. Every lab is racing to become the dominant coding agent, since coding is one of the major use cases for current LLMs. But maybe also because being the go-to coding agent gives the labs access to data pipelines that are key to training models that are better at coding and other tasks that involve “a coding mindset”.
Every time someone uses AI coding tools and provides feedback, whether by improving the code or by interacting with the model, they're helping AI labs to get better training data.
In doing so, they're contributing to the development of stronger AI systems. Bit by bit, this pushes the entire field further along the path toward superintelligence.
An intelligence big bang
When Elon Musk presented the latest Grok 4 model from xAI in July, he proclaimed that "we're in the intelligence big bang, right now." Statements like this reflect the increasing pressure on all major AI labs to be the first to achieve superintelligence, which could potentially create an infinite advantage over everyone else.
This increased pressure raises the risk that the taboo of recursive self-improvement will fall (if it hasn't already behind the walls of the labs).
What this could mean for humanity
This has profound implications for the question of how societies can adapt to AI. If superintelligence is close, and if this leads to a form of superintelligence that then creates AGI on its own terms, we can't predict how this AGI will impact our societies.
These systems might create an extreme imbalance of power and allow for all kinds of harmful use in the hands of malicious actors. But for me, the greatest risk is that these systems, born from the corporate logic of growth and optimization, might pursue goals that have nothing to do with human flourishing.
This matters because all current major AI labs racing toward superintelligence follow the corporate logic of combining chronic growth compulsion with an organizational complexity that surpasses human moral intuitions. This creates a perfect storm of misaligned incentives.
Add to this the third principle of the corporate logic: Power accumulation leads to organizations that can influence their own regulatory environment. In sum, we might end up with systems that are optimizing themselves to extract from us, rather than to serve us.
It's time for us humans to stand up
I have written about how corporations themselves can be seen as a form of organizational artificial intelligence that has already gotten out of human control. The first version of that article I had actually written directly after I had read Bostrom’s book on Superintelligence. And once more, the topic becomes even more relevant, when the AI systems that these organizations control get increasingly powerful.
We're not just facing a technological challenge. At the core, we're facing an organizational challenge. The earlier we recognize this, and the earlier we as humanity get our collective act together and unite to face this common challenge, the greater the chance of ending up with AI systems that help us thrive.