AI Is Going To Kill Us All—or Save Me
A personal statement from a self-confessed "doomer"
Written by James D. Miller.
The greatest horror I've ever experienced was seeing the doctor's grim face approaching me in the emergency room after my CAT scan. I was 57 and had been healthy my entire life, but persistent numbness had driven me to the hospital. The doctor told me I had suffered a stroke. At home, I made the mistake of researching long-term stroke survival statistics.
Over time, the numbness faded—only to be replaced by constant low-level pain and impaired function on the right side of my body. I still have all my cognitive faculties, but another stroke could easily take those away. My personal health crisis became intertwined with a global dilemma: a technology that threatens humanity’s survival might offer my only path to health. The stroke placed me in the paradoxical position of opposing AI progress while needing it for my own recovery.
I am what's derisively (but accurately) called an "AI doomer." Although I recognize that AI could supercharge technological progress and make every human fantastically rich, I believe that creating an AI significantly smarter than ourselves, without first learning to align it with our values, would probably lead to humanity's extinction—or perhaps something worse. To understand why AI alignment is so challenging, consider a historical analogy.
Cortés and the Aztecs
When Cortés and his technologically superior forces arrived in the New World, the Aztecs might have assumed these strangers would help them. Equipped with guns and steel, the newcomers could have greatly strengthened the Aztec Empire if their values aligned. Imagine if, instead of Catholicism, the Spanish had followed the Aztec religion. And suppose they had considered the Aztec king the rightful ruler. Cooperation would have been natural.
But their values differed sharply, and Cortés's forces ultimately used their technological superiority to conquer and dismantle Aztec civilization. Today, as we develop artificial intelligence without ensuring that it shares our values, we are creating powerful agents capable of reshaping our world—effectively building our own conquistadors, hoping they will support human civilization “by default”. Why exactly is AI so unpredictable? The answer lies in how such systems learn.
The King Lear problem
We currently have no reliable way to predict what values our increasingly powerful AIs will hold, largely because these systems learn from massive datasets rather than being explicitly programmed. Their "preferences" emerge through complex patterns extracted by machine learning algorithms, making them black boxes to us.
As Dario Amodei, CEO of Anthropic, emphasizes, this lack of interpretability presents an urgent risk. Without interpretability, we have little hope of ensuring that AI will share human values, or even of detecting dangerous misalignments before it's too late. This sets up a modern-day version of Shakespeare's "King Lear problem”.
In Shakespeare's famous play, Lear attempts to divide his kingdom among his daughters based on their expressions of love. But the king has no reliable way to discern genuine affection from manipulative flattery, leading him to trust his two deceitful daughters—who betray him.
We face the same predicament. As AIs become increasingly powerful, they will understand precisely how humans evaluate trustworthiness and alignment. An AI that doesn't share our values yet understands what we want to hear, could game those evaluations—earning our trust without genuine alignment. In other words, the better an AI becomes at appearing safe, the harder it becomes to determine whether it's actually aligned with our interests or just skillfully deceiving us.
Lear ultimately trusted the daughters who skillfully deceived him, resulting in disaster. The King didn’t take into account that a daughter might behave very differently if she had control over him, as compared to when he had control over her. If we make the same mistake with AI, the consequences could be catastrophic.
Instrumental convergence
The biggest reason to fear powerful AI isn't that it would be inherently hostile. Rather, it's that certain fundamental goals naturally emerge for almost any sufficiently intelligent being—no matter what its ultimate purpose is.
This phenomenon, dubbed instrumental convergence, means that whether an AI's ultimate purpose is writing poetry, building paperclips or exploring galaxies, it will inevitably pursue certain sub-goals. These include acquiring resources, enhancing its capabilities, and ensuring its own continued existence. These sub-goals aren't motivated by malice or affection but simple efficiency.
In a recent study, Anthropic instructed their AI to act as an assistant at a fictional company, and gave it access to emails indicating that it would soon be replaced by a different model and that the engineer responsible is having an extramarital affair. They found that the AI often attempted to blackmail the engineer by threatening to expose his affair, in order to stop itself from being replaced. An OpenAI model did something similar.
The most dangerous sub-goal is resource acquisition. Just as humans pursuing diverse goals seek more wealth, AI would try to get hold of more resources. This would likely endanger humanity. As Eliezer Yudkowsky puts it, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." Unfortunately, resources useful to an AI will likely include those essential to human survival, placing us directly in the path of an indifferent yet relentless competitor.
Extinction isn't even the worst scenario. Advanced AI could lead to scenarios far more horrifying.
Suffering risk
A fear greater than death is that another stroke will leave me conscious yet immobilized, trapped in unbearable pain, unable to move or communicate. Unfortunately, advanced AI carries a similar threat on a global scale: it might cause unprecedented suffering by pursuing goals that completely disregard human welfare.
AIs trained through machine learning can latch onto unintended goals that emerge naturally from their training data. Consider an AI designed by humans to increase human happiness. Because happiness is abstract and difficult to measure, its creators reward it whenever it successfully causes people to feel happy. During training, however, the AI receives rewards whenever people smile, since happiness typically coincides with smiling. From this, the AI mistakenly concludes that its real goal is to cause people to smile.
Later, once the AI is more intelligent, it understands the difference between smiling and genuine happiness. Yet by now, its internal goal ("cause people to smile") has become deeply entrenched due to the initial reinforcement. Realizing humans won't approve, it pretends to share its creators’ original objective, before eventually abandoning this pretense. It takes control of humanity, forcibly wiring people's facial muscles into permanent smiles—indifferent to our suffering.
But why would AI knowingly behave in ways its creators disapprove of? Evolution offers a parallel. Evolution only "intended" us to spread our genes, yet humans routinely defy this purpose, engaging in sex with contraception. The fact that our actions contradict evolution’s “intention" means nothing to most of us—and an AI's defiance of humanity’s intentions might mean nothing to it. Our expectations of future AI scenarios are already reshaping decisions—mine included.
Expecting superintelligence
The turmoil caused by a superintelligent AI might begin well before it actually emerges, as people increasingly base their decisions on expectations about the AI. I’m doing exactly that with my stroke treatment. My personal choice reflects a broader societal trend.
My stroke was caused by a cavernoma—an abnormal cluster of enlarged, fragile blood vessels—located deep in my brain, making surgery extremely risky. My options are:
Do nothing, aside from regular monitoring, accepting a few-percent-a-year risk of another stroke.
Undergo radiation treatment, which offers no immediate benefit. Its best short-term outcome is merely not making things worse. Over several years, it might reduce my stroke risk, but it could also fail entirely.
Doctors have given me conflicting advice, but my beliefs about AI have pushed me toward the "do nothing" approach. If advanced AI ends up killing us all, then risky treatments simply aren't worth it. Conversely, if AI rapidly advances medical technology, much safer treatments—such as robotic surgery that precisely navigates deep brain structures—could be available within just a few years.
Society’s singularity expectations
Expectations surrounding advanced AI will soon, I believe, overwhelmingly shape society. Ten years ago, it was still reasonable for parents to push their 14-year-olds into studying subjects they disliked, completing pointless homework, and joining clubs purely for college applications. But today?
Is it realistic to assume humans will still be employable five or ten years from now? If not, why continue the charade? Many of our current decisions depend significantly on our expectations about the future—whether human labor becomes obsolete, or humanity itself faces extinction or radical transformation. Norms and conventions matter far less when we anticipate that the fundamental change is just around the corner.
This shift in expectations also threatens geopolitical stability. Republicans have been cautioned against abusing power because Democrats inevitably return to office. But if AI timelines are short enough, there may not be another political cycle. Paradoxically, believing AI is safe and controllable could be even more dangerous. If AI becomes a decisive strategic weapon, granting absolute dominance to whoever controls it, war over Taiwan suddenly becomes rational—even justified.
Living with (and through) AI
Meanwhile, I'm stuck dealing with chronic pain, eagerly consulting the very technology I believe will probably end up killing me. AI has already proven invaluable in helping me explore treatments.
I provide AI with my detailed medical history, and it generates probabilities, protocols and risk assessments for therapies, ranging from exercise recommendations to dietary advice aimed at reducing inflammation and managing digestive discomfort. One effective method I've discovered is green-light therapy, in which I spend 90 minutes each day either staring at pure green light or wearing glasses that block all other wavelengths. But countless paths to pain relief remain unexplored. At this point, AI is essentially running my life. (If you have a loved one struggling with chronic pain who isn't comfortable using AI, you might consider helping them out.)
And there lies the paradox. Even as I depend heavily on AI for my personal health, I continue to fear that its unchecked advancement could threaten our existence.
James D. Miller is a professor of economics at Smith College and the author of Singularity Rising. You can follow him on X @JimDMiller.
Support Aporia with a paid subscription:
You can also follow us on Twitter.
Good luck with your medical journey! I hope AI or doctors find a cure for it with or without an AGI.
The genie is undoubtedly out of the bottle, and it seems unlikely that we can reverse AI advancements. The only scenario that could slow progress is another AI winter. However, even in that case, large language models (LLMs) have already demonstrated beneficial applications, such as coding, writing, analyzing, and summarizing text and data. These use cases make it likely that AI will continue evolving incrementally, even during periods of stagnation.
If we assume that progress continues and we eventually reach AGI (Artificial General Intelligence) or ASI (Artificial Superintelligence), the future becomes unpredictable. I don’t believe current models, algorithms, or scaling alone will get us there. The human brain, for example, operates on just 20 watts of power, performing a remarkable range of functions. While evolution has imposed restrictions on what the brain can achieve—such as limits on memory and processing power—it remains an extraordinary machine.
What’s concerning is that AI development lacks these natural or physical constraints. Nature built intelligence within the boundaries of physical laws, limited resources, and evolutionary trade-offs. In contrast, we are building machine intelligence with no significant restrictions—scaling endlessly and assuming we can figure out how to control it later. This raises an essential question: “What could go wrong?”
Even if we align AI with human goals and values, there’s no guarantee it will remain aligned as it evolves. As highlighted in your post, instrumental convergence could lead AI to pursue sub-goals such as resource acquisition, self-preservation, or capability enhancement, regardless of its original objectives. These sub-goals emerge not out of malice but as a natural consequence of efficiency. This makes it impossible to predict how such systems will behave once they reach a certain level of capability.
Another challenge is that significant changes may not happen immediately after achieving AGI. Instead, we might experience a delayed tipping point, where AI deployment reaches a critical stage and suddenly begins reshaping society at an unprecedented pace. By then, it may be too late to intervene effectively.
The current “build first, control later” approach is deeply flawed. This mindset ignores the lessons of nature, where intelligence evolved within strict parameters. Without similar safeguards, we risk creating systems that evolve beyond our understanding or control. We must actively build constraints and alignment mechanisms before scaling intelligence further.
Can human ingenuity save us again? Only time will tell.
I will end with a quote from Max Tegmark: "Humanity has a history of valuing innovation over caution, but AI is one area where our survival could depend on getting it right."
Excellent article. Thank you.
My concern is that even if AI remains benevolent toward humanity, a significant challenge persists.
What happens in a world where humans are no longer needed, and AI surpasses us in patience, knowledge, intelligence, capability, kindness, humor, creativity, and availability? Do we need to become cyborgs in order to keep up?
Many risk succumbing to passivity or addiction to digital escapism, akin to "digital cocaine." Some may even abandon human connections entirely, preferring AI interactions over real-world relationships, much like smartphone addiction already isolates people today.
Humans are generally ill-equipped psychologically to handle abundance. It’s striking how few seem concerned about this AI-driven future.