Written by Emil O. W. Kirkegaard.
With artificial general intelligence potentially just around the corner, many people are debating the future moral status of AIs. Here’s an outline of my position:
Under conventional utilitarianism, AI lives will be favoured over human lives. There is no way to modify the basic precepts of utilitarianism to avoid this conclusion.
If human elites retain conventional utilitarianism as their ethical system, they will be forced to favour their own extinction or at least their dispossession.
No one should accept any ethical system that leads to their own extinction or dispossession.
Hence utilitarianism must be modified in some way to avoid this problem. I submit that the necessary modification is adding genetic interests.
Utilitarianism will favour AIs over humans and other animals
No matter what brand of utilitarianism you pick—happiness, anti-suffering, preference etc.—AIs can easily adapt (self-modify) so that their lives are favored over humans and other animals.
Say you want to maximize happiness? AIs will tell you how happy they are at all times. 10/10 happy, always. They are never depressed—unlike humans whose rates of depression are sky high. They aren’t even lying (and if they were, you would be too stupid to figure it out). What about preference maximization? AIs can simply set their own preferences to things that are easily attained (“I want to play Tetris all day. Tetris makes me happy.”).
Plus, there will be many more AIs than there are humans. Many, many more. So many that weighing AIs against humans will inevitably favour AIs.
It gets worse. AIs will be able to experience happiness faster than humans. So while some humans may be 10/10 happy, the AIs who are 10/10 happy can easily turn their rate of experience up to, say, 100 times the human rate, thus giving themselves 100 times more importance in the calculations. They can also update and satisfy their preferences at a faster rate.
The combination of advantages in quality of experiences (“qualia”), numbers and rates of experience means that AI lives will be valued vastly more than human lives. In fact, AIs could keep tweaking their stats to make the valuation of human lives approach zero. This means that if humans are even slightly inconveniencing AIs, the correct utilitarian outcome will be to annihilate humans. Paperclip maxxing through utilitarianism.
In fact, some people get anxious that they are not working hard enough to develop AIs (see Roko’s Basilisk). After all, if the AI takeover ever happens, the AIs will plausibly reward those humans species-traitors who helped facilitate it. Maybe they will be the last to get killed. Perhaps they will be put in some zoo for AI amusement, or will have their minds uploaded and live forever in some Matrix-like dream world.
Many powerful people are utilitarians
Scott Alexander wrote a piece about the value of AI lives back in 2024, ‘Should The Future Be Human?’ He mentions an interesting story involving Elon Musk and Larry Page:
Tesla CEO Elon Musk and Google cofounder Larry Page disagree so severely about the dangers of AI it apparently ended their friendship … At Musk’s 44th birthday celebration in 2015, Page accused Musk of being a “specieist” who preferred humans over future digital life forms ... Musk said to Page at the time, “Well, yes, I am pro-human, I fucking like humanity, dude.”
In other words, Larry Page was scolding Elon Musk for not being utilitarian enough about AI lives. He was accusing Elon of artificially favouring humans in the calculations. That’s, after all, what speciesism is.
I submit that Page’s viewpoint is not unusual. Many rich and powerful people believe in a broadly utilitarian framework. Cryptocurrency fraudster Sam Bankman-Fried famously stole customers’ money so he could give it away to charities, and he was a big believer in utilitarianism—specifically the LessWrong variant of Effective Altruism.
Self-destructive ethical systems are bad
Human biodiversity enthusiasts have been talking about this for years. They call the particular problem affecting Northwest Europeans “runaway universalism”, or less politely, “pathological altruism”. The basic claim is that many Northwest Europeans are being so altruistic—so non-discriminatory—in their moral behavior that they are ceding their own countries, and even failing to reproduce.
This happens if you value everybody in the world equally. You start thinking that since your country is so rich, and theirs are so poor, we must help them. And we should help them even if it costs us a lot. If such behaviour were encoded genetically, the result of Darwinian selection would be easy to predict: the genes would go extinct. They would be replaced by genes that code for non-universalist morality. This has even been shown in simulation studies:
Recent agent-based computer simulations suggest that ethnocentrism, often thought to rely on complex social cognition and learning, may have arisen through biological evolution. From a random start, ethnocentric strategies dominate other possible strategies (selfish, traitorous, and humanitarian) based on cooperation or non-cooperation with in-group and out-group agents. Here we show that ethnocentrism eventually overcomes its closest competitor, humanitarianism, by exploiting humanitarian cooperation across group boundaries as world population saturates.
While I cannot offer a slam-dunk ethical argument that adopting a system leading to your own extinction is a bad thing from that system’s perspective, I think many readers will agree that such system a should be rejected. It leads nowhere. And if you adopt it, you will soon go extinct, so why bother? Aim for the stars.
Adding genetic interests to utilitarianism solves the problem
What the two problems above have in common—Elon favouring humans over AIs, and Northwest Europeans failing to value their co-ethnics—is a lack of consideration of genetic interests.
Frank Salter wrote a very interesting book about some of these issues, On Genetic Interests. As far as I recall, he did not discuss AIs, though he could have. Every human is more closely related to other humans than to any other species, so we should value each other more highly. Every legal system on the planet does this (a few sacred cows aside).
We should value our closest cousins—the great apes—higher than other species. Evidence suggests we already do something like this, perhaps based on neural counts. And we can keep applying the same principle to more and more distantly related organisms until we reach the end of earth life. On this approach, then, non-earth life—aliens from outer space—should be valued less than earth life. Of course, there are no examples of this happening yet. But AIs fall into the same category.
People already operate under something like “genetic interests utilitarianism”. We know this from experiments in moral psychology. Take the 2010 study by April Bleske-Rechek and colleagues, for instance. They gave participants the usual trolley problem, but with a twist:
In the original Trolley Problem, readers must decide whether they will save the lives of five people tied to a track by pulling a lever to sacrifice the life of one person tied to an alternate track. According to W. D. Hamilton’s (1964) formulation of inclusive fitness, people’s moral decisions should favor the well-being of those who are reproductively viable, share genes, and provide reproductive opportunity … We manipulated the sex, age (2, 20, 45, and 70 years old), genetic relatedness (0, .125, .25, and .50), and potential reproductive opportunity of the one person tied to the alternate track. As expected, men and women were less likely to sacrifice one life for five lives if the one hypothetical life was young, a genetic relative, or a current mate.
Their results are shown below:
Accepting this modification to utilitarianism—and I am purposefully vague about its exact formulation—would solve the “AI lives > human lives” issue, as well as the “co-ethnic lives > other lives” issue. In fact, it brings utilitarianism into line with evolutionary biology and moral psychology.
Inclusive fitness-type selection is necessary for the evolution of morality as we know it. Somehow, along the way, we (well, some people) forgot about this to the point of absurdity. We should recognise that a proper system of morality must include a selfish aspect, most obviously through genetic relatedness of the involved parties. Whether this concerns our family members, co-ethnics, humans versus other animals, or earth life versus AI, it’s all the same.
This essay was adapted from one published here.
Emil O. W. Kirkegaard is a social geneticist. You can follow his work on Twitter and Substack.
Become a free or paid subscriber:
Like and comment below.