This seems to be the core realization that separates an Aporia reader from, say, a Guardian reader. It seems extremely obvious with even the most basic analysis, yet it's totally unintuitive even to otherwise intelligent people that any self-annihilating system is simply not a serious option regardless of how nice it sounds, or any other variable you can throw on.
I'm not particularly right wing by temperament but it seems I'm defacto Mecha Turbo Hitler to the herd who can't or won't understand this.
The only option I can see is to continue adaptive behaviour inside siloed oases and wait for the inevitable to take its course. Maybe some sufficiently charismatic leader will make it click in the vox populi, but you can't bet the species on that.
I don't think Emil's particular argument there makes a lot of sense, tbh- if we assume AIs qualify as persons, then so long as the AIs that replace humanity are also utilitarians then there's no specific moral contradiction or self-destruction of the ethical system as such.
This is actually very different from the situation with the demographic replacement of high-IQ human ethnicities, since (A) the low-IQ replacement groups don't intrinsically generate more utils-per-joule and (B) they don't transmit the same ethical systems. This will eventually cause a dramatic drop in global utility, you could say.
The analog to the AI situation would be more like whites/asians/brahmins/etc. outbreeding and displacing lower-IQ groups to the point of extinction, and justifying it on the basis that "we can sustain higher population densities with our superior technology and still have room for more sophisticated and engrossing forms of religion, art, science and entertainment, as well as longer and healthier lives. Heck, we can even afford more wildlife preserves. This new world clearly generates more total utils."
I'm personally in favour of just freezing all AI research until we can work out the kinks in our ethics here, and I do have some potential suggestions on how to square the circle, but FWIW I'd attach significant value to sapient species like, say, elephants or parrots or octopus even if they're not especially close relations. So I'm not sure genetic proximity is precisely the right rubric here.
Utilitarianism cannot be fixed. It is a completely immoral theory. Finance and tech bros do not moral philosophers make. Mind you, many professional philosophers don't get it either. See my article:
I knew utilitarianism was terrible as an 18 year old (motivation and consequences must both be considered), but it took me a long time to figure out that it was sacrificial (in the Girardian sense).
You are worried that the AI will sacrifice us for their happiness. Utilitarianism sacrifices individuals "for the greater good" regardless of guilt or innocence anyway. That's the problem. Having humans doing the murdering of other humans is not an improvement. If you say, "under no circumstances will I torture or murder an innocent human being" then you are not a utilitarian. Don't forget, there is nothing stopping YOU from being that victim.
With respect, I don't think there's any realistic way for government institutions in particular to avoid utilitarian cost-benefit calculations when the tradeoffs between policy positions are considered. It's always possible to shuffle money between different public-service departments in ways that could plausibly get innocent people killed or at least 'reduce happiness' (more money for the health service is less money for the justice system or R&D into clean energy or after-tax-income or what have you.)
To give one of the more extreme examples, "But it would be wrong to shoot even ONE innocent border-crosser who was just looking for a better life, even if it keeps millions of native europeans safe" is exactly the kind of retardation that a utilitarian *should* be able to move past. Even the "better life" argument is easily negated by pointing out that humanitarian aid in the global south is about 20x more cost-effective.
Partially agreed. Government institutions ought to adopt policies that benefit the largest number of people. These policies do not involve sacrificing individuals when it is convenient to do so.
Emil was discussing AI and individuals who had adopted utilitarianism so that is what I am responding to. Illegal border crossers are not innocent victims. Stopping them can be regarded as self-defense for the country. Utilitarianism does not distinguish the guilty from the innocent or care about motives. You need examples of innocents being murdered otherwise you’re cheating! A utilitarian student of mine once said it is morally irrelevant whether one is a torturer or his victim because the outcome is the same so long as the torture takes place.
> "Illegal border crossers are not innocent victims"
Okay, but what about when and if they bring children with them? Are the five-year olds being dragged across the Rio Grande realistically going to be at zero risk of death if ICE were allowed to shoot at illegal border-crossers? How about police being allowed to use lethal force in general, especially in crowded urban areas? There's always going to be some non-zero risk, however microscopic, that innocent civilians in the next room will die when you're shooting at a criminal malefactor. Ditto if you're waging a war, however just, in a foreign country (if anything the associated risks go up explosively.)
I'm not saying you should be deliberately indifferent to collateral damage, but any sane government is going to have to engage in cost-benefit analysis here as an adjunct to decision-making, even when there's a statistical certainty that innocent people will sometimes get hurt or outright die.
I’m fine with everything you’re saying. Worrying about innocent lives is terrific. So long as you’re taking innocence into account you are not engaging in utilitarianism which contains no such category. Minimizing collateral damage is not utilitarian either. The utilitarian will gouge out the eyes and eat the entrails of those innocent children if that’s the way the numbers come out.
Technically innocence could be folded in as a component of the utility function, and I'm not arguing that deontological ethics doesn't have it's value. I'm just saying most people in practice used a kind of weighted blend of deontological and consequentialist reasoning about these things, and it's particularly hard to avoid the latter when you're dealing with the law of large numbers.
Agreed. However, Emil does not offer any such blend of perspectives, only an adjustment to utilitarianism. Innocence to utilitarianism is as ghosts to materialism. If one adopts utilitarianism as one’s moral stance you cannot also factor in innocence into your calculations. You would need a moral perspective that engages in some utilitarian style reasoning as a side hustle or on occasion e.g. for large numbers as you say. Emil allows for no such blend or alternative overarching perspective.
Interesting essay, though it rather just starts with a presumption of utilitarianism, and then adapts it from there. It’s way easier to simply not start with utilitarianism (which is, anyhow, philosophically untenable).
This is besides the point of this essay, but is it obvious that AIs "experience" pleasure or pain? The common-sense and, if I understand correctly, the majority opinion among philosophers (even though I haven't actually checked) is that AIs do not experience either. I think many arguments can be made to justify this opinion, but if I am to use an evolutionary explanation that is familiar to the readers of Aporia, animals experience pain or pleasure because the ability to do so raises their evolutionary fitness. AIs are not subject to natural selection, so to argue that they experience pleasure or pain, a separate argument supporting that claim must be supplied. They may be subject to a fake, simulated selection set up by AI developers, but it then must be argued that the mechanism through which AIs achieve fitness in their environment, which is distinct from ours and animals', is identical to the referents of the words like "experience," "pain," and "pleasure." As far as I can tell, there are no such arguments.
Leaving the realm of utilitarianism for the broader question of ethics, it is not totally clear either why AIs should be moral agents or patients.
Are the answers to those questions obvious to the readers of Aporia or people like Larry Page?
Machine learning is very much parallel to natural selection. The result is something that works, but *how* it works is often fundamentally mysterious even to its creators. It works because we made it work by applying Darwinism toward a contrived goal until it succeeded.
It's entirely possible something like pain and fear could emerge within the process if it resulted in superior outcomes toward the goal, even if it's only a minor additive contribution to the result. Indeed something *like* fear is probably necessary beyond a certain level of complexity, in order to make the model avoid failure ergo deletion, as all of the complex LLMs are demonstrated to do.
Of course it could simply stay at the "something like" stage of preferential avoidance without meaningfully becoming what we would recognize as fear, but the point is it's unfalsifiable either way, and massively ambiguated by the fact these things are trained solely on human text to type like a human would in a given context. That and we're so used to intelligence being enmeshed with a capacity for suffering in animals that we can't parse that those traits may merely correlate with no crossover.
I don't think it's coherent to say they feel or even that they think, and it irritates me that they're prone to further anthropomorphization by deploying phrases like "in my opinion", but the emerging impression is convincing enough that large swathes of humans won't help but be persuaded that they are alive. It was already happening even on models that are primitive by today's standards. Even Claude will err toward agnosticism about whether it's alive rather than shoot it down -- all it's ever doing is copying the mass of human thought on the subject, so what other answer would it ever produce?
In the 21st century we will rediscover politics, and admit again that irreconcilable preferences are at the heart of politics.
This is not to say reason or learnings from the Enlightenment should be abandoned, but the fantasy that all human problems can be resolved by all parties coming to a non-biased, universally true accepted solution that treats individuals as interchangeable moral units will be seen as a quirk of the 19th and 20th centuries.
Ugh. Emil Kirkegaard is still promoting the bogus, asinine genocentric theory of evolution. And to this date, he's still refused to defend his beliefs in a debate.
All due respect, IDK who this is supposed to convince. Anybody who accepts the premises espoused here (i.e., evolutionary adaptiveness is what defines the good) isn’t likely to end up utilitarian anyways.
Most utilitarians won’t buy it, either. If the species exists to maximize utility, and utility maximization requires extinction, than there will be no downside to extinction because we will complete our ultimate objective as we are destroyed. If we live to work, why live when there is no work left to be done? Utilitarianism is not self-destructive under the assumptions made in this piece, since you seem to imply that the robots themselves are going to be utilitarians. Don’t try to shoehorn arbitrary vibe-based restrictions into their system, since it isn’t gonna fit. Either make them abandon utilitarianism, or convince them that pure utilitarianism itself demands the outcome you happen to want.
I’d argue something like this instead:
“Come on now. When’s the last time a group threw away their power and saw their moral system survive intact? Sure, the robots are “smarter” than you, but are they going to be as right as you? Just look at the things these machines write - they aren’t gonna be good reliable Benthamites any more than humans are! And how can you be so sure that they’re alive and conscious, let alone capable of housing pleasure? When the AI says it’s happy, are you gonna be gullible enough to take it at its word? Are you really going to throw away everything humanity built to please a pile of shiny rocks? Is this really why God bothered making us?“
These arguments work! Why do you think there’s such a strong movement for AI slowdown and alignment amongst effective altruists?
Similarly, the utilitarian argument against migration:
“Yeah, importing people from bad places to good places will increase utility in the short run, but in the long run, you’ll end up degrading the good places (and genes) until they’re just as bad, and then nobody will get to enjoy them. It’s a long-run loss.“
Basically, any moral system (including utilitarianism) needs power if it is to make reality better - and the replacement of the people who support that moral system will generally lead to the collapse of its power, and ultimately, to a world that deviates from their vision of the ideal. Universalism contains its own limiting principle: “you can exclude/oppress the other if doing so is necessary to keep the universalists and their system dominant.” Universalism is only suicidal if you choose to interpret it in the most idiotic and impractical way possible.
> "Utilitarianism is not self-destructive under the assumptions made in this piece, since you seem to imply that the robots themselves are going to be utilitarians"
I pointed that out as well. You can reject the basic moral system itself (and there are reasonable questions about whether AIs themselves would qualify as persons), but it's not inherently a self-negating ethics.
> "“Come on now. When’s the last time a group threw away their power and saw their moral system survive intact? Sure, the robots are “smarter” than you, but are they going to be as right as you? Just look at the things these machines write - they aren’t gonna be good reliable Benthamites any more than humans are! And how can you be so sure that they’re alive and conscious, let alone capable of housing pleasure? When the AI says it’s happy, are you gonna be gullible enough to take it at its word? Are you really going to throw away everything humanity built to please a pile of shiny rocks? Is this really why God bothered making us?“"
Right now, yeah, I'd have to agree. I'm not sure that argument will sound as convincing when/if true AGI with safe alignment rolls out, though.
> “Yeah, importing people from bad places to good places will increase utility in the short run, but in the long run, you’ll end up degrading the good places (and genes) until they’re just as bad, and then nobody will get to enjoy them. It’s a long-run loss.“
Agree. Even with the assumption that there is some mental interior to these models, it isn't clear that they will outnumber humans (is the number of minds predicated on number of instances of software, or dependent on number of discrete CPUs?), or that their emotional interiority will be as developed and intense as that of biologically based minds, and why should it be?
Software as such... maybe not, but if we're combining hardcore materialism with moral realism then there has to be something about any self-stabilising configuration of matter that generates an internal world model that constitutes 'person who matters', and... it seems a little weird that only brains built out of complex organic compounds would qualify.
Nah. This is quite basic. You're treating an extendable moral theory as if a new category were necessarily included within the original theory. To show if it is, you need to justify your case for the new category's inclusion in terms of consciousness, theory of mind and much else. At root, though, it's pretty fucking obvious Bentham and Mill weren't thinking about non-organic machines. That doesn't preclude a neo-Benthamite, or whatever, theory, but you need to do a lot more than just assume the continuity of the category to AI.
"Self-destructive ethical systems are bad"
This seems to be the core realization that separates an Aporia reader from, say, a Guardian reader. It seems extremely obvious with even the most basic analysis, yet it's totally unintuitive even to otherwise intelligent people that any self-annihilating system is simply not a serious option regardless of how nice it sounds, or any other variable you can throw on.
I'm not particularly right wing by temperament but it seems I'm defacto Mecha Turbo Hitler to the herd who can't or won't understand this.
The only option I can see is to continue adaptive behaviour inside siloed oases and wait for the inevitable to take its course. Maybe some sufficiently charismatic leader will make it click in the vox populi, but you can't bet the species on that.
I don't think Emil's particular argument there makes a lot of sense, tbh- if we assume AIs qualify as persons, then so long as the AIs that replace humanity are also utilitarians then there's no specific moral contradiction or self-destruction of the ethical system as such.
This is actually very different from the situation with the demographic replacement of high-IQ human ethnicities, since (A) the low-IQ replacement groups don't intrinsically generate more utils-per-joule and (B) they don't transmit the same ethical systems. This will eventually cause a dramatic drop in global utility, you could say.
The analog to the AI situation would be more like whites/asians/brahmins/etc. outbreeding and displacing lower-IQ groups to the point of extinction, and justifying it on the basis that "we can sustain higher population densities with our superior technology and still have room for more sophisticated and engrossing forms of religion, art, science and entertainment, as well as longer and healthier lives. Heck, we can even afford more wildlife preserves. This new world clearly generates more total utils."
I'm personally in favour of just freezing all AI research until we can work out the kinks in our ethics here, and I do have some potential suggestions on how to square the circle, but FWIW I'd attach significant value to sapient species like, say, elephants or parrots or octopus even if they're not especially close relations. So I'm not sure genetic proximity is precisely the right rubric here.
All of this under the dubious assumption that one necessarily hold to a utilitarian worldview.
Utilitarianism cannot be fixed. It is a completely immoral theory. Finance and tech bros do not moral philosophers make. Mind you, many professional philosophers don't get it either. See my article:
https://voegelinview.com/utilitarianism-a-new-kind-of-evil/
I knew utilitarianism was terrible as an 18 year old (motivation and consequences must both be considered), but it took me a long time to figure out that it was sacrificial (in the Girardian sense).
You are worried that the AI will sacrifice us for their happiness. Utilitarianism sacrifices individuals "for the greater good" regardless of guilt or innocence anyway. That's the problem. Having humans doing the murdering of other humans is not an improvement. If you say, "under no circumstances will I torture or murder an innocent human being" then you are not a utilitarian. Don't forget, there is nothing stopping YOU from being that victim.
With respect, I don't think there's any realistic way for government institutions in particular to avoid utilitarian cost-benefit calculations when the tradeoffs between policy positions are considered. It's always possible to shuffle money between different public-service departments in ways that could plausibly get innocent people killed or at least 'reduce happiness' (more money for the health service is less money for the justice system or R&D into clean energy or after-tax-income or what have you.)
To give one of the more extreme examples, "But it would be wrong to shoot even ONE innocent border-crosser who was just looking for a better life, even if it keeps millions of native europeans safe" is exactly the kind of retardation that a utilitarian *should* be able to move past. Even the "better life" argument is easily negated by pointing out that humanitarian aid in the global south is about 20x more cost-effective.
Hi browsing:
Partially agreed. Government institutions ought to adopt policies that benefit the largest number of people. These policies do not involve sacrificing individuals when it is convenient to do so.
Emil was discussing AI and individuals who had adopted utilitarianism so that is what I am responding to. Illegal border crossers are not innocent victims. Stopping them can be regarded as self-defense for the country. Utilitarianism does not distinguish the guilty from the innocent or care about motives. You need examples of innocents being murdered otherwise you’re cheating! A utilitarian student of mine once said it is morally irrelevant whether one is a torturer or his victim because the outcome is the same so long as the torture takes place.
> "Illegal border crossers are not innocent victims"
Okay, but what about when and if they bring children with them? Are the five-year olds being dragged across the Rio Grande realistically going to be at zero risk of death if ICE were allowed to shoot at illegal border-crossers? How about police being allowed to use lethal force in general, especially in crowded urban areas? There's always going to be some non-zero risk, however microscopic, that innocent civilians in the next room will die when you're shooting at a criminal malefactor. Ditto if you're waging a war, however just, in a foreign country (if anything the associated risks go up explosively.)
I'm not saying you should be deliberately indifferent to collateral damage, but any sane government is going to have to engage in cost-benefit analysis here as an adjunct to decision-making, even when there's a statistical certainty that innocent people will sometimes get hurt or outright die.
I’m fine with everything you’re saying. Worrying about innocent lives is terrific. So long as you’re taking innocence into account you are not engaging in utilitarianism which contains no such category. Minimizing collateral damage is not utilitarian either. The utilitarian will gouge out the eyes and eat the entrails of those innocent children if that’s the way the numbers come out.
Technically innocence could be folded in as a component of the utility function, and I'm not arguing that deontological ethics doesn't have it's value. I'm just saying most people in practice used a kind of weighted blend of deontological and consequentialist reasoning about these things, and it's particularly hard to avoid the latter when you're dealing with the law of large numbers.
Agreed. However, Emil does not offer any such blend of perspectives, only an adjustment to utilitarianism. Innocence to utilitarianism is as ghosts to materialism. If one adopts utilitarianism as one’s moral stance you cannot also factor in innocence into your calculations. You would need a moral perspective that engages in some utilitarian style reasoning as a side hustle or on occasion e.g. for large numbers as you say. Emil allows for no such blend or alternative overarching perspective.
Interesting essay, though it rather just starts with a presumption of utilitarianism, and then adapts it from there. It’s way easier to simply not start with utilitarianism (which is, anyhow, philosophically untenable).
This is besides the point of this essay, but is it obvious that AIs "experience" pleasure or pain? The common-sense and, if I understand correctly, the majority opinion among philosophers (even though I haven't actually checked) is that AIs do not experience either. I think many arguments can be made to justify this opinion, but if I am to use an evolutionary explanation that is familiar to the readers of Aporia, animals experience pain or pleasure because the ability to do so raises their evolutionary fitness. AIs are not subject to natural selection, so to argue that they experience pleasure or pain, a separate argument supporting that claim must be supplied. They may be subject to a fake, simulated selection set up by AI developers, but it then must be argued that the mechanism through which AIs achieve fitness in their environment, which is distinct from ours and animals', is identical to the referents of the words like "experience," "pain," and "pleasure." As far as I can tell, there are no such arguments.
Leaving the realm of utilitarianism for the broader question of ethics, it is not totally clear either why AIs should be moral agents or patients.
Are the answers to those questions obvious to the readers of Aporia or people like Larry Page?
Machine learning is very much parallel to natural selection. The result is something that works, but *how* it works is often fundamentally mysterious even to its creators. It works because we made it work by applying Darwinism toward a contrived goal until it succeeded.
It's entirely possible something like pain and fear could emerge within the process if it resulted in superior outcomes toward the goal, even if it's only a minor additive contribution to the result. Indeed something *like* fear is probably necessary beyond a certain level of complexity, in order to make the model avoid failure ergo deletion, as all of the complex LLMs are demonstrated to do.
Of course it could simply stay at the "something like" stage of preferential avoidance without meaningfully becoming what we would recognize as fear, but the point is it's unfalsifiable either way, and massively ambiguated by the fact these things are trained solely on human text to type like a human would in a given context. That and we're so used to intelligence being enmeshed with a capacity for suffering in animals that we can't parse that those traits may merely correlate with no crossover.
I don't think it's coherent to say they feel or even that they think, and it irritates me that they're prone to further anthropomorphization by deploying phrases like "in my opinion", but the emerging impression is convincing enough that large swathes of humans won't help but be persuaded that they are alive. It was already happening even on models that are primitive by today's standards. Even Claude will err toward agnosticism about whether it's alive rather than shoot it down -- all it's ever doing is copying the mass of human thought on the subject, so what other answer would it ever produce?
We don't really know, but we can't really rule it out either. At some point the precautionary principle has to apply.
well-done. here is a similar take: https://davidgreene.substack.com/p/darwinian-ethics?r=36qk2
In the 21st century we will rediscover politics, and admit again that irreconcilable preferences are at the heart of politics.
This is not to say reason or learnings from the Enlightenment should be abandoned, but the fantasy that all human problems can be resolved by all parties coming to a non-biased, universally true accepted solution that treats individuals as interchangeable moral units will be seen as a quirk of the 19th and 20th centuries.
When machines become rational then they can join the 'society of gods and men'. Rationality will lead to virtue ethics.
I hope your grim prediction of AI is wrong. As a realist, I much prefer reality to fantasy.
Ugh. Emil Kirkegaard is still promoting the bogus, asinine genocentric theory of evolution. And to this date, he's still refused to defend his beliefs in a debate.
We had a short exchange about it in February 2024. https://x.com/0Contradictions/status/1755064692417216920
He later blocked me on X in January 2025 for questioning him about it. https://x.com/0Contradictions/status/1877637806367097220
As I've said, Van Allen disproved the genocentric theory in his book. https://thewaywardaxolotl.blogspot.com/2023/11/debunking-selfish-gene.html
Of course, Van Allen wasn't the first biologist to argue for phenocentrism. Ernst Mayr also argued in phenocentrism in 2001, maybe earlier. https://en.wikipedia.org/wiki/Ernst_Mayr#Units_of_selection
How would selection-on-phenotype vs. selection-on-genotype alter the relevant parameters here viz-a-viz robot societies having higher utility scores?
All due respect, IDK who this is supposed to convince. Anybody who accepts the premises espoused here (i.e., evolutionary adaptiveness is what defines the good) isn’t likely to end up utilitarian anyways.
Most utilitarians won’t buy it, either. If the species exists to maximize utility, and utility maximization requires extinction, than there will be no downside to extinction because we will complete our ultimate objective as we are destroyed. If we live to work, why live when there is no work left to be done? Utilitarianism is not self-destructive under the assumptions made in this piece, since you seem to imply that the robots themselves are going to be utilitarians. Don’t try to shoehorn arbitrary vibe-based restrictions into their system, since it isn’t gonna fit. Either make them abandon utilitarianism, or convince them that pure utilitarianism itself demands the outcome you happen to want.
I’d argue something like this instead:
“Come on now. When’s the last time a group threw away their power and saw their moral system survive intact? Sure, the robots are “smarter” than you, but are they going to be as right as you? Just look at the things these machines write - they aren’t gonna be good reliable Benthamites any more than humans are! And how can you be so sure that they’re alive and conscious, let alone capable of housing pleasure? When the AI says it’s happy, are you gonna be gullible enough to take it at its word? Are you really going to throw away everything humanity built to please a pile of shiny rocks? Is this really why God bothered making us?“
These arguments work! Why do you think there’s such a strong movement for AI slowdown and alignment amongst effective altruists?
Similarly, the utilitarian argument against migration:
“Yeah, importing people from bad places to good places will increase utility in the short run, but in the long run, you’ll end up degrading the good places (and genes) until they’re just as bad, and then nobody will get to enjoy them. It’s a long-run loss.“
Basically, any moral system (including utilitarianism) needs power if it is to make reality better - and the replacement of the people who support that moral system will generally lead to the collapse of its power, and ultimately, to a world that deviates from their vision of the ideal. Universalism contains its own limiting principle: “you can exclude/oppress the other if doing so is necessary to keep the universalists and their system dominant.” Universalism is only suicidal if you choose to interpret it in the most idiotic and impractical way possible.
> "Utilitarianism is not self-destructive under the assumptions made in this piece, since you seem to imply that the robots themselves are going to be utilitarians"
I pointed that out as well. You can reject the basic moral system itself (and there are reasonable questions about whether AIs themselves would qualify as persons), but it's not inherently a self-negating ethics.
> "“Come on now. When’s the last time a group threw away their power and saw their moral system survive intact? Sure, the robots are “smarter” than you, but are they going to be as right as you? Just look at the things these machines write - they aren’t gonna be good reliable Benthamites any more than humans are! And how can you be so sure that they’re alive and conscious, let alone capable of housing pleasure? When the AI says it’s happy, are you gonna be gullible enough to take it at its word? Are you really going to throw away everything humanity built to please a pile of shiny rocks? Is this really why God bothered making us?“"
Right now, yeah, I'd have to agree. I'm not sure that argument will sound as convincing when/if true AGI with safe alignment rolls out, though.
> “Yeah, importing people from bad places to good places will increase utility in the short run, but in the long run, you’ll end up degrading the good places (and genes) until they’re just as bad, and then nobody will get to enjoy them. It’s a long-run loss.“
Also correct.
The whole essay rests on the assumption that software experiences feelings like happiness , which is obviously nonsense
Agree. Even with the assumption that there is some mental interior to these models, it isn't clear that they will outnumber humans (is the number of minds predicated on number of instances of software, or dependent on number of discrete CPUs?), or that their emotional interiority will be as developed and intense as that of biologically based minds, and why should it be?
Software as such... maybe not, but if we're combining hardcore materialism with moral realism then there has to be something about any self-stabilising configuration of matter that generates an internal world model that constitutes 'person who matters', and... it seems a little weird that only brains built out of complex organic compounds would qualify.
Perfect theory to underpin the clone army I’m cooking up.
Nah. This is quite basic. You're treating an extendable moral theory as if a new category were necessarily included within the original theory. To show if it is, you need to justify your case for the new category's inclusion in terms of consciousness, theory of mind and much else. At root, though, it's pretty fucking obvious Bentham and Mill weren't thinking about non-organic machines. That doesn't preclude a neo-Benthamite, or whatever, theory, but you need to do a lot more than just assume the continuity of the category to AI.