The Goldilocks zone between inbreeding and outbreeding

Even subspecies can have trouble breeding with each other.

Written by Peter Frost.

The more closely related a man and a woman are to each other, the higher their risk of producing abnormal embryos. The risk is highest if they are siblings, lower if they are first cousins, and lower still if they are second cousins. This is well known. But did you know that the same risk then starts to increase progressively with further decreases in relatedness? The sweet spot for having healthy children seems to be marriage between third or fourth cousins.

Subspecies can breed with each other, but species cannot. So it is said. The reality is less clear-cut. “Subspecies” and “species” are arbitrary points along a continuum of decreasing genetic compatibility. As we move along that continuum, we observe more and more problems when two populations try to interbreed. But how difficult must things get before we classify them as two species? There is no right answer. Many populations have been redefined from “subspecies” to “species” and back again.

Yes, even subspecies can have trouble breeding with each other (Presgraves, 2010). When two subspecies of house mouse are crossed (Mus musculus domesticus and Mus musculus musculus), the offspring suffer not only from reduced fertility but also from reduced immune function and higher loads of intestinal parasites (Turner et al., 2011).

The reduction of fertility seems to have multiple causes: "The type and severity of fertility defects observed depends on the geographic origin of the strains and also varies among individuals within regions. This variability suggests that multiple genetic incompatibilities contribute to hybrid male sterility" (Turner et al., 2011). Moreover, one incompatibility can interact with another to produce a combined adverse effect that is worse than the sum of the two separate adverse effects.

Incompatibilities are more likely with complex traits, such as fertility, which rely on systems of genes working together. The larger the system and the greater its complexity, the higher the risk of something going wrong when two populations interbreed.

That “something” will involve only a tiny minority of the genes in any gene system — the vast majority will be essentially the same in the two parent populations. Nonetheless, that tiny minority will have an outsized impact—like a monkey wrench thrown into a threshing machine, or like a gene mutation in a developing embryo, except that new mutations are normally fewer in any one embryo.

In short, genes are team players. As the geneticist Ernst Mayr wrote:

Hybridization between species leads almost invariably to unbalance through deleterious combinations of genes. ... in [the fruit fly] Drosophila even the hybridization of races may lead to destruction of well-integrated gene combinations. (Mayr, 1970, p. 399)

The risk of incompatibility depends not only on the complexity of the trait but also on the degree of divergence between the two parent populations. The more their genomes have diverged from each other, the higher the risk of something going wrong.

Such divergence is due partly to the passage of time and partly to differences in natural selection:

• Passage of time. The longer the two parent populations have been apart, the more their genomes have diverged, if only through genetic drift and other stochastic processes.

• Differences in natural selection. The more the two parent populations have adapted to different environments, the more their genomes have diverged in evolutionarily significant ways. Unlike genetic drift, this divergence is not a linear function of time. It is faster when the two parent populations have just separated from each other—typically, when one of them has split off from the other and is adapting to a totally different environment. Later, the pace of divergence will slow down. The short period of rapid evolution will give way to a much longer period of genetic finetuning with occasional adaptive breakthroughs.

In a review of the literature, back in the late 1960s, Ernst Mayr wrote that no one had yet shown that outbreeding reduces human biological fitness. He also pointed to the lack of controlled studies on mortality and fertility in the first and second generations. Finally, he added: “nor is there much evidence on nonphysical traits” (Mayr, 1970, pp. 399-400).

Since Mayr’s time there have been two controlled studies on outbreeding and human fertility, for both the first and second generations.

The Icelandic study

A research team looked at the fertility of 160,811 Icelandic couples born between 1800 and 1964, using a database that covers 95% of all Icelanders born since 1700. Fertility varied with the relatedness of the parents. Specifically, marriages between third or fourth cousins produced the most children, who likewise enjoyed higher fertility:

... the reproductive success of the couples, as reflected by the number of their children who reproduced, followed an n-shaped curve from the relatively low reproductive success of couples related at the level of second cousins or closer, to the maximum for couples related at the level of third and fourth cousins, after which there is a steady decrease in reproductive success with diminishing kinship between spouses. A similar picture emerges when the number of grandchildren per couple is examined (Helgason et al., 2008)

The researchers proposed a biological cause:

Our results, drawn from all known couples of the Icelandic population born between 1800 and 1965, show a significant positive association between kinship and fertility, with the greatest reproductive success observed for couples related at the level of third and fourth cousins. Owing to the relative socioeconomic homogeneity of Icelanders, and the observation of highly significant differences in the fertility of couples separated by very fine intervals of kinship, we conclude that this association is likely to have a biological basis. (Helgason et al., 2008)

Could the fertility-kinship relationship be due to social changes over time? In Iceland, mean fertility decreased from 1800 to 1965, while outbreeding increased. To control for those changes, the authors divided the data into 25-year intervals. Again, “the same general relationship between kinship and reproductive outcome was observed within each 25-year subinterval” (Helgason et al., 2008).

What about differences in educational status? Perhaps Icelanders who married beyond their fourth cousins were more likely to go to university and meet a future spouse from the other side of the country. They would go on to have fewer children, but only because they had married later in life.

That potential confound has been investigated in a study that examined how Danish fertility varies with the distance between the wife’s home parish and her husband’s.

The Danish study

A research team looked at the fertility of 22,298 Danish women born in 1954 and still living in their country in 1969. The cohort was followed up to the end of 1999. The data came from the Danish Central Personal Register, which covers almost all of Denmark’s residents.

The researchers compared each woman’s fertility with the distance between her home parish and her husband's. Fertility peaked at around 75 km. The relationship between fertility and distance was not explained by education, family income, urbanicity or mother's age at first birth. Again, the cause seemed to be parental relatedness. At longer distances between the wife’s parish and her husband’s, a steady decrease in inbreeding depression gave way to a steady increase in outbreeding depression (Labouriau and Amorim, 2008).

Mean number of children as a function of geographic distance between Danish marriage partners (Labouriau and Amorim, 2008).

The Icelandic and Danish studies have their respective advantages and disadvantages. The former had a larger sample size over a longer period. The family data also came from “one of the most socioeconomically and culturally homogeneous societies in the world, with little variation in family size, use of contraceptives, and marriage practices" (Helgason et al. 2008).

The latter study was likewise done on a relatively homogeneous society, at least during the period in question. The researchers also controlled for a greater number of possible socioeconomic confounds. On the other hand, parental relatedness was not directly measured.

Outbreeding and decrease in sperm quality

Outbreeding can reduce fertility in many ways. Incompatibilities may arise during embryonic development and lead to miscarriage. They can also arise at a later stage of development, in the fetal germ cells, and thus reduce the fertility of the next generation. The epidemiologist Michael Joffe has argued that the increase in outbreeding explains a century-long decline in sperm quality and a corresponding increase in testicular cancer (Joffe, 2009).

Joffe rejects the usual explanation — the rising level of estrogenic compounds in the environment, like dioxin, DDT, PCBs, PBBs, phthalates, and others. This putative cause fails to explain why the decline in sperm quality varies so much between different geographic regions, often within the same country. Why, for instance, has it been steep in Paris and nonexistent in Toulouse? Why is it nonexistent in domestic animals that are no less exposed to estrogenic compounds? And why did the decline in sperm quality begin before the commercial production of most estrogenic compounds?

Similar doubts have been raised by Deonandan and Jaleel (2012), who point out that the decline in sperm quality seems to be confined to sperm donors from Western countries. No decline is observable in a meta-study of sperm donors from non-Western countries who “likely … were not rural, agrarian workers, but urban individuals experiencing the same environmental and occupational exposures as men from wealthier countries.”

Joffe attributes the male fertility crisis to an increase in outbreeding in both the previous generation and the current one, and a consequent increase in “D&D” — duplications and deletions of genetic material during meiosis:

On the face of it, this contradicts the earlier observation that hybrid vigour would tend to increase fertility in unrelated individuals. Yet these two ideas may be compatible: both extremes — genetic similarity (inbreeding) and genetic distance (D&D accumulation) — could decrease fertility, so that an intermediate degree of relatedness would be associated with the highest degree of fertility. (Joffe, 2009, p. 303)

A Jamaican study: impact on psychological traits

Ernst Mayr claimed there was little evidence that outbreeding has adverse effects on mind and behavior. Yet, at the time he wrote those words, there was already evidence in the form of a published study. In 1926, Charles Davenport, a Harvard zoology professor and leading biologist, went to Jamaica with his assistant Morris Steggerda to examine 300 adults and some 1,200 children of Black, White, and Biracial (“Brown”) origin. The two researchers administered anthropometric, physiological, and psychological tests.

In general, the tests placed the Biracial participants midway between the Black and White ones. But a higher proportion of the Biracial participants failed certain psychological tests, particularly when asked to make comparisons or to imagine objects in two or three dimensions:

Failure rates of Black, White, and Biracial participants for tests of comparison or 3D imagination (Davenport, 1928).

Davenport concluded:

One gains the general impression that, though on the average the Browns did not do so badly, there was among them a greater number of persons than in either Blacks or Whites who were muddled and wuzzle-headed. The Blacks may have low intelligence, but they generally can use what they have in fairly effective fashion; but among the Browns there appear to be an extra 5 per cent who seem not to be able to utilize their native endowment. (Davenport, 1928, p. 69)

Since the negative impacts of outbreeding were evident on only some of the psychological tests, one could accuse Davenport of cherry-picking. On the other hand, there was little evidence of positive impacts:

The question arises: are there any traits in which, on the average, the adult Browns are superior to the Whites? We might, theoretically, expect such, yet when we have tested all of the probably genetically distinct traits between Blacks and Whites, we find only one in which the mean of the adult Browns is clearly higher than that of either parental stock.

On the other hand, if we examine the means for children of 10 to 13 or 13 to 16 years there are a few in which the brown children grade higher than either the black or the white children. ... The fact that youthful Browns sometimes score higher than youthful Blacks or Whites suggests the conclusion that brown children develop in some mental capacities precociously; and then fall behind in development. (Davenport, 1928, pp. 69-70)

If we exclude the possibility of incompatibilities during development, we are left with two explanations:

• The Biracial participants had suffered some kind of deprivation that the Black and White ones had not.

• The researchers had oversampled the population of Biracial participants with certain mental issues.

Neither explanation seems likely. In the 1920s, Biracial Jamaicans were generally a privileged group. They dominated the middle class. Nor is it clear why those with mental issues would be oversampled. 

Conclusion

The creation of human life is a complex process that requires coordination between many genes. It is thus vulnerable to malfunctions at various stages of development, particularly at the embryonic stage. The result is usually a spontaneous abortion, most often without the mother even knowing she was pregnant. Such malfunctions are difficult to observe directly; for the most part, the only observable sign is a reduction in fertility, i.e., the ability to conceive a child.

Fertility is thus the canary in the coal mine. It is an early warning indicator of developmental problems that may otherwise go unnoticed. Among other things, it can show us the optimal degree of parental relatedness for childbearing.

Where is the sweet spot between being too closely related and being too distantly related? Apparently, it is at the degree of relatedness of third or fourth cousins. If you marry closer in or farther out, your children will be less biologically viable.

Inbreeding and outbreeding can cause embryonic development to fail in different ways:

• Inbreeding. Parents are more likely to share the same defective allele at various gene loci. Recessive disorders may develop in the embryo, leading to higher rates of miscarriage. This reduction of fertility is called inbreeding depression.

• Outbreeding. Parents are more likely to combine genetic variants that have never co-adapted to each other. Incompatibilities may develop in the embryo, leading to higher rates of miscarriage. This reduction of fertility is called outbreeding depression.

Outbreeding has been on the rise in the West. Its effects might seem beneficial, as indicated by the upward trend of height and the downward trend of menarche. Both trends, however, have come to a halt: “In Northern Europe, adult height has largely stabilised, and the age of menarche has also settled at around 13 years, while weight continues to increase due to obesity” (Cole, 2003). The rise in IQ, known as the Flynn Effect, has sometimes been attributed to outbreeding, although that claim has been disputed (Flynn, 2007, pp. 101-102; Woodley, 2011). In any case, the Flynn Effect too is coming to a halt throughout the West (Flynn, 2007, p. 143).

The Icelandic and Danish studies, together with the rise of infertility throughout the West, suggest that outbreeding has become more problematic than inbreeding. Yet most people seem to think otherwise. When I did a Google search for "inbreeding is bad," I got 15,200 hits. "Outbreeding is bad" yielded only 7.

How bad can outbreeding be for humans? Davenport examined perhaps the maximum degree of outbreeding possible within our species, and only a minority of individuals from the cross-bred population showed more dysfunction than expected. The proportion varied from one psychological test to another, from a low of about 5% to a high of 30%. That proportion should decline over successive generations within an initially cross-bred population, as natural selection removes less functional individuals through illness, reduced fertility, and inability to find a mate.

Davenport interpreted his findings cautiously, noting the difficulty in estimating the size of outbreeding’s negative effect. And he analyzed his data at a time when statistical analysis was less advanced than it is today, with no tests of significance for race, age or sex.

Davenport’s self-criticism was taken up by his critics, and the current prevailing view is that his study has been discredited, if not by methodological issues, then … “because.”

Peter Frost has a PhD in anthropology from Université Laval. His main research interest is the role of sexual selection in shaping highly visible human traits, notably skin color, hair color, and eye color. Other research interests include gene-culture coevolution. Find his Newsletter here.

---

Support Aporia with a $6 monthly subscription and follow us on Twitter.

---

References

Cole, T.J. (2003). The secular trend in human physical growth: a biological view. Economics & Human Biology 1(2): 161-168. https://doi.org/10.1016/S1570-677X(02)00033-3

Davenport, C.B. and Steggerda, M. (1928). Race Crossing in Jamaica. Washington: Carnegie Institution, Publication no. 395. http://www.velesova-sloboda.info/archiv/pdf/davenport-race-crossing-in-jamaica.pdf   

Deonandan, R., and Jaleel, M. (2012). Global decline in semen quality: ignoring the developing world introduces selection bias. International Journal of General Medicine 5: 303-306. https://doi.org/10.2147/ijgm.s30673

Flynn, J.R. (2007). What is Intelligence? Beyond the Flynn Effect. Cambridge University Press.

Helgason, A., Pálsson, S., Guðbjartsson, D.F., Kristjánsson, þ., and Stefánsson, K. (2008). An association between the kinship and fertility of human couples. Science 319(5864): 813-816. https://doi.org/10.1126/science.1150232

Joffe, M. (2010). What has happened to human fertility? Human Reproduction 25(2): 295-307. https://doi.org/10.1093/humrep/dep390  

Labouriau, R., and Amorim, A. (2008). Comment on "An Association Between the Kinship and Fertility of Human Couples." Science 322(5908): 1634. https://doi.org/10.1126/science.1161907  

Mayr, E. (1970). Populations, Species, and Evolution. Cambridge (Mass.): Belknap Press.

Presgraves, D.C. (2010). The molecular evolutionary basis of species formation. Nature Reviews Genetics 11:175-180. https://doi.org/10.1038/nrg2718

Teasdale, T.W., and Owen, D.R. (2005). A long-term rise and recent decline in intelligence test performance: The Flynn Effect in reverse. Personality and Individual Differences 39(4): 837-843. https://doi.org/10.1016/j.paid.2005.01.029

Turner, L.M., Schwahn, D.J., and Harr, B. (2011). Reduced male fertility is common but highly variable in form and severity in a natural house mouse hybrid zone. Evolution 66(2): 443-458. https://doi.org/10.1111/j.1558-5646.2011.01445.x  

Woodley, M.A. (2011). Heterosis doesn't cause the Flynn effect: A critical examination of Mingroni (2007). Psychological Review 118(4): 689-693. http://dx.doi.org/10.1037/a0024759