Economist Doug Allen’s new study, published earlier this month, found lower high-school graduation rates among children of same-sex couples than those of opposite-sex couples and even single parents. (A brief and informative summary of the study is available here.) Meanwhile, the majority of, but not all, prior studies have concluded that there are “no differences” in outcomes for children raised by same-sex couples.Obviously, both of these claims cannot be true. So what are the differences between the studies that cause them to find such different results?To put it bluntly, statistics is a messy science. Previous statistical studies have not adequately established a consensus because almost all of these studies have been substantially subpar. Unfortunately, this point has rarely been conveyed in articles reporting about these studies. By and large, earlier studies have used nonrandom sampling and sample sizes that are too small to detect even large differences. Although Allen’s study is limited in some respects, it tells us more than almost all of the previous studies on the subject, because Allen is able to analyze a large number of children from a random, representative sample. In fact, when this basic selection criterion is implemented, only one other study makes the cut.

Because of a scarcity of data on the subject, much of the social science around same-sex parenting has deviated from the ideal in order to glean what knowledge can be obtained from limited data sets. While this is understandable, it is nevertheless a concession. Researchers should not “settle” by prematurely accepting a consensus about what’s possible in select same-sex parenting communities rather than pursuing the truth about what is probablein the population of same-sex households at large. Whether it is probable that children of same-sex couples fare better or worse than those of opposite-sex couples still remains largely unknown because of the failings of the vast majority of prior studies.

While no statistical study is perfect, most of the previous literature falls well short of even basic criteria for making convincing arguments. In terms of random sampling and large sample size, which are the most basic of social scientific criteria, the vast majority of the previous literature simply fails. Meanwhile, among the small number of studies that pass even these foundational tests, the decision is still split.

How Big Is Big Enough?

Allen’s study has far more cases than most of its peers, with nearly 1400 young adults raised by same-sex couples. Meanwhile, among studies conducted prior to 2010 (which constitute forty-four of the fifty studies reviewed by Allen), the average sample size was just sixty-nine children raised by gay or lesbian couples. The largest of these early studies included only 475 children, making it just barely acceptable in terms of sample size.

Why is a small sample size a damning characteristic of such studies?

Suppose a researcher wishes to compare two groups based on their likelihood of exhibiting a certain behavior—in the case of Allen’s new study, completing high school. Further, suppose that the true unknown underlying population average for Group A is 87%, and for Group B it is 90%. This means that the failure rates for each group are 13% and 10%, respectively. Now, a 3-percentage-point difference may not seem like much, but it would represent a 30% increase in the rate at which children fail to finish high school in Group A as compared to Group B. This indicates a dramatic distinction that could have far-reaching implications for such children’s future job prospects and quality of life.