Philosophy, Science, Top Stories

The Strange Truth About Alternative Facts

Facts are overrated in political and moral debates. They’re fragile and impotent. They don’t do the work expected of them. What follows is an explanation of why no one should be persuaded by most facts. This is not a rehashing of the well known inability of facts to persuade emotional beings. Nor is it a postmodern denial of truth. My point is that facts which are absolutely true can be absolutely irrelevant in overlooked ways.

It’s obvious that facts can be irrelevant; everyone knows that. But some facts which seem relevant are not. For example, let’s say you ask me whether or not women face gender bias in grad school admissions and I tell you that hummingbirds can fly backwards. This is an obviously irrelevant fact. But if I told you that data shows female applicants were significantly less likely than male applicants to be accepted to grad school, that would also be irrelevant. How could that fact possibly be irrelevant? It seems like exactly the kind of fact that could help answer the question. But the fact is at the wrong scale; its unit of analysis is wrong, which makes it irrelevant. This requires some explanation…

The term ‘unit of analysis’ comes from the quantitative social sciences. It refers to the specific type of thing a scientist wants to measure and compare. For example, an economist comparing the GDP of Mexico to the GDP of Japan is using ‘country’ as his unit of analysis. A psychologist conducting a controlled experiment is taking ‘treatment group’ as his unit of analysis.

Unit of analysis is not to be confused with the unit of observation, which is the unit described by one’s data. For example, a study may have a unit of observation at the individual level but may have the unit of analysis at the neighborhood level, drawing conclusions on neighborhood characteristics from data collected from individuals.

When two scientists choose two different units of analysis, they may come to apparently opposite conclusions, both of which are true. In applied statistics, this is known as Simpson’s paradox. One of the best-known examples of Simpson’s paradox is a study of gender bias among graduate school admissions at University of California, Berkeley. The admission figures for the fall of 1973 showed that men applying to the university were more likely than women to be admitted. But it was found that men were not more likely than women to be admitted by the academic departments which make up the university. This is explained by the fact that women tended to apply to departments with low rates of admission among qualified applicants (such as the English Department), whereas men tended to apply to departments with high rates of admission among the qualified applicants (such as engineering and chemistry).

This table shows a simplified version of how that can work:

In this example, the statement ‘men are more likely to be accepted’ is true, taking ‘university’ as the unit of analysis. At the same time, given the same data, the statement ‘women are more likely to be accepted’ is true, taking ‘department’ as the unit of analysis. These are literally alternative facts. Both statements are true, but which is more relevant? That depends on the question the facts are meant to answer. A question about the university as an indivisible whole calls for one correct answer, while a question about the departments calls for the opposite correct answer. So the real question is, what’s the right question? Since nobody applies to the university as an indivisible whole, the right question is about the departments, and so the relevant fact is that women are admitted at a higher rate than men. The fact that women are admitted to the university at a lower rate than men is irrelevant.

Now we see that the question must specify a unit of analysis if we want a coherent answer. The same is true of questions asked in moral and political philosophy. What kind of society is most fair, or most safe, or most just, or most beneficial, or most free? No matter which question one sets out to answer, the question must specify a unit of analysis. Most fair for whom? For people. But for people at which scale? For individuals, or for families, or for neighborhoods, or for genders, or for races? Simpson’s paradox implies that two philosophers who ask the same question with different units of analysis will sometimes come to opposite conclusions.

If one were to make moral judgements about the admissions at Berkeley, for example, conclusions would depend on the units of analysis chosen as units of moral agency, and the unit of moral harm. Suppose we take the ‘individual’ to be the unit of moral harm. Then we care about outcomes for individuals, and individual applicants only apply to single departments, so we will look for evidence of departmental treatment of applicants, and find no bias against women. Or suppose we take gender as the unit of moral harm. Then we have an array of contradictory units of moral agency from which to choose. Is the whole of the university an agent which produces an outcome (bias against women)? Or are the departments agents which each produce their own outcomes (no bias against women)? Or maybe we decide that a societal system of male bias is the relevant agent, which should be counteracted by a system of localized female bias in the university. Each of these a priori units of analysis leads to different conclusions about the moral character of grad school admissions at Berkeley.

Now if a single moral judgement depends on unit of analysis, then surely a political philosophy which imagines some sort of societal structure that maximizes some moral values (whatever they may be), must also depend on unit of analysis.

As we have seen, apparently conflicting conclusions can all be correct, given different a priori units of analysis. But is there such a thing as a correct unit of analysis? Philosophers have used state, economic class, gender, individual, race, neighborhood, and many other units. And there are practically infinitely many more ways in which it is possible to group people for analysis. The choice of unit is a choice between an infinite number of options. I have argued elsewhere that ‘individual’ is the only correct unit of moral analysis. But here, my goal is just to show that true facts are irrelevant if their units of analysis are wrong, and since there are so many possible units of analysis, a person with an opinion should have a good reason for choosing one unit over the others.

Citing true facts is child’s play. If you’re interested in moving from facts to Truth, you need a good reason to believe your facts are relevant, which means you need a good reason to believe your questions are relevant.

 

Stuart Doyle is a Recon Marine with a BA in Neuroscience & Behavior from Columbia, and an MS in Criminology from UPenn. You can follow him on Twitter @philosoraptor21

20 Comments

  1. MDM says

    I agree individuals should be the unit of analysis, but at some point groups come into play when groups of individuals diverge from others. Does this negate truth or is truth a matter of select facts as a coin has two sides but is still a coin.

    • Joe says

      In my opinion, group analysis can only be used to address group problems. However, when you go and implement a policy or solution to rectify group problems, you must be willing to accept (or willfully ignore) that you’re going to end up with more individual or subgroup problems. As in the example cited by the author, if you apply a blanket policy to university admissions so that men and women have equal admission rates overall, you’ll probably find that this creates an even greater disparity at the department level in favor of women. You’ll still get the intended results, but in the process you will have effectively implemented active discrimination against men and possibly have lowered the average quality of the individuals admitted.

  2. Ian says

    Thank you, this has clarified some of my own thinking. I struggle in a male dominated occupation (policing) which aspires to 50:50. It is not going to happen unless merit is thrown out the window. No one has suggested that Nursing should aspire to 50:50 albeit there are some wonderful male nurses. My question is”How should diversity be measured?

    • ga gamba says

      An alternative question is: Why should diversity be measured?

      What’s the intent? Is it to measure a shortfall? If so, where? Of whom? Seems to me it should be applied uniformly then if we’re using it as the basis to craft policies to rejigger society. I’m sure this may upset many of the women who are about 90% of veterinary science students and will have to be expelled to jobs where women are greatly under represented. Deep sea fishing, for example. Hey, it’s working with animals still. Always a silver lining.

      No one has suggested that Nursing should aspire to 50:50 albeit there are some wonderful male nurses.

      Why not? OK, I’m not only suggesting it, I’m demanding it. End the institutionalised and systemic under representation of men in nursing.

      I think people ought to really think about the mantras they’re encouraged if not coerced to profess.

    • If ‘individual’ is the unit you care about, then ‘diversity’ is neither good nor bad.
      In a world perfectly fair to individuals, there would be no discrimination against women trying to become police officers, but there would still be far fewer of them than the men.

  3. But for people at which scale? For individuals, or for families, or for neighborhoods, or for genders, or for races?

    • LOL oops. I was going to say the answer is easy, Freedom. But seriously, good article, gave me something to really think about, which is rare these days. Thank you.

  4. Tezza says

    I enjoyed and learnt from this piece. Commendably brief too! Thanks

  5. ga gamba says

    Citing true facts is child’s play.

    Indeed. If we look at the demography game being played by activists it’s simple maths, i.e. addition, subtraction, and division, distinguishing gender, which could become very burdensome because there are 92 of them; and recognising a few colours, and perhaps shades if and when colourism takes hold.

    This is being perpetrated by people with uni educations, often possessing advanced degrees.

  6. Chris says

    refreshing read and a relevant message!

    two things that crossed my mind is how much an individual (especially young people) are actually influenced by groups and how much they try to belong to one. In other words, I wonder if the statement to assess at individual unit level isn’t based an idealized world view. My generation accepted individuality as something worth striving for, and I would certainly agree with the author, but I’m not sure to which degree that is currently the case and what it means for following generations. Think of how helpful or not it was for the average peasant or factory worker in the distant past.
    secondly, the example cited begs the question, why we ask certain questions over and over again; why is equality of outcome (code word: diversity) such an important thing and who is asking for what reasons

  7. Joe says

    Great article. You offered a satisfying explanation of your claim, and I only had to scroll twice! Simpson’s rule is quite profound, and most explanations I have encountered have been divisive persuasion essays. This was a nicely balanced explanation which addressed a charged issue with logic and tact.

    My only nitpick is regarding in the conclusions of differing units of analysis for univserity admission. At the university level, the conclusion rightly yields “bias against women” since women are accepted at a lower rate than men overall. An equal, alternative statement would be “bias for men”. At the department level, men have lower acceptance rates than women, which should lead to the conclusion of “bias against men” or “bias for women”. Instead, “no bias against women” is used as a neutral conclusion where the result is clearly (as stated earlier in the article) that women have higher department level acceptance rates.

    This inconsistency demonstrates an even deeper error made by people blinded by confirmation bias: the inability to admit or tolerate a truth which doesn’t match their biased expectations. I’m not accusing the author of unmitigated confirmation bias, I just thought it was worthwhile to point this out.

  8. anthony senatore says

    I really enjoyed reading this piece, and that is a fact.

  9. Jeff Burns says

    Some day, Western thinkers will come to grips with the idea that there is no “Truth”, only truths, and that some of these will be inherently contradictory and irreconcilable.

  10. Cnan says

    Right on. And many of the units of analysis and the reasons for prioritising them, are not spoken about, merely implied.

    Much of it will probably go back to whether and how one’s bases of reasoning include the Hebraic god or not.

  11. ccscientist says

    A classic case of wrong unit of analysis is when people talk about the “middle class” getting poorer over some time period. But there is no such thing as the middle class over time. Young people make less and gradually increase their wages over time. Longitudinal data show that a large percent of the population makes it into the top 10% for at least a year. Likewise, much of the top 1% is only in it for a year or two (due to stock options or inheritance or such). The 1% is different people over time.
    The more useful metric would involve upward mobility, not income inequality. I personally never experience income inequality except by reading about billionaires but I do personally experience my path of upward mobility.

  12. Joe Gerth says

    Fantastic article, with a great example to help explain it. I’ve often worried about the misuse of statistics, particularly in my field, education.

Comments are closed.