Psychology, recent, Recommended, Science

Bad Data Analysis and Psychology’s Replication Crisis

In 2014, a study published in JAMA Pediatrics linked playing aggressive video games to real-life aggression in a large sample of Singaporean youth. The study attracted considerable news media attention. For instance, a sympathetic article in Time magazine breathlessly reported its findings and suggested that brain imaging research found aggressive games change kids’ brains. But was the evidence from the Singapore study reliable?

In recent years, concerns about the Singapore dataset have grown. UK scholars Andrew Przybylski and Netta Weinstein recently wrote that the way the dataset had been used by the primary authors was problematic. The analyses from the same dataset kept changing across published articles in suspicious ways. These shifting analyses can be a red flag for the data massaging that may produce statistically significant outcomes and hide outcomes that didn’t work out. Such practices may be unintentional or unconscious (scholars are only human after all). But they do suggest that the results could do with further scrutiny.

When the dataset became available to my colleague John Wang and me, we re-analyzed the data using more rigorous methods. We publicly pre-registered our analyses, which meant we couldn’t subsequently alter them to fit our hypotheses. Our results were strikingly different from the 2014 paper: in fact, there was no evidence in the dataset that aggressive game play was related to later aggression at all. So, what happened? How did a dataset come to show links that don’t exist between aggressive video games and youth aggression?

This isn’t the first time this has happened in video game research. Recently, another study appeared to link violent media to irresponsible gun behavior among youth. However, an independent researcher re-analyzed that dataset and found that the questionable elimination of a few inconvenient participants had transformed non-significant results into significant ones.  Furthermore, recent brain imaging studies have not supported the claims made in Time‘s 2104 article.

The problem is larger than video game research though. For almost a decade now, psychological science has been undergoing a significant replication crisis wherein many previously held truisms are proving to be false. Put simply, many psychological findings that make it into headlines, or even policy statements issued by professional guilds like the American Psychological Association, are false and rely upon flimsy science. This happens for two reasons. The first is publication bias wherein studies that find novel, exciting things are preferred to those that find nothing at all. And second, death by press release, or the tendency of psychology to market trivial and unreliable outcomes as having more impact than they actually do.

Publication Bias

The tendency to publish only or mainly statistically significant results and not null findings is due to a perverse incentive structure within academia. Most of our academic jobs depend more on producing research and getting grants than they do teaching or committee work. This creates the infamous publish or perish structure, in which we either publish lots of science articles or we don’t get tenure, promotions, raises, prestige, etc. Scientific studies typically require an investment of months or even years. As an academic, if we invest years on a science study we must get it published or we will lose our jobs or funding.

If journals only publish statistically significant findings, the outcome of that study must be statistically significant. Typically, scholars can choose from dozens or even hundreds of ways to analyze their data. Thus, if we need statistically significant results, we can simply cycle through these different analytical options until we get the outcomes we need to publish. Variations on this include p-hacking (when analyses are run in multiple ways, but only those that produce statistically significant findings are reported) and HARKing (Hypothesis After Results are Known.) Harking occurs when scholars run multiple analyses between numerous variables without any particular theory.  When they publish their findings, they pretend the statistically significant ones were those they predicted all along. In fact, they are usually the product of random chance and therefore unreliable. This perverse incentive structure is thought to be the source of considerable scientific misconduct.

So why do journals tend to publish only statistically significant findings? Part of this has to do with the nature of traditional statistical analyses, which made it easy for scholars to dismiss non-significant results as lacking any meaning. Effectively, what is called Null-Hypothesis Significance Testing actually makes it difficult to prove a theory false, which is a way of turning science on its head. But also, statistically significant findings are more exciting, increase readership (this matters to academic journals too), and attract media interest. Dutch scholar Daniël Lakens recently said:

I agree.

Death by Press Release

Death by Press Release occurs when scholars fail to disclose the trivial or unreliable nature of some findings, often through a university press release. The recent imbroglio over whether social media leads to mental health problems in teens is an example. A recent study by Jean Twenge and colleagues claimed that social media use is associated with decreased mental health among youth. Pretty alarming! However, a re-analysis of the data by Oxford researchers found that, although statistically significant, the effect was no greater than the (also significant but obviously nonsense) correlation between eating bananas and mental health or wearing eyeglasses and mental health, neither of which produce anxious think-pieces.

In large samples (the studies mentioned above had hundreds of thousands of participants), very tiny correlations can become “statistically significant” even though the magnitude of the effect is tiny. Usually this magnitude or “effect size” is demonstrated by the proportion of variance in one variable explained by another. In other words, if the only thing you knew about people was variable X, how accurate would you be in predicting variable Y above chance. So, zero percent variance explained would be literally no better than a random coin toss, whereas 100 percent would be perfect predictive accuracy. The effects for screens on mental health suggests screens account for far less than one percent of variance in mental health, little better than a coin toss.  Dr. Twenge has published defenses of her results, arguing that many important medical effects also have small percentage of variance explained. However, these claims are quite simply based on miscalculations of the medical effects which are actually much stronger in terms of variance explained. Although these miscalculations were revealed over a decade ago, this scientific urban legend is sometimes repeated by psychologists because it makes psychological research sound much stronger than it actually is.

So, “statistical significance” doesn’t really mean very much and the first thing people should wonder when they hear about a scientific result is “What is the effect size?” Effects that are very tiny (explaining 1–4 percent of variance or less) are often not real at all, but rather the noise of survey research, participant hypothesis guessing, unreliability of respondents, and so on. They’re not just really small, they actually don’t exist in the real world. In the spirit of Dr. Lakens’s comment about publication bias, psychology’s inability to let go of tiny effect sizes or communicate them honestly to the general public is another (arguably unethical) element of psychological science that damages our credibility and grossly misinforms the public.

This phenomenon is abetted by two groups. The first group are the professional guilds, such as the American Psychological Association or the American Academy of Pediatrics, which many in the general public mistake for science organizations. They’re not. They’re dues-paying organizations that function to market a profession, not necessarily tell objective truths (disclosure: I am a fellow of the American Psychological Association). The accuracy of the policy statements on science has often turned out to be poor.

The second group are news reporters who often uncritically publish university press releases without doing any fact-checking of claims. Even when the studies they cover are later discredited, they don’t always carry the new information. For instance, Time magazine covered the early Singapore study which purported to show a link between aggressive games and youth aggression. However, they have not covered the subsequent re-analysis, even after I contacted the original reporter to inform her of the correction in science. The truth is that flashy, exciting, and novel findings tend to get news coverage. But when those findings later turn out to be unreliable, news media often drops the ball. Perhaps corrections aren’t that thrilling, or perhaps they simply don’t want to acknowledge they didn’t do any due diligence with university press releases in the first place.

In 2005, statistician John Ioannidis published an article in which he claimed that most published research findings are false. Almost 15 years later, not much has changed.  Psychological studies continue to be found to be unreliable and unstable. Addressing publication bias through more transparent science has resulted in some clear improvements. However, until psychology acknowledges that most effect sizes, even in high-quality studies, are trivial and unlikely to have an impact in the real world, we will continue to deceive more than we illuminate. Our challenge is to be more honest about how much we still don’t know about the workings of the human mind. Hopefully, the next 15 years will see more progress toward an open, honest science that acknowledges it finds nothing at all far more often than it does clear effects.

 

Christopher J. Ferguson is a professor of psychology at Stetson University in Florida. He is author of Moral Combat: Why the War on Violent Video Games is Wrong and the Renaissance mystery novel Suicide Kings. His forthcoming book How Madness Shaped History will be out in January 2020. You can follow him on Twitter @CJFerguson1111

Photo by Alexander Andrews on Unsplash

86 Comments

  1. Isaac Franklin says

    Interesting to see that we’re living in a period of crisis in mediums of sense-making(media) and truth generation(academia).

    Partly brought on by human bias and partly by Gutenberg 2.0(internet).
    Fake news and destruction of revenue structure for media and junk online journals for academia.

    • David of Kirkland says

      Newspapers suffer less from the Internet than from the fact that people don’t trust them.
      If they returned to reporting the news, questioning statements, investigating, etc. there’s no need for what could have been a successful free press.

      • tommy mc donnell says

        you know how liberals always said “question authority,” well what the american people really need to question is the news media.

      • staticnoise says

        You mean reporting instead re-printing Twitter screenshots and ignoring any critical analysis of said Twitterer? God I hate Twitter.

    • Farris says

      “Interesting to see that we’re living in a period of crisis in mediums….”

      Mediums was an interesting word choice. These types of studies and prognostications are converting psychology into present day soothsayers, fortune tellers and seers. As with any good fortune teller she tells the client what the client wishes to hear. The practices described in this article are corrupting science. Advocacy groups are paying for studies to justify policies. If psychology wants to prostitute itself, that will be it’s legacy. Chalk up another win for the skeptics.

      • Geary Johansen says

        @ Farris

        Good comment. A recent example of this the attempt by the psychological establishment to brand traditional masculinity as problematic. In a recent interview with The Epoch Times on YouTube, Dr Warren Farrell author of ‘The Boy Crisis’ explained that although there are, apparently, some issues of ‘Toxic Masculinity’- they all stem from male sacrifice, rather than male privilege. So the male tendency to put women and children first, means that we do not tend to our own emotional, material and physical needs. If you’re lucky, you get to buy yourself a big TV when you’re 30, a nice armchair at 40 and a bed that doesn’t screw your back up at 50- and that’s about it.

        • Charles R. Twardy says

          I think they made a clear distinction between traditional and toxic.

    • TarsTarkas says

      The narrative is all, and only data that supports the narrative, no matter how false, is the only data that matters.

      • Isaac Franklin says

        Data can be manipulated too, unfortunately.

  2. Howdy says

    Want to see an epic meltdown? Take electronics, especially “games”, away from someone used to having them.

    • Howdy says

      “So why do journals tend to publish only statistically significant findings?”
      Academics do this because they expect to find evidence to support their preexisting view, so they clearly see that evidence in their research, even to blindness of the things that contradict their belief.

      Their papers will reflect claims of proof if there is something that can be interpreted as evidence for their ideas as well as explanations of contradictions as mere anomalies, irrelevant to the “facts” as a means to achieve their desired reaction from peer and public.

      This is even more prevalent in the sciences among those holding to the (ever changing) evolutionary story line. Even those who have doubts will toe the line though and publish in such a way as to show allegiance to popular mainstream ideas. I say this as someone who has studied biology and see exactly this behavior. Evolutionary ideas are often tucked into a text when its inclusion offers nothing to the study; and leaving it out would take away absolutely nothing. Worst of all, a careful reading of the added text will often be in clear contradiction to the factual scientific information the interloping hypothesis is attempting to hijack for validation.

      So basically, we see what we want to see. The only way to stop this is, first, to be aware of our own bias and attempt to prove our own ideas WRONG instead of trying to prove we are right, and second, recognize and let go of the dogma of what someone else (no matter who or how many) told us was settled fact.

      • Andrew Scott says

        It’s amazing how people who pride themselves on questioning and analyzing what they’re told draw the line at anything having to do with evolution and take a step back from it. There are lines you just don’t cross.

        Sure, we get it drastically wrong when it comes to chemistry, biology, and psychology that’s right in front of us where we can put it under a microscope, watch it, take it apart, and put it back together again.

        But when someone comes up with total nonsense about how human intellect is the result of decreased foliage or complex spiderwebs are the accumulation of simultaneous mutations affecting anatomy, nervous systems, and behavior, all of which must occur at once in order to confer any survival benefit, are discussed only at a vague, high level disconnected from actual biology, and none of which are ever hypothesized in any detail whatsoever…

        (sorry, run-on sentence)

        …that’s where we draw the line. Every single research paper announces significant results. Every last one of them is supported by more just like it, creating the illusion of a mountain of evidence.

        We recognize that the need to publish and the overwhelming preference for research that claims to “discover” something lead to bias, error, and omission of contradictory evidence. But not when it comes to evolution. If you question that, something’s wrong with you. It’s used as a litmus test to determine whether someone is rational and accepts scientific conclusions.

        • Respek Wahmen says

          Are you questioning evolution just for the sake of it? What’s a better hypothesis? If it’s gods then yes, there’s something wrong with you. But it’s ok if you don’t vote.

          • Andrew Scott says

            “We have no idea” would be a better hypothesis. If the only “hypothesis” is utterly implausible and dependent on details which it never includes, we’re not obligated to take it seriously just because there’s no other.

          • Ray Andrews says

            @Respek Wahmen

            If you think that gods are not a logical possibility then on what grounds do you say that? But those of us who question evolution are far from automatically being theists as you seem to suggest and there is something wrong with you if you suppose so.

          • TarsTarkas says

            Darwinian evolution, that being lineages of organisms ‘evolve’ slowly and steadily into different lineages of organisms, does not exist in the fossil record.
            What is observed time and time again is distinctly differing organisms (yes, sometimes distinct is a bit in the eye of the beholder, but is usually measurable in some degree) replacing previous lineages of organisms. What Gould & Eldridge called ‘punctuated equilabria).
            Lineages of organism do not progress, that is, they do not necessarily become bigger, faster, stronger, or smarter. They simply are wiped out and replaced by similar but different lineages. What is ‘fit’ in an ecological niche can in an instant of meteoric impact become unfit. Kenneth Hsu pointed this out. Therefore human intelligence is not the end point of evolution, it is simply the current state of a particular evolutionary lineage that happens to able to transform its environment orders of magnitude greater than any previous set of organisms (like beavers).

        • I like how you jump from this article about how statistical methods with low explanatory power are causing a lot of issues in some of the “softer” science fields to EVOLUTION IS ALSO A LIE!!!! Not every scientific study is based on statistical analysis. You can show evolution by comparing two DNA sequences or two fossil specimens, neither of which rely on regression analysis or p-values. Just because some of the more modern and untested methods are proving problematic doesn’t mean that you throw out decades of hard studies that didn’t rely on those faulty methods to justify their findings.

        • Eddie Marcia says

          Andrew Scott wrote:

          “It’s amazing how people who pride themselves on questioning and analyzing what they’re told draw the line at anything having to do with evolution and take a step back from it.”

          That’s because (as with “climate change”) evolutionary biology is a mixture of hard science and pseudo-religious philosophizing. The modern creed of reductionist atheism is wedded to (among other axioms) the traditional understanding of Darwinian evolution. David Gelernter at Yale recently did a piece outlining the profound problems faced by Darwinian macro-evolution on purely logical and mathematical grounds. This and similar careful analytic evaluations of this sacred dogma are attacked with a fury generally associated with religious zealots. Or else they are simply ignored right up until (pace Galileo) the old emperor theory is seen by all to be naked.

        • Charles R. Twardy says

          A little too dismisive. About half of social science studies replicate, and crowds have been able to predict which ones with about 75% accuracy. Please join one of the ongoing studies to help develop reliability metrics. (Disclaimer: I’m helping run one of them, not yet recruiting but hopefully soon.)

          • Doug F says

            Half? That is a pretty pathetic percentage:)

  3. Robin says

    How did a dataset come to show links that don’t exist between aggressive video games and youth aggression?

    How did the Sokol hoax get published? (“Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity”,[3] was published in the Social Text spring/summer 1996 “Science Wars” issue. It proposed that quantum gravity is a social and linguistic construct.)

    Or how about the James A. Lindsay, Peter Boghossian, and Helen Pluckrose hoax? (The “dog park” study… rape culture among dogs… etc)

    https://www.chronicle.com/article/Sokal-Squared-Is-Huge/244714

    Pretty straightforward. The entire social studies community has been taken over by feminist activists with an agenda. First they begin with a narrative, (ie: man bad women good), then cherry pick results to arrive at their pre-determined conclusion.

    It’s not science, it’s propaganda. This is a front line of the culture war and the first casualty is anything resembling truth.

    As for the field in which the author is studying in question you don’t see the Gamergate connection? Mostly boys play games and since all boys are bad then all games must be bad. A study that ‘proves’ it makes boys aggressive is pure gold to the ideologues!

    • Brett says

      I will adjust my statement below some as I do agree with you. It isn’t just money for everyone, for many it fits their agenda.

      I would really like to know how much confirmation bias plays into the lack of fact checking / accountability vs the lack of money in it.

    • David of Kirkland says

      Caveat emptor has to do with ideas too.
      You point out “problems” that are the feature, the way things work. What next, expecting corporations to report bad news about their businesses and products and service? Politicians explaining that lobbyists wrote their laws after giving “donations”? Drug dealers informing you of the negative effects of crack?
      Without trusted institutions that can fact check each other, we are left to apply “critical thinking” to topics we simply cannot know well enough to actually be critical in a reasoned way.
      Society simply relies upon information from others, and sorting out good information from the bad remains what distinguishes the well off from the poor, and it will separate them further in the future as our ability to fake out large numbers improves.

    • Charles R. Twardy says

      Fascinating case studies that point to a real problem in quality control, but the conclusion is too extreme. Cancer research is probably not “taken over by feminist activists with an agenda” – but has a similar replication problem. Separate issues.

      Current estimates (from actual tests) put about 50% of social science as replicating.

  4. Brett says

    Very good article. From other things I’ve read, this doesn’t surprise me. It seems it boils down to money. The rewards are for cool, new, exciting, or edgy papers. Not for retractions or “nothing to see here” papers.

    Sadly, I think this is not just in media/academia, this is prevalent in its own form everywhere.0

    Actual solid integrity seems to be lacking, seriously lacking in the world today.

    • David of Kirkland says

      Integrity is as strong as ever before. It’s just that those without integrity have a much better way to reach people than before communications and transportation improved to make it easy, fast and global for nearly everyone.
      The very idea of democracy took place at a time when people learned and debated in face to face conversations. The USA knew better than to let everyone vote, but despite progress in communications and travel, we instead believe that more is better, that opinions and work are equal (like the equal pay for equal work nonsense of women’s soccer — shall the women’s and men’s teams compete to see how equal they are?), etc. It’s why classrooms spend more time on slow learners, boring the best and stressing/shaming/boring the worst.

  5. Ray Andrews says

    I was brought up to believe that if Walter C. said it on the CBS Evening News it was true. If I read it in the local paper it was true. Later on I came to believe that at least video of an event established that it was true. Now I believe nothing that I don’t see with my own two eyes. Folks tut-tut me for relying on anecdotes, but alas they are the only thing I can trust. For example, every day I hear about the horrible racism and sexism and every other ism you can name that drenches my society. But I never see it. Shouldn’t I be seeing it every day since we are drowning in it? For lack of anecdotes supporting belief I find myself disbelieving. But folks here will tell me that there’s no such thing as global warming, yet over my lifetime I’ve seen the rise in sea level with my own two eyes. And so on. Wouldn’t it be special if we had some source of information we could trust? Or maybe some sort of Better Business Bureau analogue for sources of information.

    • David of Kirkland says

      Watch out for magicians. And you might read up on the accuracy of eye witnesses. “Seeing is believing” is also out.

      • Ray Andrews says

        @David of Kirkland

        Yes, I understand that eyewitness testimony is unfashionable at the moment. But unfortunately ‘seeing is believing’ is all that I’ve got and it has generally proven reliable. I’d much prefer wider and more expert sources of information but I don’t know where to find them. And the very sociologist people who tell me that I can’t trust my own eyes are the very people who are referred to above. They want me to trust them, but I don’t.

        • D.B. Cooper says

          @Ray Andrews

          Yes, I understand that eyewitness testimony is unfashionable at the moment.

          By “unfashionable” do you mean “hands up don’t shoot” unfashionalbe, or some other type of passe verification process?

          Just wondering…

    • Raoul says

      “But folks here will tell me that there’s no such thing as global warming, yet over my lifetime I’ve seen the rise in sea level with my own two eyes”

      I doubt that there are many people here who would tell you that. Most people don’t have any issue accepting that the planet has experienced modest warming following the end of the ‘little iceage’. What many people are sceptical of is the claim that human activity, as opposed to solar activity or other natural phenomena, is the principal cause.
      Given your progression from believing everything the authorities say is true to doubting it, I would have thought you would appreciate the wisdom of scepticism in this age of almost universal deceit.

      • TarsTarkas says

        Re sea level rise you also have to be very careful which coast you are using to measure with, where, and when. On the Atlantic coasts sea level is rising partly due to the ongoing subsidence of the unconsolidated sediments of the coastal plain and continental shelf of the inactive continental margins. The sea level rise gets more pronounced as one travels north due to continued isostatic rebound from past glaciation. Whereas the sea level is falling on portions of the west coast of North America not because global warming is a hoax but due to strain build up of normal faults in front of an ocean trench (whose release will cause the next ‘Big One’). El Nino’s and La Nina’s with their warming and cooling effect on ocean surface water.

  6. Martin Lawford says

    Douglas Peters of the University of North Dakota and Steve Ceci of Cornell University decided to test the objectivity of journals of psychology. They chose twelve articles from prominent psychological journals, written by professor from well-known universities from 18-32 months before. They changed the names of the authors and their institutions to fictitious ones and submitted the articles to the journals which had previously published them. Ninety percent of the editors failed to recognize the articles. The journals rejected eight of the twelve articles for “serious methodological flaws.” Therefore, Ceci and Peters concluded that the difference in results between the time they were originally submitted and the time Ceci and Peters submitted them under fictitious names was the only thing they had changed, the names of the authors and their institutions. Peters and Ceci could not get their article published in “Science” or “American Psychologist” but when they did get it published in “Behavioral and Brain Science”, other professors ostracized them and they lost some of their grant support. from “Profscam” by Charles Sykes, 1988

  7. Sarah says

    “Harking occurs when scholars run multiple analyses between numerous variables without any particular theory. When they publish their findings, they pretend the statistically significant ones were those they predicted all along.” Huh. When I was in college (M.I.T.), we were taught that one shouldn’t make any prediction as to results before the results are obtained. (No preconceived expectations.) Having expectations can skew how one interprets the results. The results themselves will elucidate the truth.

    • markbul says

      You can’t be serious. Results of what? If you don’t know what you’re collecting data for, why are you collecting it? Does MIT have a Gender Studies department now?

    • NashTiger says

      You just described the differences in Hard Sciences that rely on The Scientific Method and Soft Sciences (Psychology, Sociology, etc ) that are ever in search of correlations – although correlation does not equal causation.

    • Richard says

      Depends on the type of research you are doing, in a way. If you are hypothesis testing, then your alternative hypothesis is usually your preconceived expectation. Yes, you were probably taught to approach the data in as close to an unbiased approach as possible (“no preconceived expectations”), but in reality you almost always have an expectation that the null hypothesis will not hold. In most cases, you would not bother with the experiment otherwise.

      You can analyse data without an existing hypothesis. “Data science” uses this technique all the time for prediction. In research, it’s usually considered exploratory data analysis. It’s useful for hypothesis generation. It can also be useful if you have whole-population data rather than a sample (not common in social science research, more so in industry).

  8. Paul A. Thompson says

    I think that this is a fair and generally reasonable summary of some of the “lack of reproducibility” issues in science, not only in psychology today. There’s a pretty large amount of work in the whole issue. It’s a combination of “p-hacking”, small sample sizes, and, in some cases, fraud. If you, Gentle Reader, are interested in following more closely these issues, I recommend the blog “Retraction Watch”.

  9. markbul says

    Psychology is taking a beating recently, while Evolutionary Biology and Ecology lie low and stay very, very quiet.

  10. JamieM says

    Adding “science” to a field of study doesn’t make it a science, or even scientific. Without testable, falsifiable hypotheses, and replicable results, all you have is opinion and fairy tails. Whether social science or creation science, none of it should be presented as anything approaching truth, and especially shouldn’t be used in decision-making or policy formation.

    The National Association of Scholars has written on the subject:
    https://www.nas.org/reports/the-irreproducibility-crisis-of-modern-science

  11. codadmin says

    All of these studies and theories exist to blame everything but the problem itself, because the problem is politically incorrect.

  12. Ross Wallgren says

    This is the result of academia becoming a rat-race for the upper-middle class, rather than the intellectual playground for the children of the aristocracy unable to hack it in more elite endeavors, the way it used to be.

    • TarsTarkas says

      Mr. Wallgren:
      Science was originally mostly the playground aristocrats because they were the only ones who had the time and leisure to engage in those pursuits. Everyone else was too busy trying to make a living or just surviving to bother with such trifles, even though in the end many of these pointless endeavors actually proved quite useful and lifesaving (the end of the cholera epidemics is a good example). The aristocrats had no competition. With the enrichment of the entire society more people became able to engage in those pursuits, and the aristocratic dilettante soon became relegated to the dustbin.

      • Richard says

        I think Wallgren’s point, if more than trolling, might have been that these problems in purportedly scientific research are the result of academia becoming “professional” in the sense that it is the source of one’s income and social position rather than something done only for the pursuit of knowledge. I’m not sure of the historical veracity of the claim, but it isn’t too wild.

  13. E. Olson says

    It isn’t just publish or perish that drives the various statistical shenanigans addressed in this interesting article, but also the interests and agenda of the researchers and research funding agencies.

    Who is going to do research on homosexuals or gender differences or global warming, or fund research on those topics? Most of the time it is going to be people and organizations with a strong personal interest in providing results that support their agenda, which means homosexuals or feminists doing research designed to show how “normal” or “superior” gays and women are at various things, and tree-huggers doing research that conclusively demonstrates how we are all going to suffer painful early deaths unless we stop burning fossil fuels and start living in densely packed cities powered by sunshine and good intentions.

    And who reviews these articles? It is almost always people who have already published on the topic, which means they share the same agenda and will accept research that supports the agenda, and reject research that questions the agenda or simply doesn’t get any positive significant results to report (see Climategate scandal or the LaCour scandal as an examples).

    The utter domination of Leftists in academia means there is rarely going to be any anti-PC or pro–Right pushback in the research or review processes, and hence little reason to trust academic research on any politically correct topic.

    • codadmin says

      The siren sounded when they called James Watson ‘unscientific’, had him sacked him and then stripped him of his honours.

    • D.B. Cooper says

      @E. Olson

      It isn’t just publish or perish… but also the interests and agenda of the… research funding agencies.

      You strike me as a fairly well-informed individual, so I doubt this will be news to you, but the predominant research funding agencies are actually just an agency – the U.S. federal government.

      The federal government – whose financial judgements are known to be handicapped without interruption by a rank ignorance – has for all intents and purposes weaponized ideological sophistries as now legitimate research domains; and in doing so, has indulged the cannibalization of civil society. I, myself, am not certain the conditions of the success of our current path can be found on earth. We may in fact be sawing off the branch we’re sitting on. I would like to think society will carry on perfectly fine under the emotional jihad of a progressive ideology. Unfortunately, I don’t think an allergy to truth or to facts is a civil right; and so, I don’t think the center will hold.

      feminists doing research designed to show how ‘normal’ or ‘superior’… women are at various things

      That may be true. Hell, it is true. But of those “various things” I think we can safely say, philosophy will not be counted among them; at least not if the feminist philosophy journal Hypatia counts for anything. While being moored in the diseased waters of progressive duplicity has its down sides (too many to list); every so often the proliferation of semantic paradoxes becomes a bit too slippery for even the relativists to negotiate, and when that happens, you at least get to sit back and watch a sinister paroxysm of bad faith hitherto seen in authoritarian societies.

      On reflection, the nexus of feminism and philosophy was a ticking-time bomb just waiting to explode. It was just a matter of time before a feminist philosopher like Rebecca Tuvel came along and published a logically defensible argument about the wrong sacred cow – sound arguments are one of the great ironies within feminist philosophy, so I’m told. In other words, a publication like Tuvel’s “In Defense of Transracialism” has been long in coming, and I for one, am thank it finally came.

    • B.Kan says

      I can think of quite a few groups with a vested interest in proving that gays are terrible and that the planet is actually better off with more CO2 up in the atmosphere. Where can I think their ground-breaking, scientifically sound results?

      • E. Olson says

        B.Kan – the social justice and environmental divestiture movements mean that organizations that support traditional family values or the fossil fuel industry can’t dare fund research that might prove gays are poor parents or prone to child molestation, or that higher CO2 saves lives and improves the food supply, because they would be immediately targeted and take a stock price hit or lose funding. Thus you will find that almost all research that is perceived to be “anti-gay” or “anti-environment” is almost always going to be unfunded or sparsely funded.

        Thus any academic who might wish to research such topics by taking a balanced or neutral approach will have a much more difficult time finding external funding for their research, which together with the lower probability of finding fair and balanced reviewers and journal editors for non-Leftist conforming results means such research efforts will almost certainly results in not publishing (or publishing less often or in less prestigious outlets) and thereby perishing, because such researchers will also not get any benefit of the doubts by their mostly Leftist colleagues and administrators when it comes time to make tenure decisions. This is one reason that the most prominent climate science skeptics in academia have almost always been retired or at least senior and tenured.

        “Nice example of this imbalance is provided by the controversy generated by the research of “anti-gay” Mark Regnerus and “anti-climate change” Bjorn Lomborg (see links).

        https://www.lifesitenews.com/opinion/mark-regnerus-and-the-storm-over-his-controversial-gay-parenting-studd

        https://en.wikipedia.org/wiki/Bj%C3%B8rn_Lomborg#Formal_accusations_of_scientific_dishonesty

        • B.Kan says

          So the fact that there are no credible studies proving that gays are horrible people and that climate change is actually a great think means that those things are probably true. Gotcha.

  14. Comrade Lisenko says

    As comrade Xi said about the study of history in China (paraphrasing): history has to provide the right answers and avoid the wrong answers. What is right and wrong is determined by the Party. All social sciences and increasingly supposedly “hard” sciences are affected by this kind of bias. The amazing thing is this is happening not in China but in the American and Western European academia. Scientists are working hard to make people distrust science.

    • tommy mc donnell says

      being done by the same people; communists.

    • Comrade, your work for Stalin was a brilliant example of pure science!

  15. Christopher Moss says

    Time was in medical journals it was absolutely acceptable to publish negative results. This or that drug doesn’t work for this or that condition? Useful to know! And the same went for surgical regimens, some negative results changing common practice 180º (like not starving patients with acute pancreatitis). I don’t see why any other discipline ought to be afraid of negative results when the fact is that they tell you just as much as positive results; both advance the body of knowledge equally. Surely we are not so superficial as to think that something must be positive or right if it is to teach us something? How else does the burned child fear the fire etc? We learn from our mistakes more than we do from our successes.

    • E. Olson says

      Christopher – you are certainly correct, but as you probably know there have recently been lots of scandals in medical research ranging from fabricated data/results to failure to publish negative results due to pressures from funding agencies (i.e. drug companies who don’t want their billion dollars of R&D to get written off because clinical trials show ineffective results). If such scandals are present in medical research, just imagine how often they occur in fields that are not so life and death. Imagine the problems created for the education industry if research confirmed beyond the shadow of any doubt the complete dominance of genetic causes of IQ, or for the gay rights communities if transgenderism or homosexuality would be clinically confirmed as mental defects that lead to dangerous sexual activities, or high rates of child molestation and suicide, or the feminist movement if research confirmed greater happiness and economic prosperity for traditional family structures or generally lower rates of female ability in certain high profile professions. How many social justice oriented politicians, bureaucrats and educators would lose their jobs if most inequalities of outcome were found to be mostly genetic and cultural and therefore not amendable to social-welfare interventions designed to correct non-existent or harmless discrimination and patriarchy? That is why such topics are deemed too dangerous to research in an unbiased way by the social justice crowd, and why such strong efforts are made to discredit or hide any negative results.

  16. This is good. Up to now the results of psychology studies have been reported in the press as of general applicability (particularly those which enforce the writer’s preferred narrative). Caveat emptor is two millenia old. Carl Sagan said it best — “Extraordinary claims require extraordinary evidence.”

    For an example of data slicing and dicing (and by Johns Hopkins), please see — https://luysii.wordpress.com/2009/10/05/low-socioeconomic-status-in-the-first-5-years-of-life-doubles-your-chance-of-coronary-artery-disease-at-50-even-if-you-became-a-doc-or-why-i-hated-reading-the-medical-literature-when-i-had-to/

  17. tommy mc donnell says

    they’re wrong because they are not searching for the truth, they are trying to advance a political agenda. if the truth will do it fine, if not we’ll lie.

  18. ccscientist says

    The problem is worse in the fields affected by post-modernism which states that there is no objective truth, that everything is just about power. In this framework, the important thing is narrative, your subjective story, and power relations. Thus writing in feminism and some social science is hilariously incoherent, with little respect for facts. No fact could refute an author’s thesis in such fields. To the extent that post-modernism infects psychology–which depends on the individual–it corrupts the scientist who will then argue their thesis rather than searching for the truth. As an example, it is forbidden to ask if teens who feel trans later decide they are simply gay (if left alone) or if those who transition later regret it. If you cannot ask the question or publish the results, then we are all poorer. Note that this is not even like the taboo of investigating IQ and race.

    • Peter from Oz says

      If there is no objective truth, then the statement that there is not objective truth is false. Further if everything is just about power, then why are the post-modernists trying to gain power through debunking all other theories?

  19. Debbie says

    Harking occurs when scholars run multiple analyses between numerous variables without any particular theory. When they publish their findings, they pretend the statistically significant ones were those they predicted all along.

    Aside from the pretending part, wouldn’t the most robust, statistically significant results be the ones you weren’t looking for?

    • Debbie says

      Wait a minute! It’s worse than I thought: the author goes on to say, In fact, they [the statistically significant results] are usually the product of random chance and therefore unreliable.

      But a result is ‘statistically significant’ only if there is a very, very low chance that it is due to random chance. So … Wtf???

      • Jay says

        95% certainty sounds good, but one time in 20 it’s wrong. If there are even small inconsistencies in how your data is gathered, you can easily have a much higher ‘certainty’ which does not match reality.

        If you take a broad test with multiple variables you are virtually certain to hit ‘statistical significance’ incorrectly on some of them. This is often a big problem in diet testing.

        The most statistical significance can do is tell you that MAYBE there is a connection

        • statistical cynic says

          And “95% certainty” is wrongly said: it’s a 95% probability.
          And that “95%” is calculated against the statistical model of the population distribution you have chosen and so depends on it being an accurate depiction of reality if that 95% is to mean anything about the real world. Choose a Gaussian distribution when reality corresponds more closely to a Boltzmann distribution, or a power law distribution, and that tiny .05 p value becomes a meaningless fantasy.

      • Richard says

        The problem is that the statistically significant finding of one study is still fairly weak evidence, even if the p value is very small. If you have multiple studies with similar findings, the evidence becomes strong fairly quickly. Of course, in the social sciences you don’t often have reproductions of previously conducted studies, but you do have studies with some degree of similarity. There is some debate as to how similar study designs should be in order to support one another’s findings.

      • Charles R. Twardy says

        @Debbie: If enough labs run essentially the same study when there’s no effect, we’d expect one of them to get (un)lucky – and be only ones to publish. So, of published studies, many “significant” results are due to random chance. In social science it appears to be roughly half.

        Likewise, if I get to try dozens of tests, I can get easily find effects that aren’t really there, just noise in this sample.

        Of course we try dozens of tests during exploration – but we really shouldn’t believe it until we gather more data in a controlled trial — where we fix the test ahead of time.

      • BrainFireBob says

        Yeah, that’s only “really” true when your experiment properly controls for variables in terms of sample size, etc.

        If I gathered 100 people, and 99 of them had average IQs, but 1 of them was 6″ taller than average and had a genius IQ, then I’d draw an incredibly strong statistical correlation of height with IQ. It’d be false because my sampling wouldn’t be a sufficiently random cross-section of the population. If I initially gathered my 100 people to not test for IQ or height, I may not have captured a truly representative sample in the first place.

  20. SiGe says

    The data is the data; most all errors are in the analyses (or lack there of).

    • Richard says

      Nah, proper data collection is and always will be a huge issue, especially when attempting to analyse any sort of complex, probabilistic system. Not just whether the data were collected in a rigorous manner, but whether the data collected is relevant and whether all (or enough) relevant factors are represented in the data. Problems with analysis are still a huge issue, too (and likely will always be).

  21. Conor says

    I think there is a typo in the 4th paragraph.

    “Furthermore, recent brain imaging studies have not supported the claims made in Time‘s 2104 article.”

    That should probably say 2014. Interesting read none the less.

  22. Peter from Oz says

    I note the author of this article uses the oxymoron ”psychological science” instead of the correct ”psychology”.
    Psychology is not a science on any definition of the word that anyone familiar with epistemology would understand. That is why it is having such a replication crisis.
    Also I’d like to point out that people of all political and social views seem to share the belief that the media can cause harm. Thus old fashioned conservatives thought that rock music would rot the brains of youth and result in more drug taking and promsicuity leading to social dissolution. The modern left scoffed at that, but then decides that Murdoch newspapers are brainwashing us all into voting for fascists. FUrther lefties believe that if the word nigger is spoken, the lynchings will return. Though one wonders if they will ”believe all women” when those women are making accusations of rape that cause the lynchings.

  23. Steve Bowden says

    How to make sense of it all? “Gutenberg 2.0” (that’s a great term by the way) shoots information at us like a fire hose into the face. The human brain can only focus on a very, very tiny fraction of information available. It has to ignore almost all of the available data, and wouldn’t you know that evolution has selected that we ignore everything that conflicts with our preexisiting world view. What a trap!

    Science (when it’s not politicized) corrects for this by having multiple brains trying to disprove claims.

    It is harder for individuals because we are all stuck inside of one (highly biased) nervous system. Personally, I try hard to gather information from differing view points, and take everyone’s claim with a huge grain of salt. In the end I try to hold my conclusions with humility and say to myself “well this is what seems true to me… but really, who the fuck knows”.

  24. Closed Range says

    Quite a few of the comments above seem to want to throw baby out with bathwater. It’s true that psychology is a mess of a field, but it can’t be compared to the cesspit of gender and racial studies, which are little more than covers for political activism to give itself a veneer of respectability. As explained by the author, psychology is ironically the victim of human weaknesses for excitement and novelty. Overall, however, it isnt (yet) just an ideology – the proof being that many psychologists dispute the claims made by various studies doing the rounds in the media.

    Another element I think is worth pointing out – in academia, anything can be published if you’re prepared to wait long enough and roll the dice enough times until you get a few lazy referees who don’t actually bother to read the paper. There’s few chores in life like doing peer review on a topic you’re not interested in, and not all referees are going to look at the fine details, where errors of methodology usually lurk.

    Another problem is that the level of understanding of what statistical tests actually mean is abysmal among academics many fields. Anyone who has encountered confidence intervals will know what I mean.

    • Richard says

      Very good points. Additionally, psychology is such a broad field, I wouldn’t say the term is meaningless, but you would definitely want to ask some follow up questions after someone told you they were a psychologist if you wanted to have much idea of what their job entailed.

  25. Tim H says

    P-Value reliance has significant problems. Recommended reading is William Briggs Uncertainty: The Soul of Modeling, Probability and Statistics. But with the institutional framework pushing the publish or perish ethic, it will be hard to move away from simple p-value analysis.

  26. Kevin Herman says

    You mean when some partisan hack media type/politician tells me studies show something I shouldn’t believe its ironclad fact? Dont worry I already was smart enough to figure that out for myself.

  27. Fred says

    All you need to know about psychology: The latest DSM has de-listed gender dysphoria as a mental illness but lists binge drinking as one. In other words, a man who has himself chemically and surgically mutilated because he thinks he’s a woman is mentally healthy, but a man who gets drunk on Saturday night is mentally ill. Psychology is the phrenology of the 21st century.

  28. Phil says

    Bad Data Analysis and Psychology’s Replication Crisis

    Sadly this is nothing new.

    Lies, Damned Lies, and Lenore Weitzman by Cristopher Rapp. Since its publication in 1985, sociologist Lenore Weitzman’s The Divorce Revolution has had a …

    Sadly the website appears to no longer be active.

    People will believe research that matches their own biases, and they will fight to the death that their research is honest and accurate.

  29. Angus says

    I think the article misses an important point that in order for an under-powered study to achieve statistical significance, it must by definition drastically over-estimate the effect size, and can often even miscalculate the direction of the effect. Even honest researchers can be total misled by this.

  30. Good article. I have been in this business for 40 years and hired therapists for 30. Nobody follows the research regardless of the outcomes. Most cannot see how it applies in the real world. We tend to pick and choose the outcomes we like. The Teapot Anger Release theory reigned for decades without serious data.

  31. Pingback: Bad Data Analysis and Psychology’s Replication Crisis | Sassy Wire

  32. Edward Erwin says

    Professor Ferguson’s paper is important and right, but the problems he discusses extend to other disciplines including medicine and are far more serious. Anyone with an extremely sick family member hopes that the treatment prescribed by the specialist is safe and effective. How good is the evidence supporting this hope? Recent research in medicine demonstrates it is generally very weak and not to be trusted.

  33. Edward Erwin is correct, in my view. One does not need to be a scientist (which I am) to realise from the evidence of many people that the medical profession often: (i) wrongly diagnoses problems, (ii) exaggerates problems, and (iii) creates medical problems by prescribing dangerous and unnecessary treatments.

    The medical profession frequently prescribes dangerous and unnecessary “treatments” for people who either have nothing (or nothing seriously) wrong with them or who need quite a different treatment for their condition.

  34. One criticism of this article is the use of the term “mental health” when the author actually means “mental illness”. It’s something that has slipped into common parlance, but is extremely inaccurate, and an important term to use correctly when we talk about what would produce better mental health (correct use of the term) and decrease mental illness (correct use of the term).

Comments are closed.