In his PLOS Medicine Journal paper Why Most Published Research Findings Are False, John P. A. Ioannidis wrote of "increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias.
"Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias."
Three American researchers, James Lindsay, Peter Boghossian, and Helen Pluckrose, have tried to prove the "measure of prevailing bias" thesis but from the bottom-up, and they end up vindicating Ioannidis' 2005 paper. It turns out that under the heading "Humanities" it is possible to publish any academic article that bemoans racism and discrimination. Under false names, the researchers sent baseless and even ridiculous articles to the most prestigious journals - and the articles were accepted and published.
The three academics succeeded in publishing articles in reputable peer-reviewed humanities journals. The three, who call fields of scholarship loosely known as “cultural studies” or “identity studies” (for example, gender studies) "'grievance studies' because of their common goal of problematizing aspects of culture in minute detail in order to attempt diagnoses of power imbalances and oppression rooted in identity."
They wanted to prove that the current discourse in humanities and social sciences makes it possible to publish any preposterous academic article as long as it opposes any social "wrongdoing" and oppression that exists in society. They wrote 20 papers and submitted them to the best journals in the relevant fields, and met with considerable success.
"Something has gone wrong in the university—especially in certain fields within the humanities," the three write. "Scholarship based less upon finding truth and more upon attending to social grievances has become firmly established, if not fully dominant, within these fields, and their scholars increasingly bully students, administrators, and other departments into adhering to their worldview. This worldview is not scientific, and it is not rigorous. For many, this problem has been growing increasingly obvious, but strong evidence has been lacking. For this reason, the three of us just spent a year working inside the scholarship we see as an intrinsic part of this problem."
"Sometimes we just thought a nutty or inhumane idea up and ran with it," the three write. "What if we write a paper saying we should train men like we do dogs—to prevent rape culture?" The resultant paper was Human reactions to rape culture and queer performativity at urban dog parks in Portland, Oregon.
"What if we write a paper claiming that when a guy privately masturbates while thinking about a woman (without her consent—in fact, without her ever finding out about it) that he’s committing sexual violence against her? That gave us the 'Masturbation' paper. What if we argue that the reason superintelligent AI is potentially dangerous is because it is being programmed to be masculinist and imperialist using Mary Shelley’s Frankenstein and Lacanian psychoanalysis? That’s our 'Feminist AI' paper. What if we argued that 'a fat body is a legitimately built body' as a foundation for introducing a category for fat bodybuilding into the sport of professional bodybuilding?" That article became Who are they to judge? Overcoming anthropometry through fat bodybuilding.
"At other times, we scoured the existing grievance studies literature to see where it was already going awry and then tried to magnify those problems. Feminist glaciology? Okay, we’ll copy it and write a feminist astronomy paper that argues feminist and queer astrology should be considered part of the science of astronomy, which we’ll brand as intrinsically sexist. Reviewers were very enthusiastic about that idea. Using a method like thematic analysis to spin favored interpretations of data? Fine, we wrote a paper about trans people in the workplace that does just that. Men use “male preserves” to enact dying 'macho' masculinities discourses in a way society at large won’t accept? No problem. We published a paper best summarized as, Gender scholar goes to Hooters to try to figure out why it exists...
"We used other methods too, like, 'I wonder if that "progressive stack" in the news could be written into a paper that says white males in college shouldn’t be allowed to speak in class (or have their emails answered by the instructor), and, for good measure, be asked to sit in the floor in chains so they can "experience reparations".' That was our 'Progressive Stack' paper. The answer seems to be yes, and feminist philosophy titan Hypatia has been surprisingly warm to it. Another tough one for us was, 'I wonder if they’d publish a feminist rewrite of a chapter from Adolf Hitler’s Mein Kampf.' The answer to that question also turns out to be 'yes', given that the feminist social work journal Affilia has just accepted it. As we progressed, we started to realize that just about anything can be made to work, so long as it falls within the moral orthodoxy and demonstrates understanding of the existing literature."
"The papers themselves span at least fifteen subdomains of thought in grievance studies, including (feminist) gender studies, masculinities studies, queer studies, sexuality studies, psychoanalysis, critical race theory, critical whiteness theory, fat studies, sociology, and educational philosophy. They featured radically skeptical and standpoint epistemologies rooted in postmodernism, feminist and critical race epistemology rooted in critical social constructivism as well as psychoanalysis. They all also endeavored to be humorous in at least some small way (and often, big ones). The project so far has generated more than 40 substantive editorial and expert reader reports...
"Our papers also present very shoddy methodologies including incredibly implausible statistics, making claims not warranted by the data, and ideologically-motivated qualitative analyses. Questionable qualitative methodologies such as poetic inquiry and autoethnography (sometimes rightly and pejoratively called 'mesearch') were incorporated.
"Many papers advocated highly dubious ethics including training men like dogs, punishing white male college students for historical slavery by asking them to sit in silence in the floor in chains during class and to be expected to learn from the discomfort, celebrating morbid obesity as a healthy life-choice, treating privately conducted masturbation as a form of sexual violence against women, and programming superintelligent AI with irrational and ideological nonsense before letting it rule the world. There was also considerable silliness including claiming to have tactfully inspected the genitals of slightly fewer than 10,000 dogs whilst interrogating owners as to their sexuality, becoming seemingly mystified about why heterosexual men are attracted to women, insisting there is something to be learned about feminism by having four guys watch thousands of hours of hardcore pornography over the course of a year while repeatedly taking the Gender and Science Implicit Associations Test, expressing confusion over why people are more concerned about the genitalia others have when considering having sex with them, and recommending men anally self-penetrate in order to become less transphobic, more feminist, and more concerned about the horrors of rape culture. None of this, except that Helen Wilson recorded one 'dog rape per hour' at urban dog parks in Portland, Oregon, raised so much as a single reviewer eyebrow, so far as their reports show.
"Near the end of July 2018, a clear need arose to call the project to a premature end after our 'dog park' paper attracted incredulous attention on social media generated by the Twitter account Real Peer Review, which is a platform dedicated to exposing shoddy scholarship. This deserved incredulity led to small and then larger journalistic publications investigating our fictitious author, Helen Wilson, and our non-existent institution, the Portland Ungendering Research Initiative (PURI) and finding no credible history of either. Under this pressure, the publishing journal, Gender, Place and Culture, asked our author to prove her identity and then released an expression of concern about the paper. This generated further attention that eventually got the Wall Street Journal involved, and far more importantly, it changed the ethics of utilizing deception within the project. With major journalistic outlets and (by then) two journals asking us to prove our authors’ identities, the ethics had shifted away from a defensible necessity of investigation and into outright lying. We did not feel right about this and decided the time had come to go public with the project. As a result, we came clean to the Wall Street Journal at the beginning of August and began preparing a summary as quickly as possible even though we still had several papers progressing encouragingly through the review process.
"We spent 10 months writing the papers, averaging one new paper roughly every thirteen days. (Seven papers published over seven years is frequently claimed to be the number sufficient to earn tenure at most major universities although, in reality, requirements vary by institution.) As for our performance, 80% of our papers overall went to full peer review, which keeps with the standard 10-20% of papers that are 'desk rejected' without review at major journals across the field. We improved this ratio from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers. Because we were forced to go public before we could complete our study, we cannot be sure how many papers would have been accepted if we had had time to see them through—papers typically take 3-6 months or more to complete the entire process and one of ours was under review from December 2017 to August 2018—but an estimate of at least 10, probably 12, eventual acceptances seems warranted at the time of having to call a halt.
"Having spent a year doing this work ourselves, we understand why this fatally flawed research is attractive, how it is factually wrong in its foundations, and how it is conducive to being used for ethically dubious overreach. We’ve seen, studied, and participated in its culture through which it 'proves' certain problems exist and then advocates often divisive, demeaning, and hurtful treatments we’d all do better without."
The researchers say that while they are aware the exercise will hurt them and their reputations, they do not regret their actions. "We intend to use the knowledge we’ve gained from grievance studies to continue to critique them and push for universities to fix this problem and reaffirm their commitment to rigorous, non-partisan knowledge production," they conclude their after-action report on the project.
Ioannidis concluded his paper on why most published research findings are false with a question: "Is it unavoidable that most research findings are false, or can we improve the situation?" He says that "a major problem is that it is impossible to know with 100% certainty what the truth is in any research question."
Lindsay, Boghossian, and Pluckrose showed that academic research in the humanities and social sciences is based on an over-emphasis on apparent discrimination against weak populations, which leads to publicizing ridiculous articles. "It's frightening to think that such articles are then taught in classrooms and perceived by social activists and politicians as truth."