In an attempt to test just how rigorous scientific research is, some researchers have undertaken the task of replicating research that’s been published in a whole range of fields. And as more and more of those attempted replications have come back, the results have been striking — it is not uncommon to find that many, many published studies cannot be replicated.

Vox published an interesting story today about the replication crisis in science. If you’re not familiar with this idea, it’s the fact that a significant percentage of all of the peer-reviewed social science that has been published in the past can not be replicated by other scientists, which is a pretty clear sign that the claims made in the original papers aren’t true. How bad is the situation. This bad:

In an attempt to test just how rigorous scientific research is, some researchers have undertaken the task of replicating research that’s been published in a whole range of fields. And as more and more of those attempted replications have come back, the results have been striking — it is not uncommon to find that many, many published studies cannot be replicated.

One 2015 attempt to reproduce 100 psychology studies was able to replicate only 39 of them. A big international effort in 2018 to reproduce prominent studies found that 14 of the 28 replicated, and an attempt to replicate studies from top journals Nature and Science found that 13 of the 21 results looked at could be reproduced.

So we’re not talking about a few bad papers we’re talking about half or more of all of the peer-reviewed papers being junk that can’t be replicated. In some cases, these bogus papers can spawn a whole cottage industry within social science or psychology which later turns out to have been a waste of time for everyone involved. In 2016 Vox published a story on this topic which included one particular example of this.

In 1998, psychologists found evidence of a tantalizing theory: We all have a finite mental store of energy for self-control and decision-making. Resisting temptations, or making tough decisions, saps this energy over time…

Over the years, the theory has been tested in hundreds of peer-reviewed studies, with countless stand-ins for the chocolate, radishes, and the quiz. Scientists have shown how diminished willpower can affect our ability to hold on to a handgrip, sap our motivation to help another in need, and even negatively impact athletic performance.

This huge body of research has helped ego depletion, as psychologists call it, and its offshoot decision fatigue, become the basis for best-selling books, TED talks, and countless life hacks. In an age where temptations and decisions pummel us at warp speed, it’s become an empowering concept. If we know how the system works, we can game it: President Obama famously doesn’t pick out his suits, for fear that it might deplete some of his decision-making capabilities.

Read More