The co-author
Fred Hasselman, a behavioural scientist and lecturer of Pedagogical Sciences at Radboud University, co-authored the Science article that created quite a stir in recent weeks. In the article, 270 scientists from around the world replicated one hundred psychological experiments that were published in three scientific journals in 2008. They found that most of the results could not be reproduced.

“Although dramatic, the project does show that science is  doing its job, with 270 psychologists working hard to replicate these experiments, ” says Hasselman, who acted as a statistics and methodology consultant for the project. However, he is hesitant to refer to psychology as a pseudoscience. “This is simply how the scientific method works. Replication is important for the credibility of scientific knowledge.”

That said, Hasselman does believe that psychology is prone to mistakes. The discipline is not known for its commitment to sound research methods. On the contrary, psychology uses what natural scientists refer to as classic methodology to study its subjects. It works more or less like this: in an experiment, a variable is manipulated to trigger a measurable response. This response is then considered proof of a new property or effect. According to Hasselman, this view is far too simplistic and not appropriate to a science that studies human beings. “While this method can be applied to simple physical systems, people are much more complex.”

Human behaviour is susceptible to a range of environmental stimuli and tends to change quite quickly. As a result, it cannot be approached in a conventional way. To Hasselman, this is the crux of the problem. “I believe that our inability to replicate research studies lies in the fact that we lack sound theories on studying humans as complex, dynamic systems.”

The methodologist
Neuroscientist and methodologist Eric Maris, who also works as a psychologist at Radboud University, was not surprised when he heard that many psychological studies did not produce significant results upon replication. Other disciplines, like pharmacology, do not fare much better when it comes to replication.
Maris believes the scientific community would benefit from a different approach, one in which replicating studies that report important new findings is standard procedure. It is equally important that researchers document the research process and share their data and analyses. The latter, which is sometimes referred to as research management data, is the ideal way to encourage researchers to check each other’s work. “That’s how we learn from each other and improve.”

Psychologists have been told repeatedly how important it is to replicate studies, but in Maris’ opinion, this is rarely the case. After all, breaking new ground is far more exciting. Nevertheless, he believes that combining original and replicated research is quite easy. “In this respect, we can learn from the cognitive research on monkeys. This often produces replicable results, despite having only two monkeys available per study. The trick is getting the new monkeys to replicate the basic phenomenon (observed in other monkeys from previous studies) and to identify new phenomena that can deepen our understanding. I’ve used this method successfully in my own research with human subjects.”

Replication studies are not popular because they generate very little scientific merit. Individual researchers as well as universities and research institutes should nevertheless continue carrying out replication studies. Maris suggests drafting a code of conduct, which is “indispensable to bringing about a cultural shift within the scientific community.”

The social psychologist
According to social psychologist and Dean of the Radboud University Faculty of Social Sciences Daniël Wigboldus, psychology is far from a pseudoscience. “After all,” he says, “a pseudoscience would never have initiated a huge replication study like this.” Its research subject – us – is what makes the discipline so difficult. “Human behaviour can be studied in all kinds of ways, from brain scans to dream interpretation. This makes it quite difficult to assess the results. Moreover, people are prone to change. We behave differently in different situations.”

This does not mean, however, that we cannot study human beings. “It’s just harder to get to the heart of the matter.” What Wigboldus is trying to say is that the basic principles of psychology may only exist thanks to replication studies. Only time will tell which findings hold true and which do not. “Those deemed replicable will become the textbook material of the future. That’s how science works, by helping us slowly acquire knowledge.”

Wigboldus believes that replication studies should carry more weight, which does not mean a study should only be published after it’s been replicated ad nauseam. “Publishing initial findings can be a good thing, as it gives colleagues something to work with. That said, the publication should always explain which conclusions can be drawn from the data.”

According to Wigboldus, psychologists have a duty – one that is currently being insufficiently met – to document their hypotheses in advance in digital form, to share their data with their colleagues and to explain their methodology in their publications. In a word: transparency. This is something the field of social psychology has been doing for a long time. “In the Netherlands, people are quick to cite the Stapel incident. And while it’s certainly been a catalyst for recent change, this process has actually been going on for quite some time.” / Martine Zuidweg

This article was also published in Vox #1 (in Dutch). You can read the Dutch version here.