Common criticisms about parapsychology: Criticism 3
Parapsychology does not have a "repeatable" experiment.
on Friday, February 11, 2011
Response: When many people talk about a repeatable psi experiment, they usually have in mind an experiment like those conducted in elementary physics classes to demonstrate the acceleration of gravity, or simple chemical reactions. In such experiments, where there are relatively few, well-known and well-controllable variables, the experiments can be performed by practically anyone, anytime, and they will work. But insisting on this level of repeatability is inappropriate for parapsychology, or for that matter, for most social or behavioral science experiments. Psi experiments usually involve many variables, some of which are poorly understood and difficult or impossible to directly control. Under these circumstances, scientists use statistical arguments to demonstrate "repeatability" instead of the common, but restrictive view that "If it's real, I should be able to do it whenever I want."
Under the assumption that there is no such thing as psi, we would expect that about 5% of well-conducted psi experiments would be declared "successful" (i.e., statistically significant) by pure chance. But suppose that in a series of 100 actual psi experiments we consistently observed that 20 were successful. This is extremely unlikely to occur by chance, suggesting that psi was present in some of those studies. However, it also means that in any particular experiment, there is an 80% probability of "failure." Thus, if a critic set out to repeat a psi experiment to see if the phenomenon was "real," and the experiment failed, it would obviously be incorrect to claim on the basis of that single experiment that psi is not real because it is not repeatable.
A widely accepted method of assessing repeatability in experiments is called meta-analysis. This quantitative technique is heavily used in the social, behavioral and medical sciences to integrate research results of numerous independent experiments. Starting around 1985, meta-analyses have been conducted on numerous types of psi experiments. In many of these analyses, results indicate that the outcomes were not due to chance, or methodological flaws, or selective reporting practices, or any other plausible "normal" explanations. What remains is psi, and in several experimental realms, it has clearly been replicated by independent investigators.