There is Madness in Their Methods
Mikel Aickin University of Arizona SEP 3rd 2005
In The Structure of Scientific Revolutions, Thomas Kuhn portrayed normal science as slipping into a moribund condition, in which it could no longer provide acceptable answers to its own questions. Then some kind of shift would come along to replace the dominant outlook with an improvement, which was in turn destined to become the new version of normal science, perpetuating the whole cycle. In Kuhn’s version, it was the discovery of new puzzles, consisting of observations that could not be satisfactorily explained in the current paradigm, that led to an apparent shift. He did not, however, consider a situation in which the methods of normal science might simply degenerate, producing the same kind of crisis, and possibly the same kind of resolution.
There is unsettling evidence that we are now in the midst of a methodologic degeneration in biomedical science. This appears to be occurring in, of all places, our fundamental approach to inference –using observation and evidence to decide how to act or believe. That it might be happening in medical research makes it of more than just academic interest.
One of the few benefits in a degeneration of conventional methods is that the normal scientists are unlikely to recognize that it is happening, and so the process will be not only made public, but actually touted as excellent science. So it is with a remarkable article published on 27aug05 in the Lancet regarding homeopathy1. The editors of Lancet are evidently proud of their publication, since they use it as the basis for a call to end homeopathy. Does this article justify the editorial, or is it in fact a betrayal of the very principles that the Lancet claims to stand for? Let us see how this specific article fares in the light of the conventional criteria that are applied to articles in clinical trials and biomedical science generally.
Treatment Comparisons. A clinical trial has investigated two therapies for a given condition. On a scale in which larger numbers are better, and zero stands for no effect, the effect of therapy A is 0.54 (SDE=0.196), while for therapy B it is .13 (SDE=0.154). The researchers conclude that treatment A is effective (p=0.006) while therapy B is not (p=0.40).
The article is, of course, not accepted for publication. The reason is that the whole point of having two groups in a study is to compare them with each other. The difference between the treatment effects in the two groups is 0.41 (SDE=0.249) with p=0.10. By the conventional criterion for making such comparisons, this result is not “statistically significant”. It means that there is no basis for saying that the two therapies have different effects. The study is null.
The data come from the abstract of the Lancet article. 0.54 is the negative log of the odds ratio (0.58) from conventional studies, and 0.13 is the same transformation of the odds ratio (0.88) in homeopathic studies. In the abstract, and in the comments elsewhere in the issue, the faulty analysis is treated if it were correct: therapy A (conventional medicine) is indeed effective, while therapy B (homeopathy) is not.
Differential Compliance. Another study has randomized 110 patients each to two therapy groups. The therapies are hard to maintain, and so only 21 patients comply in one group, while an even more disappointing 9 comply in the other group. The difference is “statistically significant” with p=0.018. The authors are surprised when their article is rejected, on the grounds that such a low rate of compliance, combined with a differential between the two groups, casts the results in serious doubt. The study has failed.
The numbers are from the abstract of the Lancet article. There were 21 “high-quality” homeopathic studies, and 9 “high-quality” conventional studies. The conclusion is clear; there has been a “statistically significant” demonstration that homeopathy articles are of higher quality than comparable conventional medical articles on the same topics. Unfortunately, this invalidates the rest of the paper. (As a footnote, it was only recently that the supposed poor quality of CAM research was being cited as the reason for a false excess of positive CAM studies. Now that the quality results are in the opposite direction, this argument is evidently no longer valid.)
Intent-to-treat. Yet another study also enrolls 110 pair-matched patients in each of two groups. One group has 8 evaluables while the other has only 6. The article is rejected on the grounds that once patients are entered into the study, they must be analyzed in their original group. This means, among other things, that if they did not contribute endpoint data, then some imputation scheme must be used. The results as presented are faulty not only because more than ninety percent of the data are missing, but because there is no guarantee that the patients actually analyzed are matched (that is, the pair matching was destroyed by the missing data, a point passed over by the authors). The process of selection that produced “evaluables” is not above question.
The data come from the abstract of the Lancet article. The odds ratios cited above are based on 8 homeopathic and 6 conventional articles (not 110 of each, as implied elsewhere in the article and in the Lancet editorial). The loss of pairing was ignored, of course. The validity of the measures used to include articles is not adequately justified, despite the fact that the results might well be almost entirely driven by them.
Post-study power computations. A study without a control group reports an apparent treatment effect of 0.13 (SDE=0.154). This is properly reported as not “statistically significant”. The article is only accepted subject to revision, since a negative study with a small sample size should provide a power computation (this is not, as often and erroneously thought, to justify the study in the first place, but to determine whether the results are worth anything at all). A conventional calculation gives a detectable effect of 0.462 (power 85%). The editors decide that this is too large to be reasonable, and reject the article.
The data come from the abstract of the Lancet article. A negative result is reported (homeopathy is no better than placebo) with a miniscule sample size, and no power calculation.
Control of confounding. A group of epidemiologists conduct an observational study of seven risk factors on a disease outcome. The issue is to determine whether the risk factors are the same in two groups of people. The data presentation consists of a series of univariate odds ratios, one for each risk factor, with p-values to test a null association. The article is rejected for two reasons. First, since the purpose was to compare risk factors across the two groups, the comparisons with null effects are not germane, and the obvious comparisons between the groups should be made. But more importantly, there is a known confounder that should have been controlled in the analysis (that is, there should have been a multivariate analysis), and moreover the risk factors that were analyzed are intercorrelated, so that again multivariate analyses should have been carried out.
The data are from Table 3 of the Lancet article. One could take quality as the confounder, or perhaps one of the other factors. There is, of course, no reason to dichotomize quality, and since this generally results in misclassification bias, there is reason not to. Obviously the analysis does not compare conventional medicine with homeopathy, but rather compares each to the null. An appropriate analysis would not only make the comparison between therapy groups, but would take the pairing into account.
Meta-analyses. There are, therefore, five areas in which the Lancet article does not meet the minimum, conventional criteria for publication in biomedicine. This is, however, not the most serious problem with the article. For this, we need to go back to recall the original aim of a meta-analysis, or overview. It is to assemble all of the obtainable, relevant literature on studies done for the purpose of comparing different therapies for a given condition. The original reasons for developing the concept were to collect scattered literature into one place, to apply uniform criteria for study selection and analysis, and to come to a conclusion about the best therapeutic approach, or to say that the evidence was not yet conclusive. Somehow this precise and useful form has degraded into an unrecognizable hash, in which any papers on any topic can be bundled together in an investigation of questions of unlimited ambiguity. A classic paper along this path has already been published in the Annals of Internal Medicine.2 Here the authors studied a single therapy (vitamin E supplementation), breaking the first rule of meta-analysis, for multiple conditions (breaking the second rule) in studies not designed to test the therapy (breaking the third rule). There is evidence that they were not sufficiently careful about the form of the various vitamin E treatments, violating a fourth rule.3 This study in effect concocted perhaps the most biased sample of human beings one could find in the biomedical literature, and then made the truly bizarre assertion that its results applied to everyone. One can only presume that the lack of a negative reaction to the Annals’s article paved the way for the Lancet article.
There are other examples, of a similar order of strangeness, but I will only mention the therapeutic touch article published in Journal of the American Medical Association.4 This article was on research carried out by a nine-year old girl, under the direction of her mother, an ardent opponent of therapeutic touch. The methodology was debunked in an article in Alternative Therapies,5 showing that it contained appalling, irremediable flaws. After the original article was published, the JAMA editor was criticized for poor judgment, by accepting a low-quality article to make a political statement. This could be seen as an unjustifiably beneficent interpretation, however, because no one seems to have noticed the very real possibility that the article, poor though it was, actually did meet JAMA’s scientific standards.
Malpractice. If you make a few simple assumptions, you can roughly compute the number of possible instances of malpractice that a physician might commit in a lifetime of practice. It is not a particularly large number. Now consider a journal that publishes papers which mislead health professionals and ordinary people about the effectiveness of medical practices. Surely one article, no less one researcher, can have a harmful effect through research malpractice that dwarfs the meager capacity of a single physician. The malpractice risk of a typical journal must be even larger.
If we are to see a continued degradation of methods in biomedical research, supported by “leading” journals, then perhaps it is time to think about the End of Biomedical Journals, as we know them. In the US at least, it would not seem unthinkable that the National Institutes of Health, through the National Library of Medicine, could undertake a web-based project to publish all funded, and much of the unfunded research, in all areas of biomedicine. The need for the current unregulated system would vanish. Editors and referees would continue to be needed, but they would operate under rational regulations, and would not be in a position to endanger the public health on the basis of personal whims. Incidentally, the job of meta-analyses would be infinitely easier, since the hunt for relevant articles would be all but accomplished. And, as we know because of PubMed, the technology is available, and the NLM knows how to apply it.
To return to Thomas Kuhn, we certainly have many biomedical puzzles that are worth working on, and which have not been addressed by normal biomedical science. Some of us are engaged in an experiment to see whether we can fashion research tools that will help us to understand more, by extending existing methods when feasible, and developing new ones when appropriate. For us, it is particularly discouraging to see normal biomedical scientists perverting their own tools for the evident purpose of attacking unconventional therapies.
1. Aijing Shang, Karin Huwiler-Müntener, Linda Nartey, Peter Jüni, Stephan Dörig, Jonathan A C Sterne, Daniel Pewsner, Matthias Egger. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet 2005; 366: 726–32
2. Miller ER, Pastor-Barriuso R, Dalal D, Riemersma RA, Appel LJ, Guallar E. Metaanalysis: High-dosage vitamin E supplementation may increase all-cause mortality. Annals of Internal Medicine 2005;142(1):37-46
3. Neustadt J, Pizzorno J. Vitamin E and All-Cause Mortality. Integrative Medicine 2005;4(1):14-17
4. Rosa L, Rosa E, Sarner L, Barrett S. A close look at therapeutic touch. JAMA 1998;279(13):1005-1010
5. Cox T. A nurse-statistician reanalyzes data from the Rosa therapeutic touch study. Alternative Therapies 2003;9(1):58-65