A full critique of the article
by Shang et al
(Lancet 2005;366 (9487):726-733)
K. Chatfield and C. Relton
The article by Shang et al (2005) can be critiqued on number of different levels and this response has therefore been broken down into sections for ease of referral. Each of the issues raised should have been considered by the authors of the paper during production and should also have been raised during the Lancet’s peer review process prior to acceptance for publication. The fact that these issues have not been fully explored by the authors or publishers could suggest either a lack of understanding of research in homeopathy or a deliberate attempt to mislead.
METHODOLOGICAL ISSUES
Matching
In their report Shang et al (2005) purport to match 110 homeopathy trials with 110 allopathic trials. Their description of how these trials were matched according to similar conditions and similar outcomes leaves many questions unanswered. What exactly do they mean by similar? How similar are the conditions that are matched? Variation between patient cohorts and severity of complaint do not appear to have been taken into consideration.
Of the total number of homeopathy trials 16% involved individualised (classical) prescribing. This involves a lengthy case-taking and hence greater chance of the therapeutic relationship influencing the effects of treatment. Hence the trials involving classical homeopathy, a complex intervention, should have been matched with corresponding trials of complex interventions rather than with simple interventions such as pharmaceutical drugs.
Selection of the final 8 homeopathy and 6 allopathy trials
Of the total 220 trials the authors identify 21 homeopathy trials and 8 allopathy trials that are of higher standard according to their own set criteria. Yet they present no comparative analysis of this subset of trials. They then proceed to further select the small subset of purportedly larger and higher methodological quality trials (8 homeopathy trials and 6 conventional medicine trials) from which the paper’s main conclusion is drawn. The authors fail to describe the weighting of the two attributes size and methodological quality. The authors do not provide an explanation as to how they chose the particular cut off point that they used to select the final 14 trials.
The choice of the two parameters of size and methodological quality at first appears to be reasonable but is problematic for homeopathy trials for the following reasons:
1. High methodological quality in meta-analyses is defined as high internal validity but ignores whether the study has any external validity – its clinical relevance and its general real world applicability of the study findings. In placebo controlled RCTs, high internal validity is gained at the expense of low external validity such that those of apparent high quality often bear little relation to real world practice or relevance.
There are difficulties in attempting to research any complex health care intervention such as individualised homeopathy. Careful modelling of complex interventions is essential to ensure that interventions fit with and reflect the complexities of the settings within which interventions will be applied, and to ensure that the outcomes chosen are those most appropriate to demonstrate any benefits or risks (Rowlands 2005).
2. The use of trial size as a parameter introduces potential bias for the homeopathy trials that is not addressed in the authors’ discussion. The largest trials are ones funded by pharmaceutical companies. Pharmaceutical companies fund trials of isopathy and complex homeopathy, usually in the area of prevention (because that will involve more people potentially buying the product). Isopathy and complex homeopathy are the crudest and least clinically effective forms of homeopathy. The largest trials are subject to the bias of pharmaceutical profit funding, where the decision to fund a study will be based on potential pharmaceutical profit rather than the clinical relevance.
The authors fail to identify specifically the particular trials that are used in the final analysis that would enable a fair critique of the validity of the included trials.
Elucidation of the rationale for choosing this small number of studies as well as a list of which studies this seemingly comprehensive interpretation was drawn from is essential.
Statistical Analysis
It would be normal practice when comparing two different treatment options to measure the statistical difference between the two options for a fair comparison. The authors fail to do this, presumably as this calculation shows clearly that there is no statistical difference between the effects of the two interventions. That is, the statistics tell us that there is no basis for saying that allopathic intervention is any better than homeopathic intervention.
The authors use particular criteria for selection of trials that they define as being of higher quality. Of the 220 trials only 21 homeopathy trials and 8 allopathy trials meet these criteria for high quality. With such a small number of trials meeting the criteria and with there being a significantly larger number of homeopathy trials, the authors should have stopped at this point and concluded that:
a. there are not enough high quality trials on which to carry out the analysis, and
b. that the difference between the quality of the homeopathy and the allopathy trials invalidates the matching.
The authors went to the great trouble of selecting 110 homeopathy trials that met their inclusion criteria, matching them with 110 allopathy trials and then ignored all but 8 trials of homeopathy and 6 of allopathy in their final statistical analysis. Moreover the original stated intention to compare trials of similar condition and outcome has been ignored in the final analysis. The final small subset of trials is not matched at all suggesting that different kinds of trials are being compared, apples are being compared with oranges – a common failing in meta-analyses.
INTERPRETATION BIAS
The authors generalise from an extremely small pool of data to draw their apparently broad and negative conclusions. Their statistical analysis and interpretation of results can be challenged in many respects and their main conclusion that their “findings are compatible with the notion that the clinical effects of homeopathy are placebo effects” is unfounded.
While sample size is small in many individualised (classical) homeopathy studies because of limited funding and the early state-of-the-science need for preliminary studies to define the proper design parameters (as in any proper research effort), the Shang et al. paper does not support the conclusion that homeopathic remedies or conventional drugs cause only placebo effects. It does support the conclusion that homeopathic as well as conventional drug treatments in clinical trials can produce placebo effects. This is no surprise and does not logically support the conclusion that we should discard either homeopathy or conventional drugs. (Bell,2005)
The authors display their own bias in interpretation when they dismiss out of hand the substantially beneficial pooled effect from 8 trials of homeopathic remedies in upper respiratory tract infections. This in spite of the fact that the trials perform well in the authors own set test of funnel plot asymmetry which demonstrated that there was no significant difference between effects in the higher quality trial and the lower quality
trials. Here the authors speak of biases prevalent in these publications to excuse the effect without specifying in any way how they relate to these trials and indicate that conclusions from these trials cannot be trusted.
META-ANALYSES
Any meta-analysis of homeopathy will inevitably be considering only a small number of trials as there are relatively few for any one condition. This leads to the pooling of heterogeneous data, a fact so often completely ignored.
Meta-Analyses as a tool are akin to public surveys to glean opinions about a politician. They are of vague value but do not reflect individual truths which may carry much more meaning. That is especially true of a meta-analysis of homeopathy, a science in which individualization of prescription and interpretation of response is fundamental to success. Meta-analyses flatten diversity and minimize nuggets of truth by imposing criteria exclusive to allopathic protocols. (Gray 2005)
In this report Shang et al (2005) different types of prescribing method for homeopathy trials are all considered equally. Of the 100 trials selected only 18 (16%) deemed classical homeopathy i.e. the type of individualised prescribing that occurs in real life and there is no indication of how many of the selected 8 trials involved this type of prescribing.
RANDOMISED CONTROLLED TRIALS AND HOMEOPATHY
The authors’ conclusions are premised upon the supposition that the placebo controlled randomised trial represents the gold standard against which all research should be judged. It is becomingly increasingly understood that there are considerable problems in using this method to test complex interventions such as homeopathy.
It is always going to be difficult to adopt the reductionist research method of the placebo controlled RCT to measure effects of complex intervention such as homeopathy. Two primary concerns for homeopaths are that the treatment is holistic and that it is individualised. Treatment cannot be standardised and patient response is unpredictable. RCTs are looking for specific effects whereas homeopathy is attempting to produce general health effects as well as specific effects – homeopathy treats the whole person.
Patients may choose homeopathy or acupuncture or other forms of complementary and alternative medicine because of preferences and perhaps even the potential for responding. Randomized controlled trials do not necessarily recruit such patients – they take patients with a conventional diagnosis, but do not screen for patients who typically may gravitate to homeopathy or other types of complementary and alternative medicine. The scientific evidence in this regard is emerging in some studies and requires open-minded, albeit sceptical, consideration toward understanding the true nature and potential benefits of these treatments for some, but not necessarily all, patients with a given conventional diagnosis.” (Bell 2005)
“Actually there is no threat to anyone. Each system has its strengths and limitations. If we work hand in hand, and not against each other, it will really benefit our patients. In order to do this it is important to understand the other system, and evaluate it according to its principles and not see it from the view point of the other system.” (Sankaran 2005)
AUTHOR CONTACTS
K. Chatfield, University of Central Lancashire
kchatfield@uclan.ac.uk
C. Relton. ScHARR, University of Sheffield
c.relton@sheffield.ac.uk