Essentials of Complementary and Alternative Medicine (June 1999)

Evaluating and Researching CAM Treatments
Complementary and alternative medicine is no different from conventional medicine when it comes to the application of research methods (
2
). In addition, the 

interaction of conventional and unconventional medicine has often resulted in improvements in the scientific method. Fifty years ago, for example, current methods of 
blinding and randomization became accepted parts of orthodox medical research after they were first developed and applied to unorthodox practices, such as 
mesmerism and psychic healing, in an attempt to disprove them (
3
). Thus, research in CAM may help us develop the decision rules for a consistent strategic 
approach to the application and interpretation of science in medicine. We illustrate the application of research methods to the CAM modalities of herbalism 
(phytotherapy), acupuncture, homeopathy, and complex naturopathic interventions, and show some of the strategies that have emerged. We then outline how these 
research strategies may help us develop decision rules for the assessment of chronic disease in both conventional medicine and CAM.
EFFECTIVE TREATMENT
Is the treatment effective? This is the most crucial question for all therapies, regardless of whether they are considered conventional or complementary. A therapeutic 
intervention is effective if it beneficially influences the course of a patient's disease. The concept of effectiveness includes a causal element (i.e. , Is it the treatment 
that makes the difference?) and a quantitative element (i.e.,  What is the degree of change?). For determining whether a therapy is effective in a single patient, one 
would essentially have to compare the course of the disease both with and without the treatment in that patient. This is rarely possible, and in most conditions the 
untreated course is not predictable.
The next most straightforward way to evaluate effectiveness is in controlled trials with groups of approximately comparable patients, where one group receives the 
treatment and the other group receives no treatment. A placebo-controlled condition should be included if the research is more narrowly focused than on the question, 
Is the treatment effective? For example, the research may aim to find the proportion of the specific effects attributable to a particular feature of the intervention under 
scrutiny (e.g., needling a specific acupuncture point in a specific way) in comparison to the  nonspecific effects from the act of treating in general (e.g., the act of 
needling in general). This type of research gives information about the treatment's effectiveness over placebo, rather than effectiveness in general. This information 
may or may not be important for the individual patient, depending on the risks, costs, feasibility, and other factors involved in the treatment. However, this information 
does not answer the question regarding the treatment's effectiveness; in actual practice, a patient is not given the option of placebo treatment or no treatment based 
on a random selection process.
Frequently, the clinician and patient want to know if the therapy they choose is the most effective treatment out of several treatment options. This is an attempt to find 
an answer to another question, Is this the most effective treatment? Determining the answer to this question requires direct comparison of two active treatments to 
evaluate their relative merits ( comparative effectiveness). In any of these tests, ensuring that the groups are comparable before treatment requires, whenever 
possible, randomization. The outcomes evaluated should be relevant primarily for the patient, but these or other outcomes may also be needed either for the provider 
or for public policy decisions. The methodology of the study should minimize the influence of chance and systematic error, but this needs to be balanced as much as 
possible with how the treatment is actually delivered in the health care system. 
Table 4.1
 illustrates these questions and their relationship to some of the research 
methods from 
Figure 4.1
.
Table 4.1. Clinical Questions and Research Designs
Evidence from research (i.e., evidence showing scientifically that a treatment works on groups of individuals who have a condition similar to the one that the physician 
is treating) is only one type of information needed for decision making in daily practice. These are some other factors:
A clinician's personal experience. The physician has seen the therapy benefit similar patients.
Plausibility. It makes sense that the therapy should benefit the patient.
Patient acceptability. The therapy will be acceptable for patients and is most likely to serve their objectives.
Economic and social reasons. The therapy costs less and is wanted by payers and policymakers.
In conclusion, both the question  “Is it effective?” and the principal approach to answer that question are the same for all therapies. However, the optimal strategy for 
applying these approaches when evaluating a therapy may be different for different types of interventions, as illustrated with the following CAM examples.
EVALUATION OF SPECIFIC CAM PRACTICES
Phytotherapy
Phytomedicine is the use of plants and their preparations for preventative or therapeutic purposes. Phytomedicine is a central part of traditional medicine in almost all 
cultures, and many drugs used today in Western medicine have been directly or indirectly derived from active plant components.
What is unconventional about phytomedicine? First, knowledge about using a specific plant or plant extract for a defined ailment is almost always based on empirical 
experience instead of systematic scientific research. As a consequence, a plant is often in widespread use  before it becomes an object of scientific investigation. In 
contrast to the development of conventional drugs—defining the substance, screening and toxicity tests in vitro and in animals, pilot trials in small patient groups, 
larger randomized trials, and postmarketing surveillance—the development of phytomedicine does not proceed in the same order.
Second, plant extracts always contain a variety of bioactive substances in relatively small concentrations. This complex composition makes the identification of a 
mechanism of action difficult. The relatively low concentrations of single components further decrease the  a priori credibility of effectiveness (at least among 
scientists). Given this situation, the strategy to evaluate a widely used plant preparation cannot be exactly the same as for a newly developed drug.
The primary questions are safety and effectiveness in actual use. In addition, there is a problem common for phytomedicine—the comparability of plant preparations. 
Echinacea, a plant originally used by North American Indians and now in widespread use for the prevention and treatment of upper respiratory tract infections, is an 
example of this comparability problem. Three different species of  Echinacea are in medicinal use (E. purpurea, E. angustifolia, E. pallida); sometimes only the roots 
are used, sometimes only the stems and leaves are used, and sometimes both are used. Some preparations contain pressed juice from fresh plants, whereas other 
preparations contain alcoholic extracts from dried plants. Depending on these variables, the composition and the concentrations of components can vary widely. Even 
within one product, the concentrations of components may vary according to the origin or quality of the plant material, the timing of collection, and other related 
factors. In some widely used plant preparations, such as St. John's Wort ( Hypericum perforatum), garlic, ginkgo, horse chestnut, or hawthorn, the problems, although 
in principle the same, are less pronounced. This is because high-quality preparations of these plants are standardized in their content of a characteristic component, 
which in most cases—but not always—is related to the clinical effects.
An efficient strategy for evaluating herbal preparations must integrate research on effectiveness and safety with a certain amount of basic research. This research 
must determine a minimum of comparability in the content of active or marker ingredients. Such a strategy is expensive, and most manufacturers of herbal 
preparations have neither the resources nor the expertise for such a program. Because plant extracts tend to have relatively moderate effects, the risk of getting 
nonsignificant results in a clinical trial is relatively high; also, it may not be possible to recruit the large numbers of subjects needed for a trial to show statistical 
significance. Furthermore, patents of plant extract processes do not secure exclusive market advantage on the product, so companies that produce herbal 

preparations have limited interest in investing large amounts of money into research. In addition, if certain herbal products come into direct competition with highly 
profitable prescription drugs, research may develop in an attempt to discredit such herbal research if the market share of the prescription drug is threatened.
C
URRENT
 R
ESEARCH
Example 1: St. John's Wort
Several hundred randomized clinical trials of herbal preparations are available. Most clinical trials of herbal preparations are placebo-controlled. No plant preparation 
has reached the status of a standard treatment on a worldwide level. A possible candidate to achieve this status is St. John's Wort ( Hypericum), which has been 
tested in approximately 40 randomized trials (
4
, 
5
, 
6
 and 
7
). There is evidence that Hypericum extracts are more effective than placebo and are as effective as 
standard antidepressants in treating mild and moderately severe depression (
4
, 
5
). However, the existing trials suffer from weaknesses in the characterization of 
patients, and the doses of standard antidepressants used in comparative trials have been low. Also, it is not clear to what extent the results obtained with one specific 
extract can be extrapolated to another. Good-quality  Hypericum preparations are standardized in their content of hypericins. Hypericins, however, are only one of the 
possibly active component groups of Hypericum extract; the existing evidence suggests that a number of components synergistically influence various metabolic 
pathways involved in depression (
8
).
Future research regarding this herb has to fill the gaps left by existing clinical trials, such as effectiveness in more severe depression, long-term effectiveness, and 
safety. Also, pharmaceutical and pharmacological research on product comparability and mechanisms of action are needed to complement the clinical data. Given the 
huge public interest and the financial interest by the pharmaceutical industry, it seems possible that sufficient resources will be available to realize such research.
Example 2: Ginkgo biloba
Another promising group of plant preparations is  Ginkgo biloba extracts. Approximately 40 controlled trials done on dementia patients have established that these 
extracts effectively enhance cerebral functioning and diminish a number of symptoms of cerebral insufficiency in the elderly (
9
, 
10
 and 
11
). These trials measured only 
cognitive functioning rather than activities of daily living, which today are considered the most important outcomes. A number of further trials suggest that  Ginkgo is 
also effective in treating certain aspects of other conditions, including cardiovascular and peripheral arterial disease (
9
, 
12
, 
13
 and 
14
). Again, several constituents 
seem to be involved in the clinical effects of  Ginkgo, but defined pharmacological effects can often be attributed to single substance groups; for example, the 
inactivation of toxic oxygen radicals is mainly caused by the action of  Ginkgo flavanoids.
The research conducted on Hypericum, Ginkgo, and other relatively well-investigated plant preparations, such as garlic (
15
, 
16
, 
17
, 
18
 and 
19
); evening primrose oil 
(
20
, 
21
 and 
22
); ginseng (
23
); Echinacea (
24
), mistletoe (
25
, 
26
); horse chestnut (
27
), Kava (
28
); Valeriana (
29
); hawthorne (
30
), saw palmetto (
31
); or feverfew (
32
), 
should not obscure the fact that most medically used plant preparations have not been evaluated in rigorous clinical trials. Although it seems realistic that some plant 
preparations will be systematically investigated in-depth over the next several years, some efficient, simplified strategies must be established to deal with the majority 
that will not.
These strategies are necessary because, although most herbal preparations have relatively few side effects if taken orally and if instructions on dosage and duration 
of use are followed, these preparations are not free of risks, and cases of severe adverse effects have been reported (see 
Chapter 6
) (
33
, 
34
). It seems that standards 
specific to phytotherapy need to address product quality, safety monitoring, and some reasonable measure of benefit. The evaluation of efficacy with expensive and 
time-consuming randomized trials is difficult in phytomedicine because of the moderate effects of these substances and their use mainly for chronic, 
non–life-threatening diseases with subjective endpoints.
Acupuncture
Most people see the logic in asking,  Is garlic effective for lowering blood pressure? and not, Is phytomedicine effective for lowering blood pressure? However, the logic 
of asking about specific treatments rather than whole systems of therapy is frequently forgotten in the case of acupuncture.  Is acupuncture effective for chronic pain? 
or Is acupuncture effective for asthma? are questions typically asked by those unfamiliar with acupuncture rather than questions as to whether a specific type of 
acupuncture treatment is effective. For example, if a group of asthma patients who meet defined inclusion criteria is randomized to receive acupuncture at certain 
points, at a certain depth, with or without achievement of  de-chi (a specific needling sensation), with or without electrical stimulation, and so forth, and if this group 
does better than the group receiving placebo (superficial needling or at different acupuncture points), does that mean that acupuncture works for treating asthma? 
Conversely, if such studies do not provide consistent evidence for an effect over placebo, does that mean that acupuncture is only placebo?
The first major problem in evaluation is that acupuncture is not a uniform treatment as is the application of a 1-mg aspirin tablet. Acupuncture, like physical therapy or 
surgery, is applied in very different ways. If a treatment strategy in a trial is interpreted as  the acupuncture treatment of asthma, it must be representative, at least for a 
defined group of providers and a defined group of patients. In a systematic review of 15 randomized trials of “acupuncture for asthma,” variable results and very 
different treatment strategies were found (
35
). Only two trials by the same author used the same acupuncture points. This makes one wonder if acupuncture treatment 
(like psychotherapy or surgery) is practitioner-dependent, which makes it difficult to determine the consistency of both acupuncture theory and practice.
However, one should not rush to conclusions based on a small sample of trials with a variety of study models in heterogeneous patient samples. What can be 
concluded from the available asthma trials? Surely, that not all acupuncture works in all asthma patients in each study model. But are some acupuncture strategies 
effective, and others not? Do we need 10 further studies with 10 further treatment strategies with uncertain generalizability? Should acupuncture be reimbursed? 
Acupuncturists should be encouraged to do research on what they are doing in actual practice and why they do it. They should monitor the outcomes of their patients 
carefully and check if there are major discrepancies and differences in findings. This type of research will not convince skeptics or answer the question,  Does 
acupuncture have specific effects? It can, however, help develop guidelines and establish clinically relevant hypotheses that can then be tested in a randomized 
clinical trial to determine the probability of an effect with a certain treatment approach.
In some of the available asthma trials, lung function, symptoms, and medication use did not change during the study. However, a simple prospective observational 
study would give us similar information and would not require a controlled trial. The idea behind this is that if you have no change at all (i.e., a nonprogressive 
condition), you do not need an RCT. A prospective case series with no differences between before treatment and after treatment is sufficient. If the results of an 
observational study and a randomized trial are conflicting (e.g., a clear improvement in the observational study versus no change, even within groups in the 
randomized trial), this information is important, but its interpretation is difficult. For example, are the discrepancies due to bias? Or did the experimental design 
interfere with the context or delivery of the normal acupuncture procedure?
A
CUPUNCTURE AND
 P
LACEBO
A second major problem with acupuncture research is the subject of placebo. Most randomized acupuncture trials include some form of placebo control (see 
references 
36
, 
37
, 
38
, 
39
, 
40
, 
41
, 
42
 and 
43
 for systematic reviews of acupuncture treatment for various conditions). Acupuncture is a complex intervention. 
Randomized trials of conventional complex interventions are rarely placebo-controlled; for example, the idea of comparing physical therapy and placebo-physical 
therapy seems at least questionable, and comparing coronary care units to placebo coronary care units is absurd. However, one reason for the extensive use of 
placebo controls in clinical trials of acupuncture is the relatively low plausibility (in the West) that needling specific points in a specific way should have specific effects 
over needling in general. Many studies are initiated and designed to disprove this general placebo hypothesis. It is definitively important to determine whether where 
and how the acupuncturist inserts a needle makes any difference. However, this question should be investigated in simple models when independent replication is 
possible. The cause and the influence of a treatment on this cause (i.e., chronic diseases) are accessible only in long-term studies. In these circumstances, 
placebo-controlled studies are rarely feasible or useful. An example of such a simple model is the stimulation of acupuncture point P6 (located centrally about 5 cm 
above the wrist) for relieving nausea and vomiting. This is discussed further in the next section.
Another more important argument for placebo control is that it allows the blinding—at least in theory—of both patients and evaluators. Because acupuncture is mainly 
used for treating non–life-threatening diseases with subjective outcomes, unblinded clinical comparisons are prone to bias.
Finding an adequate placebo control for acupuncture is difficult. Typical placebo procedures include stimulating nonacupuncture points, stimulating points thought not 
to be indicated in the condition being investigated, superficial needling, using devices for electrical or laser stimulation that can be switched off. Although placebo 
treatments that involve deep needling are unlikely to be inert and might increase the risk of false-negative results, treatments that do not involve needling are likely to 

be distinguishable. Patients know if they got “true” or “sham” treatment. It remains unclear whether the introduction of placebo controls and blinding enhances or 
confuses the conclusions of acupuncture trials. This situation makes it difficult to discern whether this feature of the treatment is effective on average for these types 
of patients (see 
Table 4.1
, question 3). Which proportion of the effects result from which specific component of the treatment is secondary to most patient care. The 
question that matters most is whether acupuncture benefits the patients (see 
Table 4.1
, question 2). With that goal in mind, careful, multidimensional measurement of 
outcomes and inclusion of independent—and, if possible, blinded—evaluators and rigorous monitoring might be better ways to minimize bias than the introduction of 
questionable placebo-controlled conditions.
B
ALANCING
 R
ESEARCH
 E
VIDENCE AND
 C
LINICAL
 R
ELEVANCE
The dynamics between evidence and relevance is nicely illustrated with two examples. The first is the study of the stimulation at acupuncture point P6 for relieving 
postoperative or chemotherapy-associated nausea and vomiting. This use was one of the few indications for acupuncture recommended by a panel at a National 
Institutes of Health Consensus Conference on Acupuncture in 1997. More than 30 trials have investigated this study model, and the overwhelming majority had 
positive results (
40
). This model has relatively little relevance for the daily practice of acupuncturists, but it is perfect for researchers looking for specific effects. 
Blinding is thought to increase validity, but whether this is truly achieved is unclear. This model is based on a simple intervention that can be standardized (almost like 
a drug treatment) and involves a condition in which outcomes are straightforward and follow-up times are short. Thus, acupuncture point P6 is the most studied area 
in acupuncture, not because it is the most important potential indication for acupuncture, but because it is simple to study. Thus, evidence of nausea and vomiting and 
its treatment approach is easily gathered. However, as previously stated, the findings are not very useful for the average acupuncture practitioner, and the indication 
is not the main one of interest for the public. Acupuncture is more frequently used to treat conditions such as chronic pain or drug addiction, for which 
placebo-controlled evidence is much less convincing (
37
, 
38
), primarily because it is difficult to develop rigorous trials in such complex situations.
In smoking cessation, there is quite clear evidence that acupuncture acts only as a placebo effect (
41
, 
42
). However, smoking cessation is of particular interest 
regarding the placebo problem previously discussed. Although there was no difference in smoking cessation rates between acupuncture and placebo-acupuncture, 
there was also no difference between acupuncture and conventional, effective smoking cessation treatments (
42
). If these results are valid, however, this would mean 
that acupuncture, despite being a placebo, would be equally effective as specifically effective, conventional interventions. These two examples show the complexity 
and inefficiency of trying to answer both questions of effectiveness and specificity (see 
Table 4.1
, question 3 and question 4) of complex treatment approaches with 
the same research strategy. In addition, it may not be reasonable to try to answer all these questions for all conditions. Again, do patients and many clinicians care if 
they stopped smoking because of a placebo effect or a specific effect with a safe and inexpensive procedure? For scientific and reimbursement purposes, however, 
these may be important questions.