The following is a contribution to the JREF’s ongoing blog series on skepticism and education. If you are an educator and would like to contribute to this series, please contact Bob Blaskiewicz.
In an effort to shake up my teaching schedule, I volunteered to lead a colloquium for the Honors department this semester. This is a small class that meets twice a week for an hour. Titled “Science versus Pseudoscience: Do you know what you think you know?”, the class covers subjects from Bigfoot to health scams. One major goal of this course is to teach critical examination of evidence, both scientific and popular.
For the health & medicine portion of the class, I chose to address the two types of evidence separately. My students are upper-level science majors, and have learned to read primary literature in other classes. We started with Greenberg et al.’s Rainy days for the society of pediatric anesthesia (2012). This is an excellent, tongue-in-cheek example of the correlation-causation heuristic. The paper positively correlates meteorological data with meeting dates of the Society, ending with the suggestion that umbrellas should be sold at meetings. The students, who were used to using primary literature in summary or lab reports, were initially baffled by the paper. Were they serious? Why would someone publish something like this? I asked the students some questions: Did this meeting cause it to rain? What would be the mechanism? Why would you come to that conclusion based on that evidence? The students saw the need for more than a correlation in the data, and we talked about the importance of prior probability analysis.
The next paper I offered them was Daryl Bem’s Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect (2011). This study contains multiple experiments supposedly showing psychic abilities in college students. Since Bem’s paper is very long and confusing, it was paired with the deconstruction by James Alcock (2011). I asked the students to critique the Bem paper on their own before reading the Alcock article. Many of the students picked up on the same problems outlined by Alcock. Overall the class’s unfamiliarity with statistics made it difficult for them to analyze the methods, but they did note that it was far more confusingly laid out than other scientific papers they had read. The class concluded this odd organization was used to obfuscate problems. The different areas the students majored in were helpful during the class discussion. A psychology student took issue with the validity of Bem’s stimulus-seeking measures. A math student pointed out that there were far better statistical tests to use to examine this type of data (ANOVA). We discussed designing proper controls, sample size effects, and experimental design. The students saw the fishing expedition that likely went on with the data prior to publication. They were all shocked that the paper got published in the first place.
The next step was to give them a choice of several papers to critique on their own. Students chose a paper from this list, then presented it to the class and led a discussion. The class chose to review Wakefield’s MMR-autism paper, the Cha et al. in vitro prayer paper, and the more recent Séralini et al. GMO paper. The students were allowed to get help with jargon and stats, but were told to avoid online reviews of or blogs about the papers – I wanted them to remain unbiased in their evaluations.
First up was Cha et al.’s Does prayer influence the success of in vitro fertilization-embryo transfer? (2001). In this study, groups of Christians in the USA, Canada and Australia were asked to pray for the fertility of couples in a Korean fertility clinic. Other groups of Christians were asked to pray for the first groups’ prayers to be effective .The student presenter had looked up the researchers’ credentials (after seeing that one was listed as a JD) and told the class about Wirth’s fraud charges. I explained that the paper shouldn’t be condemned based on that information – data is data, regardless of personal or criminal behavior. The class felt that the lawyer was there to deal with the ethics of enrolling people in a study and sharing their personal medical information with complete strangers without permission. The class recognized the strangeness of the experimental design. They were figured out that the data weren’t actually different than the normal rates of successful pregnancy at that clinic. Considering the oddness of having people pray for people who are praying, the students felt that there had likely been prior studies that hadn’t gotten good results. They deduced that this was the last in a line of experiments and the only one with some kind of positive result.
The second paper in the student-led discussion was Wakefield’s Ileal-lymphoid-nodular hypoplasia, non-specific colitis, and persuasive developmental disorder in children (1998). This is the seminal paper supposedly demonstrating a link between vaccination, autism, and gastrointestinal disorders. It took me almost an hour to track down a copy of this paper that did not indicate its retracted status. The formatting of this copy was clearly different from a normal publication, and several of the students thought that was suspicious. I assured them that it had been published in a major peer-reviewed journal. The presenting student had his mother, a nurse practitioner, sit down and go over the paper with him to explain the jargon. This enabled him to break the methods and results down for the class. The class easily picked up on the problems with sample size, proper controls, and non-uniformity of developmental delays within the study population. I explained the timing of autism symptom development and vaccines. The students saw the spurious links between the two. They also noted that the children had all already been diagnosed before the study even began, creating a selection bias.
The third paper was Séralini et al.’s Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize (2012). This study claims to show a link between GMO crops, Roundup herbicide, and cancer. The presenting student began the discussion by saying, “Data just kind of threw up all over this paper.” The class again saw through an oddly designed set of experiments, small sample sizes, confusing methodology, bizarrely designed figures and unnecessarily complex statistics for what it was – obfuscation. They interpreted the huge amounts of data presented, particularly the tables of blood and urinalysis data, as a fishing expedition. The students became lost trying to parse out the methodology, so I went to the blackboard and drew out the experimental design as the students read through the methods. It was clear that proper controls were never used – all rats were exposed, in one way or another, to Roundup. The students were unfamiliar with the Sprague-Dawley rat strain, which is practically a tumor factory. I asked them, why would the researchers pick an inappropriate model organism? The class thought the researchers probably knew about this rat strain before beginning the experiment. The students thought the Sprague-Dawley line was specifically chosen because those rats develop a lot of tumors.
After we finished discussing the papers, I let students in on the papers’ and authors’ histories. The class was surprised by the shenanigans involved. It was especially surprising to them that it took so long for the Wakefield paper to get withdrawn.
The next part of the class examined medical claims. We started by discussing Wechsler et al.’s Active albuterol or placebo, sham acupuncture, or no intervention in asthma (2012). This paper beautifully illustrates how the placebo effect works. Asthma patients were given either albuterol, a placebo inhaler, sham acupuncture or no intervention (control). All three treatments had similar ratings of patient-determined effectiveness, which was significantly better than the control. The actual measures of effectiveness showed the placebo and sham acupuncture to be no different from the control, with the albuterol being remarkably more effective. In short, placebos make you feel better, but they don’t actually change the course of your disease. This launched into a discussion of the ethics of offering patients placebo treatments, and of health treatments that offer to “harness the power of the placebo.”
With this information in hand, students were required to find two or three examples of health scams. They could be from the web, TV, print ads, or in-store displays. During class we pulled up the websites, TV spots and articles the students had found. One student even brought in some Five-Hour Energy drinks for us to critique. The whole class worked together to dissect the ads and labels, looking for the fine print and “quack Miranda warning”. If a study was mentioned, we found and evaluated it. The students were surprised at how poor the studies were – not published, poorly designed, or not showing what the labels claimed.
Primed with this discussion, the students were given two handouts to read. The handouts were information copied from websites, with the URLs printed so students could navigate the sites themselves. One was for Dr. Nicholas Gonzalez (www.dr-gonzalez.com), the other for Dr. Stanislaw Burzynski (www.burzynskiclinic.com). The class was asked to evaluate the websites and information contained within.
On the Gonzalez site, red flags abounded. The students zeroed in right away on the cost of the treatments, and the advertising for books and lectures. Other red flags included the mixture of diseases cured by the treatment (which opened up a chance to talk about “one true cause” and “one true cure” claims). The students were unimpressed by the results of the one clinical trial of the treatment, comparing the Gonzalez protocol to standard treatment. This study shows a significant increase in mortality and decrease in quality of life measures within the Gonzalez protocol group as compared to the standard treatment group (Chabot et al., 2010). The Burzynski site, by comparison, was full of what one student termed “green flags”. Burzynski does not claim to treat diseases other than cancer. There are multiple studies, including an approved Phase III clinical trial scheduled to start in 2012. Burzynski belonged to multiple appropriate professional groups. The students felt comfortable stating that this was not a quack site. I must have smiled, because one student said, “You’re going to drop a bomb on us, aren’t you?” And indeed I did. Burzynski hasn’t had a publication since about 2009, and the Phase III trial was approved several years ago and has no participants. The website doesn’t say, but he is charging people tens of thousands of dollars to enroll in his ongoing Phase II clinical trials. His treatments include standard chemotherapy along with the antineoplaston treatment, a major flaw in study design. None of this information is included anywhere on the site, which the class found to be unethically deceiving.
The students were floored. Burzynski was once a qualified, non-quack doctor. Why would he go down this road? We talked about how people can be so sure of themselves that they can’t see where they are wrong. The class discussed the multiple ways people fool themselves to keep their worldview in place. We then talked about the ethics of treatments like this – if a person is going to die anyway, what does it hurt? The students thought it was unethical to offer treatments which don’t work – time, money, and quality of life get poured down the drain by seeing these doctors.
This series of readings and class discussions taught students how to critique health claims from multiple sources. They were able to go through primary literature and look for flaws or issues, even without much background in a specific subject. The students know what trigger words or phrases to look for on supplement or other dubious product packaging and advertising. Finally, they learned that sometimes quackery is covered over with a veneer of legitimacy that must be scraped away.
Alcock, J., 2011. Back from the future: parapsychology and the Bem affair. http://www.csicop.org/specialarticles/show/back_from_the_future.
Bem, D.J., 2011. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100(3): 407 – 425.
Cha, K.Y, D.P. Wirth and R.A. Lobo, 2001. Does prayer influence the success of in vitro fertilization-embryo transfer? The Journal of Reproductive Medicine 46(9) 718 – 787.
Chabot, J.A., W.-Y. Tsai, R.L. Fine, C. Chen, C.K. Kumah, K.A. Antman and V.R. Grann, 2010. Pancreatic proteolytic enzyme therapy compared with gemcitabine-based chemotherapy for the treatment of pancreatic cancer. Journal of Clinical Oncology 28(12): 2058 – 2063.
Greenberg, R.S., M. Bembea, E. Heitmiller, 2012. Rainy days for the society of pediatric anesthesia. Anesthesia and Analgesia 114: 1102 – 1103.
Séralini, G.-E., E. Clair, R. Mesnage, S. Gress, N. Defarge, M. Malatesta, D. Hennequin, J. Spiroux de Vendômois, 2012. Long term toxicity of a Roundup herbicide and a Roundup-tolerant genetically modified maize. Food and Chemical Toxicity 50(11): 4221 – 4231.
Wakefield, A.J., S.H. Murch, A. Anthony, J. Linnell, D.M. Casson, M. Malik, M. Berelowitz, A.P. Dhillon, M.A. Thompson, P. Harvey, A. Valentine, S.E. Davies, and J.A. Walker-Smith, 1998. Ileal-lymphoid-nodular hypoplasia, non-specific colitis, and persuasive developmental disorder in children. The Lancet 351: 637 – 641 (retracted).
Wechsler, M.E., J.M. Kelley, I.O.E. Boyd, S. Dutile, G. Marigowda, I. Kirsch, E. Israel and T.J. Kaptchuk, 20120 Active albuterol or placebo, sham acupuncture, or no intervention in asthma. The New England Journal of Medicine 365(2): 119 – 126.
Dr. Karen Koy is an Assistant Professor of Geology in the Department of Biology at Missouri Western State University. This is the second in a three-part series chronicling an ongoing college-level class that takes extraordinary claims as its topic. The first part may be found here.