Having written about pseudoscience and quackery continuously for over a decade and having engaged in conversations about it online for over 15 years, I’ve come to recognize a number of traits that are virtually the sine qua non of quacks and pseudoscientists and their believers. Obviously, one of them is a severe case of the Dunning-Kruger effect, a tendency of those with low expertise in a topic to overestimate their expertise and express far more confidence in their conclusions than warranted while those with high expertise know enough to know how much they don’t know about a topic and thus tend to express more uncertainty and caveats. Basically, the Dunning-Kruger effect describes how unskilled individuals express an illusory superiority, mistakenly believing their knowledge, competence, and ability to be much higher than it really is.
As a result of the Dunning-Kruger effect, coupled with other cognitive shortcomings suffered by all human beings (but seemingly amplified in believers in quackery and pseudoscience) that lead them to believe in pseudoscience, such as confusing correlation with causation, motivated reasoning, and the like, believers in pseudoscience are often so absolutely rock-solid in their beliefs that they are virtually impossible to reason with. It is incredibly difficult to change their minds, and disconfirming evidence often causes them to dig in all the more deeply to defend their beliefs. Not uncommonly, this leads them to commission studies designed to support their beliefs. But what happens when such a study does not actually support their belief? What happens when such a study backfires spectacularly and not only fails to support their belief, but emphatically so? Skeptics were re-treated to just such a spectacle last week when SafeMinds and other antivaccinationists were burned by a study they funded (subscription required):
Between 2003 and 2013, SafeMinds provided scientists from the University of Texas Southwestern School of Medicine, the University of Washington, the Johnson Center for Child Health & Development and other research institutions with approximately $250,000 to conduct a long-term investigation evaluating behavioral and brain changes of baby rhesus macaques that were administered a standard course of childhood vaccines. (The National Autism Association, another organization that has questioned vaccine safety, also provided financial support for this research.) The latest paper in the multiyear project was published Monday in the Proceedings of the National Academy of Sciences (PNAS). In it, the researchers concluded that vaccines did not cause any brain or behavioral changes in the primates.
Astute readers will recognize that I’ve written about similar papers before reporting that pediatric vaccines cause changes in behavior and/or brain structure in macaque monkeys. Specifically, way back in 2008, I noted the initial report of this ongoing study, first when preliminary results were reported as a poster presentation and then later another publication from the same group published in 2009. Steve Novella and a certain well-known friend of the blog have also described how poor experiments published from these studies in 2010 were, the latter of whom cited several other major criticisms of the study, not the least of which was some reporting of changes in the size of a part of the brain known as the amygdala that were…hard to believe. There were also a lot of issues with the control group chosen.
Basically, these abstracts and papers reported the results of an ongoing study looking at infant vaccines in macaque monkeys to see if there was an effect on socialization or changes in brain anatomy, the key hypothesis seeming to be that thimerosal-containing vaccines cause autism. (The investigators even added thimerosal to some of the vaccines because they weren’t being made with thimerosal anymore!) You can read the links I cited just now if you want the gory details; suffice to say that these were not good studies and not particularly good evidence that vaccines cause autism, as shown by the fact that homeopaths loved the study, and ultimately the paper examining hepatitis B specifically was withdrawn. Yet these reports were flogged for quite a while by the antivaccine movement as proof positive primate data that vaccines are Evil.
Fast forward to 2015. Now we have a much larger, much better study. It’s even by the same people. And guess what? It’s as negative as negative can be. No wonder SafeMinds and other antivaccinationists are unhappy. Let’s take a look.
Three dozen dead monkeys later, vaccines still don’t cause autism
One of the limitations constraining those of us who do human subjects research is that ethical considerations often prevent us from designing our clinical trials in what would be, from a strictly scientific standpoint that ignores all other considerations, in the most methodologically rigorous way. For example, we can’t intentionally infect human beings with known inocula of deadly bacteria in order to cause a reproducible severity of disease to be treated with a new antibiotic. Also, leaving aside that such a study would be impractical from a purely practical standpoint given that it would take decades, we can’t ethically do a randomized, controlled study of smoking versus no smoking and then assess differences in the incidence of heart disease, lung cancer. Yet, from epidemiological studies, we do know that tobacco smoke does cause lung cancer, heart disease, chronic lung disease, and other cancers.
Similarly, from an ethical standpoint we can’t do a randomized, controlled clinical trial of a “vaxed versus unvaxed” group because it would leave the control group of children unprotected against vaccine-preventable diseases. There would be no clinical equipoise. Some antivaccinationists have begun to accept this reality—grudgingly—and started to propose epidemiological “vaxed-unvaxed” studies, of course without realizing how difficult it would be to control for confounders and how many subjects would be needed. (Hint: More unvaccinated children than probably exist in the US.)
One way to try to get answers when you can’t use humans is to use animals. Indeed, that is traditionally how drug and toxicity studies have been done. However, this approach has problems as well, because, depending on the animal model and the disease, the relevance of such experiments can be questioned. One way to try to maximize the relevance to human physiology is to use nonhuman primates, but such experiments are incredibly expensive to do and must be held to very high ethical standards given how human-like they are.
Here’s where the monkey study by Gadad et al. in PNAS, that has so riled antivaccinationists, comes in. Entitled “Administration of thimerosal-containing vaccines to infant rhesus macaques does not result in autism-like behavior or neuropathology“, it examines the effect of the infant vaccine schedule on macaque monkeys. Oddly enough, Laura Hewitson is one of the authors. Remember that Hewitson was a key player in doing the previous awful studies, one of which listed Andrew Wakefield as a co-author. She is now the Research Director of the Johnson Center for Child Health and Development, which is what Andrew Wakefield’s old quack clinic, the Thoughtful House Center for Children, became after he was given the boot by its board of directors in the aftermath of his losing his UK medical license and seeing his 1998 Lancet paper implicating the MMR vaccine retracted. Her history and current position just make the results of this study even more delicious.
Unfortunately, part of the justification for the current study were the previous monkey studies that purported to show a link between vaccines and neurodevelopmental disorders. Indeed, so bad were those previous studies that personally I consider this study to be highly unethical, given that there is no compelling evidence to justify subjecting 79 macaque monkeys to a bunch of injections and killing 36 of them to dissect their brains, all in the service of testing the long discredited hypothesis that vaccines cause autism or other neurodevelopmental disorders. The University of Washington IACUC, which approved this study, should be ashamed of itself. Ethically, the study is a travesty. It was also a waste of a lot of money, again, because this question did not need to be studied yet again. Despite the lack of a compelling scientific rationale for the study, it was nonetheless done, and done competently; so we have to consider its results which—surprise! surprise!—were completely negative. Let’s take a look at the details.
The design was quite simple. There were 79 infant macaque monkeys subjected to six different vaccination schedules: (i) Control (n = 16), in which animals received saline injections in place of vaccines; (ii) 1990s Pediatric schedule (n = 12), in which animals received vaccines following the pediatric schedule recommended in the 1990s; (iii) 1990s Primate (n = 12), in which animals received vaccines recommended in the 1990s but on an accelerated schedule; (iv) thimerosal-containing vaccines (TCV, n = 12), in which animals received all TCVs but no MMR vaccines following the accelerated schedule; (v) MMR (n = 15), in which animals only received the MMR vaccine but no TCVs following the accelerated schedule; and (vi) 2008 (n = 12), in which animals received vaccines recommended in 2008 but on an accelerated schedule. Infants were assigned to a peer group of four animals, with multiple study groups being tested each year for neurodevelopmental outcomes.
The investigators then assessed social behavior, with testing being carried out by a social tester blinded to the experimental group. The testers were well-trained and experienced; they were also tested for reliability and used standard testing methods:
Infants underwent testing as follows: from birth to 20 d, infants were assessed for the development of neonatal reflexes and perceptual and motor skills; from postnatal day 14 to ∼3.5 mo of age, infants were examined for the development of OCP; from ∼3 to 6 mo of age, animals underwent discrimination learning assessments; from ∼5 to 8 mo of age, animals were assessed for learning set development; and from 30 d to 12 mo of age, animals underwent assessments of behavior before group living. These developmentally appropriate tests are measures of neurodevelopment, learning, cognitive abilities, and social behavior in young macaques (45). At ∼13 mo of age, animals were transferred to juvenile caging where they were group housed (n = 4 males per group) with animals from within their peer group for the duration of the study. All subsequent behavioral data were collected while animals were in their home cage.
Behaviors were defined as passive, exploring, playing, sex, aggression, withdrawal, fear-disturbed, rock-huddle-stop-clasp (strong clasping/grasping of another monkey without play behavior, or self-clasping with arms, legs, hands, or feet, without locomotion and no active inspection of own or other’s body), and stereotypy (repetitive body movements, with or without locomotion, requiring three or more consecutive, repetitive movements). It might be a cliché, but there were no statistically significant differences detected in any of the behaviors measured in any of the experimental groups.
Then, at the conclusion of the experiment, brains from monkeys in the control (N=12), 1990s (N=12), and 2008 groups (N=8) were sectioned for histological and immunohistological examination. The authors examined the brains for neuropathology in parts of the brain previously shown in humans to have changes in autism: the cerebellum, hippocampus, and amygdala. Try as they might to find differences, Gadad et al. failed to find any differences between controls and either of the two vaccinated groups examined. There were no changes in the neurons in these regions. There were no changes in protein levels. Basically, there were no differences from control in the two experimental groups in the volume of the cerebellar hemispheres, the number or density of Purkinje cells in the cerebellum. There was no difference in the size of the Purkinje cells. Western blots (a means of detecting proteins with antibodies) failed to find differences in certain Purkinje-cell associated proteins calbindin, GAD-67, and proteins that are markers for different cell types, such as Iba1 (a microglial marker) and GFAP (astrocyte marker). Again, these were all negative. Gadad et al. measured these proteins up, down, right, left, and sideways (so to speak), but failed to find any differences.
Why, you might ask, didn’t Gadad et al examine the brains from the monkeys in the other experimental groups? The authors justify this decision thusly:
The neuroanatomical analyses were first performed in brains from the 1990s Primate and 2008 groups, as animals in these groups received the highest amount of EtHg exposure (1990s Primate) or the most extensive vaccine exposure (2008). Because no neuronal differences were found in either of these vaccine groups compared with the control group, no additional vaccine groups were fully studied.
This is a reasonable compromise to keep the number of monkeys euthanized as low as possible. If the groups that received the most extensive thimerosal exposure and the highest vaccine exposure showed no detectable differences in brain structure in regions relevant to autism pathophysiology, then there really isn’t a good reason to kill the rest of the monkeys to look at their brains. Even with that compromise, 36 monkeys paid for this information with their lives (16 control + 12-1990s schedule, and 8-2008 schedule) and brains while the rest received at least unnecessary injections and other interventions. Basically, this study’s results are inconsistent with the three main “hypotheses”—though after all the disconfirming data calling them “hypotheses” really is doing them too much honor—that thimerosal-containing vaccines, MMR vaccines, or “too many [vaccines] too soon” cause autism. Each hypothesis is represented by an experimental group.
It’s utterly ironic that so many antivaccinationists funded this study, as noted in the acknowledgements:
We thank the following for their generous financial support: The Ted Lindsay Foundation, SafeMinds, National Autism Association, and the Johnson and Vernick families. This work was also supported by WaNPRC Core Grant RR00166 and CHDD Core Grant HD02274.
It’s depressing to see that the Washington National Primate Research Center, the Center on Human Development and Disability, and the University of Washington contributed wasted funds to this unethical study.
SafeMinds is, of course, highly antivaccine. It’s particularly amusing to see SafeMinds get burned funding a study to show that vaccines cause autism when they’ve been burned before for being involved with such a study. I’m referring, of course, to the study published in 2007 in the New England Journal of Medicine by Thompson et al. (yes, that William Thompson) that failed to find a correlation between thimerosal-containing vaccines and neurodevelopmental disorders. As you might recall, Sallie Bernard of SafeMinds was on the advisory committee for the study. Why invite an antivaccine activist to be on such a committee? I don’t know. Perhaps the misguided idea that you should keep your friends close and your enemies closer. (That didn’t work out so well for Jon Snow.) Be that as it may, when Thompson et al. didn’t show what SafeMinds and antivaccinationists wanted it to show, she immediately pulled criticisms out of her nether regions to try to discredit it, to be followed by other antivaccine groups.
Here we go again, same as it ever was.
So how did antivaccine activists react to the study they had helped fund? Did they accept the results and start to wonder whether they might have to start rethinking their previous views about vaccines and autism? (That’s what real scientists and those who are science-based in their outlook would do.) Come on, this is SafeMinds we’re talking about! Of course it did nothing of the sort! Predictably, SafeMinds did exactly what it did in 2007 when Thompson et al. didn’t turn out the way it had hoped going in. It immediately went on the attack:
SafeMinds, the nonprofit that funded the research, is not happy with the results. Representatives from the group say the findings contradict both an earlier pilot study and interim progress reports the organization received from the researchers.
Naturally, no such attack would be complete without insinuations of scientific fraud:
SafeMinds also believes that the research team behind the new PNAS study may have cherry-picked their data. SafeMinds Director Lyn Redwood, a registered nurse, says she received an email in 2013 from the researchers reporting a “statistically significant” 11 percent reduction in certain types of hippocampal cells in the vaccine groups. But she says the authors did not include these findings in the new paper.
Redwood, of course, is not a scientist, and it shows. Many are the times that preliminary analyses in a scientific study look as though they will find a positive result. Many are the times that the positive result disappears as more and more of the subjects are analyzed and more measurements completed. Many are the times that scientists have been disappointed because of this phenomenon, so tantalized by a seemingly positive result only to see it disappear. Redwood seems not to understand this; she seems not to understand that a two year old progress report describing promising preliminary analyses does not matter one whit if those preliminary analyses didn’t pan out. In this case they didn’t. Laura Hewitson, who by working with Andrew Wakefield on one of the preliminary studies used to justify this study showed that at that time she was very much leaning antivaccine, appears to have pulled back from the pseudoscience and become much more scientific. She even tries to school Redwood in the very point I just made:
Dr. Laura Hewitson, director of research for the Johnson Center for Child Health & Development, an adjunct associate professor in the Department of Psychiatry at the University of Texas Southwestern Medical Center, a lead researcher on project and co-author on all four papers, says that at the time that email was sent, it was also made clear to SafeMinds “that the data should be treated as preliminary until all of the animals had completed the study.” She added that none of the study’s procedures changed once her team moved from the pilot program to a larger sample.
And to explain to Redwood basic things like blinding to experimental group so important in studies where humans are making measurements:
She added that all of the researchers, technicians and behaviorists involved in collection and analysis of data did not know which of the monkeys were in the vaccine groups or the control group. The researchers also implemented a “chain of custody” protocol once the data were collected, in which they reviewed chronological documentation that shows the control, transfer and analysis of all data sets. Hewitson says that her team used an independent statistical consultant for all data analysis, and that two additional outside investigators from two other academic institutions confirmed their findings.
“As you can see, we have done everything possible to ensure the integrity of the data. My co-authors and I stand by our published findings,” she says. “The comprehensive nature of the current study underscores why the findings from the pilot study should be interpreted with an abundance of caution, given the small number of animals included.”
One notes that this is much more rigorous than the protocols used in the pilot studies. I (and others) have previously pointed out that pilot studies are often positive, but those positives are usually due to small numbers of subjects and less rigorous methodologies, and tend to disappear with larger, more rigorous studies. This is exactly what happens with acupuncture and homeopathy studies; it is what appears to have happened with this study.
Elsewhere, antivaccinationists have taken to Twitter to claim things like:
@EMcCra2 @doritmi more on methodology. Apparently weight for the monkeys was averaged, based on an assumed 6.3:1 ratio, and thus dosing was
— Master Muppet ™ (@rblotnicky) October 3, 2015
@EMcCra2 @doritmi lowered 6.3 times from the 1990shot. Also didn’t take into account differences in human birth weight. The study assumed
— Master Muppet ™ (@rblotnicky) October 3, 2015
@EMcCra2 @doritmi all human infants that provided the 6.3:1 difference were born on time, at same weight, w/ zero genetic factors considered
— Master Muppet ™ (@rblotnicky) October 3, 2015
This is, of course, an utterly ridiculous objection when you read the supplemental data and methods section in the paper, which lays out in considerably detail exactly how doses were adjusted for the monkeys:
EtHg [thimerosal] content of primate vaccines was determined by first averaging the weight ratios for human infant boys: male infant primates across the six time points of vaccine administration. The average weight ratio was 6.3:1. The EtHg content in each pediatric vaccine was then divided by 6.3 to determine the dosing of EtHg for each primate vaccine. This method provided a similar dosing of μg EtHg/kg body weight for infant boys and primates.
Included is a table of the vaccine schedule that describes how the dosing was determined. It lists the weight ratio between human baby boys and infant macaques at the different time points where vaccines are administered. This ranges from 4.5 to 7.5, meaning that human infants weigh from 4.5 to 7.5 times what infant macaques weigh at the same time point. So basically at some ages, the macaques received somewhat more thimerosal than human babies on a per weight basis, while at other ages they received less. Since at no time is the difference more than 29% and given that the very idea behind the mercury militia’s belief that thimerosal causes autism is that the thimerosal content far surpasses the toxic threshold, it’s unlikely that averaging would have had any significant effect. Finally, given that no one has shown any genetic risk factors for vaccine injury leading to autism (mainly because vaccines do not cause autism), how could the investigators possibly know how to consider genetic risk factors in this study? They can’t. If they had found differences in neurodevelopmental correlates in one of the vaccinated groups, that would have been a topic for another study.
A particularly amusing response comes from Sallie Bernard of SafeMinds, who can’t seem to restrain herself from attacking science that SafeMinds was involved in when it doesn’t show what she thinks it should show:
But Sallie Bernard, president of SafeMinds, says she would at least like to see a re-analysis of the newest data. “We feel that embedded within these data sets there are animals that have potentially an adverse reaction to this vaccine schedule that would mirror what happens in human infants,” she says. “The majority who get vaccines are fine, but we believe there is a subset that have an adverse reaction to their vaccines. By looking at the raw data, not data in aggregate, we may be able to identify the subgroup that had that reaction.”
Ironically, one of the complaints that I’ve heard about the study from antivaccinationists is that the numbers are too small. If that were true, then doing subset analysis is a fool’s errand, because if the number of subjects in the experimental groups was too small to detect a statistically significant difference in the overall groups, it’s almost certainly too small to detect anything meaningful in subgroup analyses. Be that as it may, it is a general principle in doing studies of this type, either animal experiments or human clinical trials, that subgroup analyses, unless they are pre-specified in the protocol, is far more likely to mislead than illuminate. Indeed, whenever I see clinical trials in which post hoc subgroup analyses are done, I know it’s a (likely) futile attempt to salvage a positive result out of a negative study. That’s exactly what Sallie Bernard wants to do here. It’s also hilarious, in light of her accusation against the researchers of “cherry picking data,” that cherry picking data from the raw data set is exactly what Bernard is proposing here, although she’s certainly too scientifically ignorant to realize it.
Why do scientists keep doing these unethical, wasteful studies?
One of the arguments for studying the question of whether vaccines cause autism over and over and over again is that enough negative studies will ultimately convince antivaccine parents that vaccines are not dangerous and do not cause autism. I’ve argued again and again that such is not the case. This is yet another example demonstrating just that.
Dr. Paul Offit, after listing all the studies that have failed to find a correlation between vaccines and autism, alluded to a similar same concern in an accompanying editorial, although he was nowhere near as blunt as I have been because, well, that’s the way he rolls. Characteristically, he refrained from calling this macaque study unethical and a waste of money and tried to find something good about this waste of money and primates:
One could reasonably wonder whether it is necessary to continue to spend more money chasing this fruitless, dead-end hypothesis. However, the constant drumbeat of negative studies has made a difference. Unlike 10 y ago, the media no longer covers the vaccine– autism controversy by telling both sides of the story when only one side is supported by the science; for the most part, they have chosen perspective over false balance. Legislators are also stepping up; both California and Vermont recently eliminated their philosophical exemptions to vaccination.
I will admit that Dr. Offit has a point, nor am I in any way arguing that there isn’t value in negative studies. No one, least of all I, would dispute that negative studies can be very useful. As useful as such studies can be when evaluating issues like whether vaccines cause autism, a question—and reasonable scientists can disagree over the answer to this question is—remains: At what point can we say that enough is enough, that the question being studied has been settled to a sufficient degree of certainty that it is no longer worth spending large sums of money on and killing three dozen intelligent primates to ask and answer the question yet again? An animal study like this might be considerably less expensive and complex than large epidemiological studies, but it’s still an animal study. Human epidemiological studies, on the other hand, are much more expensive and require controlling for confounders that don’t need to be controlled for in the highly controlled world of animal studies.
Sadly, I must disagree with Dr. Offit here. The relentless drumbeat of negative studies about vaccines and autism has probably played much less of a role in how the press has changed its tune in the way it covers vaccine-autism pseudoscience to a far less credulous one than Dr. Offit credits it with. In fact, as much as it pains me to do so, I argue that this turnabout has far more to do with the recent discrediting of the chief architect of the MMR-autism scare, Andrew Wakefield. As much as I really, really wish it were the science alone that finally turned the tide and persuaded reporters and much of the general public that vaccines do not cause autism, I really do believe that was less of a factor than Wakefield’s utter disgrace and discrediting. From my perspective, seeing Wakefield lose his medical license, be dismissed as the medical director of the quack clinic he used to run in Texas, have his Lancet paper that launched the MMR scare retracted, and be shown to have been a scientific fraud practicing what Brian Deer so aptly called “Piltdown medicine” probably played a far greater role in changing public perception. Add to that the string of outbreaks of vaccine-preventable diseases that culminated in the Disneyland measles outbreak early this year. These things, more than anything else, were likely what shifted public opinion. That’s why it’s my opinion that, barring new compelling evidence that demands that we study the question further, studies such as this one are unnecessary.
In the abstract, studies like this one do have value—to a point. Given the evidence we have and what we already know about the relationship between vaccines and autism (namely that there isn’t one), however, I really have to question the very necessity for such a study. Sure, it shows what every other well-designed study asking whether vaccines, be they thimerosal-containing or not, are associated with autism, but surely the investigators should have known before they started that that’s what it would show. Worse, it’s changed no minds among antivaccine activists. They dismiss it as an animal experiment and continue to call for either an experiment even more unethical than this one (a randomized trial of “vaxed versus unvaxed”) or an epidemiological study of “vaxed versus unvaxed” children that they can dismiss because it’s an epidemiological study prone to all of the confounders that epidemiological studies are prone to.
This brings us back to how I started this post. One of the arguments for studies like this is that negative studies will change minds. They will not. They might give some reassurance to parents who were already open to being reassured about vaccines, but they have zero effect on people like Sallie Bernard and Lyn Redwood. They are true believers, and if a study is negative, even one that they funded, it must be because there was incompetence or scientific fraud. It can’t possibly be because vaccines don’t cause autism.
At least I can hope that after this study no more primates need die in the service of a dead, discredited hypothesis. A bigger challenge will be preventing children from dying because belief in this discredited hypothesis leads parents to not vaccinate their children against deadly diseases.