Andrew Leigh’s Randomistas: How Radical Researchers Are Changing Our World is a love letter to randomized controlled trials (RCTs). The book wonderfully illustrates the ubiquity of experimentation and shines a spotlight on randomization as an integral part of disentangling cause and effect. Each chapter reads as an impassioned literature review covering the latest experimental findings in the fields of medicine, education, crime, development, business, philanthropy, and politics, all while continuously demonstrating RCTs as the gold standard of truth-seeking. This latter claim is not unique to Randomistas, but it is alluringly presented. It would take a stubborn will to not at least consider the ways to add more randomization to your own research after reading this book. This, I would contend, is the main strength of Randomistas. Leigh’s knowledge and passion draw in the reader to consider the importance of randomized experimentation in their own life. Thankfully, for those who do catch the randomization-bug and wish to become a randomista, the last chapter is a step-by-step guide on how to formulate and run your own experiment on any scale.

A major appeal of the book comes from it being written by a true believer. Leigh lives his life by the latest experimental finding. He stopped taking multivitamins after a study found them to be ineffective and no longer attends annual medical check-ups after seeing evidence that suggests they do not reduce the chance of falling ill; even the subtitle of the book was chosen based on the result of an RCT. Leigh’s commitment to experimentation bleeds into what I consider the central argument of the book: randomize and experiment wherever you can, whenever you can. The simplicity of the argument is a major draw. Once convinced that randomized controlled trials are the gold standard of truth finding, it doesn’t take much to buy into Leigh’s advocacy for randomization and experimentation at the margin. Why not experiment?

Yet as a love letter, Randomistas largely glosses over the reasons not to experiment. An exception is the chapter dedicated to the ethical concerns of randomized controlled trials, whereas questions of cost, scale, and generalizability are only brought up in passing. Ethics may be the primary obstacle for convincing policy makers and their constituents to engage in randomized experimentation, yet the latter concerns are primary for the economist. RCTs may be the gold standard of truth-seeking in an unconstrained world, but, alas, we live in a world of constraints. Leigh recognizes this. Much of the book, including the limited responses to the criticisms of RCTs, focuses on advocating more experimentation rather than simply affirming RCTs as the pinnacle of truth discovery. “RCTs may not be perfect,” Leigh admits, but the alternative is worse: policy based on mere opinions (p. 125). As with all things, we must consider the tradeoff and “weigh the risk of doing a randomised trial against the risk of continued ignorance” (p. 110). Leigh’s experiment-at-the-margin argument is convincing, even if lacking, though it does limit the case for RCTs some. Yes, not all experiments are overly costly and there are ways to design experiments to get around many of the given issues; however, depending on the constraints, we may be left to toil at the margin instead of addressing questions of importance.

Other reviewers have more thoroughly addressed the lack of an adequate response in Randomistas to the major criticisms of RCTs (see, for instance, Lisbeth B. Schorr, “Solving Social Problems ‘One Coin Toss at a Time,’” Stanford Social Innovation Review [September 2018]), so I won’t dwell on it here. Instead, I would like to focus on a different sort of critique which I’ll call “OkCupid Mysticism.” The chosen name comes from an experiment mentioned in the book in which the dating company OkCupid randomly varied their matching score to find that displaying high, yet false scores produced more conversations and pairings. The OkCupid founder concluded from the experiment that “the mere myth of compatibility works just as well as the truth” (quoted on p. 130). How do false compatibility scores on a dating site constitute a critique of RCTs? Sometimes objectively false beliefs—superstitions or myths—are preferred to the truth. Leigh mentions many false beliefs throughout the book, from the effectiveness of bloodletting, to spontaneously created mice, witches, and alchemy. Such beliefs evidence how far we have progressed intellectually as a result of continued experimentation. OkCupid Mysticism points out that in addition to the risk of continued ignorance we must also consider the risk of displaced myth.

Consider two myths: the oracle of benge practiced by the Azande and the bulletproofing gri-gri in the village of Bulambika. The oracle of benge provides an ordained answer to the question of whether a person was bewitched by their neighbor. For benge to work, a fowl must be fed poison after which those seeking the oracle’s advice detail the situation, ask a yes-or-no question of guilt or innocence, and shake the fowl to ensure the poison was ingested. Depending on how the question is formulated, the death or continued living of the poisoned fowl will affirm or deny the guilt of the accused neighbor. If found innocent, the conflict is dropped. If convicted, the result of the oracle is presented to the neighbor who then apologizes, and the conflict is resolved. The bulletproofing ritualresults in the creation of a powder which, when applied to those who volunteer to protect the village from attack, makes a person bulletproof. Any bullet shot at the protected villager will either miss or bounce off. Death in battle did not dispel the myth. Instead it indicated that the complex set of conditions required for the gri-gri to work were performed incorrectly. Are these beliefs true? Can poisoned fowls determine guilt and can detailed rituals and a magical concoction make people immune to bullets? I doubt it, but then I also believed that breast cancer screenings were cost-effective before reading Leigh’s summary of the experimental evidence. As Leigh aptly points out, many common beliefs may be incorrect, but we can’t know which until a proper experiment is performed.

Now imagine a randomized controlled trial was designed to determine the accuracy of the poisoned fowl or the effectiveness of bulletproofing. (I should note that the bulletproofing gri-gri was tested on a goat and proven effective, but it is unclear if it was a properly structured experiment with a control goat and random assignment.) Would this be helpful? Perhaps, but it may also be harmful. While both myths sound about as farfetched as the spontaneous creation of mice to modern, western ears, they serve important functions. As argued by Peter Leeson (“Oracles,” Rationality and Society 24.2 [2014]: 141-169), benge served as a form of conflict resolution in the absence of courts to deal with low-grade social conflict. And the belief in the bulletproofing gri-gri, as argued by Nathan Nunn and Raul Sanchez De La Sierra (“Why Being Wrong Can Be Rights: Magical Warfare Technologies and the Persistence of False Beliefs,” American Economic Review: Papers & Proceedings 107.5 [2017]: 582-87), worked to resolve the collective action problem in motivating individuals to protect the village against raids. In these scenarios, the truth of the belief is not as valuable as the mechanism which the myth plays. And discovery of that truth could destroy the functioning institution rather than help further social cooperation.

Unlike the criticism which suggests RCTs are inefficient in some contexts due to the costs of implementing an experiment exceeding the value of the knowledge obtained, OkCupid Mythicism cannot be easily remedied by the cost-reduction strategies mentioned in Randomistas. Even if implementation were costless, the result of the experiment—the truth discovery on the accuracy of the belief—can be costly if it displaces a myth with no available substitutes. An RCT on poisoned fowls may suggest that benge doesn’t determine guilt any better than flipping a coin, but it says nothing about the overall effectiveness of the belief system. Of course, we could run an RCT to test the effectiveness of the belief system relative to other systems, but constraints limit the availability of the “other” options. A western style court system or a non-mythified randomization process which lacks the believed infallibility of benge was not an available option for dispute resolution in the short-run and a large supply of Kevlar vests or a peace treaty seemed equally unlikely for the people of Bulambika in their time of war.

OkCupid Mythicism doesn’t negate any of Leigh’s passion, advocacy, or even arguments in favor of RCTs. It isn’t so much a criticism as it is a call for humility, as Leigh does emphatically throughout Randomistas. RCTs are the gold standard of truth finding and we should add in randomization and experiment where we can. But the discovery of truth without an adequate replacement for myth can be as costly as continued ignorance.

Randomistas opens with the fascinating story of scurvy and the experimentation of James Lind. Lind discovered the importance of citrus decades before it was prescribed for every voyage, yet his experimental results were ignored to the detriment of many sailors. Leigh suggests part of the reason Lind’s findings never caught on was that his theoretical reason for why citrus worked was “hocus pocus” (p. 4). Perhaps Lind would have been better served by increasing the hocus pocus surrounding lemons and creating a better myth for those unwilling to listen science. The historical stories of the ignored randomistas is one of the highlights of the book and Leigh is correct in his admiration of the radical researchers who dedicated their lives to experimentation. Like Leigh suggests, we should encourage randomization and experimentation where we can—at least where it makes sense to do so.