Ranking Nations provides a difficult-to-classify exploration of three important indexes used to facilitate comparisons across countries. These indexes are the Human Development Index (HDI) published by the United Nations Development Programme, the Corruptions Perceptions Index (CPI) published by Transparency International, and the Environmental Performance Index (EPI) published by the Yale Center for Environmental Law & Policy. I am a co-author of another similar index, the Economic Freedom of the World index, published by the Fraser Institute.

The book is organized into seven chapters, each with a clear focus, but with common threads throughout. Chapter 1 functions as an introduction, along with several reflections, some of them a bit offbeat, on why we rank countries. Chapter 2 describes the history and calculations of the HDI, the CPI, and the EPI. Chapter 3 gives five reasons for using indices—instrumental, conceptual, tactical, symbolic, and political—and developing them conceptually, while also describing how they may be employed by the media. Chapter 4 briefly sets up statistical work that will be performed in subsequent chapters, which are the main empirical results of the book. Chapter 5 evaluates the data to explore how volatile rankings are, and how sensitive rankings are to changes in methodology. Chapter 6 provides a descriptive analysis of how the three indices are used in both traditional media and social media. Chapter 7 restates the main ideas of the book and provides a conclusion.

The two common threads this reader takes away from the analyses are that rankings are volatile at the middle of the rankings (a point which I see no reason to challenge), and that changes in methodology can cause substantial changes in rankings. Either of these problems can potentially result in very misleading headlines in the media, or in incorrect perceptions in general.

The use of rankings—especially in something like a regression—rather than index scores in academic work is generally a mistake, and many of the issues that the author raises simply disappear if one uses the index score instead of the rank. But we cannot simply provide an index score to the public: If a country were to increase its score in economic freedom from 7.05 to 7.25 out of 10.00, what does that mean? Is 7.25 “good”? A rank conveys most of the missing information. Ideally, we would provide z-scores instead of rankings, but no one would have any idea what we are talking about. The author acknowledges that the rankings could be reported for the benefit of the consumer, but then contends that we should ignore that and only report index values (p. 22). If that were to happen, either the index would be ignored, or journalists would simply calculate the rankings themselves in order to report on them, if the data are all that important. The sovereignty of the consumer will hold.

Throughout the book, the focal point for the author is the titular indices. But in an offhand remark (p. 65) he acknowledges that his claims equally apply to rankings using “unemployment rates and school exam results in mathematics.” This issue completely undercuts the rationale for the book; he states that he is using the term “index-based ranking” “largely because [his] focus in the monograph is on the three indices outlined [in an earlier chapter].” The issues he raises are not inherent to constructing indices and comparing countries: they are inherent in ranking almost anything that has something quantitative behind it.

Also strange is his criticism of how rankings may be jostled around by changes in methodology. Early on, the author cites technical notes from EPI documentation stating that their ratings should not be assembled into time series or panel data because of how different their methodologies are across time (p. 60). The author implies this is a general problem. It is not. From what I can tell, EPI is distinct in how they make it abnormally difficult in building data that is comparable across years. The website Our World in Data, which compiles different datasets into something that is more manageable for the non-specialist to explore, includes both the HDI and CPI. Despite the environment being an important area of focus for Our World in Data, they do not report the EPI. I suspect these two points are not unrelated.

The author does mention—rather awkwardly—that other indexes are “often” updated backwards with new editions of each report (that is, they will apply the new methodology to earlier years to make everything consistent), but this does not matter. Why?

Hence in this research I have worked with the rankings as initially published in these reports as these were the basis for the narratives, the stories, that the writers of the report told at the time and it is these stories that were intended to be read and digested by those ‘consuming’ the reports and acting, in their own ways, upon them. The rankings and stories were inevitably time-bound and attempts at retrofitting indices [do] not change that. Stories were not retold and actions undone because of ‘retrospective’ revaluations of the indices. In other words, I feel the curators cannot have it both ways. (p. 61)

I would suggest that constructing a narrative using rankings is something one does at one’s own peril; moreover, if a narrative is sensitive to how rankings ended up some particular year, that might be a problem with a particular narrative. One should not create a narrative that depends on whether a country should actually be ranked thirty-third or forty-second. Rankings offer the benefit of rapidly conferring information that might be central to a narrative in such a way that is comprehensible to laypeople, but at the expense of robustness. It is a tradeoff for the narrative writer to make, not for the “curators.”

Later (p. 122), the separate criticism is raised that reports of rankings that change from year to year due to methodology may be confusing for non-academics who are uncomfortable digging into technical appendices. What I am rather sure of is that less than one percent of consumers of these indices could remember where a country ranked in some previous year while also being unwilling or unable to look at the report to see what has driven the change. I could not tell you with any certainty what the exact rank of the United States in economic freedom was in the last published report, for instance.

What the author in general conveys is a very specific set of priorities: that we should put lexical importance on consistency in rankings across published reports, regardless of whether consistent, sensible scoring/ranking is still available in the actual data set, regardless of whether we want the data to be the best we can do at the moment, and regardless of whether we could do something better in the future. I do not think many share his priorities.

I am also aware that rankings in earlier years for Economic Freedom of the World will change not only because of changes in methodology, but because data from earlier years are added or corrected. In the coming year’s report, I updated government consumption data for Chile, Costa Rica, Croatia, Mexico, and Togo because the World Bank changed what it was reporting for the last twenty years. We have no reason to believe that these data are less accurate than what was reported previously. Should we not update the data because it makes certain narratives awkward when earlier ranks might move around? This is certainly also true for other indicators; the GDP data that goes into the Human Development Index are updated and corrected all the time, sometimes also going back decades.

Lastly, statistical work in the book is fine insofar as it illustrates the author’s points concerning how rankings misbehave in practice. But if I were simply to take the subtitle of the book as given (“the value of indicators and indices”), I would have expected in an academic work to have found more technical discussion using tools from, for instance, construct validity to assess EPI, CPI, and HDI. I suspect that the reason EPI has been inconsistent in its methodology is that they are struggling to balance issues like face validity (it gives numbers that look right), predictive validity (the score predicts things it is expected to predict), and content validity (it is including everything that falls under the umbrella of “environmental performance”). The author’s exploration of changing ranks across methodologies could have also been heavily informed by convergent validity and perhaps the application of Cronbach’s alpha. This is not the flavor that is found in the book.

If the central idea of Ranking Nations is that rankings are often less robust than we think, then that hypothesis is true. If the correct Straussian reading of Ranking Nations is that there are considerable issues with the Environmental Performance Index, I find that to be at least plausible. If the author believes that the issues raised regarding indexes has any bearing on how indexes ought to be constructed and reported, I disagree.