I just added Jerry Z. Muller’s The Tyranny of Metrics to my ever-growing manager-training-via-book-discussion arsenal.
As the title suggests, the basic premise is that evaluation metrics, even with the best of intentions, can lead to disastrous outcomes. As a data practitioner who lives an evidence-based lifestyle, to that I say: “Preach, dude. 🙌”
There’s a lot of good stuff in here, even if the case studies inside are bit too well-trodden. Written by an academic historian for general audiences, using familiar touch points to make his point is fine I guess. (I’m also generally well read in on pop-business concepts, armchair economics, and the books and pundits that promote them. This isn’t a book review, but I did wish he included at least one novel case studies.)
There’s also an undertone of “trust me, bro” that I find wanting and thought to address it, particularly for those to whom I will recommended this book.
Considering the Source
A high school English teacher routinely pop-quizzed us every time the first chapter of a new reading assignment was due. It was a very short quiz that asked, without fail:
- Who is the author?
- When was this book published?
- Where was this book first published?
Jerry Z. Muller. 2018. Princeton University Press.
The habit stuck. (Thanks, Dr. Schiller.)
After finishing The Tyranny of Metrics, I went to go learn more about this Muller fellow. Relevant to this book, two major points stuck out:
- His work primarily covers economics, capitalism, and their cultural context. His conclusions are nuanced, well-researched, and hew conservative. He seems deeply skeptical towards all radical political leaning and favors traditionalism.
- On the Smith/Keynes continuum, he leans Smith and has written about him extensively.
I, on the other hand, am a student of behavioral economics who can be occasionally found sporting a T-shirt with a squirrel holding a martini printed on the front, a nod towards Keynes’ “animal spirits”.
In the long run, we’re all dead. But ideas, even antiquated ones, seem to live forever.
Baby and the Bathwater
Anyway, the book’s conclusion is not to discard metrics wholesale, but to select good ones. He provides a very, very good checklist for this on p.177 (which, alone, is reason to buy the book) that underscores the tradeoffs between useful and tyrannical.
In this, The Tyranny of Metrics is an excellent introduction to the topic.
However, the miss for me (and where I start to diverge from the author) is what one ought to do about metrics that falter.
In fairness, this isn’t where he claims he will spend most of his time—his goal seems to be to educate and caution, not instruct—and so the guidance offered is pretty thin. He also claims that transparency (vis-a-vis metrics) sometimes does more harm than good.
In one example, he claims that effective governance requires backroom deals in order to create pareto-optimal outcomes. He argues that trying to publicly work out policies that affect myriad conflicting interest groups would place undo burden on the process, leading to less-good decision-making than if unencumbered by transparency. Politicians would clam up if they had to justify every decision. Muller’s suggestion is that we should trust people and their expertise as it has been done.
Dubious, maybe; but, that’s his argument.
Less careful readers might interpret this to say that metrics are expensive and counterproductive, and are more trouble than they’re worth. (As Americans are generally behind in science and mathematical literacy, the burden for worthiness is even higher.) I think Muller was careful to allow readers to draw their own conclusions and “so what?“s as he circled politically-charged discussions like police force efficacy and socioeconomic academic performance disparities.
Interestingly, Muller omits any discussion where the tyranny of metrics may be the point. He softens his criticisms of his select case studies behind a “well-intentioned but counterproductive” qualifier, allowing policy makers to save face—”oh, if they only knew better”. (e.g., No Child Left Behind, Compstat – two programs that were meant to help people, but were ultimately tainted by front-line professionals gaming the metrics.) He makes no effort to consider how bad actors might create bad metrics and instead delegates the discussion into fiction with frequent mentions of HBO’s “The Wire”. (I haven’t seen the show, and it isn’t reasonable to assume his readership has either.)
To me, this omission is incredibly naive.
People are generally good, but power tends to attract people who are not, and absolute power corrupts absolutely. Any discussion on good intentions going awry—especially around metrics-based policy evaluations at the federal level—seems incomplete without a discussion about when bad intentions disguised as agreeable metrics are used as a form of control, or as a set-up for failure and an “I told you so”.
Where some see a cautioned academic treatise, others see an instruction manual.
See also the CIA’s Simple Sabotage Field Manual, another title in my unofficial management book club.
Metric tuning
I would offer, instead of the author’s note that we perhaps allow the pendulum to swing back as false dichotomies often suggest, perhaps we look at it from another dimension.
A pendulum on a string, seen from the side, moves back and forth, left-to-right.
Observed from above, we discover there are no sides. It’s moving in a circle.
Observed from afar, we discover the pendulum is tracking a helix through space, our path of progress.
Or, perhaps we drop the metaphors all together and lean into the fact that our species has a long history of developing the tools and technology for continuous improvement. Net-net, transparency and metrics have been a boon to the safety, prosperity, and livelihoods to the bipedal earthlings on this planet, provided we empower good people to lead us there.
The bigger issue for me isn’t that metrics are tyrannical, it’s that most people are ill-equipped to understand the what our metrics mean and fail to contextualize them honestly.
People still rely on the Dow Jones Industrial Average to make investment decisions or how the economy is doing.
I mean, sheesh.
Escaping the Scapegoat
Back to what’s good about the book. (I do, after all, recommend it.)
Metrics are tools; and like any tool, they can be used to create, destroy, or modify. Well-deployed metrics can serve as a fulcrum for performance and performance management. However, when misused, they can have deleterious consequences.
This book provides a framework for understanding a metric’s limitations, with fodder on how to discuss their perceived and actual effectiveness.
In the end, responsibility lies with those who set the metrics to do so carefully and ethically. This underscores the importance of proactively supporting leaders with good intentions as metrics alone cannot tell the whole story.
After all, who decides what gets measured?
Was my high school English teacher inflating our grades with his entirely-predictable low-stakes pop-quiz, or was he conditioning us to put information in context to create knowledge?
Sometimes, it can be both.
Michael E. Gruen. 2024. Self-published on michaelgruen.com