Skip to main content

Weapons of Math Destruction - Cathy O'Neil ****

As a poacher-turned-gamekeeper of the big data world, Cathy O'Neil is ideally placed to take us on a voyage of horrible discovery into the world of systems making decisions based on big data that can have a negative influence on lives - what she refers to as 'Weapons of Math Destruction' or WMDs. After working as a 'quant' in a hedge fund and on big data crunching systems for startups, she has developed a horror for the misuse of the technology and sets out to show us how unfair it can be.
It's not that O'Neil is against big data per se. She points out examples where it can be useful and effective - but this requires the systems to be transparent and to be capable of learning from their mistakes. In the examples we discover, from systems that rate school teachers to those that decide whether or not to issue a payday loan, the system is opaque, secretive and based on a set of rules that aren't tested against reality and regularly updated to produce a fair outcome.
The teacher grading system is probably the most dramatically inaccurate example, where the system is trying to measure how well a teacher has performed, based on data that only has a very vague link to actual outcomes - so, for instance, O'Neil tells of a teacher who scored 6% one year and 96% the next year for doing the same job. The factors being measured are almost entirely outside the teacher's control with no linkage to performance and the interpretation of the data is simply garbage.
Other systems, such as those used to rank universities, are ruthlessly gamed by the participants, making them far more about how good an organisation is at coming up with the right answers to metrics than it is to the quality of that organisation. And all of us will come across targeted advertising and social media messages/search results prioritised according to secret algorithms which we know nothing about and that attempt to control our behaviour.
For O'Neil, the worst aspects of big data misuse are where a system - perhaps with the best intentions - ends up penalising people for being poor of being from certain ethnic backgrounds. This is often a result of an indirect piece of data - for instance the place they live might have implications on their financial state or ethnicity. She vividly portrays the way that systems dealing with everything from police presence in an area to fixing insurance premiums can produce a downward spiral of negative feedback.
Although the book is often very effective, it is heavily US-oriented, which is a shame when many of these issues are as significant, say, in Europe, as they are in the US. There is probably also not enough nuance in the author's binary good/bad opinion of systems. For example, she tells us that someone shouldn't be penalised by having to pay more for insurance because they live in a high risk neighbourhood - but doesn't think about the contrary aspect that if insurance companies don't do this, those of us who live in low risk neighbourhoods are being penalised by paying much higher premiums than we need to in order to cover our insurance. 

O'Neil makes a simplistic linkage between high risk = poor, low risk = rich - yet those of us, for instance, who live in the country are often in quite poor areas that are nonetheless low risk. For O'Neil, fairness means everyone pays the same. But is that truly fair? Here in Europe, we've had car insurance for young female drivers doubled in cost to make it the same as young males - even though the young males are far more likely to have accidents. This is fair by O'Neil's standards, because it doesn't discriminate on gender, but is not fair in the real world away from labels.
There's a lot here that we should be picking up on, and even if you don't agree with all of O'Neil's assessments, it certainly makes you think about the rights and wrongs of decisions based on automated assessment of indirect data.


Using these links earns us commission at no cost to you
Review by Brian Clegg


Popular posts from this blog

Patricia Fara - Four Way Interview

Patricia Fara lectures in the history of science at Cambridge University, where she is a Fellow of Clare College. She was the President of the British Society for the History of Science (2016-18) and her prize-winning book, Science: A Four Thousand Year History (OUP, 2009), has been translated into nine languages. An experienced public lecturer, Patricia Fara appears regularly in TV documentaries and radio programmes. She also contributes articles and reviews to many popular magazines and journals, including History Today, BBC History, New Scientist, Nature and the Times Literary SupplementHer new book is Erasmus Darwin.

Why history of science?
I read physics at university, but half-way through the course I realised that had been a big mistake. Although I relished the intellectual challenge, I was bored by the long hours spent lining up recalcitrant instruments in dusty laboratories. Why was nobody encouraging us to think about the big questions – What is gravity? Does quantum mechani…

The Idea of the Brain: Matthew Cobb *****

Matthew Cobb is one of those people that you can’t help but admire but also secretly hate just a little bit for being so awesome. He is professor for zoology at the University of Manchester with a sizable teaching load that he apparently masters with consummate skill. He’s a scientific researcher, who researches the sense of smell of fruit fly maggots; I kid you not!  He’s also an attentive and loving family father but he still finds time and energy to write brilliant history of science books, three to date. His first, The Egg and Sperm Race, describes the search for the secret of human reproduction in the seventeenth and eighteenth centuries and is one of my favourite history of science books, on the period. His second, Life’s Greatest Secret is a monster, both in scope and detail, description of the hunt to decipher the structure and function of DNA that along the way demolishes a whole boatload of modern history of science myths. The most recent, and the subject of this review, is

The Big Ideas in Science - Jon Evans ***

The starting point of a review like this has to be to congratulate the author on his achievement, Jon Evans, because getting all of science into one relatively short book is a massive (and thankless) task. Although inevitably the result is a fairly hectic dash through the material, with limited space for subtleness, Evans manages to make the experience readable and has a light touch that is effective without becoming too simplistic.

There is only one reason this book doesn't get four stars - it's not the quality of the writing but rather the selection of the contents. Of course, there is bound to be plenty of stuff missed out - how else could you get all of science into 269 pages? But the balance is strangely skewed. Chemistry is pretty much omitted, though aspects of chemistry occur under other headings. But for me, the real problem is that physics is really under-represented. It's interesting to use Jim Al-Khalili's recent excellent physics summary title The World Acc…