Skip to main content

Weapons of Math Destruction - Cathy O'Neil ****

As a poacher-turned-gamekeeper of the big data world, Cathy O'Neil is ideally placed to take us on a voyage of horrible discovery into the world of systems making decisions based on big data that can have a negative influence on lives - what she refers to as 'Weapons of Math Destruction' or WMDs. After working as a 'quant' in a hedge fund and on big data crunching systems for startups, she has developed a horror for the misuse of the technology and sets out to show us how unfair it can be.
It's not that O'Neil is against big data per se. She points out examples where it can be useful and effective - but this requires the systems to be transparent and to be capable of learning from their mistakes. In the examples we discover, from systems that rate school teachers to those that decide whether or not to issue a payday loan, the system is opaque, secretive and based on a set of rules that aren't tested against reality and regularly updated to produce a fair outcome.
The teacher grading system is probably the most dramatically inaccurate example, where the system is trying to measure how well a teacher has performed, based on data that only has a very vague link to actual outcomes - so, for instance, O'Neil tells of a teacher who scored 6% one year and 96% the next year for doing the same job. The factors being measured are almost entirely outside the teacher's control with no linkage to performance and the interpretation of the data is simply garbage.
Other systems, such as those used to rank universities, are ruthlessly gamed by the participants, making them far more about how good an organisation is at coming up with the right answers to metrics than it is to the quality of that organisation. And all of us will come across targeted advertising and social media messages/search results prioritised according to secret algorithms which we know nothing about and that attempt to control our behaviour.
For O'Neil, the worst aspects of big data misuse are where a system - perhaps with the best intentions - ends up penalising people for being poor of being from certain ethnic backgrounds. This is often a result of an indirect piece of data - for instance the place they live might have implications on their financial state or ethnicity. She vividly portrays the way that systems dealing with everything from police presence in an area to fixing insurance premiums can produce a downward spiral of negative feedback.
Although the book is often very effective, it is heavily US-oriented, which is a shame when many of these issues are as significant, say, in Europe, as they are in the US. There is probably also not enough nuance in the author's binary good/bad opinion of systems. For example, she tells us that someone shouldn't be penalised by having to pay more for insurance because they live in a high risk neighbourhood - but doesn't think about the contrary aspect that if insurance companies don't do this, those of us who live in low risk neighbourhoods are being penalised by paying much higher premiums than we need to in order to cover our insurance. 

O'Neil makes a simplistic linkage between high risk = poor, low risk = rich - yet those of us, for instance, who live in the country are often in quite poor areas that are nonetheless low risk. For O'Neil, fairness means everyone pays the same. But is that truly fair? Here in Europe, we've had car insurance for young female drivers doubled in cost to make it the same as young males - even though the young males are far more likely to have accidents. This is fair by O'Neil's standards, because it doesn't discriminate on gender, but is not fair in the real world away from labels.
There's a lot here that we should be picking up on, and even if you don't agree with all of O'Neil's assessments, it certainly makes you think about the rights and wrongs of decisions based on automated assessment of indirect data.


Paperback 

Kindle 
Review by Brian Clegg

Comments

Popular posts from this blog

The Art of Statistics - David Spiegelhalter *****

Statistics have a huge impact on us - we are bombarded with them in the news, they are essential to medical trials, fundamental science, some court cases and far more. Yet statistics is also a subject than many struggle to deal with (especially when the coupled subject of probability rears its head). Most of us just aren't equipped to understand what we're being told, or to question it when the statistics are dodgy. What David Spiegelhalter does here is provide a very thorough introductory grounding in statistics without making use of mathematical formulae*. And it's remarkable.

What will probably surprise some who have some training in statistics, particularly if (like mine) it's on the old side, is that probability doesn't come into the book until page 205. Spiegelhalter argues that as probability is the hardest aspect for us to get an intuitive feel for, this makes a lot of sense - and I think he's right. That doesn't mean that he doesn't cover all …

The Best of R. A. Lafferty (SF) – R. A. Lafferty ****

Throughout my high school years (1973–76) I carefully kept a list of all the science fiction I read. I’ve just dug it out, and it contains no fewer than 1,291 entries – almost all short stories I found in various SF magazines and multi-author anthologies. Right on the first page, the sixth item is ‘Thus We Frustrate Charlemagne’ by R. A. Lafferty, and his name appears another 32 times before the end of the list. This isn’t a peculiarity of my own tastes. Short stories were much more popular in those days than they are today, and any serious SF fan would have encountered Lafferty – a prolific writer of short fiction – in the same places I did.

But times change, and this Gollancz Masterworks volume has a quote from the Guardian on the back describing Lafferty as ‘the most important science fiction writer you’ve never heard of’. Hopefully this newly assembled collection will go some way to remedying that situation. It contains 22 short stories, mostly dating from the 1960s and 70s, each w…

David Beerling - Four Way Interview

David Beerling is the Sorby Professor of Natural Sciences, and Director of the Leverhulme Centre for Climate Change Mitigation at the University of Sheffield. His book The Emerald Planet (OUP, 2007) formed the basis of a major 3-part BBC TV series ‘How to Grow a Planet’. His latest title is Making Eden.

Why science?

I come from a non-academic background. None of my family, past or present, went to university, which may explain the following. In the final year of my degree in biological sciences at the University of Wales, Cardiff (around 1986), we all participated in a field course in mid-Wales, and I experienced an epiphany. I was undertaking a small research project on the population dynamics of bullheads (Cotus gobio), a common small freshwater fish, with a charismatic distinguished professor, and Fellow of the Royal Society in London. Under his guidance, I discovered the process of learning how nature works through the application of the scientific method. It was the most exciting t…