Skip to main content

Weapons of Math Destruction - Cathy O'Neil ****

As a poacher-turned-gamekeeper of the big data world, Cathy O'Neil is ideally placed to take us on a voyage of horrible discovery into the world of systems making decisions based on big data that can have a negative influence on lives - what she refers to as 'Weapons of Math Destruction' or WMDs. After working as a 'quant' in a hedge fund and on big data crunching systems for startups, she has developed a horror for the misuse of the technology and sets out to show us how unfair it can be.
It's not that O'Neil is against big data per se. She points out examples where it can be useful and effective - but this requires the systems to be transparent and to be capable of learning from their mistakes. In the examples we discover, from systems that rate school teachers to those that decide whether or not to issue a payday loan, the system is opaque, secretive and based on a set of rules that aren't tested against reality and regularly updated to produce a fair outcome.
The teacher grading system is probably the most dramatically inaccurate example, where the system is trying to measure how well a teacher has performed, based on data that only has a very vague link to actual outcomes - so, for instance, O'Neil tells of a teacher who scored 6% one year and 96% the next year for doing the same job. The factors being measured are almost entirely outside the teacher's control with no linkage to performance and the interpretation of the data is simply garbage.
Other systems, such as those used to rank universities, are ruthlessly gamed by the participants, making them far more about how good an organisation is at coming up with the right answers to metrics than it is to the quality of that organisation. And all of us will come across targeted advertising and social media messages/search results prioritised according to secret algorithms which we know nothing about and that attempt to control our behaviour.
For O'Neil, the worst aspects of big data misuse are where a system - perhaps with the best intentions - ends up penalising people for being poor of being from certain ethnic backgrounds. This is often a result of an indirect piece of data - for instance the place they live might have implications on their financial state or ethnicity. She vividly portrays the way that systems dealing with everything from police presence in an area to fixing insurance premiums can produce a downward spiral of negative feedback.
Although the book is often very effective, it is heavily US-oriented, which is a shame when many of these issues are as significant, say, in Europe, as they are in the US. There is probably also not enough nuance in the author's binary good/bad opinion of systems. For example, she tells us that someone shouldn't be penalised by having to pay more for insurance because they live in a high risk neighbourhood - but doesn't think about the contrary aspect that if insurance companies don't do this, those of us who live in low risk neighbourhoods are being penalised by paying much higher premiums than we need to in order to cover our insurance. 

O'Neil makes a simplistic linkage between high risk = poor, low risk = rich - yet those of us, for instance, who live in the country are often in quite poor areas that are nonetheless low risk. For O'Neil, fairness means everyone pays the same. But is that truly fair? Here in Europe, we've had car insurance for young female drivers doubled in cost to make it the same as young males - even though the young males are far more likely to have accidents. This is fair by O'Neil's standards, because it doesn't discriminate on gender, but is not fair in the real world away from labels.
There's a lot here that we should be picking up on, and even if you don't agree with all of O'Neil's assessments, it certainly makes you think about the rights and wrongs of decisions based on automated assessment of indirect data.


Paperback 

Kindle 
Review by Brian Clegg

Comments

Popular posts from this blog

Ancestral Night (SF) - Elizabeth Bear *****

Only a couple of weeks ago, reviewing a 1960s SF book, I bemoaned the fact that science fiction novels of ideas are less common now. Although it is correctly labelled a space opera, Ancestral Night delivers ideas with aplomb.

Let's deal with the space opera aspect first. Elizabeth Bear provides some excellent adventure scenes in space, and we've the usual mix of huge spaceships and interesting aliens. Main character Haimey Dz is an engineer on a ship that salvages wrecks - but, as we gradually discover - she also has a forgotten past. A major feature of the storyline (one that seems to link to the medieval idea of the lost wisdom of the past) is ancient technology from a long-dead race with capabilities, notably manipulating spacetime mentally (Bear has yet to point out that the travel technologies used here could manipulate time as well as space), which fit well with Arthur C. Clarke's magic definition.

I particularly liked the (surely intentional) nods to the much-misse…

The Demon in the Machine - Paul Davies *****

Physicists have a habit of dabbling in biology and, perhaps surprisingly, biologists tend to be quite tolerant of it. (I find it hard to believe the reverse would be true if biologists tried to do physics.) Perhaps one reason for that tolerance is Schrödinger’s lecture series and book What is Life?, which had a huge impact on molecular biology and with a reference to which, not surprisingly, Paul Davies begins his fascinating book. 

At the heart of the The Demon in the Machine (we'll come back to that demon in a moment) is the relationship between life and information. In essence, Davies points out that if we try to reduce life to its simple physical components it is like trying to work with a computer that has no software. The equivalent of software here is information, not just in the best publicised aspect of the information stored in the DNA, but on a far broader scale, operating in networks across the organism.
This information and its processing gives life its emergent compl…

The Creativity Code - Marcus du Sautoy *****

At first glance this might just be another 'What AI is good at and not so good at' title. And in a way, it is. But, wow, what a brilliant book! Marcus du Sautoy takes us on a tour of what artificial intelligence has achieved (and possibly can in the future achieve) in a range of fields from his own of mathematics, through game playing, music, art and more.

After a little discussion of what creativity is, we start off with the now very familiar story of DeepMind's AlphaGo and its astonishing ability to take on the hugely challenging game of Go. Even though I've read about this many times before, du Sautoy, as a Go player and mathematician, gives a real feel for why this was such a triumph - and so shocking. Not surprisingly he is also wonderful on what mathematicians actually do, how computers have helped them to date and how they have the potential to do far more in the future. After all, mathematics is by far the closest science to game playing, as it has strict rule…