Skip to main content

Weapons of Math Destruction - Cathy O'Neil ****

As a poacher-turned-gamekeeper of the big data world, Cathy O'Neil is ideally placed to take us on a voyage of horrible discovery into the world of systems making decisions based on big data that can have a negative influence on lives - what she refers to as 'Weapons of Math Destruction' or WMDs. After working as a 'quant' in a hedge fund and on big data crunching systems for startups, she has developed a horror for the misuse of the technology and sets out to show us how unfair it can be.
It's not that O'Neil is against big data per se. She points out examples where it can be useful and effective - but this requires the systems to be transparent and to be capable of learning from their mistakes. In the examples we discover, from systems that rate school teachers to those that decide whether or not to issue a payday loan, the system is opaque, secretive and based on a set of rules that aren't tested against reality and regularly updated to produce a fair outcome.
The teacher grading system is probably the most dramatically inaccurate example, where the system is trying to measure how well a teacher has performed, based on data that only has a very vague link to actual outcomes - so, for instance, O'Neil tells of a teacher who scored 6% one year and 96% the next year for doing the same job. The factors being measured are almost entirely outside the teacher's control with no linkage to performance and the interpretation of the data is simply garbage.
Other systems, such as those used to rank universities, are ruthlessly gamed by the participants, making them far more about how good an organisation is at coming up with the right answers to metrics than it is to the quality of that organisation. And all of us will come across targeted advertising and social media messages/search results prioritised according to secret algorithms which we know nothing about and that attempt to control our behaviour.
For O'Neil, the worst aspects of big data misuse are where a system - perhaps with the best intentions - ends up penalising people for being poor of being from certain ethnic backgrounds. This is often a result of an indirect piece of data - for instance the place they live might have implications on their financial state or ethnicity. She vividly portrays the way that systems dealing with everything from police presence in an area to fixing insurance premiums can produce a downward spiral of negative feedback.
Although the book is often very effective, it is heavily US-oriented, which is a shame when many of these issues are as significant, say, in Europe, as they are in the US. There is probably also not enough nuance in the author's binary good/bad opinion of systems. For example, she tells us that someone shouldn't be penalised by having to pay more for insurance because they live in a high risk neighbourhood - but doesn't think about the contrary aspect that if insurance companies don't do this, those of us who live in low risk neighbourhoods are being penalised by paying much higher premiums than we need to in order to cover our insurance. 

O'Neil makes a simplistic linkage between high risk = poor, low risk = rich - yet those of us, for instance, who live in the country are often in quite poor areas that are nonetheless low risk. For O'Neil, fairness means everyone pays the same. But is that truly fair? Here in Europe, we've had car insurance for young female drivers doubled in cost to make it the same as young males - even though the young males are far more likely to have accidents. This is fair by O'Neil's standards, because it doesn't discriminate on gender, but is not fair in the real world away from labels.
There's a lot here that we should be picking up on, and even if you don't agree with all of O'Neil's assessments, it certainly makes you think about the rights and wrongs of decisions based on automated assessment of indirect data.


Paperback 

Kindle 
Using these links earns us commission at no cost to you
Review by Brian Clegg

Comments

Popular posts from this blog

The God Game (SF) - Danny Tobey *****

Wow. I'm not sure I've ever read a book that was quite such an adrenaline rush - certainly it has been a long time since I've read a science fiction title which has kept me wanting to get back to it and read more so fiercely. 

In some ways, what we have here is a cyber-SF equivalent of Stephen King's It. A bunch of misfit American high school students face a remarkably powerful evil adversary - though in this case, at the beginning, their foe appears to be able to transform their worlds for the better.

Rather than a supernatural evil, the students take on a rogue AI computer game that thinks it is a god - and has the powers to back its belief. Playing the game is a mix of a virtual reality adventure like Pokemon Go and a real world treasure hunt. Players can get rewards for carrying out tasks - delivering a parcel, for example, which can be used to buy favours, abilities in the game and real objects. But once you are in the game, it doesn't want to let you go and is …

Uncertainty - Kostas Kampourakis and Kevin McCain ***

This is intended as a follow-on to Stuart Firestein's two books, the excellent Ignorance and its sequel, Failure, which cut through some of the myths about the nature of science and how it's not so much about facts as about what we don't know and how we search for explanations. The authors of Uncertainty do pretty much what they set out to do in explaining the significance of uncertainty and why it can make it difficult to present scientific findings to the public, who expect black-and-white facts, not grey probabilities, which can seem to some like dithering.

However, I didn't get on awfully well with the book. A minor issue was the size - it was just too physically small to hold comfortably, which was irritating. More significantly, it felt like a magazine article that was inflated to make a book. There really was only one essential point made over and over again, with a handful of repeated examples. I want something more from a book - more context and depth - that …

Where are the chemistry popular science books?

by Brian Clegg
There has never been more emphasis on the importance of public engagement. We need both to encourage a deeper interest in science and to counter anti-scientific views that seem to go hand-in-hand with some types of politics. Getting the public interested in science both helps recruit new scientists of the future and spreads an understanding of why an area of scientific research deserves funding. Yet it is possible that chemistry lags behind the other sciences in outreach. As a science writer, and editor of this website, I believe that chemistry is under-represented in popular science. I'd like to establish if this is the case, if so why it is happening - and what can be done to change things. 


An easy straw poll is provided by the topic tags on the site. At the time of writing, there are 22 books under 'chemistry' as opposed to 97 maths, 126 biology and 182 physics. The distribution is inevitably influenced by editorial bias - but as the editor, I can confirm …