Algorithms are incredible aids for making data-driven, efficient decisions. And as more industries uncover their predictive power, companies are increasingly turning to algorithms to make objective and comprehensive choices. However, while we often rely on technology to avoid inherent human biases, there is a dark side to algorithm-based decisions: the potential for homogenous data sets to produce biased algorithms.
Many of the people and companies employing algorithms hope that their use of technology in replacement of humans results in reduced unconscious bias. While it would be great if it were that simple, this mindset is often a case of “mat washing:” our tendency to attribute objectivity to technology.
Consider this example: there are more men named John than there are women named anything as S&P 5000 CEOs. If we built a predictive model of CEO performance, it’s possible that John would be a better predictor of being a successful CEO than the female gender. Is this truly reflective of a person’s potential to be a CEO or just noise from the bias in the training set? In this example, it is obvious that being named “John” is simply noise. However, when presented with similar evidence in real-world situations, it is not always as easy to spot the biases.
We’ve started to see this in less consequential but still troubling situations. Tay, Microsoft’s Twitter bot, turned misogynistic and antisemitic. If you search for images of gorillas on Google, you may get shown images of black men. If you’re a non-native English speaking student and plagiarize part of an essay, Turnitin (a plagiarism detection software) will be more likely to detect your cheating than it will native speakers.
To avoid situations like this, I strongly believe that any algorithm making decisions about opportunities that affect people’s lives requires a methodological design and testing process to ensure that it is truly bias-free. Because when you employ AI itself to remove bias from an algorithm, the results can be extraordinary.
