A hiring manager at a mid-sized firm receives 800 applications for one job. No human is going to read them all, so the company turns to software that ranks candidates and passes along the top fifty. It saves weeks. It also raises a question that has already cost some companies dearly: what if the software has learned to prefer some kinds of people over others?
How a machine learns to discriminate
Most hiring algorithms are trained on past data, such as who was hired, who was promoted, and who stayed. If a company's past decisions favoured certain groups, the model picks up the pattern and treats it as a sign of quality. Nobody has to write a biased rule. The bias arrives through the examples.
The best-known case is Amazon's experimental recruiting tool, which the company shut down after discovering that it downgraded résumés containing the word women's, as in women's chess club captain, and penalised graduates of two all-women's colleges. The system had been trained on a decade of résumés from a workforce that was mostly male. Amazon said it was never the sole basis for hiring decisions.
Proxies and hidden signals
Removing gender or race from the data doesn't fix the problem. Other details can stand in for them. A postcode can hint at ethnicity, a gap in employment can reveal parental leave, and a name or a hobby can signal background. Models are good at finding such correlations, and they find them without being told to.
Video interview tools that analyse facial expressions or tone of voice have drawn particular criticism. Researchers have questioned whether such signals say anything reliable about job performance, and some tools may score people with accents, disabilities or different communication styles lower for reasons unrelated to ability. A few vendors have dropped facial analysis after pressure from scientists and regulators.
Is human hiring any better
This is the fair counterpoint. People discriminate too, as field experiments with identical CVs have shown for decades. A well-built system can apply the same criteria to everyone and can be audited, which is harder with a human's gut feeling. Some studies find that structured, data-driven screening can widen the pool and reduce reliance on pedigree.
The risk lies in scale and opacity. One biased manager affects the people they interview. A biased model affects every applicant, quickly and without anyone noticing, and the people rejected rarely know an algorithm was involved.
Rules are arriving
Regulators have started to respond. New York City now requires employers using automated hiring tools to commission independent bias audits and notify candidates. The European Union's AI Act classifies AI used in recruitment as high-risk, which brings obligations around data quality, documentation and human oversight. In the US, the equal employment regulator has made clear that existing anti-discrimination law applies when the decision is made by software.
What good practice looks like
People who study the area suggest a few habits. Test the tool for different outcomes across groups before and after deployment. Ask the vendor how it was trained and validated. Keep a human who can override the ranking and who understands what the score does and doesn't measure. And give candidates a route to challenge a result.
The temptation is to treat an algorithm's output as neutral because it's numerical. That's the mistake to avoid. A score is only as fair as the data behind it and the people watching it.
No comments:
Post a Comment