Supermarket AI mistakes Maori shopper for thief

• Published date: 18 April 2024
• Publication title: Otago Daily Times (New Zealand)

The woman was shopping at New World in Rotorua — one of 25 Foodstuffs’ North Island supermarkets trialling the technology — when she was approached by two staff members and asked to leave

The technology scans faces and compares images with those on the store’s databases of known offenders or suspects.

Maori AI and data ethicist Karaitiana Taiuru said he was not surprised to see Maori singled out in the trial and believed other Maori had been affected but had not yet come forward.

The systems Foodstuffs was using were trained on an international dataset of people, not on people in a New Zealand context, he said.

‘‘We know from international data and international research that the systems are based on European-looking men, so it’s only in the last few years that the system has been modified to consider people of colour, women, men with beards etc.

‘‘Anyone who is Maori, Pasifika, any person of colour is not going to be recognised by the system. That’s why it’s important that we have humans double-checking the system, which in this case failed.’’

The woman was trespassed from the Rotorua supermarket despite offering three forms of photo identification to staff.

Mr Taiuru said people manually checking positive matches from facial recognition technologies often still relied on AI and not on their own common sense.

‘‘It’s a phenomenon that’s identified in America where the staff rely on the AI more than their own personal judgement. So regardless of how obvious a mismatch is, humans still rely on the AI to be correct,’’ Mr Taiuru said.

This bias could be reduced with appropriate staff training — which would have to be customised to the New Zealand population, he said.

As far as he was aware, there was no out-of-the box training...