The Department of Homeland Security (DHS) has determined that facial recognition systems can work well, although some – but not all – still show performance demographic differentials, according to a report DHS released on Feb 23.
The 2022 Biometric Technology Rally challenged industry to produce face recognition systems capable of autonomously identifying small groups of people passing through the system. In addition, the exercise required that systems must be secure, and maintain the privacy of people who opted out of the process.
A total of 575 volunteers diverse in race, gender, age, and skin tone passed through each acquisition system – four acquisition systems and ten matching systems, for a total of 40 combinations tested – in a group of two or four people each, while eight to ten others passed around the system via adjacent bypass lanes.
“System operators were tasked with capturing one best photo per person passing through and matching system operators had to identify the individual in each submitted photo?,” the report explains.
The report states that all 40 system combinations met a three-second timing mark and many of the systems were able to refrain from getting any images of bystanders or individuals who opted out. The report assessed that of the 40 combinations, 37 captured less than one percent of individuals not in the opt-in space.
The results show that “if you specify it, industry can come up with solutions to help address privacy and not capture faces not meant to be photographed efficiency, satisfaction, effectiveness, equitability, and privacy of four biometric face acquisition systems and ten biometric face matching systems,” said Arun Vemury, director of DHS’s Biometric and Identity Technology Center within the DHS Science and Technology Directorate.
The results of the facial recognition testing showed drastic improvement in algorithms and utilization possibility.
However, in some systems, there were some performance differences based on demographic –such as race, gender, and skin tone. Some systems struggle at getting the right level of exposure in photos of people with darker skin and to a lesser extent, those with extremely light skin tones.
Nine system combinations met the 95 percent identification mark for all skin tones, compared to around 26 that hit the mark for medium skin tones. However, the errors that did come up generally came from the cameras being tested, not the facial recognition algorithms.
Despite the technological challenges, DHS believes that facial recognition can work, and the technology has potential use cases in sea, land, and airport locations.
“There are technologies that will work well and work equitably, that hit that 95% performance threshold. However, as you saw, we also see some technologies that fall below that,” which “highlights the importance of testing and evaluating for specific use cases with diverse volunteers to make sure that we are picking the right technologies for specific applications,” said Vemury.
