From [HERE] and [MORE] “You good?” a man asked two narcotics detectives late in the summer of 2015.

The detectives had just finished an undercover drug deal in Brentwood, a predominately black neighborhood in Jacksonville, Florida, that is among the poorest in the country, when the man unexpectedly approached them. One of the detectives responded that he was looking for $50 worth of “hard”– slang for crack cocaine. The man disappeared into a nearby apartment and came back out to fulfill the detective’s request, swapping the drugs for money.

“You see me around, my name is Midnight,” the dealer said as he left.

Before Midnight departed, one of the detectives was able to take several photos of him, discreetly snapping pictures with his phone held to his ear as though he were taking a call.

Two weeks later, police wanted to make the arrest. The only information they had about the dealer were the smartphone pictures, the address where the exchange had taken place, and the nickname Midnight. Stumped, the Jacksonville sheriff’s office turned to a new tool to help them track down the dealer: facial recognition software.

The technology helped them pin down a suspect named Willie Lynch. Lynch, who has been described by close observers of the case such as Georgetown University researcher Clare Garvie as a “highly intelligent, highly motivated individual” despite only having graduated high school – he even filed his own case motions, which could be mistaken for ones written by an actual lawyer – was eventually convicted and sentenced to eight years in prison. He is now appealing his conviction.

Whether or not Willie Lynch is “Midnight” remains to be seen. But many experts see the facial recognition technology used against him as flawed, especially against black individuals. Moreover, the way the Jacksonville sheriff’s office used the technology – as the basis for identifying and arresting Lynch, not as one component of a case supported by firmer evidence – makes his conviction even more questionable.

The methods used to convict Lynch weren’t made clear during his court case. The Jacksonville sheriff’s office initially didn’t even disclose that they had used facial recognition software. Instead, they claimed to have used a mugshot database to identify Lynch on the basis of a single photo that the detectives had taken the night of the exchange.

An ‘imperfect biometric’

The lack of answers the Jacksonville sheriff’s office have provided in Lynch’s case is representative of the problems that facial recognition poses across the country. “It’s considered an imperfect biometric,” said Garvie, who in 2016 created a study on facial recognition software, published by the Center on Privacy and Technology at Georgetown Law, called The Perpetual Line-Up. “There’s no consensus in the scientific community that it provides a positive identification of somebody.”

The software, which has taken an expanding role among law enforcement agencies in the US over the last several years, has been mired in controversy because of its effect on people of color. Experts fear that the new technology may actually be hurting the communities the police claims they are trying to protect.

“If you’re black, you’re more likely to be subjected to this technology and the technology is more likely to be wrong,” House oversight committee ranking member Elijah Cummings said in a congressional hearing on law enforcement’s use of facial recognition software in March 2017. “That’s a hell of a combination.”

Cummings was referring to studies such as Garvie’s. This report found that black individuals, as with so many aspects of the justice system, were the most likely to be scrutinized by facial recognition software in cases. It also suggested that software was most likely to be incorrect when used on black individuals – a finding corroborated by the FBI’s own research. This combination, which is making Lynch’s and other black Americans’ lives excruciatingly difficult, is born from another race issue that has become a subject of national discourse: the lack of diversity in the technology sector.

Racialized code

Experts such as Joy Buolamwini, a researcher at the MIT Media Lab, think that facial recognition software has problems recognizing black faces because its algorithms are usually written by white engineers who dominate the technology sector. These engineers build on pre-existing code libraries, typically written by other white engineers.

As the coder constructs the algorithms, they focus on facial features that may be more visible in one race, but not another. These considerations can stem from previous research on facial recognition techniques and practices, which may have its own biases, or the engineer’s own experiences and understanding. The code that results is geared to focus on white faces, and mostly tested on white subjects.

And even though the software is built to get smarter and more accurate with machine learning techniques, the training data sets it uses are often composed of white faces. The code “learns” by looking at more white people – which doesn’t help it improve with a diverse array of races.

Technology spaces aren’t exclusively white, however. Asians and south Asians tend to be well represented. But this may not widen the pool of diversity enough to fix the problem. Research in the field certainly suggests that the status quo simply isn’t working for all people of color – especially for groups that remain underrepresented in technology. According to a 2011 study by the National Institute of Standards and Technologies (Nist), facial recognition software is actually more accurate on Asian faces when it’s created by firms in Asian countries, suggesting that who makes the software strongly affects how it works.

In a TEDx lecture, Buolamwini, who is black, recalled several moments throughout her career when facial recognition software didn’t notice her. “The demo worked on everybody until it got to me, and you can probably guess it. It couldn’t detect my face,” she said.

Unregulated algorithms

Even as the use of facial recognition software increases in law enforcement agencies across the country, the deeper analysis that experts are demanding isn’t happening.

Law enforcement agencies often don’t review their software to check for baked-in racial bias – and there aren’t laws or regulations forcing them to. In some cases, like Lynch’s, law enforcement agencies are even obscuring the fact that they’re using such software.

Garvie said she is confident that police are using facial recognition software more than they let on, which she referred to as “evidence laundering”. This is problematic because it obscures just how much of a role facial recognition software plays in law enforcement. Both legal advocates and facial recognition software companies themselves say that the technology should only supply a portion of the case – not evidence that can lead to an arrest.

“Upon review, all facial recognition matches should be treated no differently than someone calling in a possible lead from a dedicated tip line,” writes Roger Rodriguez, an employee at facial recognition vendor Vigilant Solutions, in a post defending the software. “The onus still falls on the investigator in an agency to independently establish probable cause to effect an arrest,” he continues – probable cause that “must be met by other investigatory means”.

Even if facial recognition software is used correctly, however, the technology has significant underlying flaws. The firms creating the software are not held to specific requirements for racial bias, and in many cases, they don’t even test for them.

One company I spoke to, CyberExtruder, a facial recognition technology company that markets itself to law enforcement, also said that they had not performed testing or research on bias in their software. Vigilant Solutions declined to say whether or not they tested for it. CyberExtruder did note that certain skin colors are simply harder for the software to handle given current limitations of the technology. “Just as individuals with very dark skin are hard to identify with high significance via facial recognition, individuals with very pale skin are the same,” said Blake Senftner, a senior software engineer at CyberExtruder.

As for Lynch, his case is currently playing out in the Florida courts. The clock also can’t be turned back for others like him, who may have been unfairly tried as a result of less-than-perfect software without transparent standards for its use.

Facial recognition software raises many questions that need clear answers. Obtaining those answers will take more than commissioning studies, as vital as they are. It’s also essential that laws catch up with the technology, in order to provide people like Lynch with the opportunity to know the tools that are being used against them. Most importantly, we need to take a closer look at who’s making these algorithms, and how they’re doing it.