San Francisco says it will start using an artificial intelligence tool to reduce possible racial bias among prosecutors reviewing police reports, a “first-in-the-nation” use of a technology whose applications have been criticized for compounding bias.
On Wednesday, District Attorney George Gascón announced that the city on July 1 would begin to use a “bias mitigation tool” that automatically redacts anything on the police report that might be suggestive of race, from hair color to zip code. Information about the police officer, such as badge number, will also be hidden. Currently, the district attorney’s office manually removes the first few pages of the report, but if any race details are in the narrative—the section where the police officer describes the crime—prosecutors can see them.
“This technology will reduce the threat that implicit bias poses to the purity of decisions which have serious ramifications for the accused, and that will help make our system of justice more fair and just,” Gascón said.
Between 2008 and 2014, African Americans accounted for 43% of people booked into jail despite only making up 6% of San Francisco’s population, according to a 2017 report on the DA’s office. The report found “little evidence” of overt racial bias by the prosecutors, implicit bias is still possible. That’s what the AI will try to eliminate.
Artificial intelligence’s role in amplifying racial bias has come under scrutiny both in and outside of the criminal justice system. An August 2018 MIT study found that Amazon’s facial recognition software Rekognition performed significantly worse in identifying darker-skinned individuals and women compared to white men. The controversial AI software isn’t going away— Amazon is in discussions to sell it to the government—but Amazon did take a step back Monday when Amazon Web Services CEO Andy Jassy said the company welcomed federal regulations to facial recognition.
One of the creators of San Francisco’s tool, Alex Chohlas-Wood, previously helped create Patternizr, a machine-learning tool that the NYPD began using in March to more efficiently solve crimes. Patternizr sorts through an existing database of crime files to spot patterns of criminal activity. Some civil rights advocates have cautioned against possible racial biases in the tool, though Patternizr was designed to avoid such biases.
San Francisco’s new tool, however, is quite different from Patternizr, said Chohlas-Wood, who is deputy director of the Stanford Computational Policy Lab.
“This is looking forward in order to make decisions about existing cases, rather than leaning on a bank or a data set of prior cases which may reflect racial disparities,” DA spokesman Max Szabo told Forbes.
About 80% of cases that go through the district attorney’s office will use this tool, Szabo said. Vertical cases—like homicide, domestic violence, and sexual assault—will not, at least for the time being.
The tool will be used by attorneys to review an incident often hours after it occurred. Even then, it won’t be used to make any final decisions about charging a person. Attorneys will write down an initial, unofficial decision after using the tool. Then, they’ll have access to race information in the unredacted report and body camera footage to make an official charging decision. If their final decision is different from their initial one, prosecutors are required to explain why.
The Stanford Computational Policy Lab developed the tool and an accompanying lightweight web platform for the DA’s office to review the reports. The technology involves a combination of machine-learning techniques and more standard computational techniques, Chohlas-Wood said.
The code will be released for free for other cities to implement. Prosecutors will also be able to provide input on the platform about whether the tool is redacting too much or not enough information about race.