Part I of II: Check out this story I saved on Pocket May 22, 2019

“SUBSCRIBE Long Reads Business Culture Gear Science Security Video MATT BURGESSSECURITY06.08.2020 12:02 PM Police built an AI to predict violent crime. It was seriously flawed A Home Office-funded project that used artificial intelligence to predict gun and knife crime was found to be wildly inaccurate GETTY IMAGES / WIRED A flagship artificial intelligence system designed to predict gun and knife violence before it happens had serious flaws that made it unusable, police have admitted. The error led to large drops in accuracy and the system was ultimately rejected by all of the experts reviewing it for ethical problems. The prediction system, known as Most Serious Violence (MSV), is part of the National Data Analytics Solution (NDAS) project. The Home Office has funded NDAS with at least £10 million during the last two years with the aim to create machine learning systems that can be used across England and Wales. ADVERTISING As a result of the failure of MSV, police have stopped developing the prediction system in its current form. It has never been used for policing operations and has failed to get to a stage where it could be used. However, questions have also been raised around the violence tool’s potential to be biased towards minority groups and whether it would ever be useful for policing. The MSV tool was designed to predict whether people would commit their first violent offence with a gun or knife in the next two years. People who had already come into contact with the two police forces involved in developing the tool, West Midlands Police and West Yorkshire police, were given risk scores. The higher the score, the more likely they would be to commit one of the crimes. Historic data about 2.4 million people from the West Midlands database and 1.1 million from West Yorkshire was used in the development of the system, with data being pulled from crime and custody records, intelligence reports and the Police National computer database. But as NDAS was starting to “operationalise” the system earlier this year, problems struck. Documents published by the West Midlands’ Police Ethics Committee, which is responsible for scrutinising NDAS work as well as the force’s own technical developments, reveal the system contained a coding “flaw” that made it incapable of accurately predicting violence. “A coding error was found in the definition of the training dataset which has rendered the current problem statement of MSV unviable,” a NDAS briefing published in March says. A spokesperson for NDAS says the error was a data ingestion problem that was discovered during the development process. No more specific information about the flaw has been disclosed. “It has proven unfeasible with data currently available, to identify a point of intervention before a person commits their first MSV offence with a gun or knife, with any degree of precision,” the NDAS briefing document states. ADVERTISEMENT Before the error was found, NDAS claimed its system had accuracy, or precision levels, of up to 75 per cent. Out of 100 people believed to be at high risk of committing serious violence with a gun or knife in the West Midlands, 54 of these people were predicted to carry out one of these crimes. For West Yorkshire, 74 people from 100 were predicted to commit serious violence with a gun or knife. “We now know the actual level of precision is significantly lower,” NDAS said in July. “Rare events are much harder to predict than common events,” says Melissa Hamilton, a reader in law and criminal justice at the University of Surrey, who is currently focussing on police use of risk prediction tools. Hamilton wasn’t surprised there were accuracy issues. “While we know that risk tools don’t perform the same in different jurisdictions, I’ve never seen that big of a margin of difference – particularly when you talk about the same country,” Hamilton says, adding the original estimations appeared to be too high based on other systems she had seen. As a result of the flaw, NDAS reworked its violence prediction system and its results showed the significant accuracy drop. For serious violence with a gun or knife, the accuracy dropped to between 14 and 19 per cent for West Midlands Police and nine to 18 per cent for West Yorkshire. These rates were also similar whether the person had committed serious violence before or if was going to be their first time. NDAS found its reworked system to be most accurate when all of the initial criteria it had originally defined for the system – first-time offence, weapon type and weapon use – were removed. In short, the original performance had been overstated. In the best case scenario the limited system could be accurate 25-38 per cent of the time for West Midlands Police and 36-51 per cent of the time for West Yorkshire Police. The police’s proposal to take this system forward was unanimously refused. “There is insufficient information around how this model improves the current situation around decision making in preventing serious youth violence,” the ethics committee concluded in July as it rejected the proposal for the system to be further developed. The committee, which is a voluntary group consisting of experts from different fields, said it did not understand why the revised accuracy rates were sufficient and raised concerns about how the prediction system would be used. “The Committee has expressed these concerns previously on more than one occasion without sufficient clarity being provided, and therefore as the project stands, it advises the project is discontinued,” the group said in its minutes. Committee members approached for this story said they were not authorised to speak on the record about the work. Superintendent Nick Dale, the NDAS project lead, says those behind the project do “agree that the model cannot proceed in its current form” and points out that it has so far been experimental. “We cannot say, with certainty, what the final model will look like, if indeed, we are able to create a suitable model. All our work will be scrutinised by the ethics committee and their deliberations will be published.” But multiple people who have” Police built an AI to predict violent crime. It was seriously flawed

%d bloggers like this: