영국 경찰의 범죄 예측 시스템, 신뢰성 논란

Comment Loader Save Story Save this story Comment Loader Save Story Save this story The Think Family Database holds records on close to half a million people who live in the city of Bristol, England. For many years, few of them knew anything about it. Launched in 2016 by the Bristol City Council and the regional Avon and Somerset Police, the database has stored all manner of sensitive information—police intelligence reports, housing status, mental health records, teenage pregnancies, enrollment in parenting courses, free school meals. On top of this sensitive data, officials built machine-learning models to assign scores to thousands of adults and children. They hoped to build what they called a “picture of threat, harm, and risk” in the region. At an event in early 2022 to help officials tackle child exploitation crimes, one police data scientist described part of the approach this way: “I essentially dump all that data in a big bucket and stir it with a data-science spatula, and we come out with a lovely risk score for everybody.” This risk scoring inside the Think Family Database was just one part of Avon and Somerset Police’s sprawling predictive analytics program. Among at least 23 separate models the force created were algorithms to identify the risk that people would commit burglary, fail to turn up in court, go missing, or become a victim of domestic abuse. One senior officer described creating a “league table” of the area’s most dangerous criminals—an apparent reference to the Offender Management App, which was designed to hold data on around 300,000 people in the region. How the police have developed and used their predictive tools hasn’t always been clear to the public. John Pegram, the leader of a local police accountability group in Bristol, says he didn’t hear about the Offender Management App until 2023, years after it had been created. When he did learn about it, he began to suspect he might be included. “I think I knew I was on the app,” Pegram says. In early 2024, Pegram filed a request to find out how the police were using his data. The police refused to say. Months later, after Pegram had hired solicitors to work on his case, the police confirmed he was on the app but declined to elaborate further. Like others across Bristol, the UK, and, increasingly, around the world, Pegram didn’t know whether he had been scored by an algorithm, what that score might be, or how it could affect his interactions with the authorities. WIRED, working in partnership with the nonprofit newsroom Liberty Investigates , plus the Bristol Cable and Lighthouse Reports, obtained hundreds of pages of documentation from public records requests to build the most comprehensive picture to date of Avon and Somerset’s regional experiment with data collection and predictive analytics. (Liberty, the parent organization of Liberty Investigates, had some early involvement in a potential legal challenge to the program and continues to support Pegram’s litigation.) The investigation reveals that at least two of these risk-scoring models were quietly abandoned after Bristol City Council staff deemed they could no longer trust them. Previously unreported documents show government inspectors and independent reviewers highlighting a startling lack of transparency about some elements of the program and warning that the systems could undermine public trust. Police data disclosed to WIRED—comprising more than 36,000 model performance scores—appear in some cases to show “genuinely poor predictive performance,” according to an independent analyst who reviewed the data for WIRED. These findings come as the UK appears poised to embrace predictive analytics and artificial intelligence across the criminal justice system. A familiar face is helping lead the charge: the former chief constable of Avon and Somerset, Andy Marsh, who now heads the national standard-setting body for forces across England and Wales. As CEO of the College of Policing, Marsh has said that effective AI should be “injected like heroin” to speed up British police work. In a recent interview , Marsh said his organization was examining around 100 currently deployed AI tools, including for predictive policing. “Our job is to test the ones that work properly, test them with rigorous evaluation, and then spread them like wildfire through policing.” In 2014, Avon and Somerset Police was under pressure on multiple fronts. The force, like others across the UK, had seen its budgets slashed. Its chief constable had been suspended . An official report had highlighted its failure to stick to procedures to protect some victims of domestic abuse. After that report was published, the force’s head of performance said , “We believe predictive analytics is the solution.” Gary Davies, a former police chief superintendent who had moved to a role at the Bristol City Council two years earlier, was thinking along similar lines. Davies led a team at the council supporting children and families. When families were in crisis, “it was blatantly obvious,” he says. It was much harder to spot those who were at the top of a downward spiral. Davies believed the answer lay in data. A child’s school might hold a record of increasing absences, while the police might know if the child had recently witnessed domestic abuse for the first time. On their own, these might not be enough to trigger an intervention from social services. But together? “If you could see the whole picture, you would realize that the trajectory they were on was going in the wrong direction,” he says. Starting in 2015, a small group of Bristol City Council and Avon and Somerset Police staff moved into one of the city’s police stations to work on a solution to that problem together . The Insight Bristol team, headed by Davies, started pulling together data from across the public sector to provide frontline workers with all the information they might need about children and families. The Insight Bristol team didn’t seek residents’ consent to use their data in the Think Family Database. Instead, Davies explains, the team relied on “legal gateways”—a term that describes when data sharing is deemed necessary to meet an agency’s legal obligations, such as the need to protect children. “If you were to give the impression that people had consent, then it creates a false illusion, because, actually, as [a] local authority or police or whoever, we have to keep those records.” Initially, residents could not opt out of the database; later, the council included an opt-out option in its tax letters to residents. Davies, who recently retired, believes the project did help protect children. “It improved the understanding of risk and vulnerability for children and families,” he says. “It provided that information in a far more efficient way.” When it came to communicating that to the public, Davies says, “it was fairly difficult to get any enthusiasm or interest from groups of people.” Those who did engage said they understood the need to use personal data, he recalls, summarizing the feedback as “We don't mind you using it to support us, but we don't want you to use it against us.” While the Insight Bristol team was busy creating the Think Family Database, Avon and Somerset Police had begun exploring the potential of predictive analytics. In March 2016, the force’s ethics committee met to consider how the work should proceed. Members advised that “careful consideration had to be given to what data is used” and “the variables that are used in the process,” concluding: “The use of the system must be treated with some caution and it must be ensured that there is no bias.” The committee advised that, if the force’s predictive analytics work was to proceed, “the public must be informed as to why and how you are carrying out such processes.” Once work to compile the Think Family Database was completed, a police data scientist spearheaded the development of predictive risk models for the