AI Reads Police Logs to Count Vulnerability, But Only With Heavy Human Correction

Large language models can be fine-tuned to read thousands of routine UK police incident logs and estimate how often frontline encounters involve mental ill health, substance misuse, alcohol dependence or homelessness, according to a study published in the journal Crime Science. But the research also delivers a stark warning: the raw output of these models is unreliable, systematically over-flags vulnerability, and becomes a defensible measurement only after repeated runs, structured human review and statistical correction.

The logs came from STORM, a computer-aided dispatch system widely used across British policing, in which control room operators and attending officers write free-text entries in real time under operational pressure. The result is messy, idiosyncratic text full of shorthand, overlapping updates and embedded personal information, precisely the kind of unstructured narrative that structured police databases cannot capture.

The team faced strict data governance constraints: the sensitive material could not be processed through external APIs or cloud infrastructure. Classification therefore ran on a locally hosted open-weight model inside a university secure research environment, on a single NVIDIA Tesla T4 graphics card with 16 GB of memory, which limited them to models of roughly 12 billion parameters or fewer. They settled on the Llama 3 family, training lightweight LoRA adapters for the 8B, 3B and 1B variants. The 8B model substantially outperformed the smaller versions, and the final configuration used 4-bit quantisation with a rank-16 adapter trained for a single epoch.

A key ingredient was knowledge distillation. Because the local models struggled to follow the classification instructions out of the box, the researchers generated 10,000 training examples by running GPT-4o-mini through their two-step classification procedure on a purposive subsample of publicly available Boston Police Department field incident reports from an earlier study. The more capable teacher model demonstrated the full reasoning pattern, and the smaller student models learned to reproduce it. During training, the instruction text itself was masked out, so the models learned to classify rather than simply parrot the prompts.

The classification itself worked in two steps. First, the model was asked to highlight up to two quotes from a narrative that might indicate a given vulnerability; if it found nothing, it responded “EVIDENCE: NONE” and processing ended, which saved computation on the majority of cases. Where evidence appeared, a second step asked the model to grade it as strong, moderate or weak. Each of the resulting 3,850 text segments was classified five times per vulnerability type at a temperature of 0.7, producing 77,000 individual classification events. The repeated runs were designed to expose uncertainty: a confident model gives the same answer every time, while one sitting near a decision boundary does not.

The variability analysis revealed a consistent pattern. The “no evidence” label dominated and was highly stable, with the vast majority of such segments receiving unanimous agreement across all five runs. Where the model did find evidence, labelling was far less consistent, with moderate labels both rare and the most variable. The four-level scheme proved too granular for the model to sustain, so the team collapsed it to three categories: no evidence, strong evidence, and uncertain. They then conducted a purposive human review of 250 text segments, sampled to cover the full label distribution, with each reviewed segment coded as agree, upgrade, downgrade or invalid.

Human review confirmed the model’s strength as a negative filter. Agreement rates for “no evidence” labels exceeded 96 percent across all four vulnerability types. Strong evidence labels were accurate for mental ill health, alcohol and homelessness, but substantially less so for substance misuse, where 13 of 34 strong labels were downgraded and five marked invalid, often because a single instance of drug use was assumed to indicate longer-term abuse, or illegal drug use was confused with alcohol consumption. Uncertain labels were the noisiest, with recurring errors such as hospital visits read as mental ill health, alcohol consumption read as drug use, and anyone away from home, whether in a hotel or hospital, flagged as homeless.

To turn these labels into prevalence estimates, the researchers aggregated segment classifications to the incident level, then used the human review data to correct for known classification errors through Monte Carlo simulation, drawing 10,000 bootstrap samples to estimate transition probabilities from model labels to corrected labels. The adjustment mattered enormously. Raw estimates suggested mental ill health indicators in 29.4 percent of incidents; after correction this fell to 22.9 percent, with a 95 percent confidence interval of 19.4 to 26.7 percent. Substance misuse showed the most dramatic correction, dropping from 12.8 percent to 5.0 percent (95 percent CI 3.7 to 6.5). Alcohol dependence fell from 11.9 percent to 8.2 percent (95 percent CI 6.3 to 10.3), and homelessness from 9.7 percent to 2.9 percent (95 percent CI 2.1 to 4.0). The confidence intervals, particularly for the less prevalent vulnerabilities, remain wide.

A comparison with the force’s own administrative qualifier flags, assigned by control room operators at the point of recording, showed substantial disagreement for all three vulnerabilities where both sources existed. For mental ill health, 121 of 358 qualifier-present incidents showed no narrative evidence according to the model, while 220 incidents with strong narrative evidence carried no qualifier. Of 34 drug-flagged incidents, only seven showed strong narrative evidence; for alcohol, just two of 82 flagged incidents did. The researchers deliberately did not treat the qualifiers as ground truth, noting that flags may draw on information outside the log, or may simply be applied inconsistently in pressurised call-handling environments.

The authors conclude that population-level estimates from LLM classification are achievable but resource-intensive, requiring repeated inference, domain-informed human review and statistical adjustment, while individual-level classifications remain too error-prone for operational decisions such as triage or prioritisation. They also acknowledge limitations: the review was conducted by a single reviewer without a formal inter-rater reliability exercise, only one pipeline configuration was tested, narratives were split at a 400-word boundary set by compute constraints, and the model was fine-tuned on US data and applied to whitelisted UK text. The reported figures, they stress, should be read as indicative orders of magnitude rather than precise measurements, and extending the approach across multiple forces and years would be needed to build a robust evidence base for resourcing and multi-agency policy decisions.

Subject of Research: Using fine-tuned large language models to estimate the prevalence of vulnerability indicators in UK police incident logs

Article Title: Using fine-tuned LLMs to identify indicators of vulnerability in UK police incident logs

Article References: Relins, S., & Birks, D. (2026). Using fine-tuned LLMs to identify indicators of vulnerability in UK police incident logs. Crime Science, 15(1), Article 42. https://doi.org/10.1186/s40163-026-00304-x

Image Credits: AI Generated

DOI: 10.1186/s40163-026-00304-x

Keywords: large language models, police incident logs, vulnerability, mental ill health, substance misuse, alcohol dependence, homelessness, knowledge distillation, LoRA fine-tuning, Monte Carlo correction, Crime Science, UK policing