AI in Social Services Demands Human Oversight and Bias Safeguards, Study Warns

Artificial intelligence is quietly becoming a fixture in the machinery of social welfare. Across the United States, agencies responsible for protecting children, survivors of intimate partner violence, and older adults facing abuse are turning to algorithmic tools to triage cases, flag risk, and manage mountains of data. The promise is seductive: faster decisions, fewer missed warnings, and scarce resources aimed where they matter most. But a new study from the University of Iowa argues that without deliberate safeguards, these same systems could deepen the very inequities social services exist to remedy, disproportionately burdening families already struggling with poverty, housing instability, and systemic neglect.

The research, published online on October 9 in the Journal of the Society for Social Work and Research, was conducted by Aislinn Conrad, an associate professor in the university’s School of Social Work and a former child protective services investigator. Her paper, titled “Centering survivor agency in AI for violence prevention,” lays out a framework for how agencies should govern the deployment of machine learning in high-stakes, life-altering decisions. The study arrives at a moment when the gap between technological adoption and ethical oversight in public agencies is widening, and it offers one of the most concrete sets of recommendations yet for closing that gap.

Conrad’s argument rests on a technical reality that data scientists know well but that policymakers often overlook: machine learning models are only as fair as the historical data on which they are trained. When an algorithm is fed decades of child welfare records, it does not merely learn patterns of abuse. It learns patterns of surveillance. Families living in poverty, families of color, and families already entangled with public assistance systems appear in those records far more often, not necessarily because they are more likely to harm their children, but because they are more likely to be watched, reported, and investigated in the first place. A model trained on this skewed record will reproduce the skew, and it will do so at a scale and speed no human caseworker ever could.

“When historical data shows that poor families and families of color have been disproportionately surveilled by child welfare systems, the algorithm learns that pattern and repeats it at scale, and does so invisibly,” Conrad says. “A ‘neutral’ algorithm trained on biased data is just biased automation.” The word “invisibly” carries particular weight here. Unlike a human investigator, whose reasoning can be questioned, documented, and appealed, an algorithmic risk score often arrives as an opaque number. Families flagged by such systems may never learn why they were flagged, and caseworkers themselves may not understand the model’s inner workings well enough to challenge its output.

Conrad’s perspective is shaped by years in the field. As a social worker in a rural region of south-central Kansas, an area where drug use and poverty were prevalent, she conducted child protective services investigations and helped children in foster care safely reunite with their families. It was there that she began to notice a troubling correlation between economic hardship and system involvement. “I saw the link between poverty and abuse over and over, and I began wondering, ‘Is it really that poor kids are just more likely to be abused? Or is it a surveillance issue because they come to the attention of social services more often?'” she recalls. That question, born of frontline experience, now sits at the heart of her academic work on algorithmic governance in welfare systems.

The framework she proposes rests on three pillars. The first is what she calls putting people before automation: families and survivors must have meaningful input into how AI systems are used, must understand why an algorithm has flagged them, and must be able to contest decisions that affect them. This principle, often described in the technical literature as procedural transparency and contestability, transforms the algorithm from an unaccountable judge into an advisory tool whose outputs remain subject to human review. The second pillar is systematic bias auditing. Before an agency deploys an AI system, Conrad argues, it should be evaluated for fairness across demographic groups, and those evaluations should be repeated on a regular schedule, because model performance can drift as populations and data distributions change over time.

The third pillar is perhaps the most provocative: use AI to reduce surveillance rather than expand it. Instead of deploying machine learning to detect and monitor supposedly risky families, agencies could train models to identify communities where housing instability, caregiver stress, or inadequate services are most severe, and then direct resources toward those needs. Under this vision, the same computational power that might otherwise generate watchlists becomes an engine for proactive support, connecting vulnerable people with housing, childcare, and services before crises occur. “The goal shouldn’t be more monitoring of vulnerable families,” Conrad says. “It should be whether the technology genuinely improves access to services and outcomes for families.”

The recommendations could not be more timely for Iowa. In February, the state’s Department of Health and Human Services announced a partnership with Google to develop VISION, a new information system that uses AI to modernize how child welfare agencies manage cases and data. Conrad warns that systems like VISION, however well intentioned, carry the risk of unintentional misuse. Data points such as receipt of public assistance, housing instability, or previous contacts with social service agencies could be interpreted by an algorithm as indicators of risk, subjecting families already facing economic or social hardship to heightened scrutiny. What begins as an efficiency upgrade could, without guardrails, become an amplifier of existing disparities.

The stakes extend well beyond one state. AI-assisted decision-making in child welfare, elder abuse detection, and intimate partner violence response can trigger investigations, family separations, and long-term involvement with state systems, consequences that are difficult to reverse. Researchers in the emerging field of algorithmic fairness have documented similar dynamics in predictive policing, credit scoring, and hiring, where models trained on historically biased outcomes perpetuate discrimination under a veneer of mathematical objectivity. Social services present an especially acute version of the problem because the populations involved are among the least equipped to challenge automated decisions, lacking the legal resources, technical literacy, and institutional power that contestability requires.

“AI can make us more efficient and effective, but only if we’re willing to do the hard work of developing responsive systems that keep families safe without reproducing bias,” Conrad says. “That’s why I believe AI-safeguard frameworks are so critical right now.” Her study, which she authored as sole researcher and self-funded, offers agencies a practical starting point: keep humans in the loop for consequential decisions, audit algorithms for discriminatory outcomes before and after deployment, and orient the technology toward connecting people with resources rather than watching them more closely. As public agencies across the country race to modernize with AI, the study serves as a reminder that efficiency without equity is not progress, and that the families these systems are built to protect deserve both the benefits of new technology and the preservation of human judgment, dignity, and voice.

Subject of Research: Safeguards and ethical frameworks for artificial intelligence use in social service and child welfare agencies

Article Title: Social service agencies need safeguards when using AI

Article References: Social service agencies need safeguards when using AI. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: artificial intelligence, social work, child welfare, algorithmic bias, surveillance, machine learning fairness, University of Iowa, violence prevention, public policy, survivor agency, child protective services, AI ethics