Construction safety AI that knows what it doesn’t see: a Sentry–Judge hybrid for PPE absence detection

Construction sites are among the most dangerous workplaces in the world, and one of the most persistent, preventable causes of injury is the failure to wear personal protective equipment such as hard hats and high-visibility vests. Computer vision systems have become central to automating safety monitoring on site, and modern deep learning detectors can spot workers, machinery, and protective gear in video with impressive accuracy. Yet these systems share a striking blind spot: they are excellent at recognizing what is present in a frame and surprisingly poor at confirming what is missing. A hard hat resting on a hook is detected almost instantly, but deciding that a particular worker is not wearing one is a fundamentally harder inferential problem.

The reason for this asymmetry lies in the nature of visual evidence. Evidence of presence is usually local and unambiguous: the detector simply finds the object. Evidence of absence, by contrast, is circumstantial. The system must reason about what should be present on a specific person’s body and conclude that it is not there. Detectors trained to find objects therefore tend to make two characteristic mistakes: they mistake background clutter for protective equipment, or they flag workers whose gear is merely occluded, blurred, or too small in the frame to be resolved reliably. In a safety-critical setting, a monitor that is either too conservative or too alarm-prone quickly loses the trust of the supervisors who are supposed to act on its alerts.

Recent construction-specific research has attacked the problem mainly by strengthening a single detector, training explicit classes such as “non-helmet” or “non-vest,” adapting models to scarce on-site violation data through domain adaptation, and redesigning networks to catch the small, multi-scale cues that characterize missing equipment. These efforts have raised per-frame detection accuracy substantially, but the remaining errors concentrate precisely in the ambiguous cases: partial occlusion, unusual posture, distant or overlapping workers. A new study published in Frontiers in Built Environment by Bhargav Vaghani of Arizona State University takes a different route. Rather than asking one detector to solve absence detection alone, it builds a two-tier system, a “Sentry–Judge” hybrid, that decouples the task into rapid screening and deliberative verification.

The Sentry is a YOLOv11m detector, a lightweight network that processes incoming video frames with low latency and attaches an uncertainty score to every worker it provisionally flags as lacking PPE. It operates on the principle that most frames are unambiguous and can be classified with high confidence. The Judge is a SAM 3 segmentation model, a far more computationally intensive system that is invoked only when the Sentry’s uncertainty exceeds a predefined threshold. Given a candidate region, the Judge segments the worker’s body and any protective equipment within it, producing precise masks rather than rough boxes. It then computes a coverage ratio between the equipment area and the body area, confirming a violation only if that ratio falls below a threshold. Because the check is performed at the level of pixel-accurate masks, it resolves many of the ambiguities that cause false alarms at the box level, such as shadows, occlusion by machinery, and awkward poses.

The gating logic is mathematically transparent. The uncertainty score is a monotone function of the worker detection confidence alone, so routing a case to the Judge is exactly equivalent to applying a cut-off on that confidence. A worker detected with low confidence is typically blurred, partially visible, or truncated at the frame edge, which are precisely the conditions under which its PPE is also likely to be missed. The authors show that the score is bounded and saturates smoothly, and that the two system thresholds jointly determine a single confidence cut-off selected on validation data. Under typical operating conditions the Judge is called for only about one in every five candidate detections, which is what keeps the pipeline fast.

The performance gains on real construction-site footage are substantial. The hybrid reduces the false alarm rate from 18.6 percent for a standalone YOLO model to 4.1 percent, an absolute reduction of 14.5 percentage points and a relative reduction of 78 percent, while retaining a detection rate of 89.2 percent, close to the 91.4 percent of the unverified baseline. The full pipeline sustains 29.8 frames per second on an NVIDIA RTX 4090, barely slower than the 31.2 frames per second of the detector alone. If the segmentation model were run on every frame, throughput would collapse to roughly 2 frames per second. An ablation study confirms that each component matters: removing the Judge returns the false alarm rate to 18.6 percent, removing the gating mechanism restores accuracy only marginally while destroying speed, and replacing the uncertainty-based gate with a fixed threshold slightly worsens both error rates.

Perhaps the most technically interesting contribution is a controlled simulation of the geometric rule that links protective equipment to workers. The standard approach uses intersection over union, a measure designed to compare two boxes describing the same object, not a small object with the larger body that carries it. The simulation of 4,000 synthetic worker–hard hat pairs shows why this fails: for a typical hard hat worn correctly on the head, the median overlap with the worker box is only about 0.037, so any sensible threshold rejects it. At its best operating point the IoU rule accepts every hat held in the hand and most hats lying on the ground near a worker’s feet, achieving an association precision of just 66.6 percent. Checking containment within the anatomically relevant head region instead raises precision to 97.3 percent, rejecting handheld and ground-level hats almost entirely. The lesson is that PPE must be verified on the body part where it is required, not merely somewhere near the worker.

The system also closes the loop between detection and action. A reporting module automatically converts confirmed violations into formatted PDF documents following OSHA reporting conventions, extracting timestamps, frame numbers, detection boxes, and segmentation masks into a structured report with an annotated snapshot and a summary of the worker’s compliance history. Report generation is fully automated and requires no human intervention beyond final verification. The authors are careful to note that the decision producing a report is made within the per-frame processing cycle, averaging 33.6 milliseconds, and that for deployments requiring immediate intervention they recommend decoupling lightweight alerts from asynchronous PDF documentation so that report generation never delays detection.

The study is candid about its limitations. The Judge requires a GPU capable of running SAM 3, the method depends on accurate worker detection and may degrade in dense crowds or severe occlusion, the uncertainty score currently cannot react to independent evidence such as a weak but correctly placed PPE detection, and the thresholds were set empirically and may need adjustment for other sites or lighting conditions. The system has also not yet been compared directly with human observers. To address that gap, the authors propose a controlled protocol in which the same periods of held-out site video would be monitored by a supervisor on walk-through, a trained observer watching the live feed, the system alone, and the system with supervisor-confirmed alerts, reporting violation recall, false alarm rates, detection latency, and coverage in worker-minutes per supervisor-hour.

The central finding resonates far beyond construction safety. Effective monitoring AI, the study argues, does not require applying the most complex model to every frame; it requires knowing when closer scrutiny is warranted and when to rely on a rapid initial judgment. That principle of spending computation only where doubt exists, familiar from decades of cascade architectures in computer vision, here delivers a rare combination: a fourfold reduction in false alarms, near-real-time throughput, and documentation that supervisors can actually use. The authors plan to integrate body-region association, a calibrated multi-term uncertainty score, an event-driven alerting path, and tests of the framework’s transferability to manufacturing and logistics, where confirming the absence of required safety equipment is equally important. If those plans succeed, the quiet revolution in site safety may come not from bigger models, but from smarter ones that know when to look twice.

Subject of Research: A hybrid Sentry–Judge AI architecture for detecting absence of personal protective equipment on construction sites

Article Title: Construction safety AI that knows what it doesn’t see: a Sentry–Judge hybrid for PPE absence detection

Article References: Vaghani, B. (2026). Construction safety AI that knows what it doesn’t see: a Sentry–Judge hybrid for PPE absence detection. Frontiers in Built Environment, 12, Article 1976288. https://doi.org/10.3389/fbuil.2026.1976288

Image Credits: AI Generated

DOI: 10.3389/fbuil.2026.1976288

Keywords: construction safety, computer vision, PPE detection, YOLOv11m, SAM 3, absence detection, false alarm reduction, uncertainty gating, segmentation, real-time monitoring, OSHA reporting, deep learning