New open-source toolkit reveals where vision models look, and flags when they are guessing

Deep neural networks that classify images and video can be startlingly accurate while quietly learning the wrong things. A model may score highly not because it recognizes the object in front of it, but because it latched onto a background texture, a watermark, or some other shortcut buried in its training data. Researchers at Gyeongsang National University have now released SSAT, an open-source toolkit designed to expose exactly this kind of hidden spatial dependence, and to do so in a way that other scientists can reproduce rather than take on faith. The software, published in the journal SoftwareX, is available under an MIT license and runs on ordinary Python installations with an optional GPU.

The core idea behind SSAT is deceptively simple: occlusion-based analysis. Take an input image, cover one region of it, run the model again, and measure how much the prediction changes. If blanking out a patch causes the model’s confidence to collapse, the model was probably relying on whatever was in that patch. This perturbation primitive has existed since the early days of convolutional network visualization, and libraries such as Captum already provide the underlying feature-removal operations. What has been missing, the authors argue, is a standardized workflow that scales this experiment to entire datasets while guarding against the many ways such an analysis can mislead.

A large change in the model’s output, it turns out, does not by itself prove meaningful dependence. Occluding an equally large area at an arbitrary location may produce a comparable shift in confidence, and the direction of the change can even reverse depending on how the occluded region is filled in. SSAT therefore pairs every region-level intervention with matched random controls of the same area, repeats the perturbation under multiple fill strategies and random seeds, and grades the resulting evidence rather than reporting raw magnitudes alone. Each comparison between a sample and a target region, called an anchor, receives a reliability grade of HIGH, MODERATE, LOW, or UNRELIABLE based on four checks: whether the degradation has the same sign under every operator, whether it exceeds the control mean by more than two control standard deviations, whether at least two operators reproduce the dominant sign, and whether the 95 percent bootstrap interval of the region mean excludes zero.

The toolkit is organized as a five-stage pipeline driven by a single YAML configuration. An estimate stage performs a bounded preflight of configuration validation, workload and storage estimation, and sanity checks. A run stage combines a data source, a model adapter, a region generator, and a perturbation operator to perform clean and perturbed inference, writing full model outputs to a resumable Parquet dump with a manifest that records configuration hashes, checkpoint hashes, adapter identity, and code version. A metrics stage normalizes task responses, an analyze stage performs matched-control and stability analysis, and a report stage produces HTML, CSV, and JSON outputs including dataset-wide heatmaps and per-sample drill-downs. Because the raw observations are preserved before any metrics are derived, researchers can regrade thresholds, recompute intervals, or apply new aggregations to stored interventions without running any new inference.

To validate the workflow, the authors first turned to a synthetic dataset in which a shortcut patch was inserted at a known location, together with a negative-control model that does not exploit the shortcut. Under pre-registered questions, SSAT recovered the patch as the most sensitive region of the shortcut model and ranked it last in the negative-control model. The patch effect was 175.65 times the same-area sibling-region baseline, far above the 3.0 times threshold, and the conclusion reproduced across all five fill strategies. Independently of the region audit, removing the patch caused a 95.75 percentage-point accuracy-drop margin between the two test conditions, confirming that the shortcut dependence found by the spatial analysis corresponds to an actual generalization gap.

The second case study audited 10,000 samples from the ImageNet-1k validation set across four pretrained classifiers: MobileNetV2 at two width multipliers, ConvNeXt-T, and the Vision Transformer DeiT-S. Each model was audited under its official preprocessing and a crop-free variant, giving eight runs that shared the same 4 by 4 grid, the same three perturbation operators, three matched random controls per anchor, and the same aggregation protocol. The results showed a strikingly consistent spatial profile across architectures: in all eight runs, the four central cells of the grid held the top four ranks in mean margin degradation, followed by the eight edge cells and the four corner cells, with 16-cell profiles correlating at Spearman 0.89 to 0.95 between ConvNeXt-T, DeiT-S, and the MobileNetV2 models.

Yet the ImageNet study also exposed a genuine confound that had been distorting region-level comparisons. Under the official center-crop pipeline, corner cells projected onto the model input plane have a smaller effective area than center cells, a ratio of 1.78 for a square image and a median of 4.4 between the largest and smallest cell of an audited image. An area effect can therefore be conflated with a genuine difference in sensitivity across locations. The area-matched controls themselves failed this check for 99.3 percent of anchors under the official pipeline. Switching to crop-free preprocessing restored uniform cell area within 4 percent and matched every control within 5 percent, while preserving the dataset-level profile at Spearman 0.96 to 0.98. The finding motivated a preprocessing-aware geometry diagnostic that now warns when the effective area fraction of a region deviates from its intended fraction beyond a tolerance.

The third case study extended the same protocol to video, auditing TSM-ResNet50 on the NTU RGB+D 60 action recognition dataset using skeleton-guided body-part regions that vary from frame to frame. Aggregated over all body parts, the upper body showed the largest mean degradation, followed by the arm and torso, whereas the head showed the smallest. For the class-conditional view, a third-party annotator blind to SSAT’s results rated the relevance of five body-part groups to each of the 60 classes before any scores were computed. SSAT’s standardized class-level group scores ranked relevant groups above non-relevant ones with a mean class AUROC of 0.69, and a perfect AUROC of 1.0 in 25 of the 60 classes, including sitting down, kicking, and brushing teeth.

The authors also examined how sensitive the grading design itself is to its thresholds and control counts. Regrading stored runs showed that the z threshold moves anchors only between HIGH and LOW, leaving at least 93 percent of ImageNet grades unchanged across thresholds from 1 to 4. The HIGH share decreased smoothly as the control spread was estimated from more controls, from 14.5 percent at three controls to 6.8 percent at twenty, and 97.6 percent of grades agreed between ten and twenty controls. In terms of cost, the toolkit achieved about 3.6 milliseconds per model evaluation on an RTX 4090 host with 3.9 kilobytes of storage per evaluation, so each 3.2-million-evaluation run takes roughly four hours. A coarse-to-fine strategy, scanning a 2 by 2 grid and refining only the most sensitive cell, reaches 8 by 8 resolution with 31 percent of the evaluations of an exhaustive grid.

Compared with hand-written Captum-based reference implementations of the same audits, SSAT required roughly half the user-authored code for the synthetic setting, with one explicit design decision instead of eight, and moved to ImageNet with only two changed configuration lines rather than 419 changed lines of code. The authors are careful to note the limits of what the tool measures: sensitivity is a model’s response to a change in an input region, not a causal explanation or a statement about human semantic importance, and results depend on the region definition, perturbation operator, fill strategy, and preprocessing. Anchor-level grades agreed poorly across preprocessing pipelines and architectures, so they are model-specific, whereas dataset-level profiles remained similar. Future work includes a GPU pipeline, matched controls for frame-varying video regions, and task adapters beyond classification. For now, SSAT offers researchers something the field has lacked: a shared, verifiable protocol for asking where their models look, and an honest accounting of when the answer is trustworthy.

Subject of Research: SSAT: A reproducible toolkit for task-conditioned spatial sensitivity auditing of vision models

Article Title: SSAT: A reproducible toolkit for task-conditioned spatial sensitivity auditing of vision models

Article References: Lim, D., Jin, T., & Kang, C. (2026). SSAT: A reproducible toolkit for task-conditioned spatial sensitivity auditing of vision models. SoftwareX, 36, Article 103130. https://doi.org/10.1016/j.softx.2026.103130

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103130

Keywords: SSAT, reproducible, toolkit, task-conditioned, spatial, sensitivity, auditing, vision, models, scientific research