AI Spots Cracks, Stains and Creeps in Ancient Paintings With New Precision

Centuries-old paintings and works of calligraphy are among the most fragile treasures held by museums and archives. Over long periods of storage and display, their silk and paper supports fall victim to a quiet catalogue of injuries: insects gnaw at fibers, pigments flake away, cracks spider across surfaces, creases fold into the material, and stains bloom where moisture or contaminants have crept in. For generations, conservators have catalogued this damage by eye, a process that is slow, exhausting, and inevitably shaped by individual judgment. Now a team at Shanxi University in China has shown that a computer vision system can take over much of that burden, automatically finding and classifying deterioration in high-resolution images of painting and calligraphy relics with a level of consistency that manual inspection struggles to match.

The research, published in npj Heritage Science, centers on a technique known as instance segmentation, built on the YOLO-seg architecture. Unlike a simple object detector that draws a box around something interesting, instance segmentation traces the exact outline of each damaged region, pixel by pixel. That distinction matters enormously in conservation work. A stain that seeps across a third of a scroll cannot be meaningfully described by a rectangle; conservators need to know its true shape, its area, and how it relates to other damage nearby. By outputting precise masks for every instance of deterioration, the system preserves the spatial fingerprint of the damage rather than a coarse approximation of it.

The team focused on five deterioration categories that recur across painting and calligraphy collections: animal damage, missing parts, cracks, folding creases, and stains. Each category presents its own detection challenge. Animal damage tends to produce irregular, ragged losses with soft edges. Cracks are thin, elongated structures that can be nearly invisible at low magnification. Folding creases follow straight or gently curved lines that intersect the pictorial content, making them easy to confuse with intentional brushwork. Stains vary wildly in color, size, and transparency. Missing parts may blend into the background of an image. Building a single model that handles all five reliably required careful thinking about how the training data itself was prepared.

That preparation is one of the study’s most instructive contributions. High-resolution scans of scrolls and paintings are far too large to feed directly into a neural network, so the researchers adopted a patch-based strategy. Full images were cropped into 1024 by 1024 pixel patches, a size that balances the network’s memory demands against the need to see fine detail. But naive cropping creates its own problems: a crack might be sliced in half across a patch boundary, and most patches would contain nothing but undamaged background. To address this, the team used target-centered supplementary sampling, generating additional crops centered on annotated damage so that each deterioration category was adequately represented. Crucially, they also retained background patches containing no damage at all as negative samples, teaching the model what healthy material looks like and reducing false alarms on intact areas.

At inference time, the pipeline reverses the process. The system performs sliding-window inference across the full 2K-resolution image, analyzing overlapping patches and then fusing the results back together. Because adjacent windows can each detect the same piece of damage, duplicate detections must be reconciled. The researchers employed mask-based non-maximum suppression, a fusion step that merges overlapping instance masks so that a single crack spanning two patches is reported as one continuous defect rather than two fragments. The result is instance-level identification at the scale of the complete artwork, with the spatial distribution of every deterioration region preserved in the final output.

The performance numbers reported in the study are striking for a task of this difficulty. Across the five categories, the model achieved an overall box mean average precision at an intersection-over-union threshold of 0.50, commonly written as box mAP50, of 0.880, and a mask mAP50 of 0.875. In practical terms, this means that in roughly nine out of ten cases the system correctly identified and localized a deterioration instance, and its pixel-accurate masks were nearly as reliable as its bounding boxes. The narrow gap between the two metrics suggests that the model is not merely finding damage but genuinely understanding its extent, a prerequisite for any quantitative condition assessment.

That quantitative capability is where the approach moves beyond novelty into genuine conservation practice. Because the system outputs instance masks rather than boxes, it can compute the area of each damaged region, the proportion of the artwork affected, and statistics on connected components, meaning it can distinguish a single large stain from a scatter of small ones. These measurements feed directly into preservation condition assessment. A conservator or collection manager can track whether a particular scroll’s stain coverage has grown between imaging sessions, compare the severity of damage across an entire collection, and prioritize treatment based on objective, repeatable numbers instead of subjective impressions recorded in prose.

The implications reach well beyond a single laboratory. Cultural heritage institutions worldwide face a mounting backlog of condition surveys, and the sheer volume of digitized collections, accelerated by large-scale scanning initiatives, has created a data environment that manual inspection cannot keep pace with. Automated deterioration mapping offers a way to triage: flagging the works that need urgent attention, monitoring others over time, and building longitudinal records of how damage evolves under different storage conditions. Standardized, machine-generated annotations also remove a source of inconsistency between surveyors, making it possible to compare condition reports across institutions and across decades, something handwritten survey notes were never designed to support.

There are, of course, limits worth keeping in view. The model was trained on annotated examples of five deterioration types, and real collections contain other threats, from pigment discoloration to previous restoration interventions, that fall outside its vocabulary. Performance on damage types or materials underrepresented in the training data would need to be validated before deployment in new contexts. The patch-based pipeline also introduces engineering considerations, since extremely large or irregularly shaped artworks may require adapted sampling strategies. Yet the framework is deliberately general: new categories can be added by annotating polygon instances and retraining, and the sliding-window and mask-fusion machinery does not depend on the specific five classes studied here.

What makes this work resonate beyond its technical achievements is the picture it paints of conservation’s future. The most delicate objects in our museums, the ones too fragile to handle frequently or display for long, can now be monitored through their digital surrogates with a rigor that was previously impossible. Every scan becomes not just a record of appearance but a measurable health report, with each crack and crease logged, quantified, and timestamped. As imaging resolution grows and models improve, the gap between what a conservator sees under raking light and what an algorithm extracts from a flatbed scan will continue to narrow. For the scrolls and paintings that have survived centuries of insects, humidity, and careless hands, that growing digital vigilance may prove to be one of the most powerful preservation tools yet devised, ensuring that the damage of the past is documented and the deterioration of the future is caught early, precisely, and without a single subjective note.

Subject of Research: Automated identification and classification of deterioration in painting and calligraphy relics using YOLO-seg instance segmentation

Article Title: Research on the identification and classification of deterioration in painting and calligraphy relics based on YOLO-seg

Article References: Fan, K., Wang, H., Wang, Y., & Ren, Y. (2026). Research on the identification and classification of deterioration in painting and calligraphy relics based on YOLO-seg. npj Heritage Science. https://doi.org/10.1038/s40494-026-03041-1

Image Credits: AI Generated

DOI: 10.1038/s40494-026-03041-1

Keywords: YOLO-seg, instance segmentation, cultural heritage, painting conservation, calligraphy relics, deterioration detection, deep learning, computer vision, condition assessment, npj Heritage Science, mask-NMS, digital conservation