The project, named Fragmented Legacy: Turning Broken 3D Materials into Archaeological Interpretations, is one of 19 selected for this year’s Humanities and AI Virtual Institute awards from Schmidt Sciences. It is led by Chen Feng, Institute Associate Professor at NYU’s Tandon School of Engineering and founding co-director of the NYU Center for Robotics and Embodied Intelligence, together with Jing Zhang, a faculty fellow at the same center. On the archaeological side, the team includes Radu Iovita and Scott Williams of NYU’s Department of Anthropology and Raiko Krauß, Professor of Central and Southeast European Prehistory at Tübingen, with additional collaborators at Leiden University, Arizona State University, and the Max Planck Institute for Evolutionary Anthropology.

The problem the group is attacking is deceptively simple to state and brutally hard to solve. Most archaeological collections are never systematically refitted at all, because the work is so slow that it simply does not get done. Reported success rates tell the story: roughly 1 percent for ceramics, and between 4 and 26 percent for bones. A human specialist may spend days or weeks trying to find the matching edge for a single flake, and in a museum setting the physical constraints make it worse. Drawers can only be opened one at a time, so an entire collection can never be laid out for comparison at once. The result is that vast stores of information sit locked inside unmatched fragments.

Why does a single match matter so much? Because when two fragments fit, they carry information that neither piece holds alone. If a stone flake found in one part of a site fits back onto a core recovered from another, researchers can reconstruct how people moved through that space, how a tool was manufactured step by step, or whether the site’s layers were disturbed after deposition. Refitted animal bones can show how a carcass was butchered and how meat was shared across a group, a direct window into social behavior tens of thousands of years old. Refitted human fossils can reveal anatomical features that no single fragment preserves. Without refits, many of those questions simply go unanswered.

Feng’s lab specializes in teaching machines to understand three-dimensional shapes in the real world, where conditions are messy and incomplete. Broken archaeological material, he notes, is among the hardest versions of that problem: the pieces are worn, many are missing entirely, and there is no picture on the box showing what the finished object should look like. The AI systems his team is building must reason through that uncertainty, generating plausible reconstructions while flagging where the evidence is thin, with the archaeologists on the team judging whether the answers actually make sense in light of the archaeological record.

The technical plan unfolds in stages. First, the team will assemble a dataset of at least 100,000 real and digitally simulated broken objects, pairing 3D models with photographs, descriptions, and catalog data. Crucially, the simulated breaks will not be random: they will follow how stone, clay, and bone actually fracture, so that the synthetic fragments behave like the real ones archaeologists pull from the ground. Next, the researchers will develop AI models that work out where each fragment belongs while simultaneously generating a plausible version of the complete object. The models will reason across shape, images, and text, so that archaeological knowledge — typologies, raw material properties, contextual records — can rule out reconstructions that are geometrically possible but archaeologically implausible.

Archaeologists will remain in the loop throughout, comparing competing reconstructions and teaching the system which ones hold up. This human-in-the-loop design reflects a core philosophy of the project: the AI is not meant to replace expert judgment but to multiply its reach. As Zhang puts it, archaeologists are like Sherlock Holmes, piecing together clues to understand the past, and the goal is for Fragmented Legacy to be their Watson — matching fragments and suggesting possible reconstructions so that researchers have more time to interpret the evidence and ask new questions.

The project did not appear out of nowhere. It grew out of a collaboration between Feng’s AI4CE Lab and Iovita’s lab that began about four years ago with LUWA, an earlier system that used AI to classify microscopic wear on stone tools. Zhang, then a postdoctoral researcher working in both labs, connected the two groups. In 2024, she asked Iovita what archaeology’s most pressing technical problem was, and he pointed straight at refitting. That conversation produced GARF, a system that reassembles broken objects, and then CRAG, which goes further by also filling in missing pieces — work that laid the direct groundwork for the HAVI award now funding Fragmented Legacy.

Iovita’s ambitions for the technology extend well beyond reassembling objects within a single site. Refitting objects between sites, to demonstrate direct relationships between different locations, has only ever been done twice in the history of the discipline, he notes. His stated dream is to use AI to do this at scale, which he argues will revolutionize the field. If fragments from one site can be matched against fragments from another, researchers could trace the movement of raw materials, tools, and even the people who carried them across landscapes in ways that have previously been impractical or impossible.

Accessibility is built into the project from the start. All datasets, model weights, and benchmarks will be released under permissive open-source licenses, so that university researchers and cultural resource management firms — the companies that carry out much of the archaeological work required ahead of construction in the United States and abroad — can use the tools directly in their own workflows. The team also expects the methods to travel beyond archaeology, finding applications in other scientific fields and in practical repair and restoration work, wherever broken three-dimensional objects need to be reassembled and their missing parts intelligently reconstructed. For a discipline built on fragments, the promise is that the pieces sitting unmatched in museum drawers may finally begin to speak.

Subject of Research: AI-based refitting of fragmented archaeological artifacts such as stone tools, pottery, and bones

Article Title: NYU researchers are building AI to piece together broken archaeological finds

Article References: NYU researchers are building AI to piece together broken archaeological finds. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: artificial intelligence, archaeology, refitting, 3D modeling, stone tools, pottery, bones, NYU, Schmidt Sciences, machine learning, cultural heritage, open source