Sloan Grant Targets AI Reliability in Scientific Software

The rapid integration of artificial intelligence into scientific research workflows is reshaping how computational tools are developed, tested, and validated. While these technologies promise unprecedented efficiency, they also introduce complex challenges regarding the accuracy and reproducibility of scientific results. A new initiative funded by the Alfred P. Sloan Foundation aims to address these concerns by ensuring that AI-generated research software remains transparent, traceable, and scientifically sound. This effort represents a critical step in balancing the speed of modern development tools with the rigorous standards required for credible scientific inquiry.

Assistant Professor Nasir Eisty, a research software engineering and quality assurance expert at the University of Tennessee, Knoxville, has received a $430,000 grant to lead this project. Eisty serves in the Min H. Kao Department of Electrical Engineering and Computer Science, where he specializes in the intersection of software engineering and scientific computing. His work focuses on the practical application of AI agents in the creation and verification of research software, a field that is evolving rapidly as autonomous systems become more capable of interacting with external tools and executing complex tasks.

The core of the problem lies in the nature of agentic AI systems, which can autonomously interact with external tools to achieve programmed goals. In the context of scientific research, these systems are increasingly used to generate code from scratch, execute tests, and validate outputs. However, Eisty points out a significant risk: an AI can generate a test that appears sophisticated, runs successfully, and passes, yet it may be testing the wrong scientific assumption. This discrepancy is particularly dangerous because different scientific domains, such as climate modeling and biological simulation, have distinct definitions of correctness and underlying theoretical constraints.

Traditionally, research software engineers (RSEs) verified simulations by manually identifying important scenarios, encoding them as tests, and evaluating whether the results matched known outcomes. This process was time-consuming and required deep expertise in both the specific scientific domain and software engineering. With the advent of generative AI, scientists can now ask AI systems to analyze scenarios and generate tests, or even have agentic AIs run and modify tests based on results. This capability is valuable for researchers who are domain experts first and software developers second, as it makes good software engineering practices accessible to teams without dedicated engineering resources.

Despite these advantages, the convenience of AI-generated tests comes with a cost. AI systems may fail to capture important edge cases or domain-specific theoretical constraints, leading to incorrect assumptions and irreproducible results. If the scientific assumptions underlying a simulation are not properly accounted for, the credibility of the entire research output is compromised. Eisty’s Software Analytics and Intelligence Lab (SAIL) is investigating how to leverage AI in the scientific software pipeline while maintaining reproducibility, correctness, and long-term utility. The goal is to determine where AI is useful, where it is unreliable, and how humans can remain appropriately involved in the validation process.

The project will involve a detailed study of current RSEs and scientists to understand how they are using generative agentic AIs to design and test their software. The team will then deliberately introduce multiple types of defects into existing research software to evaluate how well AI-generated tests detect these injected issues. By working with domain scientists, the researchers will assess whether the tests reflect the scientific assumptions and questions underlying the software. This approach allows for a controlled environment in which the reliability of AI-generated tests can be measured against known ground truths.

One of the most challenging aspects of this research is defining what it means for an AI-generated test to be scientifically correct. To address this, Eisty and his team plan to study 30 to 40 research software projects across at least five scientific domains, including computational biology, astronomy, climate modeling, machine learning, and computational social science. This diverse set of projects will provide a broad basis for understanding how AI performance varies across different scientific contexts. The findings will help identify patterns in AI reliability and highlight areas where human oversight is most critical.

Upon completion of the experiments, the team will create a curated benchmark suite, open-source tools, and guidelines for responsible AI-assisted software testing. These resources will include a tool to help researchers provide AIs with appropriate scientific context during prompting and a test-development framework that records important information about the AI model and other factors involved in creating each test. The framework will be model-agnostic, allowing RSEs to continue creating reliable and reproducible research software as new AI systems emerge. This ensures that the solutions developed are not tied to a specific technology but are adaptable to future advancements.

The project also includes support for two PhD students who will work with Eisty throughout the research process. This aspect of the grant is crucial for training the next generation of researchers who can help shape the responsible use of AI in scientific software development. Eisty expressed gratitude to the Sloan Foundation for supporting this work, noting that AI is changing software engineering very quickly. The project provides an opportunity not only to study this transformation but also to equip researchers with the skills needed to navigate it responsibly.

Ultimately, this initiative is about finding the right balance between leveraging AI’s ability to accelerate software development and preserving the human judgment, transparency, and rigorous validation that science requires. As AI continues to play a larger role in scientific discovery, ensuring the integrity of the tools used to generate and verify results will be essential. By developing robust frameworks and guidelines, Eisty and his team aim to future-proof research software, ensuring that it remains a reliable foundation for scientific progress in an era of rapid technological change.

Subject of Research: Validation of AI-generated research software for scientific reproducibility

Article Title: With Sloan Foundation Award, Eisty future-proofs research software

Article References: With Sloan Foundation Award, Eisty future-proofs research software. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: Artificial Intelligence, Research Software Engineering, Scientific Reproducibility, Alfred P. Sloan Foundation, University of Tennessee, Quality Assurance, Agentic AI, Computational Science, Software Testing, Data Integrity, Sloan, Foundation