AI Learns to Picture Minerals From Words Using a Vast Open Database

For more than a century, picturing a mineral you have never seen meant flipping through field guides or begging a museum curator for a glimpse of a drawer-bound specimen. Now a team of computer scientists and mineralogists has taught an artificial intelligence to do something stranger and more useful: type in a description of a mineral’s chemistry and physical behavior, and watch a plausible portrait of the crystal appear on screen. The work, published in Earth Science Informatics, turns one of the world’s great open science databases into a training ground for generative AI, and it may change how geologists, teachers and collectors think about what a mineral can look like before anyone ever digs one up.

The foundation of the project is Mindat, the sprawling online mineral database that has quietly become the reference shelf of the entire mineralogy community. As of February 2025, Mindat cataloged 6,114 mineral species approved by the International Mineralogical Association, accompanied by nearly 1.41 million high-quality photographs and more than 140 distinct mineral attributes. That ocean of images has always been searchable by name or locality, but the relationships between the pictures and the text describing them remained largely unexploited. The researchers, led by Quanli Fu and Xiang Que of Fujian Agriculture and Forestry University, together with Xiaogang Ma of the University of Idaho and colleagues, saw in those unexplored connections an invitation: if every photograph is tied to a structured description of hardness, luster, cleavage and chemistry, could a machine learn to run the relationship in reverse?

Their answer builds on the OpenMindat project, an effort funded by the U.S. National Science Foundation to expose the Mindat database through a machine-readable interface that complies with the FAIR principles of findability, accessibility, interoperability and reusability. Earlier work in the OpenMindat ecosystem produced R and Python packages, a mobile application, and visualization frameworks for exploring associations among minerals, elements and localities. What had never been attempted, the authors note, was the integration of Mindat’s textual attribute data with its images to generate entirely new mineral pictures. Text-to-image synthesis, the technology behind tools like DALL-E and Midjourney, had already proven itself in medical imaging, remote sensing and design education; nobody had pointed it at the mineral kingdom.

The first task was data engineering, and it was substantial. The team pulled 10 to 20 images for each of the 6,114 IMA-approved mineral species, collecting every available photograph for species with fewer than ten, and ended up with a dataset of 43,745 images, which they have released publicly on Hugging Face. Each image was paired with a standardized textual description assembled from ten attributes drawn from four categories: chemical composition, physical characteristics, crystal system and optical behavior. The template reads like a telegraphic mineralogist’s shorthand: a mineral’s transparency, constituent elements, Mohs hardness range, luster type, streak color, crystal system, cleavage, fracture and optical type are stitched into a single sentence. Attributes with missing values are simply omitted, and the image-to-text pairing was verified by matching image folder names against mineral names returned by the Mindat API.

With the dataset in hand, the researchers fine-tuned Stable Diffusion, a prominent text-to-image model built on the Latent Diffusion framework. Stable Diffusion’s architecture has three moving parts that matter here: a variational autoencoder that compresses images into a compact latent space and reconstructs them, a U-Net that learns to strip Gaussian noise away step by step, and a CLIP text encoder that converts prompts into vector representations injected into the U-Net through cross-attention layers. During training, the model learns to predict the noise corrupting a latent image, conditioned on the text prompt, so that at generation time it can start from pure noise and denoise its way toward an image that matches the description. Fine-tuning concentrated on the U-Net, leaving the pre-trained VAE and text encoder frozen, since those components already carry strong general-purpose capabilities.

The team compared two fine-tuning strategies. Full fine-tuning updates every parameter of the U-Net, a comprehensive adjustment that in this study involved 859.52 million trainable parameters. Low-Rank Adaptation, or LoRA, takes a leaner path: instead of rewriting the weight matrices outright, it freezes them and learns small low-rank residual matrices whose product approximates the needed update, cutting the trainable parameter count to just 0.79 million. LoRA’s efficiency has made it the darling of the open-source AI community, but the mineral experiment delivered a clear verdict. Evaluated with the Fréchet Inception Distance, the Kernel Inception Distance and CLIPScore, the fully fine-tuned model outperformed both the LoRA variant and the untouched baseline, achieving an FID of 48.83 and a KID of 0.02, indicating generated images whose statistical distribution sits closest to that of real mineral photography.

The qualitative differences were just as telling. The baseline Stable Diffusion model, asked for a mineral, tended to produce geometric patterns with excessive specular reflections and a synthetic, manufactured sheen, while the Kandinsky model drifted toward an artistic style with morphologically concentrated structures. The fully fine-tuned model, by contrast, produced images with irregular granular textures, heterogeneous color distributions and distinct crystalline structures that echo real specimens. LoRA outputs improved on the baseline but remained unstable, sometimes over-smoothed and incompletely conditioned on the target attributes. The authors caution that CLIPScore, being trained on general-domain corpora, may understate the gains on specialized mineralogical vocabulary, a known limitation of automatic metrics when domain-specific terminology enters the picture.

One of the study’s most practical findings concerns prompt detail. When the researchers varied how many attributes appeared in a prompt, from a terse three-attribute description to a comprehensive one matching the training template, generation quality climbed steadily with specificity. Sparse prompts leave the model under-constrained, allowing attribute combinations that could correspond to many different minerals, while richer descriptions tighten the visual target. The team validated the approach against reality by querying Mindat’s own advanced search portal for images of chrysoberyl matching a given attribute combination and finding the AI-generated counterparts strikingly similar. They then pushed further, feeding the model attribute combinations belonging to no known IMA-approved species, generating portraits of hypothetical minerals that may exist in nature but remain unclassified, or may only form transiently under extreme temperatures and pressures.

To put the technology in users’ hands, the researchers built an interactive tool, available online, in which anyone can select mineral attributes, have them converted automatically into a prompt, adjust parameters such as random seed, image size, guidance ratio and inference steps, and receive a watermarked generated image. The watermark is a deliberate safeguard against misuse and protects the integrity of the underlying dataset. The authors envision the tool serving mineralogists seeking a visual reference for attribute combinations, enthusiasts exploring the diversity of the mineral world, and science educators making abstract properties like cleavage and streak tangible for students. They are candid about limitations: synthetic images cannot be verified as depictions of real entities, the uniform description template may constrain image-text alignment, and only ten of Mindat’s more than 140 attributes were used, partly due to the text encoder’s input limits.

The road ahead is ambitious. The team proposes supplementing fine-tuning with supervised fine-tuning on higher-quality pairs and with preference-based optimization methods such as direct preference optimization to sharpen alignment between images and descriptions. They also sketch the inverse problem: fine-tuning a generative image-to-text model so that a photograph of an unknown specimen yields a description of its compositional and optical attributes, bypassing traditional classification pipelines in favor of reasoning from visual cues. And they suggest benchmarking against newer generators beyond Stable Diffusion. If mineral formation is a story of temperature, pressure, redox state and fluid chemistry unfolding over billions of years, a model that has learned the visual grammar of that story may one day help predict what minerals on other planets look like, or what fleeting intermediate phases in a crystal’s birth might have been, long before any camera could capture them.

Subject of Research: Fine-tuning the Stable Diffusion text-to-image model with Mindat mineral data to generate mineral images from textual descriptions of attribute combinations.

Article Title: Fine-tune the stable diffusion model using mindat data to generate mineral images from textual descriptions of attribute combinations

Article References: Fu, Q., Que, X., Ma, X., Lin, M., Sun, S., & Chen, M. (2026). Fine-tune the stable diffusion model using mindat data to generate mineral images from textual descriptions of attribute combinations. Earth Science Informatics, 19(11), Article 190. https://doi.org/10.1007/s12145-026-02235-2

Image Credits: AI Generated

DOI: 10.1007/s12145-026-02235-2

Keywords: Mindat, Stable Diffusion, text-to-image generation, mineralogy, mineral informatics, fine-tuning, LoRA, OpenMindat, generative AI, crystallography, image-text dataset, Earth Science Informatics