Smart Microscopy Runtime Analysis Reveals Variable Time Savings

Published in the journal Applied Microscopy in August 2025, the study demonstrates that the time savings achieved by smart microscopy are not uniform but depend heavily on specific workflow parameters such as tile size, imaging speed, and computation time. The team developed a mathematical model and a graphical user interface application to help users predict these savings based on their own experimental conditions, providing a practical tool for optimizing imaging workflows in connectomics and other fields.

Smart microscopy, also known as SmartEM, addresses the challenge of capturing high-quality images of biological specimens without wasting time on low-information areas. The approach involves a three-step process: a rapid initial scan provides an overview of the sample, a neural network analyzes this fast image to predict which subregions contain important features such as synapses, and a slower, higher-resolution scan is then performed only on those selected regions. This method has previously been validated for reducing beam time in electron microscopy connectomics, but the new study focuses on the end-to-end time of the complete pipeline, including stage movements, computation, and data transfer, rather than just the beam-on time.

The researchers constructed a model of the complete SmartEM workflow that incorporates nine key parameters, including the number of sections, tiles per section, transition times between tiles and sections, and the time required for fast and slow imaging. They compared this smart pipeline against a traditional imaging pipeline, which images the entire sample at a fixed rate without selective rescanning. By performing a sensitivity analysis on these parameters, the study revealed that the theoretical time savings of smart microscopy can vary drastically. For instance, larger imaging batches, defined by an increased number of sections or tiles per section, generally lead to greater time savings, while slower imaging rates or higher quality standards can diminish these benefits.

To make these insights accessible, the team developed a Shiny-based Python application that allows users to input their specific acquisition workflow parameters. The app visualizes how much acquisition time could be saved using either serial or parallel versions of SmartEM. It calculates the ideal rescan fraction for a given quality standard and indicates whether smart microscopy is faster than traditional imaging for that specific setup. This tool helps researchers determine if the overhead of machine learning inference and selective rescanning is justified for their particular experimental goals, such as achieving a certain level of segmentation accuracy or signal-to-noise ratio.

The study also investigated methods to accelerate the smart microscopy pipeline itself. The authors demonstrated two strategies for faster acquisition: using mixed-precision neural networks and parallelizing the operations of the microscope and the support computer. Parallelization allows the microscope to perform fast scans on subsequent tiles while the support computer simultaneously computes rescan masks for previous tiles, effectively hiding the latency of computation behind the time spent on imaging.

Experimental validation of these strategies was conducted on a Thermo Fisher Verios electron microscope system. The researchers acquired three datasets from mouse cortex samples under four different acquisition settings. The first setting involved traditional imaging at a dwell time of 1000 nanoseconds per pixel. The other three settings utilized SmartEM with fast scan dwell times of 100 nanoseconds per pixel and slow scan dwell times of 800 nanoseconds per pixel, with a rescan rate of 10%. The results showed that in the first setting, autocasting for mixed-precision inference accelerated the computation of rescan maps by 43%. Furthermore, parallelization sped up the total acquisition time by 224 seconds, which was equivalent to 76% of the total computation time.

However, the study highlighted that the benefits of parallelization are not universal. In a second experimental setting with smaller tile sizes, stage movements comprised a greater fraction of the overall runtime, which diminished the advantage of parallelization. Additionally, mixed-precision inference showed less of an advantage at these smaller tile sizes. This finding underscores the importance of considering hardware-specific overheads, such as stage movement times, when evaluating the efficiency of smart imaging workflows. The third setting, which used larger tile sizes similar to the first, confirmed that mixed-precision, parallel smart imaging remained the fastest acquisition method.

The researchers also addressed the concept of image quality in their model. They represented quality as a continuous quantity ranging from 0 to 1, where a quality standard Q defines the minimum acceptable level. The model assumes a quality function that maps the fraction of the image that is rescanned to a quality value. In their experiments, they used a U-Net neural network to perform binary segmentation of cell membranes on mixed dwell time images. The quality function was defined as the membrane segmentation accuracy, allowing them to determine the lowest rescan fraction that satisfied a specific quality standard. This approach provides a quantitative framework for balancing imaging speed against the desired level of detail in the final images.

Limitations of the study include the assumption that users have access to a function that maps rescan ratios to quality values, which may require estimation from a subset of data. The model also does not fully account for potential bottlenecks in parallelization, such as synchronization issues during data transfer, although the authors noted that their empirical experiments did not reveal significant synchronization bottlenecks. Additionally, the current SmartEM method involves two U-Nets in series, and the compute time was extrapolated from measurements on a specific GPU, which may not generalize to all hardware configurations.

The authors concluded that smart imaging is faster than traditional imaging in a variety of settings, particularly when imaging and computation are parallelized. They emphasized that the primary bottleneck in their experiments was the overhead costs associated with loading and executing rescans, which kept the measured rescan times above the theoretical minimum. They suggested that future work could explore other patterning techniques in electron microscopy control software, such as stream patterning, to further reduce these overheads. The study provides a foundational framework for understanding the runtime dynamics of smart microscopy, offering a path toward more efficient imaging in connectomics and potentially other fields where image quality varies across the specimen.

The code associated with the timing analysis is freely available on GitHub, allowing other researchers to replicate the findings or adapt the model to their own workflows.

Subject of Research: Runtime analysis and optimization of smart electron microscopy workflows

Article Title: Analysis of smart imaging runtime

Article References: Athey, T., Sawmya, S., Meirovitch, Y., Schalek, R., Potocek, P., Chandok, I., Peemen, M., Lichtman, J., Samuel, A., & Shavit, N. (2025). Analysis of smart imaging runtime. Applied Microscopy, 55(1), Article 10. https://doi.org/10.1186/s42649-025-00115-5

Image Credits: AI Generated

DOI: 10.1186/s42649-025-00115-5

Keywords: smart microscopy, electron microscopy, connectomics, machine learning, imaging runtime, parallelization, mixed-precision inference, neural networks, workflow optimization, Applied Microscopy, Analysis, smart