Convolutional optical neural networks herald a new era for AI imaging

Convolutional Neural Networks (CNNs), with its exceptional image recognition capabilities, have performed outstandingly in the field of AI and notably within platforms like ChatGPT. Recently, a team of Chinese researchers from University of Shanghai for Science and Technology have successfully introduced the concept of CNNs into the field of optics and realized convolutional all-optical neural network, bringing revolutionary progress to AI imaging technology.

Led by Prof. Min Gu and Prof. Qiming Zhang from School of Artificial Intelligence Science and Technology (SAIST) at the University of Shanghai for Science and Technology (USST), the research team has developed an ultrafast convolutional optical neural network (ONN), which achieves efficient and clear imaging of objects behind scattering media without relying on the optical memory effect. This finding was published on June 14th in the journal Science Advances, with the title of “Memory-less scattering imaging with ultrafast convolutional optical neural networks”. Dr. Yuchao Zhang, a researcher at SAIST, is the first author. Prof. Min Gu and Prof. Qiming Zhang serve as the corresponding authors.

The core of CNNs, convolutional operations, extracts local features from images and constructs more complex and abstract feature representations layer by layer, greatly advancing the fields of image processing and pattern recognition. However, applying the concept of convolution network operations to the field of optics faces the challenge of converting electronic signals into optical signals. The research team ingeniously designed an all-optical solution, performing convolution network operations directly in the optical domain, eliminating the cumbersome signal conversion process, and achieving true optical computing speeds.

The key to this research is the construction of a multi-stage convolutional ONN, composed of multiple parallel cores capable of operating at the speed of light, directly extracting features from scattered light for rapid image reconstruction. This process not only greatly improves imaging speed but also significantly enhances image quality, making imaging in complex scattering environments possible. Moreover, the convolutional ONN’s computational speed reaches 1.57 Peta operations per second (POPS), providing robust support for real-time dynamic imaging.

Another highlight of this research is its multitasking capability. By simply adjusting the network structure, the same convolutional ONN can perform a variety of different image processing tasks, such as classification and reconstruction, concurrently—a first in the field of optical artificial intelligence. Prof Qiming Zhang noted, “This combination of flexibility and efficiency not only underscores the importance of convolutional networks in artificial intelligence but also opens up new avenues for optical imaging technology.”

The advent of this research is not only a successful transplantation of convolutional neural networks into the optical field but also a significant boost to AI imaging technology. Prof Min Gu stated, “In the near future, convolutional optical neural networks will play an increasingly vital role in autonomous driving, robotic vision, and medical imaging.”