AI Reshapes How Machines Capture the Full Dimensionality of Light

Light is an astonishingly rich carrier of information. Every beam arriving at a camera or sensor encodes not just brightness, but phase, spectrum, polarization, spatial structure and temporal dynamics. Yet the photodetectors that sit at the heart of nearly every imaging system strip almost all of that richness away, condensing a multidimensional optical field into a simple scalar photocurrent. A new review published in Nature Reviews Electrical Engineering argues that artificial intelligence is now transforming this fundamental mismatch, enabling a class of technologies known as computational light field detection that could redefine how machines see the world.

The core idea behind computational light field detection is deceptively simple. Instead of trying to measure every property of light directly with dedicated hardware, researchers encode multiple optical dimensions into a compact set of measurements, then use algorithms to computationally reconstruct the full picture. The success of this approach depends on two things working in concert: an optical front end that captures genuinely information-rich, distinguishable measurements, and a reconstruction algorithm capable of recovering multidimensional data from those compressed observations. The review, authored by teams from Shanghai Jiao Tong University, the University of Cambridge, the University of Hong Kong, Hangzhou Dianzi University, Zhejiang University and Aalto University, maps how AI is reshaping both sides of this equation.

On the hardware side, the challenge has always been design. Nanophotonic encoders such as metasurfaces, disordered photonic structures and engineered heterojunctions can manipulate light in extraordinary ways, but discovering the right geometry for a given encoding task traditionally requires repeated, computationally expensive electromagnetic simulations. AI-based surrogate models are changing that calculus. By learning to predict the optical behavior of candidate structures from a training set of simulations, these models replace the slow forward-solving process with fast learned predictions, allowing designers to explore vastly larger design spaces. Combined with generative models and differentiable optimization, researchers can now discover complex, non-intuitive structures that human intuition or exhaustive search would never have uncovered.

The reconstruction side presents a different kind of problem. Recovering a spectral cube, a polarization state, an optical phase map or an ultrafast temporal sequence from sparse or compressed measurements is a mathematically ill-posed task: many possible light fields could explain the same detector output. Classical approaches relied on regularization and iterative optimization, but machine learning has opened flexible new routes. Deep neural networks trained on large datasets can learn priors about natural scenes and exploit them to stabilize reconstructions, while physics-informed networks embed the governing equations of light propagation directly into the learning process. Importantly, the review emphasizes that no single model class is optimal across all sensing regimes; the right architecture depends on the availability of training data, the fidelity of the forward model, latency requirements and the tolerance for reconstruction errors.

Progress is uneven across the different dimensions of light. Spectral reconstruction, from miniaturized computational spectrometers to snapshot hyperspectral imaging, is arguably the most mature field, with deep learning models already enabling video-rate hyperspectral cameras and on-chip spectrometers smaller than a coin. Temporal reconstruction has seen spectacular advances as well: compressed ultrafast photography techniques, enhanced by machine learning, have captured events at trillions of frames per second in a single shot. Phase retrieval and polarization detection, by contrast, remain more challenging, though learned models for holography, lensless imaging and full-Stokes polarimetry are closing the gap rapidly.

Perhaps the most forward-looking concept in the review is the differentiable digital twin. Today, most optical hardware and most reconstruction algorithms are designed and optimized separately, a workflow that leaves substantial system-level performance on the table. A differentiable digital twin instead creates a computational replica of the entire sensing pipeline, from the physics of the encoder to the neural decoder, through which gradients can flow. This allows the encoder parameters and the reconstruction model to be co-optimized jointly, producing hardware and software that are matched to each other from the ground up. When the digital twin is grounded in real physics, it can go further still, incorporating experimental error sources such as fabrication imperfections and noise, and even estimating the uncertainty of its own reconstructions.

The review is careful to note that trustworthy detection cannot rest on accurate reconstruction alone. Deep learning models are known to produce instabilities and hallucinations, generating plausible-looking but incorrect outputs, particularly when deployed outside their training distribution. Reliable real-world deployment demands physics-based models and hardware-in-the-loop optimization, in which measurements from actual physical devices are folded directly into the training loop. The authors highlight generalization, physical consistency, interpretability and robustness to the inevitable discrepancies between digital models and physical hardware as the critical criteria that will determine whether these systems make the leap from laboratory demonstrations to practical instruments.

The potential applications are broad and compelling. Compact, adaptable detectors capable of sensing multidimensional light could transform medical diagnostics, where hyperspectral and polarimetric imaging reveal tissue properties invisible to conventional cameras. They could enhance remote sensing, autonomous navigation, industrial inspection, agriculture and astronomy, where polarization and spectral signatures carry crucial physical information. Miniaturized computational spectrometers, for instance, promise to bring laboratory-grade chemical analysis onto drones, smartphones and lab-on-a-chip platforms, while ultrafast single-shot imagers open windows into phenomena from femtosecond laser dynamics to neural signaling.

What emerges from the analysis is a picture of a field at an inflection point. The individual ingredients, learned surrogate models for photonic design, machine learning decoders for compressed reconstruction and differentiable frameworks for joint optimization, have each matured considerably. The review argues that their integration into coherent, physics-grounded systems is the next great opportunity, one that could yield a new generation of compact, intelligent detectors capable of perceiving light the way nature does: not as a single scalar, but as a full, high-dimensional field. If researchers can satisfy the demands of generalization and physical robustness, the marriage of AI and light field detection may prove to be one of the defining developments in optical sensing for the decade ahead.

Subject of Research: AI-driven computational light field detection for multidimensional optical sensing

Article Title: Light field detection in the age of artificial intelligence

Article References: Cai, W., Zhang, Y., Yang, E., Chen, Z., Song, Z., Chen, N., Jin, L., Yang, Z., Sun, Z., & Hasan, T. (2026). Light field detection in the age of artificial intelligence. Nature Reviews Electrical Engineering. https://doi.org/10.1038/s44287-026-00328-0

Image Credits: AI Generated

DOI: 10.1038/s44287-026-00328-0

Keywords: light field detection, artificial intelligence, computational imaging, photodetectors, metasurfaces, hyperspectral imaging, polarization, inverse design, deep learning, digital twin, compressed sensing, nanophotonics