The new work, published in Nature Sensors, centers on a heterostructure formed from antimony telluride and molybdenum disulfide, two layered materials that can be stacked together with atomic precision. The resulting two-terminal device is deceptively simple in appearance: two electrodes, a vertical stack of semiconductors, and nothing else. Yet when the researchers applied a low bias voltage across the terminals, they discovered that the device could be toggled, reversibly and in situ, between two fundamentally different optoelectronic operating modes. In one mode it behaves as a photodiode, generating a current that scales faithfully and linearly with the intensity of incident light, exactly what is needed for conventional image sensing. In the other mode it behaves as an opto-synaptic element, in which light triggers a response that persists, decays, and accumulates over time, mirroring the way a biological synapse integrates incoming spikes before deciding whether to pass a signal onward.

The significance of this switchability is difficult to overstate. In existing optoelectronic computing arrays, each pixel is typically hardwired to a single function. If a designer wants an array that can both sense images and perform synaptic computation, the usual approach is to build separate hardware for each task, multiplying device count, fabrication complexity, and cost. The new sensor sidesteps this trade-off entirely. Because a single pixel can be reconfigured between photodiode and opto-synaptic modes simply by adjusting the applied bias voltage, the same physical array can act as a classical image sensor at one moment and a neuromorphic processing fabric the next. Functional diversity, the researchers argue, no longer has to be purchased at the price of integration scale. The array can grow, and its capabilities can grow with it.

The physics underlying this multi-responsiveness lies in the band alignment between the two constituent materials. Antimony telluride and molybdenum disulfide form a junction whose potential barrier can be modulated by the external bias. At certain bias conditions, photoexcited carriers are swept across the junction in a drift-dominated regime, producing the fast, linear, reset-every-frame response characteristic of a photodiode. Shift the bias, and the transport regime changes: charge generated by light becomes trapped and released slowly at interfacial states, so each optical pulse leaves behind a lingering conductance change that fades over time. It is precisely this fading memory that allows the device to emulate synaptic behavior, with the weight of the “synapse” effectively set by the recent history of illumination.

Perhaps the most striking demonstration in the study is the device’s ability to reproduce the leaky integrate-and-fire dynamics of biological neurons, the canonical model of how real neurons accumulate inputs and spike when a threshold is crossed. When the researchers illuminated the sensor with focused light pulses, the device’s internal state integrated the successive pulses, but also “leaked,” losing accumulated charge between pulses. Once the integrated response reached a threshold, the device fired, producing a discrete output event before resetting. This is not a software simulation of a neuron running on conventional hardware; it is a physical two-terminal device exhibiting the intrinsic dynamics of a neuron in response to light alone. Such light-driven leaky integrate-and-fire behavior is a cornerstone capability for spiking neural networks, a class of computing systems prized for their extreme energy efficiency because they only transmit information when events occur, rather than continuously.

To show that these single-device behaviors could scale into a usable computing system, the team integrated an array of the sensors with diffractive optical components, passive optical elements that shape and route light by diffraction rather than refraction. This combination enabled what the researchers describe as multi-mode optoelectronic computing. In one configuration, the array operated as a conventional imager, capturing scenes for standard image processing. Switching operating modes, the same hardware performed convolution-like operations directly in the optical and optoelectronic domain, preprocessing images before any digital computation was needed. The architecture also handled video streams, where the temporal persistence of the opto-synaptic mode allowed the system to exploit information across frames rather than treating each frame in isolation.

The diffractive components add a crucial spatial dimension to this capability. By patterning light before it reaches the sensor array, they allow certain linear operations, effectively optical convolutions, to be performed at the speed of light and at essentially no energy cost, since passive optics consume no power. The sensor array then transduces and further processes the patterned light in its reconfigurable modes. The result is a computing pipeline in which sensing, analog preprocessing, and neuromorphic event generation are woven together into a single front-end, with the back-end digital processor relieved of the bulk of the workload.

The team went beyond image and video processing to demonstrate transfer learning enabled by optical spike encoding. In this scheme, visual information is converted into trains of optical spikes whose timing and statistics encode features of the input scene. Because the sensor’s synaptic response naturally integrates and fires on these spikes, downstream learning layers can be trained efficiently on the encoded representations and adapted to new tasks without retraining from scratch. This kind of transfer learning, performed on representations extracted physically by the front-end hardware rather than computed digitally, points toward vision systems that can adapt to new environments with minimal additional computation, a property that becomes essential when the computing platform must operate under strict power constraints.

The researchers emphasize that the architecture achieves a superior spatiotemporal dimensionality compared with existing approaches. Conventional image sensors operate in a purely spatial domain, capturing two-dimensional snapshots at fixed frame rates. Neuromorphic event cameras add temporal richness but typically sacrifice conventional imaging capability. The multi-responsive sensor spans both worlds: in photodiode mode it delivers spatial fidelity, while in opto-synaptic and spiking modes it captures temporal dynamics and event-driven information. This expanded representational space, the authors argue, is precisely what demanding real-world applications require, and they point to autonomous driving, satellite remote sensing, and robotics as fields poised to benefit.

The appeal for those domains is concrete. An autonomous vehicle must simultaneously perform conventional object recognition, which favors photodiode-mode imaging, and react to sudden temporal events such as a pedestrian stepping into the road, where spike-based, low-latency processing excels. A satellite remote-sensing platform, with limited power and bandwidth, would gain enormously from a front end that compresses and encodes visual information before transmission. Robots operating in unstructured environments need vision that is fast, adaptive, and frugal with energy. A single sensor platform that can be reconfigured to serve all these roles, without multiplying hardware, could reshape how such systems are engineered.

The broader context for this work is a vigorous international effort to move computation out of the back end and into the front end of vision systems, whether into the sensor itself or into free-space optics. Brain-inspired computing has matured rapidly over the past decade, with memristive devices, phase-change materials, and two-dimensional heterostructures all competing to deliver analog memory and neuromorphic dynamics in compact form factors. What distinguishes the present advance is the combination of simplicity and versatility: a two-terminal device, requiring no complex three-terminal gating or embedded memory elements, that nonetheless offers two distinct photonic operating modes plus intrinsic neuronal firing dynamics, all switchable under low-voltage control.

Challenges remain before such sensors find their way into commercial systems. Array-scale uniformity, long-term endurance under repeated mode switching, integration with readout electronics, and compatibility with large-area fabrication will all need to be demonstrated at production quality. The energy savings promised by front-end computing are only realized in full when the surrounding system, from optics to readout circuits to back-end processors, is co-designed around the sensor’s capabilities. Nevertheless, the demonstration of reversible, in-situ reconfiguration within a single heterostructure marks a meaningful step toward vision hardware that behaves less like a passive camera and more like the retina and early visual cortex it takes as its model.

If the approach scales, the implications extend well beyond efficiency. A sensor that computes blurs the line between perception and cognition, suggesting machines whose first contact with the visual world already carries meaning: edges weighted by context, motion encoded in spike timing, salient events flagged before a single digital multiplication occurs. In a future where cameras multiply into billions of devices, from vehicles to satellites to embedded robots, pushing even a fraction of that interpretive work into the sensor itself could save staggering amounts of energy. The retinomorphic sensor described in this study offers a glimpse of that future, one in which the eye does not merely see but begins, at the moment of seeing, to think.

Vision transformers, commonly known as ViTs, have become a powerful alternative to convolutional neural networks for image recognition. Instead of scanning an image through layers of fixed filters, a ViT divides the image into small regions, or patches, and represents each patch as a numerical token. These tokens are then processed like a sequence of words in a language model. The transformer learns relationships between distant parts of an image, enabling it to recognize objects and patterns that may depend on the entire visual scene. Yet the standard method for creating those tokens is highly digital: a camera first captures the image, electronic circuits convert the signal into data, a processor divides the image into patches and a separate operation calculates an embedding for every patch.

That conventional chain creates a bottleneck between the physical world and artificial intelligence. Moving raw pixels from an image sensor to memory and then to a processor consumes energy, particularly when the same data must be repeatedly transferred between separate hardware components. Data movement can be more costly than the arithmetic itself, a problem that becomes increasingly severe as cameras gain higher resolution and AI models demand more input. The researchers behind the new system sought to collapse these stages into one physical process. Their prototype performs what they describe as direct light-to-token conversion, using the properties of photosensitive memory devices to preserve optical information and combine it into the representations required by a transformer.

At the heart of the system is a 32-by-32 array of monolayer molybdenum disulfide, or MoS₂, floating-gate phototransistors. MoS₂ belongs to a family of atomically thin materials known as two-dimensional semiconductors. In a monolayer, the active material is only a few atoms thick, giving it strong interactions with light and enabling electronic behavior that can be engineered at a very small scale. In the prototype, the phototransistors respond to incoming illumination and store the resulting information as non-volatile electrical states. Non-volatile memory retains its programmed state even after the light is removed or power conditions change, allowing the captured visual information to remain available for subsequent computation.

The floating-gate structure is particularly important because it enables light-induced changes to be held inside the device rather than immediately converted into a stream of digital values. When an optical image falls on the array, different locations receive different amounts of light and develop corresponding memory states. Peripheral addressing circuitry can then select individual devices or groups of devices and electrically combine their stored signals. In this way, the array does not merely record an image. It participates in the early mathematical transformation of that image, carrying out analogue operations close to the point where photons are first detected.

For a vision transformer, an image patch must be translated into an embedding, a compact vector of numbers that captures the patch’s visual content. In an ordinary digital pipeline, this involves reading pixel values, organizing them into blocks and applying learned or fixed projection operations. The new hardware instead uses the photosensitive memory array to combine optical signals in situ. By selectively addressing the stored states, the system can generate patch-level representations without separately performing image sensing, patch division and patch embedding in different components. The result is a physical tokenizer: a device that transforms light into the sequence of tokens expected by a downstream ViT.

The researchers integrated the physical tokenizer with a standard vision-transformer pipeline and tested its performance on CIFAR-10, a widely used dataset containing 60,000 small colour images across ten object categories. According to the study, the hardware-assisted system reached software-comparable accuracy, indicating that the analogue conversion process preserved enough information for reliable classification. That result is significant because analogue devices can be affected by variation, noise, imperfect switching and limited precision. Maintaining useful recognition performance despite those non-idealities suggests that the physical tokenizer can provide representations compatible with existing AI architectures rather than requiring an entirely new model design.

The most striking result was its reported energy advantage. The physical tokenizer reduced energy consumption by 14.3-fold compared with a digital tokenizer under the researchers’ test conditions. The savings arise from avoiding repeated transfers of raw image data and combining sensing, storage and computation within the same hardware layer. Instead of converting every captured signal into a conventional digital representation before beginning token formation, the device retains optical information locally and performs selected operations through the electrical behavior of the memory array. Such an architecture could be especially valuable in edge systems, where battery capacity, heat dissipation and wireless bandwidth are limited.

The prototype remains an early demonstration rather than a complete replacement for high-resolution camera systems or general-purpose processors. Its 32-by-32 array is modest compared with the sensors used in smartphones, scientific instruments and autonomous machines, and practical deployment would require larger arrays, improved uniformity, reliable calibration and efficient interfaces with full-scale transformer models. Researchers would also need to examine how the devices perform under changing illumination, long operating periods and manufacturing variation. Even so, the experiment points toward a broader change in the design philosophy of machine vision. Rather than treating the sensor as a passive source of pixels and leaving intelligence to later digital stages, future cameras could be designed to produce task-specific representations from the outset.

By bringing tokenization into the sensor, the work addresses one of the central tensions in modern AI: models are becoming more capable, but the movement of data needed to feed them is becoming increasingly expensive. Physical tokenization could allow visual systems to discard irrelevant detail, compress information or create model-ready features before data leaves the imaging surface. That possibility could reduce latency as well as energy use, while supporting intelligent devices that operate locally and preserve privacy by avoiding transmission of raw images. The MoS₂ floating-gate array is therefore more than a new type of image sensor; it is a demonstration of how emerging materials and analogue memory could reshape the boundary between the physical world and artificial intelligence. If the approach can be scaled and integrated with larger transformer systems, the first step of visual understanding may begin not inside a processor, but in the very material that catches the light.

Subject of Research: Direct light-to-token conversion for vision transformers using an integrated two-dimensional photosensitive memory array.

Article Title: Direct light-to-token conversion with integrated 2D photosensitive memory

Article References: Wang, S., Yang, N., Yangdong, XJ. et al. Direct light-to-token conversion with integrated 2D photosensitive memory. Nature Sensors (2026). https://doi.org/10.1038/s44460-026-00122-3

Image Credits: AI Generated

DOI: https://doi.org/10.1038/s44460-026-00122-3

Keywords: Vision transformers, physical tokenizer, light-to-token conversion, MoS₂, two-dimensional materials, photosensitive memory, floating-gate phototransistors, analogue computing, edge artificial intelligence, energy-efficient vision systems