Artificial intelligence may soon begin processing images before digital data ever reaches a conventional computer. In a study published in Nature Sensors, researchers report a prototype that converts light directly into the numerical tokens used by vision transformers, potentially eliminating several energy-intensive steps in modern machine-vision systems. The device combines image sensing, memory and analogue computation in a single hardware platform, allowing visual information to be captured and transformed into patch embeddings at the sensor itself. The approach could offer a new route toward faster and more energy-efficient artificial intelligence for cameras, robots, autonomous vehicles and other edge devices that must interpret large amounts of visual data without relying constantly on the cloud.
Vision transformers, commonly known as ViTs, have become a powerful alternative to convolutional neural networks for image recognition. Instead of scanning an image through layers of fixed filters, a ViT divides the image into small regions, or patches, and represents each patch as a numerical token. These tokens are then processed like a sequence of words in a language model. The transformer learns relationships between distant parts of an image, enabling it to recognize objects and patterns that may depend on the entire visual scene. Yet the standard method for creating those tokens is highly digital: a camera first captures the image, electronic circuits convert the signal into data, a processor divides the image into patches and a separate operation calculates an embedding for every patch.
That conventional chain creates a bottleneck between the physical world and artificial intelligence. Moving raw pixels from an image sensor to memory and then to a processor consumes energy, particularly when the same data must be repeatedly transferred between separate hardware components. Data movement can be more costly than the arithmetic itself, a problem that becomes increasingly severe as cameras gain higher resolution and AI models demand more input. The researchers behind the new system sought to collapse these stages into one physical process. Their prototype performs what they describe as direct light-to-token conversion, using the properties of photosensitive memory devices to preserve optical information and combine it into the representations required by a transformer.
At the heart of the system is a 32-by-32 array of monolayer molybdenum disulfide, or MoS₂, floating-gate phototransistors. MoS₂ belongs to a family of atomically thin materials known as two-dimensional semiconductors. In a monolayer, the active material is only a few atoms thick, giving it strong interactions with light and enabling electronic behavior that can be engineered at a very small scale. In the prototype, the phototransistors respond to incoming illumination and store the resulting information as non-volatile electrical states. Non-volatile memory retains its programmed state even after the light is removed or power conditions change, allowing the captured visual information to remain available for subsequent computation.
The floating-gate structure is particularly important because it enables light-induced changes to be held inside the device rather than immediately converted into a stream of digital values. When an optical image falls on the array, different locations receive different amounts of light and develop corresponding memory states. Peripheral addressing circuitry can then select individual devices or groups of devices and electrically combine their stored signals. In this way, the array does not merely record an image. It participates in the early mathematical transformation of that image, carrying out analogue operations close to the point where photons are first detected.
For a vision transformer, an image patch must be translated into an embedding, a compact vector of numbers that captures the patch’s visual content. In an ordinary digital pipeline, this involves reading pixel values, organizing them into blocks and applying learned or fixed projection operations. The new hardware instead uses the photosensitive memory array to combine optical signals in situ. By selectively addressing the stored states, the system can generate patch-level representations without separately performing image sensing, patch division and patch embedding in different components. The result is a physical tokenizer: a device that transforms light into the sequence of tokens expected by a downstream ViT.
The researchers integrated the physical tokenizer with a standard vision-transformer pipeline and tested its performance on CIFAR-10, a widely used dataset containing 60,000 small colour images across ten object categories. According to the study, the hardware-assisted system reached software-comparable accuracy, indicating that the analogue conversion process preserved enough information for reliable classification. That result is significant because analogue devices can be affected by variation, noise, imperfect switching and limited precision. Maintaining useful recognition performance despite those non-idealities suggests that the physical tokenizer can provide representations compatible with existing AI architectures rather than requiring an entirely new model design.
The most striking result was its reported energy advantage. The physical tokenizer reduced energy consumption by 14.3-fold compared with a digital tokenizer under the researchers’ test conditions. The savings arise from avoiding repeated transfers of raw image data and combining sensing, storage and computation within the same hardware layer. Instead of converting every captured signal into a conventional digital representation before beginning token formation, the device retains optical information locally and performs selected operations through the electrical behavior of the memory array. Such an architecture could be especially valuable in edge systems, where battery capacity, heat dissipation and wireless bandwidth are limited.
The prototype remains an early demonstration rather than a complete replacement for high-resolution camera systems or general-purpose processors. Its 32-by-32 array is modest compared with the sensors used in smartphones, scientific instruments and autonomous machines, and practical deployment would require larger arrays, improved uniformity, reliable calibration and efficient interfaces with full-scale transformer models. Researchers would also need to examine how the devices perform under changing illumination, long operating periods and manufacturing variation. Even so, the experiment points toward a broader change in the design philosophy of machine vision. Rather than treating the sensor as a passive source of pixels and leaving intelligence to later digital stages, future cameras could be designed to produce task-specific representations from the outset.
By bringing tokenization into the sensor, the work addresses one of the central tensions in modern AI: models are becoming more capable, but the movement of data needed to feed them is becoming increasingly expensive. Physical tokenization could allow visual systems to discard irrelevant detail, compress information or create model-ready features before data leaves the imaging surface. That possibility could reduce latency as well as energy use, while supporting intelligent devices that operate locally and preserve privacy by avoiding transmission of raw images. The MoS₂ floating-gate array is therefore more than a new type of image sensor; it is a demonstration of how emerging materials and analogue memory could reshape the boundary between the physical world and artificial intelligence. If the approach can be scaled and integrated with larger transformer systems, the first step of visual understanding may begin not inside a processor, but in the very material that catches the light.
Subject of Research: Direct light-to-token conversion for vision transformers using an integrated two-dimensional photosensitive memory array.
Article Title: Direct light-to-token conversion with integrated 2D photosensitive memory
Article References: Wang, S., Yang, N., Yangdong, XJ. et al. Direct light-to-token conversion with integrated 2D photosensitive memory. Nature Sensors (2026). https://doi.org/10.1038/s44460-026-00122-3
Image Credits: AI Generated
DOI: https://doi.org/10.1038/s44460-026-00122-3
Keywords: Vision transformers, physical tokenizer, light-to-token conversion, MoS₂, two-dimensional materials, photosensitive memory, floating-gate phototransistors, analogue computing, edge artificial intelligence, energy-efficient vision systems
